Ontology Matching in Practice: Facilitating Automated

Ontology Matching in Practice:Facilitating Ontology Alignments for

Interoperable Research Data

Author:

Philip van Damme

Master’s thesisMaster of Medical Informatics

August 2020

2

Ontology Matching in Practice: Facilitating OntologyAlignments for Interoperable Research Data

Author: Philip van DammeStudent number: 10742549Email: [email protected]

Tutors:Ronald Cornet, PhD Nicolette de Keizer, PhDAssociate professor Full [email protected] [email protected]

Mentor:Jesualdo Tomas Fernandez Breis, PhDFull [email protected]

Location:Universidad de Murcia Amsterdam UMC - AMCFacultad de Informatica Dept. of Medical InformaticsCampus de Espinardo Meibergdreef 9CP 30100 Murcia, Spain 1105 AZ Amsterdam, The Netherlands

Period:December 2019 - August 2020

3

Abstract

Objective

This study analyzes the performance of automated ontology matching techniques in the domain of rarediseases. Current efforts in the rare disease community focus on integrating data distributed over multiplesources, which requires the need for automated ontology matching. Additionally, this study analyzes howthese techniques can be applied to a practical use case of creating a machine-readable classification ofvascular anomalies. An ontology based on this classification was matched to the Orphanet Rare DiseaseOntology (ORDO), SNOMED CT, and the NCI Thesaurus (NCIt). The key rationale of this study is tocontribute to interoperable data, which is part of the Findable, Accessible, Interoperable, and Reusable(FAIR) data principles.

Methods

Three ontologies (NCIt, SNOMED CT, ORDO) and matching systems (AgreementMakerLight 2.0,FCA-Map, LogMap 2.0) were used in an experimental study. Pairwise alignments containing equivalencemappings between NCIt-ORDO, NCIt-SNOMED CT, and ORDO-SNOMED CT were created. Mod-ularization techniques were applied to extract a locality-based module from each ontology, using seedsignatures based on rare disease data items. The performance of the matching systems was evaluatedagainst reference alignments from BioPortal and the Unified Medical Language System (UMLS). Inaddition, mappings were evaluated by examining their top-level hierarchies. Finally, the matching systemswere applied to a new ontology based on a classification of vascular anomalies.

Results

Evaluation of the NCIt-ORDO pair resulted in F1-scores of 0.53 (UMLS) and 0.42 (BioPortal). TheNCIt-ORDO pair had F1-scores of 0.43 (UMLS) and 0.58 (BioPortal). The NCIt-SNOMED CT pair hadthe highest F1-scores of 0.60 (UMLS) and 0.78 (BioPortal). AgreementMakerLight 2.0, FCA-Map, andLogMap 2.0 had F1-scores of 0.55, 0.46, 0.55 for BioPortal and 0.66, 0.53, 0.58 for the UMLS, respectively.Evaluation with manually created top-level hierarchy mappings showed an average of 10% of mappingswhich classes belonged to an incorrect hierarchy. Application of the matching systems to the ontologybased on the classification of vascular anomalies resulted in a coverage of 44% with ORDO, 35% withNCIt, and 36% with SNOMED CT. The precision of those alignments varied between 0.11 and 0.31.

Discussion

Measuring the performance of ontology matching systems is not a trivial task, as the requirementsstrongly depend on the practical application. This study found that currently available matching systemscan produce meaningful mappings without user intervention. Moreover, combining universal referencealignments (BioPortal, UMLS) and analyzing top-level hierarchies seems to be promising for the automaticselection of useful mappings. This study concludes that available ontology matching systems can contributeto achieving interoperable data, considering limitations regarding the evaluation of mappings. Futureresearch should focus on the automatic selection of useful mappings following real-life use cases.

Keywords

Ontology matching, FAIR data, rare disease research

4

Samenvatting

Doelstelling

Deze studie onderzoekt de prestaties van systemen die automatisch concepten uit verschillende ontologieenmet elkaar kunnen matchen. De aanleiding van deze studie is het streven vanuit het onderzoek naarzeldzame ziekten om data uit verschillende bronnen met elkaar te kunnen combineren. Daarnaastanalyseert deze studie hoe matching van ontologieen toegepast kan worden op een voorbeeld uit depraktijk, door een ontologie voor vasculaire afwijkingen te matchen met de Orphanet Rare DiseaseOntology (ORDO), SNOMED CT en de NCI Thesaurus (NCIt). Het doel van deze studie is om bij tedragen aan de realisatie van uitwisselbare en herbruikbare data, als onderdeel van het FAIR principe(Findable, Accessible, Interoperable, Reusable).

Methoden

Een experimenteel onderzoek werd uitgevoerd met drie ontologieen (NCIt, SNOMED CT, ORDO) endrie matching systemen (AgreementMakerLight 2.0, FCA-Map, LogMap 2.0). De paren NCIt-ORDO,NCIt-SNOMED CT en ORDO-SNOMED CT werden met elkaar gematcht. Uit alle drie de ontologieenwerd tevens een kleinere module gegenereerd. De prestaties van de systemen werd onderzocht op basisvan een referentiestandaard afkomstig uit BioPortal en het Unified Medical Language System (UMLS).Daarnaast werden de gematchte concepten geevalueerd op basis van hun hierarchie. Als laatste werdende matching systemen toegepast op een nieuwe ontologie voor vasculaire afwijkingen.

Resultaten

De evaluatie van NCIt-ORDO resulteerde in F1-scores van 0.53 (UMLS) en 0.42 (BioPortal). NCIt-ORDOhad F1-scores van 0.43 (UMLS) en 0.58 (BioPortal). NCIt-SNOMED CT had de hoogste F1-scores,namelijk 0.60 (UMLS) en 0.78 (BioPortal). De systemen AgreementMakerLight 2.0, FCA-Map en LogMap2.0 hadden F1-scores van respectievelijk 0.55, 0.46 en 0.55 op basis van BioPortal. De F1-scores voorUMLS waren 0.66, 0.53 en 0.58. Evaluatie met de handmatig gematchte hoogste hierarchieen leverdeeen gemiddelde op van 10% van het totaal aantal gematchte concepten die als incorrect konden wordenaangemerkt. Het toepassen van de systemen op de ontologie voor vasculaire afwijkingen wees uit datdeze een overlap had van 44% met ORDO, 35% met NCIt en 36% met SNOMED CT. De precisie vandeze gematchte concepten lag tussen de 0.11 en 0.31.

Discussie

Het meten van de prestaties van matching systemen is niet eenvoudig, omdat de interpretatie van deresultaten sterk samenhangt met de praktische toepassing van de gematchte concepten. Dit onderzoeklaat zien dat beschikbaar matching systemen in staat zijn om automatisch bruikbare matches te vinden.Daarbij lijken het gebruik van referentiestandaarden en de analyse van hierarchieen potentie te bieden.Dit onderzoek concludeert dat matching systemen kunnen bijdragen aan uitwisselbare en herbruikbaredata. Een uitdaging die nog open ligt is het automatich selecteren van bruikbare matches gebaseerd ophun toepassing, toekomstig onderzoek zal zich daar op moeten richten.

Trefwoorden

Matching van ontologieen, FAIR data, onderzoek naar zeldzame ziekten

Contents

1 Introduction 7

1.1 Context of this study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.1.1 Reusable research data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.1.2 Ontologies and ontology matching . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.1.3 Combining FAIR data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

1.2 Problem statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

1.2.1 Use case: a classification for vascular anomalies . . . . . . . . . . . . . . . . . . . . 9

1.3 Objective and research questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.4 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

2 Background 13

2.1 The Semantic Web and interoperability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.2 Ontologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.2.1 Module extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2.3 Ontology matching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.3.1 Matching biomedical ontologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

2.4 Matching techniques and state-of-the-art systems . . . . . . . . . . . . . . . . . . . . . . . 16

2.4.1 Classification of matching techniques . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.4.2 Matching systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

2.4.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3 Methods 21

3.1 Ontologies and matching systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.1.1 Selection of ontologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.1.2 Module extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.1.3 Selection of matching systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

5

6 CONTENTS

3.2 Alignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.3.1 Reference alignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

3.3.2 Hierarchy analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

3.3.3 Rare disease use case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4 Results 27

4.1 Rare disease data elements and modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.2 Alignments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.3 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.3.1 Reference alignments: BioPortal and UMLS . . . . . . . . . . . . . . . . . . . . . . 28

4.3.2 Evaluation results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

4.3.3 Hierarchy analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

4.3.4 Rare disease use case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

5 Discussion and conclusion 33

5.1 Principal findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

5.2 Strengths and limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

5.3 Relation to other work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

5.4 Future research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

5.5 Implications of this study and conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

References 39

List of abbreviations 43

Appendices 45

Appendix A Rare disease data items 47

Appendix B Full results BioPortal and UMLS 49

Chapter 1

Introduction

1.1 Context of this study

1.1.1 Reusable research data

The generation, collection, and usage of data is crucial for scientific research. Consequently, the sharingand reuse of research data have become more and more important. In Europe, a report published in2010 from the European High-Level Expert Group on Scientific Data urged immediate action to an openapproach to science and the related digital infrastructure [1]. Furthermore, the European Union’s Horizon2020 programme focuses on improving access to scientific publications and research data [2]. On the onehand, data sharing could be as simple as researcher A sending a dataset to researcher B, but would becomemore challenging as soon as researchers do not know each other personally or have no knowledge aboutwhich data is available. On the other hand, for data reuse, one would also need to have information aboutthe data in order to be able to reuse it. Researcher A could give such information to researcher B and thenB can reuse the data that has been shared by A. However, when considering multiple research groups,universities, countries, or even continents, specific arrangements are needed to make data reuse possible.Such arrangements can be included in a data management plan (DMP), DMPonline being an exemplartool for researchers to create and share DMPs [3]. To support the harmonization of data management inEurope, Science Europe and the Dutch Research Council (NWO) published core requirements for DMPs[4]. Such requirements help individual researchers, research organizations, funding agencies, and researchcommunities to align their DMPs.

Reusable data does not only relate to humans but also to machines, as machine-readable data enablesboth humans and machines to browse and discover information. In 2016, a group of researchers andother stakeholders with an interest in the findability and reuse of research data published a set ofprinciples to propagate the reuse of research data for machines and humans [5]. These principles werepresented as the FAIR Guiding Principles for scientific data management and stewardship (Findable,Accessible, Interoperable, and Reusable). The FAIR acronym is since then widely acknowledged by theresearch community to refer to reusable data. The process of making data FAIR is often referred toas FAIRification, which has been described in seven steps by the GO FAIR initiative (see Figure 1.1)[6]. The FAIRification process can also be described by ‘pre-FAIRification’ (identify objective, analyze(meta)data), FAIRification (define semantic model, make (meta)data linkable, and host (meta)data), and‘post-FAIRification’ (assess the FAIR data) [7]. It is important to note that FAIR data, and so reusabledata, does not equal open data. Hence, accessibility is part of the FAIR Guiding Principles [5].

1.1.2 Ontologies and ontology matching

In information science, an ontology can be defined as a ‘formal, explicit specification of a sharedconceptualization’ [8]. An ontology describes concepts and the relationships between those concepts of a

7

8 Context of this study

Figure 1.1. Seven steps of the FAIRification process. Adapted from GO FAIR [6]. Thisstudy focuses on step 3 (define the semantic model) and step 4 (make data linkable).

domain and can be made available in multiple languages. BioPortal is an open repository of biomedicalontologies and includes at the time of writing 835 ontologies [9]. An example of a biomedical ontology isSNOMED CT [10], which includes healthcare terminology on multiple domains and in multiple languages.Another example is the Orphanet Rare Disease Ontology (ORDO) [11], which focuses on rare diseases.When making data FAIR, a semantic model of the data needs to be created (step 3), and the dataneeds to be made linkable (step 4) (Figure 1.1), for both of which ontologies are useful as they provide aconsensus about some domain.

Different ontologies can contain classes describing the same domain or concepts which makes themoverlap. Using an ontology should improve the interoperability of data, however, no interoperabilitywould be accomplished if researchers use a different ontology for describing the same concept(s). Toovercome the problem of having to deal with multiple ontologies, the field of ontology matching comesinto play. Ontology matching aims to make ontologies interoperable by matching semantically relatedconcepts from different ontologies, resulting in alignments between ontologies. Such ontology alignmentsallow for interoperability even when multiple ontologies are involved.

A specific community (e.g. the rare disease community) could define standards to which the entirecommunity should adhere, including specific ontologies. Ontology matching would then be of use toachieve interoperability within a community and between communities, as different communities might usedifferent standards. Examples of achieving interoperability by defining community standards, adhering tothe FAIR data principles, can already be found [12].

1.1.3 Combining FAIR data

Combining FAIR data, thus creating the ability to ask questions across multiple sources (e.g. datasets),should be the outcome of the FAIRification process [6]. Ontology matching can make data interoperablein case data(sets) are annotated using different ontologies [13]. Several initiatives embrace the FAIRprinciples to create an environment where existing data becomes reusable, such as the Personal HealthTrain [14]. The Personal Health Train focuses on health data and assumes data to be FAIR. Then, datastays at their source (‘stations’) while users can send queries or use cases (‘trains’) to interrogate dataacross multiple sources.

In January 2019 the European Joint Programme on Rare Diseases (EJP RD) has started, which isa large Europe-wide project focusing on creating an ecosystem for rare disease research and care [15].One of the objectives of the EJP RD is to build a FAIR-compliant virtual platform that describes raredisease resources and enables researchers to interrogate data from multiple resources at different locations.Consequently, these sources can use different ontologies which requires ontology matching in order toenable interoperability.

Chapter 1. Introduction 9

1.2 Problem statement

Making data interoperable involves annotating the data with ontology classes which, as mentioned inSection 1.1.2, introduces the need for ontology alignments when multiple (overlapping) ontologies areinvolved. Ontology alignments could be created manually but that would rapidly become infeasible dueto the large size of biomedical ontologies (e.g. SNOMED CT contains over 350.000 classes). Moreover,datasets change over time, as do ontologies, which creates the need for dynamic ontology matchingservices [13]. Lots of matching techniques and systems have been developed but there is a lack of papersdescribing real-life applications [16]. An example of a real-life application could be the implementation ofontology matching for a web service that queries data over multiple sources. Most matching systems,as a consequence, do not consider a specific real-life use case but rather focus on tasks concerning thealignment of specific ontologies. Consider two datasets containing data about rare diseases: the firstdataset annotates diseases using SNOMED CT and the second one with ORDO, an alignment betweenORDO and SNOMED CT would then be necessary to achieve interoperability without human intervention.A systematic analysis of term overlap and term reuse across biomedical ontologies in BioPortal foundan approximate overlap of over 25% of ontology concepts and less than 9% of reuse [17]. Another workstudied the reuse of logical axioms in biomedical ontologies and discovered that 49 out of 123 ontologiesdid not apply any type of reuse [18]. Hence, even when data is annotated with ontology classes it is notper definition interoperable, as such an ontology would need to be matched to other (similar) ontologies.Matching classes from different ontologies is a non-trivial task to which this study wants to contribute.

1.2.1 Use case: a classification for vascular anomalies

European Reference Networks (ERNs) are networks (24 in total) that aim to connect medical expertiseacross Europe and exchange information [19]. Each ERN focuses on a different group of rare diseases.VASCERN [20] is the ERN on rare multisystemic vascular diseases and aims to build a Registry ofVascular Anomalies (VASCA) based on the FAIR data principles. The VASCA registry is being builtusing the common data elements for rare disease registries, part of which is the set of data elements for thediagnosis of a patient. For the latter, VASCA has to register vascular tumors and malformations accordingto a classification of the International Society for the Study of Vascular Anomalies (ISSVA) [21]. Currently,this classification is only available in PDF-format and is non-machine-readable. Therefore, efforts weremade by VASCA to transform this classification into an ontology. Ultimately the classes in this ontologyshould be mapped to ORDO and other relevant ontologies, to which this study wants to contribute usingontology matching. Ontology matching should help the ISSVA ontology to be interoperable with otherontologies, and to ensure that existing ontologies are reused instead of building a new ontology fromscratch.

10 Objective and research questions

1.3 Objective and research questions

This study aims at analyzing the performance of existing automated ontology matching techniques, andtheir practical application in the domain of rare disease research. In addition, it studies the usage ofontology matching techniques for combining and/or relating data for querying over multiple sources. SeeFigure 1.2 for a simplified example. Hence, this project contributes to the facilitation of interoperable,thus FAIR, research data. The following research questions will be discussed:

1. What is the performance of automated ontology matching techniques to expose mappings betweenontologies used in the rare disease research domain?

2. To what extent are currently available techniques for ontology matching useful for implementationin FAIR-related projects that focus on the integration of multiple data sources?

3. How can ontology matching be used in practice while creating new and using existing ontologies tocreate machine-readable data, with regard to term overlap and reuse?

Figure 1.2. Example of an ontology matching service. Two sources are annotated withdifferent ontologies (SNOMED CT and ORDO). An ontology matching service can make

these two sources interoperable by matching equivalent annotations. Questions like‘Which countries have data available about multiple sclerosis?’ could then be answered.

Chapter 1. Introduction 11

1.4 Outline

This thesis is divided into five chapters. The first chapter gives background information on the topic,states the problem, and describes the objective and research questions of this study. The second chapterprovides all the necessary background information. The third chapter explains the methods that wereused. In the fourth chapter the obtained results are presented. The fifth chapter discusses the results,puts the research in context of other related work, and presents the conclusions drawn from the workthat has been done.

12 Outline

Chapter 2

Background

2.1 The Semantic Web and interoperability

Making data machine-readable is the goal of the Semantic Web [22]. The Semantic Web describes a setof technologies to create a collection of interlinked data (Linked Data) on the web, those technologies areset as standards by the World Wide Web Consortium (W3C) [23]. All things on the Semantic Web needto be unambiguously identified by Unified Resource Identifiers (URIs). W3C standards for the SemanticWeb include:

• Resource Description Framework (RDF), enabling to represent data usingsubject/predicate/object triples. RDF allows for graph representations.

• RDF Schema (RDFS), an extension of RDF providing a vocabulary to create data models of RDFdata.

• SPARQL, an RDF query language that allows querying RDF represented data.

• Web Ontology Language (OWL), a Semantic Web language designed to specify knowledge in aprecise way. OWL extends the RDF and RDFS standards and allows for reasoning.

Semantic interoperability means that machines can exchange data unambiguously and with the samemeaning [23]. The mentioned techniques of the Semantic Web support semantic interoperability. TheFAIRification process as described by GO FAIR (Figure 1.1) includes creating a semantic model andmaking data linkable, which can be done using Semantic Web technologies.

2.2 Ontologies

As mentioned in Chapter 1, ontologies provide a formal description of concepts and the relationshipsbetween those concepts. OWL is the international standard of W3C to represent ontologies. Ontologiesform an important part of the Semantic Web and the FAIRification process (Figure 1.1). Creating asemantic model of data includes the usage of appropriate ontological entities and making data linkableinvolves the application of Semantic Web technologies such as RDF and OWL. Figure 2.1 shows anexample of the concept Cystic fibrosis screening (procedure) in the hierarchy of SNOMED CT, which isdefined as a screening of respiratory disease and has three properties.

2.2.1 Module extraction

Due to the typically large size of (biomedical) ontologies, it can be useful to extract a smaller module.Such a smaller part of a larger ontology makes it easier to use and understand the ontology. The OWL

13

14 Ontology matching

171191008

Cystic fibrosis screening (procedure)

171228002

Respiratory disease screening (procedure)

260686004

Method (attribute)

129265001

Evaluation - action (qualifier value)

363702006

Has focus (attribute)

190905008

Cystic fibrosis (disorder)

363703001

Has intent (attribute)

360156006

Screening - procedure intent (qualifier value)

Figure 2.1. Example of the concept Cystic fibrosis screening (procedure) in SNOMED CT,from the SNOMED CT Browser [10]. The concept a subclass of ‘respiratory disease screening’,and has three properties (called attributes in SNOMED CT): 1) ‘Method’ is ‘Evaluation - ac-tion’, 2) ‘Has focus’ is ‘Cystic fibrosis’, and 3) ‘Has intent’ is ‘Screening - procedure intent’.

API includes a syntactic locality module extractor [24]. This module extractor uses a seed signature asinput, which contains a list of classes from the parent ontology which the module(s) should be basedon. Using this seed signature, the module extractor can extract three different types of modules: star,bottom, and top. A top module includes all subclasses and (sub)properties (e.g. the properties mentionedin Figure 2.1) of the classes in the seed signature, a bottom module does the opposite by including thesuperclasses and (super)properties. Disjointness between classes is also included, e.g. if class A is disjointwith B then B will be included if A is included and vice versa. A star module combines both strategiesby including the intersection of the top and bottom modules. Both the asserted and inferred versions ofthe ontologies are used. The difference between a top, bottom, or star module is shown as a simplifiedexample in Figure 2.2.

2.3 Ontology matching

Ontology matching aims to make ontologies interoperable by matching semantically related concepts fromtwo or more ontologies, resulting in alignments between ontologies. Such ontology alignments allow forinteroperability even when multiple ontologies are involved, as mentioned in Chapter 1. For example,the Unified Medical Language System (UMLS) [25] integrates over 100 vocabularies and several keybiomedical ontologies including SNOMED CT and ORDO. First, some basic definitions are given below,adopted from [26] and modified when desired for the scope of this work.

Definition 2.3.1. (Ontology matching) Matching is the process of finding relationships between differentconcepts of different ontologies. The matching process can be defined as a function f :

A′ = f(O,O′, A, p, r)

where A′ is the output alignment of a pair of ontologies O and O′, A is an optional reference alignmentthat would be extended or completed, p is a set of parameters (e.g. weights) used during the matchingprocess, and r the possible external resources that are used to match the ontologies (e.g. vocabularies).

Definition 2.3.2. (Alignment) An alignment A is a set of mappings between classes of two ontologies Oand O′ and is the output of the matching process.

Definition 2.3.3. (Mapping) A mapping m is the relation, according to an alignment, between differentclasses of two ontologies. Some papers refer to a mapping as a correspondence. Formally a mapping can

Chapter 2. Background 15

(a) Whole ontology (b) Top module

(c) Bottom module (d) Star module

Figure 2.2. Example of different module types. The classes B and F (red) are used as seedsignature, classes marked in blue are included in a module. The top module contains all sub-classes of the seed signature classes (and properties), the bottom module contains all super-

classes (and properties), and the star module contains the intersection of top and bottom. Sib-lings classes are disjoint, meaning that C is also included because B has a disjoint axiom with C.

be defined as a triple by a pair of ontologies O and O′ and a set of mapping relations Θ:

m = 〈e, e′, r〉

wheree ∈ O and e′ ∈ O′;r ∈ Θ.Moreover, a mapping can include metadata such as a confidence value and identifiers. Although metadatawill not be used for this work.

Example 2.3.3. (Mapping) The concepts ‘Physician’ in SNOMED CT and ‘Doctor’ in LOINC share thesame Concept Unique Identifier (CUI) in the UMLS, which means that they are semantically equivalent.An equivalence mapping m between ‘Physician’ and ‘Doctor’ could then look like:

m = 〈 http://purl.bioontology.org/ontology/SNOMEDCT/309343006, http://purl.bioontology.org/ontology/LNC/LA18968-0, ≡〉

The English word ‘doctor’ is a homonym that can refer to either a medical practitioner or an academicdegree for someone who has obtained a doctorate. This example shows that context of the domain isimportant for ontology matching. That is, whether or not the ontologies describe the same domain (e.g.medicine).

2.3.1 Matching biomedical ontologies

A work from 2011 [27] studied the state of biomedical ontologies and concluded that half of the ontologiesin BioPortal fit the manageable OWL 2 EL profile (e.g. SNOMED CT). Classes in biomedical ontologies

http://purl.bioontology.org/ontology/SNOMEDCT/309343006

http://purl.bioontology.org/ontology/LNC/LA18968-0

http://purl.bioontology.org/ontology/LNC/LA18968-0

16 Matching techniques and state-of-the-art systems

are often described using labels and several synonyms, which makes that their vocabulary is importantfor ontology matching [28]. Hence, lexical matching is often the main approach for matching biomedicalontologies. Structural matching is less common [28]. Most state-of-the-art matching systems depend onlexical similarity to either discover or evaluate mappings [29].

It is important to know certain requirements before performing ontology matching (description of thematching problem), as described by [26]. Such requirements involve:

• The type of input that is available for the matching system. E.g. if input is available as XML, orRDF/OWL, or in another format. The matching systems need to support the type of input that isavailable.

• If the matching systems need to be automatic or can receive feedback from the user, if the systems’mappings need to be always correct, if the system needs to find all possible mappings, and if thereis a limit in run time for the system.

• How the output alignment is going to be used.

2.4 Matching techniques and state-of-the-art systems

2.4.1 Classification of matching techniques

Ontology matching techniques can be classified using a classification model, which organizes techniquesbased on how they interpret the input [26]. Figure 2.3 shows an adapted version of this classificationmodel. This model can be used to classify the matching systems based on their granularity and inputinterpretation. The granularity of a matching system can be defined by two levels: element-level andstructure-level. Element-level matching techniques focus on a class without considering its relations toother classes (e.g. comparing the label strings using some similarity measure), structure-level matchingtechniques focus on a class within the structure of the ontology (e.g. super- or subclass similarities, pathdistance in the hierarchy, or property similarity). At each level, the model makes a distinction betweensemantic and syntactic matching techniques. Syntactic matching techniques use only the information of aclass without interpretation, for example, the textual label or synonyms. Semantic matching techniquesadd meaning to the structural information (e.g. using a reasoner or external resources). Then, theinput of a matching system can be interpreted using nine different techniques. Formal resource-based(e.g. an upper-level ontology), informal resource-based (e.g. external knowledge from encyclopedias),string-based (e.g. some string similarity metric), language-based (e.g. usage of a lexicon), constraint-based(e.g. comparing types, attributes, or the cardinality of two classes), taxonomy-based (e.g. if class A is asubclass of B, then their neighbors may also be similar), graph-based (e.g. comparing the depth of a classin the graph), instance-based (e.g. formal concept analysis: creating a concept hierarchy from a group ofclasses and their properties), model-based (e.g. reasoning techniques using description logic).

2.4.2 Matching systems

A matching system can be described using three dimensions [26]: input, process, and output. Theinput dimensions refer to which information a matching system uses. For example, which informationis extracted from the ontology and if the system uses any external resources. The process dimensionsinclude the algorithms and techniques that are used (the basic matching techniques), and how thesystem interprets the input (semantically and syntactically). The output dimensions regard the type ofmappings that the system produces, the relationship(s) that are exploited, if the system matches classesone-to-one/one-to-many/many-to-many, and how mappings are delivered (with a confidence value, aprobability, or binary as true/false). Several basic matching techniques together form a matching system.Matching systems often combine several matching techniques in order to be effective [26]. Table 2.1


Figure 2.3. Matching techniques classifications. Adapted from [26]. Be-low each category an example of a possible implementation is given.

shows an overview, based on the classification of matching approaches, of three state-of-the-art matchingsystems used in this study. These three systems will be explained in more detail down below.

A literature review from 2014 [16] included 694 articles about ontology matching, of which 302 wererelated to the development of matching systems and the enhancement of existing systems. Accordingly,many matching systems have been developed over the years. Some matching systems implement severalbasic matching techniques and offer customization of the matching process [30, 31]. Others use a two-stepapproach and use the output of an element-level matcher as the input for a structural-level matcher [32].Some systems focus on automating the aggregation of different matching techniques, as the selection ofeffective techniques is known as a major problem in ontology matching [33]. Machine learning techniquesare also used, such as neural networks [34], and word embedding combined with a random forest classifier[35]. Moreover, the authors of [35] argue that standard machine learning approaches fail, because 1)feature engineering fails due to features that cannot be generalized, and 2) supervised learning suffers aclass imbalance problem (the number of true mappings between two ontologies is much smaller than thenumber of all possible mappings).

Automatically matching large ontologies raised the need for ontology alignment repair, since alignmentscan be logically incoherent [36]. For example, two classes that cannot share an instance (disjointness).Moreover, the authors of [36] state that it is impossible to confirm the correctness of mappings automatically.Some matching systems include mapping repair algorithms [37].

AgreementMakerLight 2.0

AgreementMakerLight 2.0 (AML) matches a pair of ontologies (target and source) based on the followingsteps [30, 31, 38]. AML uses only element-level matching techniques.

1. Ontology loading: the creation of a lexicon data structure that includes the names of all classes,their labels, and synonyms. This lexicon also includes a weight system that can be used by thematching algorithms.

2. Primary matching: lexical matcher (literal name matches), mediating matcher (same as thelexical matcher, but matches each input ontology to an external ontology first, using the HumanDisease Ontology, and the Uber-anatomy ontology), word matcher (word-based string similarityalgorithm, using a weighted Jaccard index between the words of the class names).

3. Secondary matching (tries to extend the mappings from step 2): parametric string matcher


Table 2.1. Classification of three state-of-the-art matching systems:AgreementMakerLight 2.0 [38], FCA-Map [39], and LogMap 2.0 [40].

AgreementMakerLight 2.0 FCA-Map LogMap 2.0

Element level

Semantic: Formal resource-based X - -Syntactic: Informal resource-based - - -Syntactic: String-based X X XSyntactic: Language-based X X XSyntactic: Constraint-based - X -

Structure level

Semantic: Model-based - X XSyntactic: Instance-based - X -Syntactic: Graph-based - - XSyntactic: Taxonomy-based - - X

(all-to-all string similarity algorithm, using the ISub similarity metric).

4. Selection and repair: combines the output of all matchers, in the case of duplicate mappings itdiscards the mapping with the lowest similarity score. It makes sure that each class appears in atmost one mapping.

FCA-Map

FCA-Map matches a pair of ontologies based on the following steps [39].

1. Token-based formal context: uses the names, labels, and synonyms of the classes and performsnormalization. Synonyms are retrieved using the UMLS Sub-Term Mapping Tools, lexical variationsfrom the UMLS Lexicon. Then, two types of initial mappings are created.

(a) Exact match (e.g. SNOMED:muscle, ORDO:muscle).

(b) Partial match (e.g. SNOMED:left lung disease, ORDO:right lung disease).

2. Relation-based formal context: the initial mapping set from step 1 are validated with structuralrelations. It considers positive evidence (when a class of one mapping shares a relationship with aclass in another mapping), and negative evidence (disjointness relationships between two classes).

(a) Conflicts between mappings are repaired by removing all conflicted mappings based on negativeevidence.

(b) Mappings with no structural evidence are considered to be structurally isolated or incorrect.All partial matches from step 1 without positive evidence are removed.

3. Positive relation-based formal context: new structural mappings are identified based on thealready discovered mappings.

(a) One-to-one structural mappings, e.g. ‘humerus bone’ - ‘upper arm bone’.

(b) One-to-many (e.g. ‘ear’ - ‘left ear’, ‘right ear’) and many-to-many mappings (vertebra 1,2,3,4,5- L1,L2,L3,L4,L5 vertebra).

4. Property-based formal context: only the structural information of the class properties is usedto match properties between the source and target ontology.


5. Restriction-based formal context: based on the property mappings an ‘anonymous ancestors’formal concept analysis is created. For example: if SNOMED:Hemangioma of liver and NCIt:HepaticHemangioma are matched, and SNOMED CT has the property ‘finding site→ blood vessel structure’and NCIt ‘has associated anatomic site → blood vessel’, then the classes SNOMED:blood vesselstructure and NCIt:blood vessel are anonymous ancestors of the original classes.

LogMap 2.0

LogMap 2.0 (LogMap) matches a pair of ontologies based on the following steps [37, 40].

1. Overlapping estimation: in this step, an overestimation of pairs of concepts based on their labelsis made, based on lexical similarity. Then, a module is extracted from each input ontology basedon this set of overestimated mappings. This should reduce the number of classes that need to bematched.

2. Lexical indexation: indexation of the labels of the two extracted modules. Performs tokenizationand uses the UMLS Lexicon to find synonyms and lexical variations.

3. Computation of candidate mappings: the lexical index from each module is intersected to geta set of initial mappings. Then, two groups are created using the ISub string similarity metric onthe neighbors of the initial mappings. If the neighbors of the classes in a mapping are similar, thenthe mapping is probably correct.

(a) Fixed mappings, mappings that have similar neighbors (e.g. ‘stenosis’ and ‘stenosis’ that areboth classified as ‘disease’) and are therefore probably correct.

(b) Active mappings, mappings that probably need expert curation (e.g. ‘stenosis’ and ‘stenosis’where the first is a ‘disease’ and the second a ‘body structure’).

4. Mapping repair: mappings in the two groups (fixed and active) are represented in Horn propo-sitional logic. A reason algorithm (Dowling-Gallier) is used to detect unsatisfiable classes in the‘active’ group. A diagnosis algorithm then deletes those unsatisfiable active mappings. Leftoveractive mappings are finally included as fixed mappings.

5. Structural indexation: the inferred versions of the ontologies are used for structural indexation.Two direct acyclic graphs (DAGs) are created.

(a) Descendants DAG (subclass relationships)

(b) Ancestors DAG (superclass relationships)

Each class is represented as a node in the DAG and has information about the location of the classin the graph.

6. Conflict detection using the structural index: mappings that are disjoint are removed fromthe active mappings set.

7. (Optional: remaining active mappings can be presented to the user for evaluation, otherwise thematching process is finalized automatically using heuristics.)

2.4.3 Evaluation

After obtaining an alignment between two ontologies, the mappings should be evaluated. This evaluationcan be done manually (which is often done by domain experts) or automatically. The results of anevaluation is often expressed using measures such as precision and recall, which provide informationabout the correctness and completeness of an alignment [26]. The evaluation of ontology matchingsystems is being coordinated internationally by the Ontology Alignment Evaluation Initiative (OAEI)since 2004 [41]. The OAEI facilitates the comparison of the performance of different matching systems.


Automatic evaluation of an ontology matching system can be done using a reference alignment that couldbe identified as [42]:

• Gold standard (complete alignments that are created manually by domain experts)

• Silver standard (alignments that are not necessarily complete or correct)

• Baseline (alignments that are highly incomplete)

Determining whether the evaluation results of an alignment and/or its mappings are good or bad, dependson the situation in which the alignment will be used. For example, if some application requires thatmappings are 100% correct, precision would be more important than recall. On the other hand, if it isrequired that all mappings are retrieved, recall would be more important instead. Moreover, apart fromthe alignment itself, other factors might also be considered. Such as the run time of the matching system,the required computational resources, and if the system is fully automatic or needs intervention from theuser.

Chapter 3

Methods

An experimental study was performed for measuring the performance of ontology matching systems andtechniques using two types of reference alignments, including an analysis of the top-level hierarchies ofclasses in the mappings. The matching systems were also applied to a practical use case in the rare diseasedomain. The following steps were carried out in chronological order: selecting rare disease data elements,selecting relevant biomedical ontologies, module extraction, selecting available ontology matching systems,calculating alignments between ontology pairs, and a two-part evaluation of the alignments that wereobtained. Part one of the evaluation was done by measuring the performance using two types of referencealignments, the second part consisted of a top-level hierarchy analysis and application of the acquiredalignments to a practical use case. Figure 3.1 shows a visual overview of these steps. All developmentwork was done using Java 8, data analysis was done in R version 4.0.1 [43].

Figure 3.1. Visual overview of the performed steps. Highlighted blocks rep-resent tools that were used from BioPortal (blue) or the UMLS (orange).

21

22 Ontologies and matching systems

3.1 Ontologies and matching systems

3.1.1 Selection of ontologies

The ontologies used for this study needed to be appropriate for usage in the rare disease domain. Thisimplies that their content should be useful for annotating datasets in this domain. Therefore, a set of raredisease data items was used as input for the BioPortal Recommender [44]. This is a tool that receivesfree-text or keywords as input and outputs a list of recommended ontologies based on annotations of theinput text. The BioPortal Recommender uses a ranking algorithm that takes into account the coverage(which ontology has the best coverage after annotating the textual input), the acceptance (based on thenumber of visits on BioPortal and the presence of the ontology in the UMLS), the detail of knowledge(how many synonyms, definitions, and properties do the annotations provide), and lastly the specialization(using the position of the annotations in the hierarchy of the ontology). For retrieving the annotationsbased on the input, the Recommender uses the BioPortal Annotator [45]. The input text was extractedfrom the set of common data elements for rare disease registries (the element and coding names) [46], andthe classifications of rare diseases from Orphanet (all categories and one random disease per category)[47]. Items, i.e. one or multiple words, could have multiple annotations. The Recommender was runusing the default configuration and the first two ontologies were selected from the list (SNOMED CT(International Edition release 26-02-2020), NCIt (version 20.02d) [48]). ORDO (version 2.9.1) was thenadded as a third ontology as it specifically targets the rare disease domain.

3.1.2 Module extraction

After selecting the input ontologies, the module extractor included in the OWL API, as explained inSubsection 2.2.1, was used for creating modules based on the original ontologies. The module extractortakes the whole ontology and a seed signature as input, and outputs a module based on the signature. Themodularization of the ontologies firstly served to obtain manageable chunks of data in terms of availablecomputational resources, and secondly to perform the experiments with parts of the ontologies basedon the rare disease data items described above. Each seed signature, one per ontology, contained theannotations of the data elements described in Subsection 3.1.1, as returned by the BioPortal Annotator.To make sure the modules included the entire top-level hierarchy of the original ontology, the seedsignature also included all ancestors of those annotated classes.

3.1.3 Selection of matching systems

This study aimed to include existing ontology matching systems that had a proven history of matchingbiomedical ontologies. Hence, the matching systems were chosen from the list of participants of theOAEI disease and phenotype track [42]. This track focuses on using disease and phenotype ontologiesfor practical use cases, such as data integration. The matching systems were chosen based on if theywere actively developed, published with an open license, and had available source code. Following thosecriteria, three matching systems were selected: AgreementMakerLight 2.0 (AML) [38], FCA-Map [39],and LogMap 2.0 (LogMap) [40]. A description of each system and the underlying matching techniques isgiven in Section 2.4.2.

3.2 Alignments

All matching systems were run with their default configuration and no changes were made to the systems’parameters. The output from the matching systems, the alignments, were saved in the general Alignmentformat of the Alignment API [49]. Each run was assigned 64GB of RAM. The matching systems did notneed any user-input during the matching process, i.e. they provided automated ontology matching. All

Chapter 3. Methods 23

possible ontology pairs were used as input: ORDO-SNOMED CT, NCIt-ORDO, NCIt-SNOMED CT(matching A to B is equivalent to matching B to A). All output alignments contained pairwise equivalencemappings and included the URI of each class. Figure 3.2 shows an example of a mapping between NCItand ORDO.

Figure 3.2. Example mapping of the class Polyplidy between NCIt and ORDO. Shown area chunk of the RDF output from the alignment and a visual representation of the mapping.

3.3 Evaluation

3.3.1 Reference alignments

The alignments produced by the matching systems were evaluated using two reference alignments, basedon mappings from BioPortal and the UMLS Metathesaurus [50]. Both were chosen because they are usedas reference alignments in the OAEI disease and phenotype track and large BioMed track respectively.The BioPortal mappings for ORDO, NCIt, and SNOMED CT are lexical mappings based on the LexicalOWL Ontology Matcher (LOOM). LOOM is a simple string matching algorithm that compares thepreferred names and synonyms of classes in both ontologies. BioPortal mappings were retrieved using theBioPortal API.

Unlike the matching systems and BioPortal, the UMLS does not provide pairwise mappings. Instead,the UMLS groups all classes from all included ontologies with the same meaning. Classes with the samemeaning share a code: the concept unique identifier (CUI). Moreover, a single class can have multipleCUI codes. The UMLS reference alignment was extracted from a subset of the UMLS Metathesaurus(version 2020AA), which was done earlier by [51]. This subset was obtained using the MetamorphoSystool of the UMLS by retrieving the MRCONSO.RRF file. Then, the subset was installed locally usingMySQL Community Server version 5.6.48. The MRCONSO subset contains every concept of the UMLSand specifies the CUI, its language, its code in the source ontology, the term status, and other detailsthat are not relevant for extracting the reference alignment. Pairwise mappings were retrieved by firstgetting all available CUIs for every class in each ontology. Then, all classes of ontology A and B that hadat least one correspondent CUI were included as a mapping in the reference alignment. ORDO is notpresent in the UMLS but does include CUI code mappings as annotations in the ontology. Hence, ORDOCUIs were not retrieved from the UMLS but instead directly from the ontology itself.

Reference alignments for the modules were derived from their full-size counterparts. Namely, mappingscontaining classes which were not present in the module were deleted. Alignments were evaluated byclassifying each mapping as true positive (TP, present in both the alignment as the reference alignment),false positive (FP, only present in the alignment) or false negative (FN, only present in the referencealignment). See Figure 3.3. True negatives were not considered in this study, as there was no goldstandard available that contained 100% of all possible correct mappings.

24 Evaluation

Figure 3.3. Categories for evaluation with BioPortal and UMLS reference alignments, adapted from [26].

Three performance measures were calculated, precision (Equation 3.1), recall (Equation 3.2), andF-measure (F1-score) (Equation 3.3). Precision shows the proportion of mappings in the alignment thatare classified as true positive. Recall shows the proportion of mappings in the reference alignment thatare also present in the alignment. F-measure combines precision and recall by calculating their harmonicmean.

Precision =TP

TP + FP(3.1) Recall =

TP

TP + FN(3.2)

F-measure = 2× precision× recall

precision + recall(3.3)

3.3.2 Hierarchy analysis

For the use case of this study, equivalence mappings containing classes of the same type are of specialinterest. For example, a matching system should detect that two classes are both diseases. Each top-levelhierarchy of an ontology contains classes that are of similar types, and every descendent of a top-levelclass shares the IS-A relationship with its ancestor(s). Biomedical ontologies have a high amount ofinformation in their lexical labels, hence, ontology matching systems often primarily use lexical matchingtechniques [28]. However, a mapping containing classes that originate from different top-level hierarchiescan be incorrect, even if the classes have labels that are lexically similar. For example, two classes eachlabeled as bone fracture where the first is part of the hierarchy clinical finding and the second of bodystructure. Therefore, the content of the alignments was analyzed by comparing the top-level hierarchiesof matched classes. Mappings between the top-level hierarchies of NCIt-ORDO, NCIt-SNOMED CT,and ORDO-SNOMED CT were created manually. This manual creation of top-level hierarchy mappingswas done by inspecting the top-level hierarchies of true positive mappings for the UMLS and UMLS+ BioPortal. The class descriptions were also used to determine whether or not two top-level classesshould be matched. True positives for only BioPortal were not utilized as they were known before to behighly incomplete [42]. Selected mappings for the top-level hierarchies were considered to be semanticallyequivalent. Figure 3.4 shows an example of how such a manual mapping was created.

False positive mappings (not in the UMLS nor BioPortal) were marked as incorrect if their top-levelhierarchy classes were not present in the set of manually created mappings. The precision and F-measurewere recalculated after discarding such incorrect mappings from the alignments.

Chapter 3. Methods 25

Figure 3.4. Example of manually created top-level hierarchy mappings. The four classes from NCIt andSNOMED CT were matched as equivalent by the matching system, all four mappings were present in the refer-ence alignment of the UMLS and BioPortal (true positive). Analyzing the top-level hierarchies revealed that allNCIt classes were descendants of Anatomic Structure, System, or Substance and all SNOMED CT classes ofBody structure. A manual mapping between those top-level classes was then created for NCIt-SNOMED CT.

3.3.3 Rare disease use case

Finally, the aforementioned experiments were applied to a use case from the VASCA registry (Section1.2.1). Alignments for the ISSVA ontology to ORDO, NCIt, and SNOMED CT were created usingAML, LogMap, and FCA-Map. Then, the previously created alignments and their evaluations betweenORDO, NCIT, and SNOMED CT were used to specify mappings in the ISSVA-ORDO alignments astrue mappings. This practical evaluation served two purposes.

1. Determine the overlap between the ISSVA ontology and ORDO/SNOMED CT/NCIt, and evaluatebased on the prior matching and evaluation between ORDO and NCIt + SNOMED CT whichmappings would be marked as true positives. Figure 3.5 shows an example of this process, which isbased on transitivity.

2. The top-level hierarchies of the ISSVA mappings in ORDO, NCIt, and SNOMED CT can be usedby VASCA to evaluate the structure of their ontology and hence improve future iterations.

When the classes a, b, c are considered to be in a set of classes S, with the equivalence relationship R:a, b, c ∈ S, if (a, b) ∈ R and (a, c) ∈ R, then (b, c) ∈ R (Figure 3.5).

Figure 3.5. Example evaluation of an ISSVA-ORDO mapping, using transitivity. Class A is mappedto B in ORDO and C in SNOMED CT (by either one or more matching systems). If a true pos-itive mapping exists between B and C, the mapping A-B is also considered to be true positive.

26 Evaluation

Chapter 4

Results

4.1 Rare disease data elements and modules

A total of 117 data items were extracted from the set of common data elements for rare disease registries andthe Orphanet rare disease classifications (full list included in Appendix A). The BioPortal Recommenderannotated 42% of the input with ORDO classes, 52% with SNOMED CT classes, and 65% with classesfrom NCIt. The seed signatures contained 471 classes for SNOMED CT, 74 for ORDO, and 547 for NCIt.Table 4.1 shows the details of the extracted modules.

Table 4.1. Details of the ontologies and extracted modules.

ORDO (74 classes in seed signature) SNOMED CT (471 classes in seed signature) NCIt (547 classes in seed signature)

Module type Star Bottom Top Star Bottom Top Star Bottom Top

Classes (% of total) 299 (2%) 306 (2%) 13,436 (93%) 1.408 (0.4%) 1,410 (0.4%) 345,169 (98%) 1,014 (0.7%) 1,014 (0.7%) 156,172 (100%)Axioms (% of total) 2,227 (0.9%) 2,288 (1%) 216,295 (92%) 7,105 (0.4%) 7,146 (0.4%) 1,597,246 (98%) 19,017 (0.7%) 19,017 (0.7%) 2,542,770 (99.9%)Object properties (% of total) 7 (39%) 8 (44%) 17 (94%) 16 (13%) 18 (15%) 93 (78%) 40 (41%) 40 (41%) 97 (100%)

Total classes 14,502 352,449 156,172Total axioms 234,982 1,629,354 2,543,710Total object properties 18 120 97

For each ontology, its star and bottom module had the same size. The star and bottom modulescontained between 0.4-2% of the total amount of classes and axioms of the original ontologies. Thetop modules were significantly larger and contained over 90 % of the classes and axioms of the originalontologies. The NCIt top module had the same size as the whole NCIt ontology.

4.2 Alignments

Alignments were created between the ontology pairs ORDO-SNOMED CT, NCIt-ORDO, and NCIt-SNOMED CT. A total of 12 OWL files were used as input for each matching system, four files perontology (three modules, and the whole ontology). Hence, 36 alignments were created (12 per matchingsystem). All alignments contained mappings with an equivalence relationship. In terms of run time, AMLand LogMap were the fastest (NCIt-SNOMED CT full alignment within a few hours), FCA-Map wasslower (6-8 hours for NCIt-SNOMED CT). Table 4.2 shows the number of mappings per alignment, andthe overlap of equivalent mappings between the star/bottom modules and the top module/whole ontology.

The star and bottom module had a similar amount of mappings, as does the top module in comparisonwith the whole ontology. Furthermore, the overlap between all similar-sized alignments, i.e. star/bottomand top/whole ontology, had an overall overlap of almost 100%. Therefore, only results from the starmodules (being the smallest modules) and the whole ontologies will be presented from now onwards. Thefull list of results for the whole ontologies and star modules are included in Appendix B.

27

28 Evaluation

Table 4.2. Details of the created alignments. The alignments for the star modules/bot-tom modules and top modules/whole ontologies are shown together. The percent-

age of equal mappings (overlap, percentage of the smallest alignment that is presentin the other) between the star/bottom and top/whole alignments is also included.

Star / bottom AgreementMakerLight 2.0 FCA-Map LogMap 2.0

Mappings (star) Mappings (bottom) Overlap Mappings (star) Mappings (bottom) Overlap Mappings (star) Mappings (bottom) Overlap

ORDO - SNOMED CT 42 42 100% 46 45 100% 53 53 100%NCIt - ORDO 36 36 100% 47 47 100% 31 31 100%NCIt - SNOMED CT 193 194 99% 220 225 100% 214 215 100%

Top / whole AgreementMakerLight 2.0 FCA-Map LogMap 2.0

Mappings (top) Mappings (whole) Overlap Mappings (top) Mappings (whole) Overlap Mappings (top) Mappings (whole) Overlap

ORDO - SNOMED CT 6,373 6,463 99% 4,784 4,973 99% 3,854 5,742 92%NCIt - ORDO 2,544 2,543 99% 4,556 4,663 98% 2,717 2,679 95%NCIt - SNOMED CT 18,481 18,887 99% 26,078 26,630 99% 23,444 23,885 99%

4.3 Evaluation

4.3.1 Reference alignments: BioPortal and UMLS

Table 4.3 shows the sizes of the reference alignments that were extracted from BioPortal and the UMLS.All reference alignments from the UMLS contained more mappings than the ones from BioPortal. Theoverlap between the NCIt-ORDO and NCIt-SNOMED CT reference alignments was the largest witha weighted overlap of 45% and 57% respectively (whole ontologies). The lowest overlap was 14% (starmodules) and 25% (whole ontologies) for the ORDO-SNOMED CT pair.

Table 4.3. Reference alignments size, BioPortal and UMLS. The overlapand harmonic mean of the overlap between the alignments is also shown.

Ontology pair Ontology type Mappings UMLS Mappings BioPortal Overlap Harmonic mean overlap

ORDO-SNOMED CT Star module 35 7 3 14%NCIt-ORDO Star module 27 18 12 53%NCIt-SNOMED CT Star module 127 90 56 52%ORDO-SNOMED CT Whole ontology 3,861 1,750 776 28%NCIt-ORDO Whole ontology 1,484 1,450 656 45%NCIt-SNOMED CT Whole ontology 19,309 16,290 10,195 57%

4.3.2 Evaluation results

Table 4.4 shows the evaluation results for the whole ontologies. For all ontology pairs, the recall withregard to BioPortal is higher than the recall with regard to the UMLS. The precision with regard to theUMLS is higher for ORDO-SNOMED CT (0.45) than that of BioPortal (0.28). The opposite is true forNCIt-ORDO and NCIt-SNOMED CT. AML had the highest F1-score for BioPortal (0.66), all matchingsystems had an overall higher F1-score for BioPortal than the UMLS. LogMap had a higher precision forthe UMLS (0.47) than for BioPortal (0.45). AML had a higher precision for BioPortal (0.54) than theUMLS (0.47). NCIt-SNOMED CT had the highest recall and precision among all ontology pairs andsystems.

Table 4.5 shows the results for using the star modularization algorithm. The recall for all ontologypairs was higher for BioPortal than for the UMLS, which corresponds to the results of the whole ontologies.The UMLS precision was higher than the BioPortal precision for all ontology pairs. Overall, all matchingsystems had a higher F1-score for the UMLS than for BioPortal.

Chapter 4. Results 29

Table 4.4. Evaluation results for using the whole ontologies, mean precision/recall/F1-scorefor both the UMLS and BioPortal. The scores for the ontology pairs indicate the mean of all

matching systems, the scores for the matching systems indicate the mean of all ontology pairs.

Pair or matching system Precision UMLS Precision BioPortal Recall UMLS Recall BioPortal F1-score UMLS F1-score BioPortal

ORDO - SNOMED CT 0.45 0.28 0.66 0.89 0.53 0.42NCIt - ORDO 0.33 0.44 0.67 0.91 0.43 0.58NCIt - SNOMED CT 0.55 0.67 0.66 0.94 0.60 0.78

AgreementMakerLight 2.0 0.47 0.54 0.66 0.96 0.55 0.66FCA-Map 0.39 0.39 0.64 0.90 0.46 0.53LogMap 2.0 0.47 0.45 0.69 0.88 0.55 0.58

Table 4.5. Evaluation results using the star modularization algorithm, mean precision/recall/F1-score for both the UMLS and BioPortal. The scores for the ontology pairs indicate the mean of

all matching systems, the scores for the matching systems indicate the mean of all ontology pairs.

Pair or matching system Precision UMLS Precision BioPortal Recall UMLS Recall BioPortal F1-score UMLS F1-score BioPortal

ORDO - SNOMED CT 0.45 0.14 0.60 0.95 0.51 0.25NCIt - ORDO 0.49 0.47 0.67 0.96 0.56 0.62NCIt - SNOMED CT 0.51 0.42 0.84 0.98 0.64 0.59

AgreementMakerLight 2.0 0.49 0.37 0.67 0.97 0.57 0.51FCA-Map 0.42 0.31 0.68 1.00 0.52 0.46LogMap 2.0 0.53 0.35 0.77 0.92 0.62 0.49

4.3.3 Hierarchy analysis

The manual top-level hierarchy mappings are shown in Table 4.6. A total of three mappings were createdfor ORDO-SNOMED CT, six mappings for NCIt-ORDO, and 13 mappings for NCIt-SNOMED CT.Table 4.7 shows the results of the analysis. On average 10% (whole ontologies) of the mappings in analignment had classes which top-level hierarchy was not present in the manual top-level mappings set.Mappings that were true positive for either BioPortal and/or the UMLS were not discarded from thealignments. Hence, on average 4.6% of the mappings in the alignments contained classes of which thetop-level hierarchy was not present in the manual mappings set and were false positive. The star modulealignments had an average of 19% of incorrect hierarchy mappings, and 8.7% of the mappings in thealignments were false positives with an incorrect top-level hierarchy.

The results of the whole ontologies for recalculating the precision and F1-score, after discarding falsepositive mappings with an incorrect top-level hierarchy, are shown in Table 4.8. Precision and F1-scorevalues rose between 0.01 and 0.05, for all ontology pairs and matching systems. Except FCA-Map, forwhich the BioPortal precision increased from 0.39 to 0.45 (+0.06). Table 4.9 shows the new precisionand F1-scores for the star modules. The scores rose between 0 and 0.06 points overall, except theNCIt-SNOMED CT precision and F1-score for BioPortal (+0.29 and +0.22 respectively).

30 Evaluation

Table 4.6. Manual top-level hierarchy mappings.

ORDO SNOMED CT

clinical entity Clinical finding (finding)genetic material Substance (substance)geography Environment or geographical location (environment / location)

NCIt ORDO

Disease, Disorder or Finding clinical entityGene Product genetic materialConceptual Entity geographyConceptual Entity inheritanceProperty or Attribute age of onset

NCIt SNOMED CT

Structure, System, or Substance Body structure (body structure)Disorder or Finding Clinical finding (finding)Property or Attribute Qualifier value (qualifier value)Anatomic Structure, System, or Substance Substance (substance)Activity Procedure (procedure)Organism Organism (organism)Drug, Food, Chemical or Biomedical Material Substance (substance)Drug, Food, Chemical or Biomedical Material Pharmaceutical / biologic product (product)Manufactured Object Physical object (physical object)Property or Attribute Observable entity (observable entity)Conceptual Entity Environment or geographical location (environment / location)Conceptual Entity Social context (social concept)Conceptual Entity Observable entity (observable entity)

Table 4.7. Hierarchy analysis results. For each system and ontology pair the amount of map-pings is shown that contain classes of which their top-level hierarchy is not included as man-

ual mapping (Table 4.6). The amount and percentage of false positives (FP) refers to the map-pings that were discarded from the alignment for recalculation of both the precision and F1-score.

Whole ontology Star module

Matching system Ontology pairIncorrect hierarchy mappings(of which FP)

Proportion of totalalignment (FP)

Incorrect hierarchy mappings(of which FP)

Proportion of totalalignment (FP)

AgreementMakerLight 2.0 ORDO-SNOMED CT 494 (318) 8% (5%) 9 (6) 21% (14%)FCA-Map ORDO-SNOMED CT 489 (310) 10% (6%) 11 (8) 24% (17%)LogMap 2.0 ORDO-SNOMED CT 193 (106) 3% (2%) 5 (3) 9% (6%)AgreementMakerLight 2.0 NCIt-SNOMED CT 3,055 (252) 16% (1%) 46 (13) 24% (7%)FCA-Map NCIt-SNOMED CT 6,868 (3,299) 26% (12%) 60 (23) 27% (10%)LogMap 2.0 NCIt-SNOMED CT 3,790 (1,180) 16% (5%) 42 (9) 20% (4%)AgreementMakerLight 2.0 NCIt-ORDO 127 (102) 5% (4%) 4 (1) 11% (3%)FCA-Map NCIt-ORDO 1,229 (1,170) 3% (3%) 12 (8) 26% (17%)LogMap 2.0 NCIt-ORDO 130 (92) 5% (3%) 3 (0) 10% (0%)

Table 4.8. Evaluation results using the whole ontologies after removing false positive mappings withan incorrect top-level hierarchy, mean precison/F1-score for both the UMLS and BioPortal. The

scores for the ontology pairs indicate the mean of all matching systems, the scores for the matchingsystems indicate the mean of all ontology pairs. Recall has not changed and is therefore not shown.

Pair or matching system Precision UMLS Precision BioPortal F1-score UMLS F1-score BioPortal

ORDO - SNOMED CT 0.47 (+0.02) 0.29 (+0.01) 0.55 (+0.02) 0.44 (+0.02)NCIt - ORDO 0.36 (+0.03) 0.48 (+0.04) 0.46 (+0.03) 0.62 (+0.04)NCIt - SNOMED CT 0.59 (+0.04) 0.71 (+0.04) 0.62 (+0.02) 0.81 (+0.03)

AgreementMakerLight 2.0 0.49 (+0.02) 0.56 (+0.02) 0.56 (+0.01) 0.68 (+0.02)FCA-Map 0.44 (+0.05) 0.45 (+0.06) 0.51 (+0.05) 0.59 (+0.06)LogMap 2.0 0.48 (+0.01) 0.47 (+0.02) 0.56 (+0.01) 0.60 (+0.02)

Chapter 4. Results 31

Table 4.9. Evaluation results star modules after removing false positive mappings with an incor-rect top-level hierarchy, mean precision/F1-score for both the UMLS and BioPortal. The scores

for the ontology pairs indicate the mean of all matching systems, the scores for the matching sys-tems indicate the mean of all ontology pairs. Recall has not changed and is therefore not mentioned.

Pair or matching system Precision UMLS Precision BioPortal F1-score UMLS F1-score BioPortal

ORDO - SNOMED CT 0.51 (+0.06) 0.16 (+0.02) 0.55 (+0.04) 0.28 (+0.03)NCIt - ORDO 0.52 (+0.03) 0.50 (+0.03) 0.58 (+0.02) 0.65 (+0.02)NCIt - SNOMED CT 0.59 (+0.08) 0.71 (+0.29) 0.62 (+0.02) 0.81 (+0.22)

AgreementMakerLight 2.0 0.54 (+0.05) 0.39 (+0.02) 0.59 (+0.02) 0.52 (+0.01)FCA-Map 0.49 (+0.07) 0.37 (+0.06) 0.57 (+0.05) 0.52 (+0.06)LogMap 2.0 0.55 (+0.02) 0.36 (+0.01) 0.64 (+0.02) 0.49 (+0.00)

4.3.4 Rare disease use case

Table 4.10 shows the results of aligning the ISSVA ontology to ORDO, NCIt, and SNOMED CT. Thetotal coverage, i.e. the percentage of ISSVA classes that has been mapped to one or more classes in theother ontology, is 44% for ORDO, 35% for NCIt, and 36% for SNOMED CT. These percentages representthe union of the alignments of all matching systems.

Table 4.10. Alignments by AgreementMakerLight 2.0, FCA-Map, and LogMap 2.0 for the ISSVAontology to ORDO, NCIt, and SNOMED CT. Shown are the number of mappings in each align-

ment, the number of unique ISSVA classes in the alignment, and the percentage of the total amountof ISSVA classes that is present in the alignment. The total coverage is the number of uniqueISSVA classes over all systems divided by the total number of classes in the ISSVA ontology.

AgreementMakerLight 2.0 FCA-Map LogMap 2.0

Ontology pairISSVAcoverage

Total mappings(unique)

ISSVAcoverage


ISSVAcoverage


Total coverage

ISSVA - ORDO 23% 43 (40) 35% 77 (61) 15% 26 (26) 44% (75/172)ISSVA - NCIt 19% 37 (33) 30% 58 (52) 22% 37 (37) 35% (61/172)ISSVA - SNOMED CT 23% 52 (40) 31% 64 (53) 24% 45 (42) 36% (62/172)

Table 4.11 shows the precision per matching system for the ISSVA-ORDO and ISSVA-NCIt alignments.The ISSVA-ORDO alignment had a mean precision of 0.19 for its evaluation with the NCIt-ORDOalignment, meaning that on average 19% of the mappings were flagged as true positive. The ISSVA-NCItalignment had the same mean precision. The ISSVA-ORDO alignments contained 48 mappings on average,whereas the ISSVA-NCIt alignments contained 44 mappings. Between 10 and 13 mappings found in theISSVA-ORDO or ISSVA-NCIt alignments were also present in the NCIt-ORDO alignment.

Table 4.11. ISSVA-ORDO and ISSVA-NCIt results (NCIt-ORDO as reference alignment).

Matching systemPrecision ISSVA-ORDO(true positive mappings/total)

Precision ISSVA-NCIt(true positive mappings/total)

False positiveTrue positiveUMLS and/or BioPortal

Correct hierarchy

AgreementMakerLight 2.0 0.19 (8/43) 0.22 (8/37) 2 out of 10 8 out of 10 10 out of 10FCA-Map 0.12 (9/77) 0.16 (9/58) 4 out of 13 9 out of 13 13 out of 13LogMap 2.0 0.27 (7/26) 0.19 (7/37) 2 out of 11 7 out of 11 11 out of 11

Mean precision: 0.19 0.19

Table 4.12 shows the results for the ISSVA-ORDO and ISSVA-SNOMED CT alignments, evaluatedusing the ORDO-SNOMED CT alignment. The ISSVA-ORDO alignment had a mean precision of 0.14,the ISSVA-SNOMED CT alignment a mean precision of 0.11. The ISSVA-ORDO and ISSVA-SNOMEDCT alignments contained 49 and 54 mappings on average respectively. Between 10 and 13 mappings werealso present in the ORDO-SNOMED CT alignment.

Table 4.13 shows the results for the ISSVA-NCIt and ISSVA-SNOMED CT alignments, evaluatedusing the NCIt-SNOMED CT alignment. The ISSVA-NCIt alignment had a precision of 0.25, the

32 Evaluation

Table 4.12. ISSVA-ORDO and ISSVA-SNOMED CT (ORDO-SNOMED CT as reference alignment).

Matching systemPrecision ISSVA-ORDO(true positive mappings/total)

Precision ISSVA-SNOMED CT(true positive mappings/total)


Correct hierarchy



ISSVA-SNOMED CT alignment had a precision of 0.33. On average, the alignments contained 44 and 54mappings respectively. Between 13 and 21 mappings were present in the NCIt-SNOMED CT alignment.

Table 4.13. ISSVA-NCIt and ISSVA-SNOMED CT (NCIt-SNOMED CT as reference alignment).

Matching systemPrecision ISSVA-NCIt(true positive mappings/total)

Precision ISSVA-SNOMED CT(true positive mappings/total)


Correct hierarchy



Chapter 5

Discussion and conclusion

This study analyzed the performance of existing ontology matching systems using reference alignmentsfrom the UMLS and BioPortal. Besides, it studied the top-level hierarchies of mappings using manuallycreated mappings between top-level classes. Finally, the matching systems were applied to a practicaluse case from the ISSVA classification ontology. This study wanted to contribute to solving the problemof making annotated data interoperable when multiple ontologies are used. It specifically targeted thisproblem in the context of rare disease research, where projects like the EJP RD focus on the integrationand interrogation of data over multiple sources. The study’s principal findings are discussed first, followedby its strengths and weaknesses, and the relation of this study to other work in the field. Finally, futureresearch is discussed followed by the implications and conclusion of this study.

5.1 Principal findings

What is the performance of automated ontology matching techniques to expose mappingsbetween ontologies used in the rare disease research domain?Three ontology matching systems (AML, FCA-Map, LogMap) were evaluated against reference alignmentsfrom the UMLS and BioPortal using three biomedical ontologies (NCIt, ORDO, SNOMED CT). Thesystems exposed, on average, 5.726 mappings between ORDO-SNOMED CT, 3.295 mappings betweenNCIt-ORDO, and 23.134 mappings between NCIt-SNOMED CT. AML had F1-scores of 0.55 (UMLS)and 0.66 (BioPortal), FCA-Map had F1-scores of 0.46 (UMLS) and 0.53 (BioPortal), and LogMap hadF1-scores of 0.55 (UMLS) and 0.58 (BioPortal). Comparing the results between systems may not besignificant, as results may differ for other ontologies or different samples from the current ontologies.Hence, no conclusions can be drawn about whether one system outperformed the other.

The results of the OAEI 2019 Large BioMed track (SNOMED CT-NCIt task) are the most comparableto this work, as the other tracks and tasks use other ontologies. For the SNOMED CT-NCIt task, usingan UMLS-based reference alignment (inconsistent mappings were flagged to be ignored), AML had anF1-score of 0.69, FCA-Map 0.65, and LogMap 0.70. These are better results, although its not a 1:1comparison. Overall, the systems are considered to be good performing candidates when applied to therare disease domain. The systems’ mappings can be combined into one alignment, hence, the practicalapplication is not limited to the use of a single system. The combination of alignments from differentmatching systems potentially improves performance, as likely more mappings are discovered when systemsuse different matching techniques. Additionally, determining performance for a practical applicationshould also be based on other factors such as usability, speed, and the desired precision/recall.

To what extent are currently available techniques for ontology matching useful for imple-mentation in FAIR-related projects that focus on the integration of multiple data sources?

All systems were able to generate alignments without intervention from the user, which is important for

33

34 Strengths and limitations

the on-the-fly matching of ontologies. Run times varied between minutes up to a few hours, depending onthe input size. The matching systems were also able to expose equivalence relationships between classesof the different ontologies, which allow for the integration of data. This means that the application ofthese systems in practice would be a sensible decision. Moreover, the systems accept the input ontologiesin OWL-format and can export the alignments as a machine-readable RDF-file. This allows to joinalignments from several matching systems. In addition, analyzing the top-level hierarchies of matchedclasses was shown to be effective to reveal mappings with classes from the same hierarchy. For instance,mappings with classes that are both ‘diseases’ can be exploited for answering questions like ‘Which diseaseshave a prevalence higher than 1:1000?’. All systems and techniques can be used in situations where noreference alignments are available. The use of modularization helps for faster development and testingof matching techniques, in comparison to working with the whole ontologies. Using modules instead ofthe whole ontologies could be considered if speed or resources are important factors. As modularizationremoves content from the ontologies it should be noted that this could limit the results of matchingsystems that use structural matching techniques, although this study did not test this hypothesis. Thestar/bottom modules and top module/whole ontologies were similar in size and content.

How can ontology matching be used in practice while creating new and using existing on-tologies to create machine-readable data, with regard to term overlap and reuse?The ISSVA classification was used as a practical use case in this study. The created ISSVA ontologywas matched to NCIt (35% coverage), ORDO (44% coverage), and SNOMED CT (36% coverage). Theevaluation using transitivity explored a way to automatically evaluate alignments between new andexisting ontologies, as the nature of this task means there are no existing (reference) alignments available.Such alignments can be used by the domain experts and ontology engineers to improve their ontologyand to expose classes that can be reused from other ontologies. The latter would avoid term overlap andimprove reuse.

5.2 Strengths and limitations

Several strengths and limitations can be identified for this study. First of all, this study approachedontology matching from a practical point of view of the rare disease domain. This approach benefitsfrom its hands-on usage of ontology matching (systems) and their application to the ISSVA classificationuse case. Furthermore, the used matching systems had a proven track record at the OAEI and wereproven to be effective matching systems in specific matching tasks with curated reference alignments.AML, FCA-Map, and LogMap are not the only matching systems available, however, they cover most ofthe matching techniques as specified by the classification model (Table 2.1) defined by [26]. Likewise,other ontologies exist that would be of use in the rare disease domain, especially considering the largenumber of ontologies present in, among other repositories, BioPortal. Nonetheless, only NCIt, ORDO,and SNOMED CT were chosen for this study due to practical limitations. The BioPortal Annotator andRecommender were used to select the ontologies and create the seed signatures, but other similar toolingcould also be of use.

The top, star, and bottom module extraction using seed signatures based on rare disease data elementsmade it easier to assess the mappings manually while performing the experiments. However, since the listof rare disease data elements was not validated, it was not possible to extract any additional conclusionsfrom the modules versus whole ontology results. For instance, whether or not the classes contained in themodules were to be used more often in the rare disease domain.

Evaluation with BioPortal and UMLS reference alignments

Precision, recall, and F1-score were used as performance measures for the evaluation with the referencealignments and provided uniform metrics for comparing the results of the different matching systemsand ontologies. Those measures were chosen because they are widely known and used by the OAEI.

Chapter 5. Discussion and conclusion 35

However, both precision and recall introduce a problem when used in the context of ontology matchingthat should be mentioned. As stated by [52], both precision and recall are set-theoretic measures thatdo not discriminate between mappings that may be semantically equivalent but not identical. Thus,when a mapping is not present in the reference alignment it is per definition considered to be incorrect(false positive). Semantic precision and recall would solve this problem by considering mappings that are,semantically speaking, close to a mapping in the reference alignment. For example, when a mappings’class is a super- or subclass of a correspondent class in the reference alignment.

The BioPortal mappings were considered to be a baseline alignment, as previously mentioned bythe OAEI [42]. The precision and recall for BioPortal were both the highest for AML (0.54 and 0.96respectively, whole ontology). This corresponds to the fact that both AML and BioPortal (LOOM) basetheir mappings on lexical techniques only. On the other hand, the mappings derived from the UMLS wereconsidered to be a silver standard since the UMLS is being maintained by domain experts. A limitationof the UMLS reference alignment used for this study is the CUI codes from ORDO, as ORDO is notincluded in the UMLS those CUIs were extracted from ORDO itself. The BioPortal F1-scores for the starmodules were 15% lower on average, which could be due to the low number of mappings in the referencealignments. Finally, earlier research mentioned that the UMLS reference alignment contains incoherentmappings [51], which are mappings that contain logical errors following from the union of the inputontologies and the mappings [53]. Moreover, logical incoherences were also found in BioPortal mappings[54]. Such mappings were not removed and/or examined during the evaluation performed in this study.

Hierarchy analysis

The hierarchy analysis provided results that are of interest considering the scope of this study. Table4.7 in Section 4.3.3 shows that on average 10% of mappings in the alignments had an incorrect top-levelhierarchy, meaning that on average 90% were mappings which top-level hierarchies matched based on themanual mappings. The precision was not much improved (up to 0.06 points higher) by discarding falsepositive mappings from the alignments that did not have a top-level hierarchy listed in the set of manualtop-level mappings. Nonetheless, analyzing the top-level hierarchies of matched classes can be of valuewhen applying ontology matching in practice. First of all, hierarchies can exploit information about theorigin of a class. E.g. if ‘pneumonia’ and ‘asthma’ are both part of ‘diseases’ they can be classified assuch, even if it remains unknown if the classes themselves can be used interchangeably due to the lackof a reference alignment. This can be useful when querying data over multiple sources (example Figure1.2). Additionally, some classes were present in multiple mappings (one class mapped to multiple otherclasses), whereas in such cases the hierarchy analysis was able to detect incorrect mappings.

A limitation of this analysis is the manually created top-level mappings, as they might be incomplete.However, the performed experiments have shown that the hierarchies expose information about thecorrectness of a mapping. Table 5.1 shows four different potentially incorrect mappings (as returned bythe matching systems) and their top-level hierarchies. The first example mapping in Table 5.1 regardsSoft tissue and Disorder of soft tissue, the first refers to the anatomic structure of soft tissue, the secondrefers to a disorder of this soft tissue. The second example is Aneurysmal Bone Cyst and Abneurysmalbone cyst, in which the labels are lexically identical. However, the first refers to the disease and thesecond to the body structure. The fourth mappings is an example where Cell Proliferation is matched toHyperplasia, the top-level hierarchies reveal that the first class is a Biological Process and the second aBody structure. Now, this example amplifies the importance of having domain experts evaluate mappings.Hyperplasia is the result of cell proliferation [55], thus the mapping could be considered correct dependingon the application. In this study, only equivalence mappings were considered of interest.

During the manual exploration of the alignments, mappings were found that were true positive foreither the BioPortal or UMLS reference alignment, but with an incorrect hierarchy. These mappings werenot flagged as incorrect because they were included in either one of the reference alignments. Yet thosemappings could reveal incorrect mappings in the reference alignment, which is out of the scope of thiswork.

36 Relation to other work

Table 5.1. Four examples of mappings (NCIt-SNOMED CT)that are potentially incorrect based on their top-level hierarchies.

# Label class A Label class B Top level hierarchy class A Top level hierarchy class B

1 Soft tissue Disorder of soft tissue (disorder) Anatomic Structure, System, or Substance Clinical finding (finding)2 Aneurysmal Bone Cyst Aneurysmal bone cyst (morphologic abnormality) Disease, Disorder or Finding Body structure (body structure)3 Abnormality Abnormal (qualifier value) Disease, Disorder or Finding Qualifier value (qualifier value)4 Cell Proliferation Hyperplasia (morphologic abnormality) Biological Process Body structure (body structure)

Application to the ISSVA classification use case

The practical use case provided a novelty to this study by applying the matching systems to a real-lifeapplication. Namely, making a classification for vascular anomalies machine-readable, following the FAIRdata principles. The results of matching the ISSVA ontology to NCIt, ORDO, and SNOMED CT revealedthat combining the alignments from all three matching systems can provide information on how muchoverlap the ISSVA ontology has with existing ontologies. Classes that did not overlap with ORDO, NCIt,or SNOMED CT would probably be caused by their content to be very specific to the vascular anomaliescontext. The determination of true positive mappings was limited, both due to the same limitations asthe BioPortal/UMLS evaluation and the use of transitivity. However, it allowed for the mappings tobe evaluated automatically and its alignments can be used by domain experts to improve the ontologyand/or decide on using an existing ontology instead due to overlap.

5.3 Relation to other work

The evaluation of the ontology matching systems relates to earlier research on matching disease andphenotype ontologies, and large biomedical ontologies (Large BioMed), both of which are tracks ofthe OAEI [42, 29]. In addition to the BioPortal baseline reference alignments, a consensus alignmentbased on a voting mechanism (multiple systems returning the same mapping), and manually curatedmappings, were used to evaluate the matching systems. The large biomedical ontologies track of theOAEI uses the UMLS as reference alignment, which is based [51] where the authors extracted pairwisemappings from the UMLS. Moreover, this study relates to a paper published in 2020 which presented ageneric workflow for making data FAIR [7]. Step four of this workflow describes the process of defininga semantic (meta)data model, whereas step five describes the process of making (meta)data linkable.Both steps correspond to step three (define the semantic model) and four (make data linkable) of theFAIRification process presented in Figure 1.1 [6]. Annotating data using an ontology can be a challengingprocess and will sometimes require one to add classes to an existing ontology or create a new ontology(e.g. the ISSVA classification use case). Consequently, making (meta)data linkable concerns also theinteroperability of different ontologies, to which automated ontology matching could provide a solution.The main contribution of this work, concerning earlier research about FAIR data and ontology matching,is the exploration of how available ontology matching systems and techniques can be of use for creatinginteroperable data.

5.4 Future research

A problem that has not yet been solved is how mappings can be evaluated as useful or correct for theirspecific application. The (automatic) evaluation of mappings for usage in an on-the-fly matching servicewithin a FAIR data environment will be an important task. Therefore, future research should focus ondeveloping methods for evaluating mappings that can be used by such matching services. Moreover,situations where no reference alignments are available should be considered. Those developments shouldbe driven by multiple specific use cases. In addition, the ISSVA use case raised new questions on howontology matching can be used for creating a semantic model. In the case of the ISSVA ontology, theresults should be reported to domain experts for improving the ontology and matching approaches.

Chapter 5. Discussion and conclusion 37

5.5 Implications of this study and conclusion

This study focused on matching three ontologies (NCIt, ORDO, SNOMED CT), which content relatesto the rare disease domain. The ontologies were matched using three state-of-the-art matching systems(AgreementMakerLight 2.0, FCA-Map, LogMap 2.0). All systems were able to automatically producemeaningful mappings using different ontology matching techniques. Challenges lie ahead for choosingthe right mappings for the right application, which should be solved by use case driven solutionsfocusing on creating on-the-fly matching services. The analysis of top-level hierarchies and the evaluationwith universal reference alignments (BioPortal and the UMLS) are methods that seem promising forimplementation in such services. All of the above contributes to step four of the FAIRification process(making data linkable). This study concludes that currently available ontology matching systems aresuitable for implementation in FAIR-related projects, although considering that work has to be doneregarding the automatic selection of useful mappings. The ISSVA use case shows that ontology matchingalso has potential for usage during step three of the FAIRification process (defining the semantic model),assuming that the ontologies themselves also adhere to the FAIR principles.

38 Implications of this study and conclusion

References

[1] European Commission High Level Expert Group on Scientific Data. Riding the Wave: How Europecan gain from the rising tide of scientific data; 2010.

[2] Directorate-General for Research and Innovation (European Commission). Guidelines to the Ruleson Open Access to Scientific Publications and Open Access to Research Data in Horizon 2020; 2017.March.

[3] Digital curation Center. DMPonline; 2020. Available from: https://dmponline.dcc.ac.uk.

[4] Science Europe. Practical Guide To the International Alignment of Research Data Management;2018. Available from: www.scienceeurope.org.

[5] Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, et al. Comment: TheFAIR Guiding Principles for scientific data management and stewardship. Scientific Data. 2016;3:1–9.

[6] GO FAIR Initiative. FAIRification Process; 2020. Available from: https://www.go-fair.org/

fair-principles/fairification-process/.

[7] Jacobsen A, Kaliyaperumal R, Bonino da Silva Santos L, Mons B, Schultes E, Roos M, et al. AGeneric Workflow for the Data FAIRification Process. Data Intelligence. 2020;2:56–65.

[8] Guarino N, Oberle D, Staab S. Handbook on Ontologies. Handbook on Ontologies. 2009;(May2009):0–17.

[9] Whetzel PL, Noy NF, Shah NH, Alexander PR, Nyulas C, Tudorache T, et al. BioPortal: Enhancedfunctionality via new Web services from the National Center for Biomedical Ontology to access anduse ontologies in software applications. Nucleic Acids Research. 2011;.

[10] SNOMED International. SNOMED CT; 2020. Available from: http://www.snomed.org.

[11] Vasant D, Chanas L, Malone J, Hanauer M, Olry A, Jupp S, et al. ORDO: An Ontology ConnectingRare Disease, Epidemiology and Genetic Data. Phenotype data at ISMB2014. 2014;.

[12] Dzale Yeumo E, Alaux M, Arnaud E, Aubin S, Baumann U, Buche P, et al. Developing datainteroperability using standards: A wheat community use case. F1000Research. 2017;6:1843. Availablefrom: https://f1000research.com/articles/6-1843/v1.

[13] Harrow I, Balakrishnan R, Jimenez-Ruiz E, Jupp S, Lomax J, Reed J, et al. Ontology mapping forsemantically enabled applications. Drug Discovery Today. 2019;.

[14] Dutch Techcentre of Life Sciences. Personal Health Train;. Available from: https://www.dtls.nl/fair-data/personal-health-train/.

[15] European Commission. European Joint Programme on Rare Diseases;. Available from: https:

//www.ejprarediseases.org/.

[16] Otero-Cerdeira L, Rodrıguez-Martınez FJ, Gomez-Rodrıguez A. Ontology matching: A literaturereview. Expert Systems with Applications. 2015;42(2):949–971. Available from: http://dx.doi.

org/10.1016/j.eswa.2014.08.032.

39

https://dmponline.dcc.ac.uk

www.scienceeurope.org

https://www.go-fair.org/fair-principles/fairification-process/

https://www.go-fair.org/fair-principles/fairification-process/

http://www.snomed.org

https://f1000research.com/articles/6-1843/v1

https://www.dtls.nl/fair-data/personal-health-train/

https://www.dtls.nl/fair-data/personal-health-train/

https://www.ejprarediseases.org/

https://www.ejprarediseases.org/

http://dx.doi.org/10.1016/j.eswa.2014.08.032

http://dx.doi.org/10.1016/j.eswa.2014.08.032

40 REFERENCES

[17] Kamdar MR, Tudorache T, Musen MA. A systematic analysis of term reuse and term overlap acrossbiomedical ontologies. Semantic Web. 2017;8(6):853–871.

[18] Quesada-Martınez M, Tomas Fernandez-Breis J. Studying the Reuse of Content in BiomedicalOntologies: An Axiom-Based Approach. In: Conference on Artificial Intelligence in Medicine inEurope, AIME 2017. vol. 10259. Springer, Cham; 2017. p. 3–13.

[19] European Commission. European Reference Networks; 2017. Available from: https://ec.europa.eu/health/ern_en.

[20] VASCERN. European Reference Network on Rare Multisystemic Vascular Diseases; 2020.

[21] International Society for the Study of Vascular Anomalies. ISSVA Classification of Vascular Anomalies;2018. Available from: issva.org/classification.

[22] Berners-Lee T, Hendler J, Lassila O. The semantic web. Scientific American. 2001;p. 279.

[23] World Wide Web Consortium. Semantic Web W3C; 2020. Available from: https://www.w3.org/standards/semanticweb/.

[24] Horridge M, Bechhofer S. The OWL API: A Java API for OWL ontologies. Semantic Web.2011;2(1):11–21.

[25] Bodenreider O. The Unified Medical Language System (UMLS): Integrating biomedical terminology.Nucleic Acids Research. 2004;(D267-70).

[26] Euzenat J, Shvaiko P. Ontology matching. Second edi ed.; 2007.

[27] Horridge M, Parsia B, Sattler U. The state of bio-medical ontologies. Bio-Ontologies 2011. 2011;.

[28] Faria D, Pesquita C, Mott I, Martins C, Couto FM, Cruz IF. Tackling the challenges of matchingbiomedical ontologies. Journal of Biomedical Semantics. 2018;9(1):1–19.

[29] Jimenez-Ruiz E, Agibetov A, Samwald M, Cross V. Breaking-down the Ontology Alignment Task witha Lexical Index and Neural Embeddings; 2018. Available from: http://arxiv.org/abs/1805.12402.

[30] Cruz IF, Antonelli FP, Stroe C. AgreementMaker: Efficient matching for large real-world schemasand ontologies. Proceedings of the VLDB Endowment. 2009;2(2):1586–1589.

[31] Faria D, Pesquita C, Santos E, Palmonari M, Cruz IF, Couto FM. The AgreementMakerLightOntology Matching System. Springer Berlin Heidelberg. 2013;p. 527–541.

[32] Laadhar A, Ghozzi F, Megdiche I, Ravat F, Teste O, Gargouri F. POMap: An effective pairwiseontology matching system. IC3K 2017 - Proceedings of the 9th International Joint Conference onKnowledge Discovery, Knowledge Engineering and Knowledge Management. 2017;2(Keod):161–168.

[33] Gulic M, Vrdoljak B, Banek M. CroMatcher: An ontology matching system based on automatedweighted aggregation and iterative final alignment. Journal of Web Semantics. 2016;41:50–71.Available from: http://dx.doi.org/10.1016/j.websem.2016.09.001.

[34] Peng Y, Munro P, Mao M. Ontology mapping neural network: An approach to learning and inferringcorrespondences among ontologies. CEUR Workshop Proceedings. 2010;658:65–68.

[35] Nkisi-Orji I, Wiratunga N, Massie S, Hui KY, Heaven R. Ontology alignment based on wordembedding and random forest classification. Lecture Notes in Computer Science (including subseriesLecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2019;11051 LNAI:557–572. Available from: http://dx.doi.org/10.1007/978-3-030-10925-7_34.

[36] Santos E, Faria D, Pesquita C, Couto FM. Ontology alignment repair through modularization andconfidence-based heuristics. PLoS ONE. 2015;10(12):1–19.

[37] Jimenez-Ruiz E, Cuenca Grau B. LogMap: Logic-based and scalable ontology matching. LectureNotes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and LectureNotes in Bioinformatics). 2011;7031 LNCS(PART 1):273–288.

https://ec.europa.eu/health/ern_en

https://ec.europa.eu/health/ern_en

issva.org/classification

https://www.w3.org/standards/semanticweb/

https://www.w3.org/standards/semanticweb/

http://arxiv.org/abs/1805.12402

http://dx.doi.org/10.1016/j.websem.2016.09.001

http://dx.doi.org/10.1007/978-3-030-10925-7_34

REFERENCES 41

[38] Faria D, Pesquita C, Santos E, Cruz IF, Couto FM. AgreementMakerLight 2.0: Towards efficientlarge-scale ontology matching. CEUR Workshop Proceedings. 2014;1272:457–460.

[39] Zhao M, Zhang S, Li W, Chen G. Matching biomedical ontologies based on formal concept analysis.Journal of Biomedical Semantics. 2018;9(1):1–27.

[40] Jimenez-ruiz E, Grau BC, Zhou Y. LogMap 2.0: towards logic-based, scalable and interactiveontology matching. Nature Precedings. 2011;p. 2–3.

[41] Jimenez-Ruiz E. OAEI (Ontology Alignment Evaluation Initiative); 2020. Available from: http:

//oaei.ontologymatching.org/.

[42] Harrow I, Jimenez-Ruiz E, Splendiani A, Romacker M, Woollard P, Markel S, et al. Matching diseaseand phenotype ontologies in the ontology alignment evaluation initiative. Journal of BiomedicalSemantics. 2017;8(1):1–13.

[43] R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria; 2020.Available from: https://www.r-project.org/.

[44] Martınez-Romero M, Jonquet C, O’Connor MJ, Graybeal J, Pazos A, Musen MA. NCBO OntologyRecommender 2.0: An enhanced approach for biomedical ontology recommendation. Journal ofBiomedical Semantics. 2017;8(1):1–22.

[45] Shah NH, Bhatia N, Jonquet C, Rubin D, Chiang AP, Musen MA. Comparison of concept recognizersfor building the open biomedical annotator. BMC Bioinformatics. 2009;10(SUPPL. 9):1–9.

[46] European Commission. Set of common data elements for Rare Diseases Registration; 2019. Availablefrom: https://eu-rd-platform.jrc.ec.europa.eu/set-of-common-data-elements_en.

[47] Orphanet. Rare diseases and classifications; 2020. Available from: http://www.orphadata.org/

cgi-bin/rare_free.html.

[48] Fragoso G, de Coronado S, Haber M, Hartel F, Wright L. Overview and utilization of the NCIThesaurus; 2004.

[49] World Wide Web Consortium. Alignment API; 2014. Available from: https://www.w3.org/2001/sw/wiki/Alignment_API.

[50] US National Library of Medicine. UMLS Metathesaurus; 2020. Available from: https://www.nlm.nih.gov/research/umls/index.html.

[51] Jimenez-Ruiz E, Grau BC, Horrocks I, Berlanga R. Logic-based assessment of the compatibilityof UMLS ontology sources. Journal of Biomedical Semantics. 2011;2(1):S2. Available from: http://www.jbiomedsem.com/content/2/S1/S2.

[52] Euzenat J. Semantic precision and recall for ontology alignment evaluation. IJCAI InternationalJoint Conference on Artificial Intelligence. 2007;p. 348–353.

[53] Jimenez-Ruiz E, Meilicke C, Grau BC, Horrocks I. Evaluating mapping repair systems with largebiomedical ontologies. CEUR Workshop Proceedings. 2013;1014:1000–1010.

[54] Faria D, Jimenez-Ruiz E, Pesquita C, Santos E, Couto FM. Towards annotating potential incoherencesin bioportal mappings. Lecture Notes in Computer Science (including subseries Lecture Notes inArtificial Intelligence and Lecture Notes in Bioinformatics). 2014;8797:17–32.

[55] Wikimedia Foundation. Hyperplasia; 2020. Available from: https://en.wikipedia.org/wiki/

Hyperplasia.

http://oaei.ontologymatching.org/

http://oaei.ontologymatching.org/

https://www.r-project.org/

https://eu-rd-platform.jrc.ec.europa.eu/set-of-common-data-elements_en

http://www.orphadata.org/cgi-bin/rare_free.html

http://www.orphadata.org/cgi-bin/rare_free.html

https://www.w3.org/2001/sw/wiki/Alignment_API

https://www.w3.org/2001/sw/wiki/Alignment_API

https://www.nlm.nih.gov/research/umls/index.html

https://www.nlm.nih.gov/research/umls/index.html

http://www.jbiomedsem.com/content/2/S1/S2

http://www.jbiomedsem.com/content/2/S1/S2

https://en.wikipedia.org/wiki/Hyperplasia

https://en.wikipedia.org/wiki/Hyperplasia

42 REFERENCES

List of abbreviations

AML AgreementMakerLight 2.0

CUI Concept Unique Identifier

EJP RD European Joint Programme on Rare Diseases

FAIR Findable, Accessible, Interoperable, and Reusable

FCA Formal Concept Analysis

NCIt National Cancer Institute thesaurus

OAEI Ontology Alignment Evaluation Initiative

ORDO Orphanet Rare Disease Ontology

SNOMED CT SNOMED Clinical Terms

UMLS Unified Medical Language System

43

44 LIST OF ABBREVIATIONS

Appendices

45

Appendix A

Rare disease data items

Table A.1. Rare disease data items, 117 in total. Item are extracted from the commondata elements for rare diseases [46], and the Orphanet rare disease classifications [47]

.Data items 1-31 Data items 32-62 Data items 63-93 Data items 94-117

Pseudonym Consent to the reuse of data Thoracic malformation Rare Infectious DiseasesPersonal information Biological sample Rare Urogenital Diseases CholeraDate of birth Link to a biobank Urogenital tract malformation Rare IntoxicationsDate Biobank Rare Surgical Thoracic Diseases Radiation myelitis

Female Disability Thoracic outlet syndromeRare Gynaecological AndObstetric Diseases

Male Classification of functioning Rare Skin Diseases Vaginal carcinomaFoetus Classification of disability Ichthyosis Rare Surgical Maxillo-facial DiseasesSex Disability profile Rare Renal Diseases Cleft palatePatient status Disability score Multicystic dysplastic kidney Rare Allergic DiseaseAlive Rare diseases Rare Eye Diseases Acquired angioedemaDead Rare Cardiac Diseases Retinoblastoma Rare Teratologic DisordersLost in follow-up Rare cardiomyopathy Rare Endocrine Diseases Infectious embryofetopathy

Opted-outRare Developmental AnomaliesDuring Embryogenesis

Neuroendocrine neoplasmChromosomal AnomaliesSorted By Chromosomes

Opt-out Hydrops fetalis Rare Haematological Diseases Polyploidy

Date of death Rare Cardiac Malformations MastocytosisRare Rheumatologic DiseasesOf Childhood

Care pathway Congenital pericardium anomaly Rare Immunological Diseases Kawasaki diseaseFirst contact withspecialised centre

Rare Sucking Swallowing Disorders Graft versus host diseaseRare Disorders PotentiallyIndicated For Transplant

Disease history Stickler syndromeRare Systemic AndRhumatological Diseases

Systemic primary carnitinedeficiency

Age at onset Rare Inborn Errors Of Metabolism Hereditary angioedema PrevalenceAntenatal MPI-CDG Rare Odontological Diseases Cases/familiesAt birth Rare Gastroenterological Diseases Bruck syndrome CaseAge at diagnosis Eosinophilic gastroenteritis Rare Circulatory System Diseases WorldwideDiagnosis Rare Genetic Diseases Congenital renal artery stenosis ValidatedDiagnosis of the rare disease Noonan syndrome Rare Bone Diseases GeographicGenetic diagnosis Rare Neurological Diseases Aneurysmal bone cyst

Undiagnosed case Spinal cord injuryRare OtorhinolaryngologicalDiseases

Phenotype Rare Abdominal Surgical Diseases Familial nasal aciliaGenotype Adenoma of pancreas Rare InfertilityResearch Rare Hepatic Diseases TuberculosisPatient permission Rare vascular liver disease Rare Neoplastic DiseasesAgreement to be contactedfor research purposes

Rare Respiratory Diseases Germ cell tumor

47

48

Appendix B

Full results BioPortal and UMLS

Table B.1. AgreementMakerLight 2.0

Matching system Ontology pair Ontology type Precision UMLS Precision BioPortal Recall UMLS Recall BioPortal F1-score UMLS F1-score BioPortal

AgreementMakerLight 2.0 NCIt-ORDO Whole 0.42 0.57 0.69 0.96 0.52 0.71AgreementMakerLight 2.0 NCIt-ORDO Star 0.51 0.49 0.67 0.94 0.58 0.64AgreementMakerLight 2.0 NCIt-SNOMED CT Whole 0.60 0.82 0.58 0.94 0.59 0.88AgreementMakerLight 2.0 NCIt-SNOMED CT Star 0.54 0.49 0.76 0.98 0.63 0.65AgreementMakerLight 2.0 ORDO-SNOMED CT Whole 0.46 0.28 0.72 0.98 0.56 0.44AgreementMakerLight 2.0 ORDO-SNOMED CT Star 0.56 0.19 0.57 1.00 0.56 0.33

Table B.2. FCA-Map


FCA-Map NCIt-ORDO Whole 0.27 0.37 0.63 0.88 0.37 0.52FCA-Map NCIt-ORDO Star 0.46 0.46 0.67 1.00 0.55 0.63FCA-Map NCIt-SNOMED CT Whole 0.58 0.67 0.70 0.97 0.63 0.79FCA-Map NCIt-SNOMED CT Star 0.57 0.46 0.88 1.00 0.69 0.63FCA-Map ORDO-SNOMED CT Whole 0.48 0.32 0.58 0.85 0.52 0.47FCA-Map ORDO-SNOMED CT Star 0.45 0.18 0.49 1.00 0.47 0.31

Table B.3. LogMap 2.0


LogMap 2.0 NCIt-ORDO Whole 0.39 0.50 0.68 0.89 0.49 0.64LogMap 2.0 NCIt-ORDO Star 0.58 0.55 0.67 0.94 0.62 0.69LogMap 2.0 NCIt-SNOMED CT Whole 0.58 0.64 0.69 0.90 0.63 0.75LogMap 2.0 NCIt-SNOMED CT Star 0.55 0.42 0.89 0.96 0.68 0.58LogMap 2.0 ORDO-SNOMED CT Whole 0.47 0.27 0.69 0.85 0.56 0.41LogMap 2.0 ORDO-SNOMED CT Star 0.52 0.12 0.74 0.86 0.61 0.21

Table B.4. NCIt-ORDO


AgreementMakerLight 2.0 NCIt-ORDO Whole 0.42 0.57 0.69 0.96 0.52 0.71AgreementMakerLight 2.0 NCIt-ORDO Star 0.51 0.49 0.67 0.94 0.58 0.64FCA-Map NCIt-ORDO Whole 0.27 0.37 0.63 0.88 0.37 0.52FCA-Map NCIt-ORDO Star 0.46 0.46 0.67 1.00 0.55 0.63LogMap 2.0 NCIt-ORDO Whole 0.39 0.50 0.68 0.89 0.49 0.64LogMap 2.0 NCIt-ORDO Star 0.58 0.55 0.67 0.94 0.62 0.69

Table B.5. NCIt-SNOMED CT


AgreementMakerLight 2.0 NCIt-SNOMED CT Whole 0.60 0.82 0.58 0.94 0.59 0.88AgreementMakerLight 2.0 NCIt-SNOMED CT Star 0.54 0.49 0.76 0.98 0.63 0.65FCA-Map NCIt-SNOMED CT Whole 0.58 0.67 0.70 0.97 0.63 0.79FCA-Map NCIt-SNOMED CT Star 0.57 0.46 0.88 1.00 0.69 0.63LogMap 2.0 NCIt-SNOMED CT Whole 0.58 0.64 0.69 0.90 0.63 0.75LogMap 2.0 NCIt-SNOMED CT Star 0.55 0.42 0.89 0.96 0.68 0.58

49

50

Table B.6. ORDO-SNOMED CT


AgreementMakerLight 2.0 ORDO-SNOMED CT Whole 0.46 0.28 0.72 0.98 0.56 0.44AgreementMakerLight 2.0 ORDO-SNOMED CT Star 0.56 0.19 0.57 1.00 0.56 0.33FCA-Map ORDO-SNOMED CT Whole 0.48 0.32 0.58 0.85 0.52 0.47FCA-Map ORDO-SNOMED CT Star 0.45 0.18 0.49 1.00 0.47 0.31LogMap 2.0 ORDO-SNOMED CT Whole 0.47 0.27 0.69 0.85 0.56 0.41LogMap 2.0 ORDO-SNOMED CT Star 0.52 0.12 0.74 0.86 0.61 0.21

Table B.7. Whole ontologies


AgreementMakerLight 2.0 NCIt-ORDO Whole 0.42 0.57 0.69 0.96 0.52 0.71AgreementMakerLight 2.0 NCIt-SNOMED CT Whole 0.60 0.82 0.58 0.94 0.59 0.88AgreementMakerLight 2.0 ORDO-SNOMED CT Whole 0.46 0.28 0.72 0.98 0.56 0.44FCA-Map NCIt-ORDO Whole 0.27 0.37 0.63 0.88 0.37 0.52FCA-Map NCIt-SNOMED CT Whole 0.58 0.67 0.70 0.97 0.63 0.79FCA-Map ORDO-SNOMED CT Whole 0.48 0.32 0.58 0.85 0.52 0.47LogMap 2.0 NCIt-ORDO Whole 0.39 0.50 0.68 0.89 0.49 0.64LogMap 2.0 NCIt-SNOMED CT Whole 0.58 0.64 0.69 0.90 0.63 0.75LogMap 2.0 ORDO-SNOMED CT Whole 0.47 0.27 0.69 0.85 0.56 0.41

Table B.8. Star modules


AgreementMakerLight 2.0 NCIt-ORDO Star 0.51 0.49 0.67 0.94 0.58 0.64AgreementMakerLight 2.0 NCIt-SNOMED CT Star 0.54 0.49 0.76 0.98 0.63 0.65AgreementMakerLight 2.0 ORDO-SNOMED CT Star 0.56 0.19 0.57 1.00 0.56 0.33FCA-Map NCIt-ORDO Star 0.46 0.46 0.67 1.00 0.55 0.63FCA-Map NCIt-SNOMED CT Star 0.57 0.46 0.88 1.00 0.69 0.63FCA-Map ORDO-SNOMED CT Star 0.45 0.18 0.49 1.00 0.47 0.31LogMap 2.0 NCIt-ORDO Star 0.58 0.55 0.67 0.94 0.62 0.69LogMap 2.0 NCIt-SNOMED CT Star 0.55 0.42 0.89 0.96 0.68 0.58LogMap 2.0 ORDO-SNOMED CT Star 0.52 0.12 0.74 0.86 0.61 0.21

Documents

Ontology Matching in Practice: Facilitating Automated