Introduction Dimensions Results

Comparison of Reasoners for large Ontologies inthe OWL 2 EL Profile

WAI

Kathrin Dentler

November 8th 2010

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 1/26

Introduction Dimensions Results

Outline

1 Introduction

2 Dimensions

3 Results

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 2/26

Introduction Dimensions Results

Outline

1 Introduction

2 Dimensions

3 Results

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 3/26

Introduction Dimensions Results

PhD ProjectSupervisors

Ronald Cornet Annette ten Teije

Nicolette de Keizer Frank van Harmelen

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 4/26

Introduction Dimensions Results

PhD ProjectSEMI4

SEMantic Interoperability for Integration of clinicalInformation and Indicators

Quality Indicators

General Problem: growing number, labour intensiveApproach:

formalize indicators;

integrate data sources;

investigate which adjustments are needed to facilitatedata-capture at the required quality levels

→ More in next WAI

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 5/26

Introduction Dimensions Results

SNOMED CTSystematized Nomenclature of Medicine, Clinical Terms

SNOMED CT

used to represent information in electronic health records

around 300,000 active concepts

available as DB tables and as OWL ontology

in tractable OWL 2 profile EL

Goal: Classify SNOMED CT with a reasoner to obtain theclosure, i.e. all possible inferences.

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 6/26

Introduction Dimensions Results

An example OWL 2 EL ontology

Fracture v Traumatic abnormality by morphologyTraumatic abnormality by morphology v Traumatic abnormalityTraumatic abnormality v Damage

Fracture of foot ≡ Fracture of lower limb u ∃.rolegroup(∃associated morphology.Fractureu ∃finding site.Bone structure of foot)

Fracture of lower limb ≡ Fracture of bone u ∃rolegroup.(∃associated morphology.Fractureu ∃finding site.Bone structure of lower limb)

Inferred:Fracture v Damage

Fracture of foot v Fracture of lower limbPellet’s explanation:Fracture of foot ≡ Fracture of bone u ∃.rolegroup

(∃associated morphology.Fractureu ∃finding site.Bone structure of foot)

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 7/26

Introduction Dimensions Results

Included Reasoners

CB

CEL

FaCT++

HermiT

Pellet

Racer

Snorocket

TrOWL

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 8/26

Introduction Dimensions Results

Related WorkEarlier Comparisons of Classification Performance for SNOMED CT

10

100

1000

10000

2005 2006 2008 2009 2010

Cla

ssific

ation T

ime in S

econds (

log-s

cale

)

Timeline

CBCEL

FaCTRacer

Snorocket

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 9/26

Introduction Dimensions Results

Outline

1 Introduction

2 Dimensions

3 Results

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 10/26

Introduction Dimensions Results

Dimensions

1 Reasoning Characteristics

2 Practical Usability

3 Performance Indicators

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 11/26

Introduction Dimensions Results

Dimension Reasoning CharacteristicsIndependent of Ontology

Methodology

Soundness and Completeness in Theory

Expressivity and Computational Complexity

Incremental Classification

Rule Support

Justifications

Support of ABox Reasoning Tasks

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 12/26

Introduction Dimensions Results

Dimension Practical UsabilityIndependent of Ontology

OWL API

OWLlink

Availability as Protege Plugin

License

Open or Closed Source

Programming Language

Supported Platforms

Jena

Kind of Institution

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 13/26

Introduction Dimensions Results

Dimension Performance IndicatorsRelated to Ontology

Classification Performance

TBox Consistency Checking Performance

Concept Satisfiability Checking Performance

Subsumption Query Performance

Required Heap Space

Soundness and Completeness in Practice

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 14/26

Introduction Dimensions Results

Outline

1 Introduction

2 Dimensions

3 Results

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 15/26

Introduction Dimensions Results

Results: Categorization of ReasonersDimension Reasoning Characteristics

CB CEL FaCT++ HermiT Pellet RP SR TrOWL(REL)

Methodology consequ.-based

compl.rules

tableau-based

hypertableau

tableaubased

tableaubased

compl.rules

approx.(compl.rules)

SoundX X X X X X X X (X)

CompleteX X X X X X X x (X)

IC(addition/removal)

x/x X/x x/x x/x X/X x/x X/x x/x

RuleSupport

x x x X(SWRL)

X(SWRL)

X(SWRL,nRQL)

x x

Justifications x X x x X X x x

ABoxReasoning

x X X X X(SPARQL)

X(SPARQL,nRQL)

x X(SPARQL)

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 16/26

Introduction Dimensions Results

Results: Categorization of ReasonersDimension Practical Usability

CB CEL FaCT++ HermiT Pellet RP SR TrOWL

OWL API x X X X X X X XOWLlink API x X X X X X x xProtege Plugin x X X X X x X XLicense DuLi: GLGPL AP 2.0 GLGPL GLGPL DuLi: AGPL own own DuLi: AGPLOpen Source X X X X X x x xLanguage OCaml Common Lisp C++ Java Java Lisp Java JavaPlatforms all Linux all all all all all allJena x x x x X x x xInstitution a a a a c c g a

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 17/26

Introduction Dimensions Results

Results: Categorization of ReasonersDimension Performance Indicators

Experiments on GaljasWith: (GO, NCI and) SNOMED CT

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 18/26

Introduction Dimensions Results

Classification Performance vs. Supported ExpressivityDimension Performance Indicators

0.1

1

10

100

1000

CEL Pellet Snorocket TR CB RacerPro HermiT FaCT++

EL EL EL EL Horn SHIQ SHIQ(D-) SROIQ(D) SROIQ(D)C

lassific

ation T

ime in S

econds (

log-s

cale

)

Reasoners ordered by increasing Supported Expressivity

Supported Expressivity

G0NCI

SNOMED CT

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 19/26

Introduction Dimensions Results

Consistency Checking PerformanceDimension Performance Indicators

CB CEL FaCT++ HermiT Pellet RP SR TR

OSNOMED CT x 0.88 15.3 0.00 16.78 x 0.00 0.00

CEL and TrOWL REL need to classify first, but return trueanyway, Snorocket outputs a warning.

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 20/26

Introduction Dimensions Results

Concept Satisfiability Checking PerformanceDimension Performance Indicators

CB CEL FaCT++ HermiT Pellet RP SR TR

OS CT BC x 38.42 22.37 5,276.85 56.91 273.45 0.07 5.17OS CT AC x 34.59 1.76 1.36 6.07 0.00 0.06 0.46

E.g TrOWL returns reliable results only after the ontology isclassified.→ benchmarks with inconsistent ontologies / unsatisfiableconcepts needed!

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 21/26

Introduction Dimensions Results

Subsumption Query PerformanceDimension Performance Indicators

CB CEL FaCT++ HermiT Pellet RP SR TR

NC BC# results x 1 20 20 20 20 x 0seconds x 0.96 701.79 6,649.85 2,793.31 3,380.67 NPE 0.17

AC BC# results x 1 1 20 20 20 x 0seconds x 0.00 0.06 16.94 0.49 0.74 NPE 0.00

NC AC# results x 20 20 20 20 20 20 20seconds x 0.00 0.00 0.00 0.00 0.70 0.00 0.28

AC AC# results x 1 1 20 20 20 20 0seconds x 0.00 0.06 17.12 0.00 0.92 6.97 0.00

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 22/26

Introduction Dimensions Results

Minimum Heap Space for Java ReasonersDimension Performance Indicators

CB CEL FaCT++ HermiT Pellet RP SR TR

n/a n/a n/a 4.5 10 n/a 2.5 4

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 23/26

Introduction Dimensions Results

Soundness and Completeness in PracticeDimension Performance Indicators

CB: missing SubClassOf owl:Thing for top-level classes

CEL: additional EquivalentClasses and EquivalentPropertiesfor each class and property

RacerPro: additional sub class owl:Nothing for all leaf concepts

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 24/26

Introduction Dimensions Results

Soundness and Completeness in PracticeDimension Performance Indicators

Table: SNOMED CT: Missing / Additional inferred SubClassOfstatements in regard to the Relationships Table

CB CEL FaCT++ HermiT Pellet RP SR TR

Missing 0 0 0 0 386 0 86 0Additional 0 0 0 0 546 0 34 0

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 25/26

Introduction Dimensions Results

Questions ?

Kathrin Dentlerk.dentler@amc.uva.nl

k.dentler@few.vu.nl

http://reasonerbench.net/

Kathrin Dentler — Comparison of Reasoners for large Ontologies in the OWL 2 EL Profile 26/26