BioHealthInformaticsGroup

QA of OntologiesQA of Ontologies

OWL Tutorial

December 6th 2005

BioHealthInformaticsGroup

The GIGO ProblemThe GIGO Problem

Logic may be necessary

But it is not sufficient

Can not validate all possibleinferences in advance

Instead must prove:Reasoner is soundAxioms are correctInduction does the rest

BioHealthInformaticsGroup

Barriers to Ontology QABarriers to Ontology QA

►Absolute Measures►No ‘Gold Standard’

►Mutual cross validation only as good as the parts

►Manual checking error-prone

►and can’t measure HOW error prone because of (1)

►Comparitive Measures►Better than worse does not imply good

►Relative Measures►Provides unequivocal evidence of improvement

►But not of proximity to goal

►Falling error detection rate does not imply none exist

BioHealthInformaticsGroup

Types of Axiom QualityTypes of Axiom Quality

Philosophical Rigour

Ontological commitment

Content correctness

Fitness for purpose

BioHealthInformaticsGroup

Philosophical RigourPhilosophical Rigour

►2500 years of research

►Theories of time, mereonomy, containment►Often FOL, so not computable

►Similar upper level ontologies ►DOLCE, BFO

►But not 100% agreement: Realist vs Cognitivist

BioHealthInformaticsGroup

Ontological commitmentOntological commitment

►Formally specified semantic equivalence►Logical transformation to canonical form

►Semantically equivalent but no logical transform

► ‘Fixation of femur by means of inserting pins’

► ‘Insertion of pins to fixate the femur’

►Metamodel rules/commitments►Arbitrary choice of preferred form

►Conventions to be applied throughout ontology

►And by all applications that use it

BioHealthInformaticsGroup

Content correctnessContent correctness

► Metadata Provenance, lexical annotations etc

► Truth► ‘Structure of labial vein’ is-a ‘Superficial vein of face’

► Completeness – ambiguity and omission► Thymus secretes Thymosin; Thymosin is-a Hormone

…but omits Thymus is-a endocrine gland

► Conciseness► Redundant inclusion of inferrable axioms

► Consistency – contradictory, duplicated, circular► endocrine surgery vs endocrine surgeons

► Traumatic unilateral amputationUnilateral traumatic amputation

BioHealthInformaticsGroup

Fitness for PurposeFitness for Purpose

► Best theories no guarantee of usability or utility

► Lab experiences no predictor of field behaviour

► All for nothing, if user can’t use it► Interrater variability

BioHealthInformaticsGroup

Fitness for Purpose:Fitness for Purpose: Inter-rater variabilityInter-rater variability

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

X

Headcloth

Cloth

Scarf

Model Person

Woman

Adults

Standing

Background

Brown

Blue

Chemise

Dress

Tunics

Clothes

Suitcase

Luggage

Attache case

Brass Instrument

French Horn

Horn

Tuba

BioHealthInformaticsGroup

Fitness for Purpose:Fitness for Purpose: Inter-rater variabilityInter-rater variability

► Miscoding ► Code meaning is inappropriate to thing being described

► Instrument definitely not a french horn

► Missed coding► Not coding something that could be coded

► No code for the table/platform

► Overcoding► Code meaning is more detailed than justified

► Can the gender really be determined?

► Undercoding► Code meaning is less detailed than justified

► Brass Instument vs Tuba

BioHealthInformaticsGroup

Fitness for PurposeFitness for Purpose

What ontological properties..What ontological properties..

► Increase usablility and utility?

► Are a prerequisite for them?

► Decrease usability or utility?

BioHealthInformaticsGroup

Ontology QAOntology QAHow much quality do we need?How much quality do we need?

Perfection is unattainable

Trade-offs between quality and…Performance

Cost

Maintainability

Usability

Acceptance

Utility

BioHealthInformaticsGroup

Need for CQI, not final QANeed for CQI, not final QAAssure the Assure the pprocess rocess notnot the product the product

Use/Test cases & exemplars

REVIEWExplore

consequences

APPLYQI

Algorithm

DEVISEQI

Algorithm

ASSESSIdentifyproblems