Upload
aron-randall
View
217
Download
0
Embed Size (px)
DESCRIPTION
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal Background Developed within the Kyoto Project Kyoto performs deep semantic searches Models and shares knowledge across different domains and different language communities Spanish, Dutch, English, Italian, Basque, Chinese, Japanese Initially targeted to the domain of ecology, but domain neutral
Citation preview
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Evaluating Ontologies with Rudify
Axel Herold, Amanda Hicksherold, [email protected]
Berlin-Brandenburg Academy of Sciences and Humanities
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Introduction
This presentation describes Rudify, a set of tools used for semi-automatically tagging concepts with ontological meta-properties.– Theory– Development– Evaluation– Application
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
BackgroundDeveloped within the Kyoto Project• Kyoto performs deep semantic searches
• Models and shares knowledge across different domains and different language communities
• Spanish, Dutch, English, Italian, Basque, Chinese, Japanese
• Initially targeted to the domain of ecology, but domain neutral
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
For More InformationSee• www.kyoto-project.eu• Vossen P., et al. 2008. KYOTO: A system for Mining, Structuring and Distributing Knowledge Across Languages and Cultures. In Proceedings of LREC 2008, Marrakech, Morocco, May 28-30, 2008.
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Theory:OntoClean
• Developed by Gaurino and Welty Guarino N., and Welty, C., (2004)
• Evaluates ontological hierarchies• Based on meta-properties of concepts– Rigidity– Unity– Dependence– Identity
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
OntoClean, cont.• Does
– show logical consequences of modeling choices after meta-properties have been assigned concepts in the ontology.
• Does not – assign meta-properties – proscribe a method for assigning meta-properties
This is where Rudify comes in!
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Rigidity in Particular
• We focus on Rigidity because– Importance of type-role distinction
– Easy to find lexical patterns– AEON focused on Rigidity, compare data (Völker et al. 2008)
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Rigidity
• A concept is rigid if it is essential to all of its instances.
• An essential concept is one that necessarily holds for all of its instances.
• “Cat” is rigid, “Pet” is not.
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Roles vs. Types• Types are rigid sortals, roles are non-rigid sortals.
• Sortals describe what sort of thing a concept represents, – “cat”, “milk”, “hurricaine” are sortals,
– “red”, “heavy”, and “singing” are not– Sortal concepts usually correspond to nouns.
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
The RuleNouns tagged as non-rigid should not subsume nouns tagged as rigid.
Conclusion: if Fluffy ceases to be a pet, Fluffy ceases to be a cat.
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Rudfiy Development:Basic Idea
Basic idea described in Völker et al. (2005), Völker et al. (2008)
• Talking about ontological metaproperties:– Unconsciously and implicitly: everybody (at times) (a) "Fluffy stopped being a CAT.” (b) "I wonder if this squirrel would make a
good PET.”– assumption: on the internet (e.g. WWW): lots of
instances like (a), (b)
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
What Rudify Does
• Queries the internet for complex patterns that might indicate meta-properties of the concept
• Uses the number of correct hits to assign scores to rigidity of the concept
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Input
• list of lexical representations (LR) for concepts– e.g. CAT <-- cat (English), gata (Spanish), ...– e.g. PET <-- pet (English), animal doméstico
(Spanish), ...• list of complex linguistic patterns with slots for
LRs:– e.g. "... would make a (good|bad|perfect) X”– e.g. "... is not (|a|an) X anymore"
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Input: cont
• additional feature descriptions for LR-slots,– e.g. part-of-speech(X) == NOUN
• additional feature descriptions for LR-contexts,– e.g. part-of-speech (right_neighbor(X)) !=
NOUN
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Sample output
(a) "is no longer (|a|an) X" (b) && !"there is no longer (|a|an) X" (c) && !"is no longer (|a|an) X Y/NOUN" 1. "He was once a 'loved' family member, but now
has gone feral as he is no longer a pet." (ok)2. "It is guaranteed only if there is no longer a pet
in the home." (violates (b))3. "Diversity is no longer a pet project on the
sidelines of corporate life." (violates (c))
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
What do patterns indicate?
For Rigidty• negative evidence: discover concepts that
might be non-rigid– 25 different linguistic patterns
• positive evidence: discover concepts that might be rigid– No patterns so far.
• hypothesis: all concepts rigid unless evidence against rigidity found (problem with sparse data!)
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Training Data
100 manually annotated monosemous LRs (gold standard)– WordNet 3.0– 50 rigid– 50 non-rigid
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Clasification
• 4 classifiers trained on gold standard– decision tree (J48, a C4.5 implementation)– multinomial logistic regression– nearest neighbor with generalization
(NNge)– locally weighted learning, instance based
• problem: polysemous LRs
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Test Data: Base Concepts
• 297 Base Concepts (Izquierdo et al. (2007))• LRs from WordNet-3.0 (Fellbaum (1998))
– for each path from leaf to root: first node with at least 50 connections pointing downwards
– roughly: cheap computational model for basic level concepts (Rosch et al. (1976))
– set depends on structure and coverage of WordNet
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Test Data, cont.• We tested Rudify on English language data sets:– 50 region terms (hand selected by environmental domain specialist)
– 236 latin species names (hand selected by environmental domain specialist)
– 201 common species names (hand selected by environmental domain specialist)
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Domain Specific Terms• Classifiers correctly classified all region terms and all Latin species names as rigid.
• Classified all common English species names correctly with three exceptions: – "wildcat" was misclassified as denoting a non-rigid concept by all four classifiers
– "wolf" and "apollo" (a butterfly) were mis-classified by all classifiers except NNge.
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Examples from Log Files
• Mount Si High School teacher Kit McCormick is no longer a Wildcat. (generalization from a school mascot to a school member)
• Also the 400 CORBON is no longer a wildcat. (a handgun)
• For example, the dog is no longer a wolf, and is now a whole seperate species. (polysemy)
• For four years, the space agency had been planning, defining, or defending some facet of what led up to and became Apollo. (a space mission's name)
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Rudify Output on Base Concepts
Rudify output Number of results
Decisive 4/0 215Difficult 82Indecisive
3/156
Undecided 2/2
26
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Decisive Rudify OutputInstances Percentages
Rigid Rudify decides
6 4%
incorrect 20 12%correct 142 85%
Non-rigid incorrect 12 25%correct 36 75%
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Summary of Output
Instances Decisive Total Term
Rudify decides
7 3% 2%
Incorrect 32 14% 11%Correct 178 83% 60%Difficult 82 Na 28%
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Overall evaluation
• Decisive output yields a high level of accuracey.– Many errors either come from high level concepts, e.g., “Artifact” and “Unit-of-measurement”, or polysemy.
• We do not regard indecisive output as usable data.
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Application of Outputamount of matter
-R drug +R antibiotic
+R chemical compound+R oil
-R nutriment (a source of material to nourish the body)
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Resulting Hierarchiesamount-of-matter
antibioticchemical compound
oil
substance-roledrugnutriment
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Conclusions• Rudify fills a gap in the process of ontology evaluation and development by semi-automatically tagging concepts with meta-properties that are useful in ensuring clean ontological hierarchies.
• In particular, Rudify can be productively used to distinguish types from roles in order to produce clean ontologies.
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Works CitedFellbaum, C., editor (1998). WordNet: An Electronic Lexical Database. The MIT Press.Guarino N., and Welty, C., 2004. An Overview of OntoClean,
Handbook on Ontologies, ed. S. Staab and R. Studer. pp. 151-172.
Izquierdo, R., Suárez, A., and Rigau, G. (2007). Exploring the automatic selection of basic level concepts. In Proceedings of the International Conference on Recent Advances on Natural Language Processing (RANLP'07), Borovetz, Bulgaria.
Kilgarriff, A. (2007). Googleology is bad science. Computational Linguistics, 33:147 151.Rosch, E., Mervis, C. B., Gray, W. D., Johnson, D. M., and Boyes-Braem, P. (1976). Basic
objects in natural categories. Cognitve Psychology, 8:382 439.Völker et al. (2005): Automatic Evaluation of Ontologies (AEON). In: Gil et al.: Proceedings
of the 4th International Semantic Web Conference (ISWC2005), volume 3729 of LNCS, pp. 716-731, Berlin/Heidelberg
Völker et al. (2008): AEON: An approach to the automatic evaluation of ontologies. In: Applied Ontology 3, pp. 41 --62.
Vossen P., et al. 2008. KYOTO: A system for Mining, Structuring and Distributing Knowledge Across Languages and Cultures. In Proceedings of LREC 2008, Marrakech, Morocco, May 28-30, 2008.
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Thank you for your attention.
Evaluating Ontologies with Rudify - KEOD 2009, Madeira Portugal
Should you use Google?
• results unstable over time, thus not exactly reproducible
• no linguistic search (extensive post processing needed)
• arbitrary limitations on results and meta data• WWW: uncontrolled repository, Google:
unknown filter• see e.g. Kilgarriff (2007)BUT: We need vast amounts of data given the rare
occurrence of LRs in our patterns