Upload
skyler
View
23
Download
0
Embed Size (px)
DESCRIPTION
Extracting a Lexical Entailment Rule-base from Wikipedia Eyal Shnarch, Libby Barak, Ido Dagan Bar Ilan University. Entailment - What is it and what is it good for?. Question Answering: Information Retrieval: “The Beatles”. luxury cars. are produced in ?”. Britain. “Which. - PowerPoint PPT Presentation
Citation preview
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 1 /20
Extracting a Lexical Entailment Rule-base from Wikipedia
Eyal Shnarch, Libby Barak, Ido Dagan
Bar Ilan University
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 2 /20
Entailment - What is it and what is it good for?
• Question Answering:
• Information Retrieval: “The Beatles”
“Which are produced in ?”luxury cars Britain
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 3 /20
Lexical Entailment
• Lexical Entailment rules model such lexical relations
• Part of the Textual Entailment paradigm – a generic
framework for semantic inference
• Encompasses a variety of relations:– Synonymy: Hypertension Elevated blood-pressure
– IS-A: Jim Carrey actor
– Predicates: Crime and Punishment Fyodor Dostoyevsky
– Reference: Abbey Road The Beatles
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 4 /20
What was done so far?
• Lexical database, made for computational consumption, NLP resource - WordNet– Costly, need experts, many years of development (since 1985)
• Distributional similarity– Country and State share similar contexts
– But also Nurse and Doctor, Bear and Tiger - Low precision
• Patterns:– NP1 such as NP2 luxury car such as Jaguar
– NP1 and other NP2 dogs and other domestic pets
– Low coverage, mainly IS-A patterns
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 5 /20
Our approach – Utilize Definitions
• Pen: an instrument for writing or drawing with ink.– pen is-an instrument– pen used for writing / drawing– ink is part of pen
• Source of definitions: – Dictionary: describes language terms, slow growth – Encyclopedia: contains knowledge, proper names, events, concepts,
rapidly grow
• We chose Wikipedia– Very dynamic, constantly growing and updating– Covers a vast range of domains– Gaining popularity in research - AAAI 2008 workshop
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 6 /20
Extraction Types
•Be-complimentnoun in the position of a compliment of a verb ‘be’
•All-Nounsall nouns in the definition
different likelihood to be entailed
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 7 /20
• The likelihood of entailment depends greatly on the syntactic path connecting the title and the noun.– Path in a parsed tree
• An unsupervised entailment likelihood score for a syntactic path p within a definition:
• Split Def-N into Def-Ntop and Def-Nbot– Indicative for rule reliability - Def-Ntop rules’ precision is much higher
than Def-Nbot’s.
Ranking All-Nouns Rules
)(
)(
pC
pCT
film
title directed
by
noun
subj vrel
by-subj
pcomp-n
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 8 /20
Extraction Types
•Redirectnoun in the position of a
•Parenthesisall nouns in the definition
•Linkall nouns in the definition
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 9 /20
Ranking Rules by Supervised Learning
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 10 /20
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 11 /20
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 12 /20
Ranking Rules by Supervised Learning
Extraction Types
• An alternative approach for deciding which rules to select out of all extracted rules.
• Each rule is represented by:– 6 binary features: one for each extraction type
– 2 binary features: one for each side of the rule indicating whether it is NE
– 2 numerical features: rule sides’ co-occurrence & count extracted
– 1 numeric feature: the score of the path for Def-N extraction type
• Manually annotated set used to train SVMlight
– Varied the J parameter in order to obtain different recall-precision tradeoffs
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 13 /20
Results and Evaluation
• The obtained knowledge base include:– About 10 million rules
• For comparison: Snow’s extension to WordNet includes 400,000 relations.
– More than 2.4 million distinct RHSs
– 18% of the rules extracted by more than one extraction type
– Mostly named entities and specific concepts, as expected from encyclopedia
• Two Evaluation types:– Rule-based: rule correctness relative to human judgment
– Inside real application: the utility of the extracted rules for lexical expansion in keyword-based text categorization
Results & Evaluations
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 14 /20
Rule-base Evaluation• Randomly sampled 830 rules and annotated them for correctness
– inter annotators agreement achieved Kappa of 0.7
• Precision: the percentage of correct rules
• Est. # of correct rules: number of rules annotated as correct multiply by the
sampling proportion.
Extraction Type
Per TypeAccumulated
PEst. # RulesPR
Redirect0.872,232,8770.870.31
Be-Comp0.82,740,9570.820.6
Def-Ntop0.722,179,3950.770.71
Parenthesis0.7166,8530.770.72
Link0.7708,6380.760.8
Def-Nbot0.471,657,9440.661
Results & Evaluations
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 15 /20
Supervised Learning Evaluation
• 5-fold cross validation on the annotated sample:
• Although considering additional information, performance is almost identical to considering only extraction types.
• Further research is needed to improve our current feature set and classification performance.
Results & Evaluations
J0.30.40.50.91.11.3
R0.860.820.750.750.70.66
P0.320.590.730.810.911
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 16 /20
Text Categorization Evaluation
• Represent a category by a feature vector of characteristic terms for it.– The characteristic terms should entail the category name.
• Compare the term-based feature vector of a classified document with the feature vectors of all categories. – Assign the document to the category which yields the highest cosine
similarity score (single-class classification).
• 20-News Groups collection
• 3 baselines: No expansions, WordNet, WikiBL, [Snow]
• Also evaluated the union of Wikipedia and WordNet
Results & Evaluations
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 17 /20
Text Categorization Evaluation
Results & Evaluations
Rule BasePRF1
Baselines
No Expansion0.530.190.28
WordNet0.460.290.36
WikiBL0.530.190.28
Extraction
Types
Redirect only0.540.210.3
+ Be-comp0.550.210.3
+ Parenthesis and Link0.410.30.35
+ Def-Ntop0.420.30.35
+ Def-Nbot (all rules)0.390.320.35
SVMJ = 0.30.550.210.31
J = 1.10.310.280.3
UnionWN + Wiki (all)0.40.340.37
WN + Wiki (redir + Be-comp)0.50.330.39
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 18 /20
Promising Directions for Future Work
• Learning semantic relations in addition to Taxonomical relations (hyponym, synonyms) :
• Fine-grained relations of LE is important for inference
RelationRulePath Pattern
LocationLovek CambodiaLovek city in Cambodia
OccupationGeorge Bogdan Kistiakowsky chemistryGeorge Bogdan Kistiakowsky chemistry professor
CreationCrime and Punishment Fyodor Dostoyevsky
Crime and Punishment is a novel by Fyodor Dostoyevsky
OriginWillem van Aelst DutchWillem van Aelst Dutch artist
AliasDean Moriarty Benjamin Linus Dean Moriarty is an alias of Benjamin Linus on Lost
SpellingEgushawa AgushawayEgushawa, also spelled Agushaway...
Conclusions & Future Work
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 19 /20
Promising Directions for Future Work
• Natural Types, naturally phrased entities:– 56,000 terms entail Album
– 31,000 terms entail Politician
– 11,000 terms entail Footballer
– 20,000 terms entail Actor
– 15,000 terms entail Actress
– 4,000 terms entail American Actor
Conclusions & Future Work
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 20 /20
Conclusions
Conclusions & Future Work
• First large-scale rule base directed to cover LE.
• Learning ontology which is a very important knowledge for reasoning systems (one of the conclusions of the first 3 RTE benchmarks).
• Automatically extracting lexical entailment rules from an unstructured source
• Comparable results, on a real NLP task, to a costly manually crafted resource such as WordNet.
Textual Entailment | Learning Lexical Entailment | Wikipedia | Extraction Types | Results & Evaluations | Conclusions & Future Work 21 /20
Inference System
t: Strong sales were shown for Abbey Road in 1969.grammar rule: passive to active
Abbey Road showed strong sales in 1969.
lexical entailment rule: Abbey Road The Beatles
The Beatles showed strong sales in 1969.
lexico-syntactic rule: show strong sales gain commercial success
h: The Beatles gained commercial success in 1969.
Textual Entailment