Upload
benjamin-lucas
View
218
Download
0
Tags:
Embed Size (px)
Citation preview
Search Engines & Question Search Engines & Question AnsweringAnswering
Giuseppe AttardiGiuseppe Attardi
Dipartimento di InformaticaDipartimento di Informatica
Università di PisaUniversità di Pisa
Università di Pisa
Question AnsweringQuestion Answering
IR: find documents relevant to queryIR: find documents relevant to query– query: boolean combination of
keywordsQA: find answer to questionQA: find answer to question
– Question: expressed in natural language
– Answer: short phrase (< 50 byte)
Trec-9 Q&A trackTrec-9 Q&A track
693 fact-based, short answer questions693 fact-based, short answer questions– either short (50 B) or long (250 B) answer
~3 GB newspaper/newswire text (AP, WSJ, ~3 GB newspaper/newswire text (AP, WSJ, SJMN, FT, LAT, FBIS)SJMN, FT, LAT, FBIS)
Score: MRR (penalizes second answer)Score: MRR (penalizes second answer) Resources: top 50 (no answer for 130 q)Resources: top 50 (no answer for 130 q) Questions: 186 (Encarta), 314 (seeds from Questions: 186 (Encarta), 314 (seeds from
Excite logs), 193 (syntactic variants of 54 Excite logs), 193 (syntactic variants of 54 originals)originals)
CommonalitiesCommonalities
Approaches:Approaches:– question classification– finding entailed answer type– use of WordNet
High-quality document search High-quality document search helpful (e.g. Queen College)helpful (e.g. Queen College)
Sample QuestionsSample Questions
Q: Who shot President Abraham Lincoln?Q: Who shot President Abraham Lincoln?
A: John Wilkes BoothA: John Wilkes Booth
Q: How many lives were lost in the Pan Am crash in Lockerbie?Q: How many lives were lost in the Pan Am crash in Lockerbie?
A: 270A: 270
Q: How long does it take to travel from London to Paris through the Q: How long does it take to travel from London to Paris through the Channel?Channel?
A: three hours 45 minutesA: three hours 45 minutes
Q: Which Atlantic hurricane had the highest recorded wind speed?Q: Which Atlantic hurricane had the highest recorded wind speed?
A: Gilbert (200 mph)A: Gilbert (200 mph)
Q: Which country has the largest part of the rain forest?Q: Which country has the largest part of the rain forest?
A: Brazil (60%)A: Brazil (60%)
Question TypesQuestion Types
Class 1Class 1 Answer: single datum or list of itemsAnswer: single datum or list of items
C: who, when, where, how (old, much, large)C: who, when, where, how (old, much, large)
Class 2Class 2 A: multi-sentenceA: multi-sentence
C: extract from multiple sentencesC: extract from multiple sentences
Class 3Class 3 A: across several textsA: across several texts
C: comparative/contrastiveC: comparative/contrastive
Class 4Class 4 A: an analysis of retrieved informationA: an analysis of retrieved information
C: synthesized coherently from several retrieved C: synthesized coherently from several retrieved fragmentsfragments
Class 5Class 5 A: result of reasoningA: result of reasoning
C: word/domain knowledge and common sense C: word/domain knowledge and common sense reasoningreasoning
Question subtypesQuestion subtypes
Class 1.AClass 1.A About subjects, objects, manner, time or About subjects, objects, manner, time or locationlocation
Class 1.BClass 1.B About properties or attributesAbout properties or attributes
Class 1.CClass 1.C Taxonomic natureTaxonomic nature
Results (long)Results (long)
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
SMU
Queens
Wat
erlo
oIB
MLIM
SINTT IC
Pisa
MRRUnofficial
Falcon: ArchitectureFalcon: Architecture
Question
Question Semantic Form
ExpectedAnswer
TypeAnswer
Paragraphs
Answer Semantic Form
Answer
Answer Logical Form
Paragraph Index
Question ProcessingQuestion Processing Paragraph ProcessingParagraph Processing Answer ProcessingAnswer Processing
Paragraph filtering
Paragraph filtering
Collins Parser + NE Extraction
Collins Parser + NE Extraction
Abduction Filter
Abduction Filter
Coreference Resolution
Coreference Resolution
Question Taxonomy
Question ExpansionWordNet
Collins Parser + NE Extraction
Collins Parser + NE Extraction
Question Logical Form
Question parseQuestion parse
Who was the first Russian astronaut to walk in space
WP VBD DT JJ NNP NP TO VB IN NN
NP NP
PP
VP
S
VP
S
Question semantic formQuestion semantic form
astronaut
walk space
Russianfirst
PERSON
first(x) astronaut(x) Russian(x) space(z) walk(y, z, x) PERSON(x)
Question logic form:Question logic form:
Answer type
Expected Answer TypeExpected Answer Type
size Argentina
dimension
QUANTITYWordNet
Question: Question: What is the size of Argentina?What is the size of Argentina?
Questions about definitionsQuestions about definitions
Special patterns:Special patterns:– What {is|are} …?– What is the definition of …?– Who {is|was|are|were} …?
Answer patterns:Answer patterns:– …{is|are}– …, {a|an|the}– … -
Question TaxonomyQuestion Taxonomy
Reason
Number
Manner
Location
Organization
Product
Language
Mammal
Currency
Nationality
Question
Game
Reptile
Country
City
Province
Continent
Speed
Degree
Dimension
Rate
Duration
Percentage
Count
Question expansionQuestion expansion
Morphological variantsMorphological variants– invented inventor
Lexical variantsLexical variants– killer assassin– far distance
Semantic variantsSemantic variants– like prefer
Indexing for Q/AIndexing for Q/A
Alternatives:Alternatives:– IR techniques– Parse texts and derive conceptual
indexesFalcon uses paragraph indexing:Falcon uses paragraph indexing:
– Vector-Space plus proximity– Returns weights used for abduction
Abduction to justify answersAbduction to justify answers
Backchaining proofs from questionsBackchaining proofs from questionsAxioms:Axioms:
– Logical form of answer– World knowledge (WordNet)– Coreference resolution in answer text
Effectiveness:Effectiveness:– 14% improvement– Filters 121 erroneous answers (of 692)– Requires 60% question processing time
TREC 13 QATREC 13 QA
Several subtasks:Several subtasks:– Factoid questions– Definition questions– List questions– Context questions
LCC still best performance, but LCC still best performance, but different architecturedifferent architecture
LCC Block ArchitectureLCC Block Architecture
PassageRetrieval
PassageRetrieval
Answer Extraction
Theorem Prover
Answer Justification
Answer Reranking
Axiomatic Knowledge Base
Answer Extraction
Theorem Prover
Answer Justification
Answer Reranking
Axiomatic Knowledge Base
WordNetNER WordNetNER
DocumentRetrieval
DocumentRetrieval
Keywords Passages
Question Semantics
Captures the semantics of the questionSelects keywords for PR
Extracts and ranks passagesusing surface-text techniques
Extracts and ranks answersusing NL techniques
Q AQuestion Parse
Semantic Transformation
Recognition of
Expected Answer Type
Keyword Extraction
Question Parse
Semantic Transformation
Recognition of
Expected Answer Type
Keyword Extraction
Question Processing Answer Processing
Question ProcessingQuestion Processing
Two main tasksTwo main tasks– Determining the type of the answer– Extract keywords from the question and
formulate a query
Answer TypesAnswer Types
Factoid questions…Factoid questions…– Who, where, when, how many…– The answers fall into a limited and
somewhat predictable set of categories• Who questions are going to be answered
by… • Where questions…
– Generally, systems select answer types from a set of Named Entities, augmented with other types that are relatively easy to extract
Answer TypesAnswer Types
Of course, it isn’t that easy…Of course, it isn’t that easy…– Who questions can have organizations
as answers• Who sells the most hybrid cars?
– Which questions can have people as answers
• Which president went to war with Mexico?
Answer Type TaxonomyAnswer Type Taxonomy Contains ~9000 concepts reflecting expected Contains ~9000 concepts reflecting expected
answer typesanswer types Merges named entities with the WordNet hierarchyMerges named entities with the WordNet hierarchy
Answer Type DetectionAnswer Type Detection
Most systems use a combination of Most systems use a combination of hand-crafted rules and supervised hand-crafted rules and supervised machine learning to determine the machine learning to determine the right answer type for a question.right answer type for a question.
Not worthwhile to do something Not worthwhile to do something complex here if it can’t also be done complex here if it can’t also be done in candidate answer passages.in candidate answer passages.
Keyword SelectionKeyword Selection
Answer TypeAnswer Type indicates indicates whatwhat the the question is looking for:question is looking for:– It can be mapped to a NE type and used
for search in enhanced indexLexical terms (keywords) from the Lexical terms (keywords) from the
question, possibly expanded with question, possibly expanded with lexical/semantic variations provide lexical/semantic variations provide the required context.the required context.
Keyword ExtractionKeyword Extraction
Questions approximated by sets of Questions approximated by sets of unrelated keywordsunrelated keywords
Question (from TREC QA track)Question (from TREC QA track) KeywordsKeywords
Q002: Q002: What was the monetary value What was the monetary value of the Nobel Peace Prize in 1989?of the Nobel Peace Prize in 1989?
monetary, value, monetary, value, Nobel, Peace, PrizeNobel, Peace, Prize
Q003: Q003: What does the Peugeot What does the Peugeot company manufacture?company manufacture?
Peugeot, company, Peugeot, company, manufacturemanufacture
Q004: Q004: How much did Mercury spend How much did Mercury spend on advertising in 1993?on advertising in 1993?
Mercury, spend, Mercury, spend, advertising, 1993advertising, 1993
Q005: Q005: What is the name of the What is the name of the managing director of Apricot managing director of Apricot Computer?Computer?
name, managing, name, managing, director, Apricot, director, Apricot, ComputerComputer
Keyword Selection AlgorithmKeyword Selection Algorithm
1.1. Select all non-stopwords in quotationsSelect all non-stopwords in quotations2.2. Select all NNP words in recognized Select all NNP words in recognized
named entitiesnamed entities3.3. Select all complex nominals with their Select all complex nominals with their
adjectival modifiersadjectival modifiers4.4. Select all other complex nominalsSelect all other complex nominals5.5. Select all nouns with adjectival modifiersSelect all nouns with adjectival modifiers6.6. Select all other nounsSelect all other nouns7.7. Select all verbsSelect all verbs8.8. Select the answer type wordSelect the answer type word
Passage RetrievalPassage Retrieval
Extracts and ranks passagesusing surface-text techniques
PassageRetrieval
PassageRetrieval
Answer Extraction
Theorem Prover
Answer Justification
Answer Reranking
Axiomatic Knowledge Base
Answer Extraction
Theorem Prover
Answer Justification
Answer Reranking
Axiomatic Knowledge Base
WordNetNER WordNetNER
DocumentRetrieval
DocumentRetrieval
Keywords Passages
Question Semantics
Q AQuestion Parse
Semantic Transformation
Recognition of
Expected Answer Type
Keyword Extraction
Question Parse
Semantic Transformation
Recognition of
Expected Answer Type
Keyword Extraction
Question Processing Answer Processing
Passage Extraction LoopPassage Extraction Loop
Passage Extraction ComponentPassage Extraction Component– Extracts passages that contain all selected keywords– Passage size dynamic– Start position dynamic
Passage quality and keyword adjustmentPassage quality and keyword adjustment– In the first iteration use the first 6 keyword selection
heuristics– If the number of passages is lower than a threshold
query is too strict drop a keyword– If the number of passages is higher than a threshold
query is too relaxed add a keyword
Passage ScoringPassage Scoring Passages are scored based on keyword windowsPassages are scored based on keyword windows
– For example, if a question has a set of keywords: {k1, k2, k3, k4}, and in a passage k1 and k2 are matched twice, k3 is matched once, and k4 is not matched, the following windows are built:
k1 k2 k3k2 k1
Window 1
k1 k2 k3k2 k1
Window 2
k1 k2 k3k2 k1
Window 3
k1 k2 k3k2 k1
Window 4
Passage ScoringPassage Scoring
Passage ordering is performed using Passage ordering is performed using a sort that involves three scores:a sort that involves three scores:– The number of words from the question
that are recognized in the same sequence in the window
– The number of words that separate the most distant keywords in the window
– The number of unmatched keywords in the window
Answer ExtractionAnswer Extraction
Extracts and ranks answersusing NL techniques
PassageRetrieval
PassageRetrieval
Answer Extraction
Theorem Prover
Answer Justification
Answer Reranking
Axiomatic Knowledge Base
Answer Extraction
Theorem Prover
Answer Justification
Answer Reranking
Axiomatic Knowledge Base
WordNetNER WordNetNER
DocumentRetrieval
DocumentRetrieval
Keywords Passages
Question Semantics
Q AQuestion Parse
Semantic Transformation
Recognition of
Expected Answer Type
Keyword Extraction
Question Parse
Semantic Transformation
Recognition of
Expected Answer Type
Keyword Extraction
Question Processing Answer Processing
Ranking Candidate AnswersRanking Candidate Answers
Answer type: Person Text passage:
“Among them was Christa McAuliffe, the first private citizen to fly in space. Karen Allen, best known for her starring role in “Raiders of the Lost Ark”, plays McAuliffe. Brian Kerwin is featured as shuttle pilot Mike Smith...”
Q066: Name the first private citizen to fly in space.
Ranking Candidate AnswersRanking Candidate Answers
Answer type: Person Text passage:
“Among them was Christa McAuliffe, the first private citizen to fly in space. Karen Allen, best known for her starring role in “Raiders of the Lost Ark”, plays McAuliffe. Brian Kerwin is featured as shuttle pilot Mike Smith...”
Best candidate answer: Christa McAuliffe
Q066: Name the first private citizen to fly in space.
Features for Answer RankingFeatures for Answer Ranking
Number of question terms matched in the answer passageNumber of question terms matched in the answer passage Number of question terms matched in the same phrase as the Number of question terms matched in the same phrase as the
candidate answercandidate answer Number of question terms matched in the same sentence as Number of question terms matched in the same sentence as
the candidate answerthe candidate answer Flag set to 1 if the candidate answer is followed by a Flag set to 1 if the candidate answer is followed by a
punctuation signpunctuation sign Number of question terms matched, separated from the Number of question terms matched, separated from the
candidate answer by at most three words and one commacandidate answer by at most three words and one comma Number of terms occurring in the same order in the answer Number of terms occurring in the same order in the answer
passage as in the questionpassage as in the question Average distance from candidate answer to question term Average distance from candidate answer to question term
matchesmatches
Lexical ChainsLexical Chains
QuestionQuestion: When was the internal combustion engine : When was the internal combustion engine invented?invented?
AnswerAnswer: The first internal combustion engine was built in : The first internal combustion engine was built in 1867.1867.
Lexical chainsLexical chains::(1)(1) invent:v#1 invent:v#1 HYPERNIM HYPERNIM create_by_mental_act:v#1 create_by_mental_act:v#1
HYPERNIM HYPERNIM create:v#1 create:v#1 HYPONIM HYPONIM build:v#1 build:v#1
QuestionQuestion: How many chromosomes does a human zygote : How many chromosomes does a human zygote have?have?
AnswerAnswer: 46 chromosomes lie in the nucleus of every normal : 46 chromosomes lie in the nucleus of every normal human cell.human cell.
Lexical chainsLexical chains::(1)(1) zygote:n#1 zygote:n#1 HYPERNIM HYPERNIM cell:n#1 cell:n#1 HAS.PART HAS.PART
nucleus:n#1nucleus:n#1
Theorem ProverTheorem Prover
Q: What is the age of the solar system?QLF: quantity_at(x2) & age_nn(x2) & of_in(x2,x3) & solar_jj(x3) &
system_nn(x3)Question Axiom: (exists x1 x2 x3 (quantity_at(x2) & age_NN(x2) &
of_in(x2,x3) & solar_jj(x3) & system_nn(x3))Answer: The solar system is 4.6 billion years old.Wordnet Gloss: old_jj(x6) live_vb(e2,x6,x2) & for_in(e2,x1) &
relatively_jj(x1) & long_jj(x1) & time_nn(x1) & or_cc(e5,e2,e3) & attain_vb(e3,x6,x2) & specific_jj(x2) & age_nn(x2)
Linguistic Axiom: all x1 (quantity_at(x1) & solar_jj(x1) & system_nn(x1) of_in(x1,x1))
Proof: ¬quantity_at(x2) | ¬age_nn(x2) | ¬of_in(x2,x3) | ¬solar_jj(x3) | ¬system_nn(x3)
Refutation assigns value to x2
Is the Web Different?Is the Web Different?
In TREC (and most commercial In TREC (and most commercial applications), retrieval is performed applications), retrieval is performed against a smallish closed collection of against a smallish closed collection of texts.texts.
The diversity/creativity in how people The diversity/creativity in how people express themselves necessitates all express themselves necessitates all that work to bring the question and the that work to bring the question and the answer texts together.answer texts together.
But…But…
The Web is DifferentThe Web is Different
On the Web popular factoids are On the Web popular factoids are likely to be expressed in a gazillion likely to be expressed in a gazillion different ways.different ways.
At least a few of which will likely At least a few of which will likely match the way the question was match the way the question was asked.asked.
So why not just grep (or agrep) the So why not just grep (or agrep) the Web using all or pieces of the Web using all or pieces of the original question.original question.
AskMSRAskMSR
Process the question by…Process the question by…– Forming a search engine query from the
original question– Detecting the answer type
Get some resultsGet some resultsExtract answers of the right type Extract answers of the right type
based onbased on– How often they occur
Step 1: Rewrite the questionsStep 1: Rewrite the questions
Intuition: The user’s question is Intuition: The user’s question is often syntactically quite close to often syntactically quite close to sentences that contain the answersentences that contain the answer
– Where is the Louvre Museum located? • The Louvre Museum is located in Paris
– Who created the character of Scrooge?• Charles Dickens created the character of
Scrooge.
Query rewritingQuery rewriting
Classify question into seven categoriesClassify question into seven categories
– Who is/was/are/were…?– When is/did/will/are/were …?– Where is/are/were …?
a. Hand-crafted category-specific transformation rulesa. Hand-crafted category-specific transformation rulese.g.: For e.g.: For wherewhere questions, move ‘is’ to all possible questions, move ‘is’ to all possible locationslocations
Look to the Look to the rightright of the query terms for the of the query terms for the answer.answer.
““Where Where isis the Louvre Museum located?” the Louvre Museum located?” ““isis the Louvre Museum located” the Louvre Museum located” ““the the isis Louvre Museum located” Louvre Museum located” ““the Louvre the Louvre isis Museum located” Museum located” ““the Louvre Museum the Louvre Museum isis located” located” ““the Louvre Museum located the Louvre Museum located isis””
Step 2: Query search engineStep 2: Query search engine
Send all rewrites to a Web search Send all rewrites to a Web search engineengine
Retrieve top N answers (100-200)Retrieve top N answers (100-200)For speed, rely just on search For speed, rely just on search
engine’s “snippets”, not the full text engine’s “snippets”, not the full text of the actual documentof the actual document
Step 3: Gathering N-GramsStep 3: Gathering N-Grams
Enumerate all N-grams (N=1,2,3) in all Enumerate all N-grams (N=1,2,3) in all retrieved snippetsretrieved snippets
Weight of an n-gram: occurrence count, Weight of an n-gram: occurrence count, each weighted by “reliability” (weight) of each weighted by “reliability” (weight) of rewrite rule that fetched the documentrewrite rule that fetched the document– Example: “Who created the character of
Scrooge?”Dickens 117Christmas Carol 78Charles Dickens 75Disney 72Carl Banks 54A Christmas 41Christmas Carol 45Uncle 31
Step 4: Filtering N-GramsStep 4: Filtering N-Grams
Each question type is associated Each question type is associated with one or more “data-type filters” = with one or more “data-type filters” = regular expressions for answer typesregular expressions for answer types
Boost score of n-grams that match the Boost score of n-grams that match the expected answer type.expected answer type.
Lower score of n-grams that don’t Lower score of n-grams that don’t match.match.
Step 5: Tiling the AnswersStep 5: Tiling the Answers
Dickens
Charles Dickens
Mr Charles
Scores
20
15
10
merged, discardold n-grams
Mr Charles DickensScore 45
ResultsResults
Standard TREC contest test-bed Standard TREC contest test-bed (TREC 2001): 1M documents; 900 (TREC 2001): 1M documents; 900 questionsquestions– Technique does ok, not great (would have
placed in top 9 of ~30 participants)– But with access to the Web… they do
much better, would have come in second on TREC 2001
Harder QuestionsHarder Questions
Factoid question answering is really Factoid question answering is really pretty silly.pretty silly.
A more interesting task is one where A more interesting task is one where the answers are fluid and depend on the answers are fluid and depend on the fusion of material from disparate the fusion of material from disparate texts over time.texts over time.– Who is Condoleezza Rice?– Who is Mahmoud Abbas?– Why was Arafat flown to Paris?
IXE FrameworkIXE Framework
Object Store
Indexer
OS Abstraction Text
MaxEntropy
Sent. Splitter
Readers
POS Tagger
NE Tagger
Passage Index
Clustering
Crawler Search
Web Service WrappersUnicodeRegExp
TokenizerSuffix Trees
FilesMem Mapping
ThreadsSynchronization
PythonPerlJava
EventStreamContextStream
GIS
Language Processing ToolsLanguage Processing Tools
Maximum Entropy classifierMaximum Entropy classifierSentence SplitterSentence SplitterMulti-language POS TaggerMulti-language POS TaggerMulti-language NE TaggerMulti-language NE TaggerConceptual clusteringConceptual clustering
Maximum EntropyMaximum Entropy
Machine Learning approach to classification:Machine Learning approach to classification:– System trained on test cases– Learned model used for predictions
Classification problem described as a number of Classification problem described as a number of featuresfeatures
Each feature corresponds to a constraint on the Each feature corresponds to a constraint on the modelmodel
Maximum entropy model: the model with the Maximum entropy model: the model with the maximum entropy of all the models that satisfy maximum entropy of all the models that satisfy the constraintsthe constraints
Choosing a model with less entropy, would add Choosing a model with less entropy, would add ‘information’ constraints not justified by the ‘information’ constraints not justified by the empirical evidence availableempirical evidence available
MaxEntropy: example dataMaxEntropy: example data
Features Outcome
Sunny, Happy Outdoor
Sunny, Happy, Dry Outdoor
Sunny, Happy, Humid Outdoor
Sunny, Sad, Dry Outdoor
Sunny, Sad, Humid Outdoor
Cloudy, Happy, Humid Outdoor
Cloudy, Happy, Humid Outdoor
Cloudy, Sad, Humid Outdoor
Cloudy, Sad, Humid Outdoor
Rainy, Happy, Humid Indoor
Rainy, Happy, Dry Indoor
Rainy, Sad, Dry Indoor
Rainy, Sad, Humid Indoor
Cloudy, Sad, Humid Indoor
Cloudy, Sad, Humid Indoor
MaxEnt: example predictionsMaxEnt: example predictions
ContextContext OutdoorOutdoor IndoorIndoor
Cloudy, Happy, HumidCloudy, Happy, Humid 0.7710.771 0.2280.228
Rainy, Sad, HumidRainy, Sad, Humid 0.0010.001 0.9980.998
MaxEntropy: applicationMaxEntropy: application
Sentence SplittingSentence Splitting Not all punctuations are sentence Not all punctuations are sentence
boundaries:boundaries:– U.S.A.– St. Helen– 3.14
Use features like:Use features like:– Capitalization (previous, next word)– Present in abbreviation list– Suffix/prefix digits– Suffix/prefix long
Precision: > 95%Precision: > 95%
Part of Speech TaggingPart of Speech Tagging
TreeTagger: statistic package based TreeTagger: statistic package based on HMM and decision treeson HMM and decision trees
Trained on manually tagged textTrained on manually tagged textFull language lexicon (with all Full language lexicon (with all
inflections: 140.000 words for Italian)inflections: 140.000 words for Italian)
Training CorpusTraining Corpus
IlIl DET:def:*:*:masc:sgDET:def:*:*:masc:sg _il_ilpresidentepresidente NOM:*:*:*:masc:sgNOM:*:*:*:masc:sg _presidente_presidentedelladella PRE:det:*:*:femi:sgPRE:det:*:*:femi:sg _del_delRepubblicaRepubblica NOM:*:*:*:femi:sgNOM:*:*:*:femi:sg _repubblica_repubblicafrancesefrancese ADJ:*:*:*:femi:sgADJ:*:*:*:femi:sg _francese_franceseFrancoisFrancois NPR:*:*:*:*:*NPR:*:*:*:*:* _Francois_FrancoisMitterrandMitterrand NPR:*:*:*:*:*NPR:*:*:*:*:* _Mitterrand_Mitterrandhaha VER:aux:pres:3:*:sgVER:aux:pres:3:*:sg _avere_averepropostoproposto VER:*:pper:*:masc:sgVER:*:pper:*:masc:sg _proporre_proporre……
Named Entity TaggerNamed Entity Tagger
Uses MaxEntropyUses MaxEntropy NE categories:NE categories:
– Top level: NAME, ORGANIZATION, LOCATION, QUANTITY, TIME, EVENT, PRODUCT
– Second level: 30-100. E.g. QUANTITY:• MONEY, CARDINAL, PERCENT,
MEASURE, VOLUME, AGE, WEIGHT, SPEED, TEMPERATURE, ETC.
See resources at CoNLL See resources at CoNLL (cnts.uia.ac.be/connl2004)(cnts.uia.ac.be/connl2004)
NE FeaturesNE Features
Feature types:Feature types:– word-level (es. capitalization, digits, etc.)– punctuation– POS tag– Category designator (Mr, Av.)– Category suffix (center, museum, street, etc.) – Lowercase intermediate terms (of, de, in)– presence in controlled dictionaries (locations,
people, organizations)
Context: words in position -1, 0, +1Context: words in position -1, 0, +1
Sample training documentSample training document
<TEXT> Today the <ENAMEX TYPE='ORGANIZATION'>Dow Jones</ENAMEX>
industrial average gained <NUMEX TYPE='MONEY'>thirtyeight and three quarter points</NUMEX>.
When the first American style burger joint opened in <ENAMEX TYPE='LOCATION'>London</ENAMEX>'s fashionable <ENAMEX TYPE='LOCATION'>Regent street</ENAMEX> some <TIMEX TYPE='DURATION'>twenty years</TIMEX> ago, it was mobbed.
Now it's <ENAMEX TYPE='LOCATION'>Asia</ENAMEX>'s turn.</TEXT><TEXT> The temperatures hover in the <NUMEX
TYPE='MEASURE'>nineties</NUMEX>, the heat index climbs into the <NUMEX TYPE='MEASURE'>hundreds</NUMEX>.
And that's continued bad news for <ENAMEX TYPE='LOCATION'>Florida</ENAMEX> where wildfires have charred nearly <NUMEX TYPE='MEASURE'>three hundred square miles</NUMEX> in the last <TIMEX TYPE='DURATION'>month</TIMEX> and destroyed more than a <NUMEX TYPE='CARDINAL'>hundred</NUMEX> homes.
</TEXT>
ClusteringClustering
Classification: assign an item to one Classification: assign an item to one among a among a givengiven set of classesset of classes
Clustering: find groupings of similar Clustering: find groupings of similar items (i.e. items (i.e. generate the classesgenerate the classes))
Conceptual Clustering of resultsConceptual Clustering of results
Similar to VivisimoSimilar to Vivisimo– Built on the fly rather than from– Predefined categories (Northern Light)
Generalized suffix tree of snippetsGeneralized suffix tree of snippetsStemmingStemmingStop words (articulated, essential)Stop words (articulated, essential)Demo: Demo: pythonpython, , upnpupnp
PiQASso: Pisa Question Answering PiQASso: Pisa Question Answering SystemSystem
““Computers are useless, they can Computers are useless, they can only give answers”only give answers”
Pablo PicassoPablo Picasso
PiQASso ArchitecturePiQASso Architecture
SentenceSplitter
SentenceSplitter
IndexerIndexer
QueryFormulation/Expansion
QueryFormulation/Expansion
WordNet
MiniParMiniPar
?
Documentcollection
MiniParMiniPar
TypeMatching
TypeMatching
RelationMatchingRelationMatching
Answer Pars
AnswerScoringAnswerScoring
PopularityRanking
PopularityRanking
Answer
found?
Answer
found?
Answer
Questionanalysis
Answer analysis
WNSenseWNSense
QuestionClassification
QuestionClassification
Linguistic toolsLinguistic tools
• extracts lexical knowledge from WordNet
• classifies words according to WordNet top-level categories, weighting its senses
• computes distance between words based on is-a links
• suggests word alternatives for query expansion
What metal has the highest melting point?
subj lex-mod
obj
mod
WNSense Minipar [D. Lin]
Example: Theatre
Categorization: artifact 0.60, communication 0.40
Synonyms: dramaturgy, theater, house, dramatics
• Identifies dependency relations between words (e.g. subject, object, modifiers)
• Provides POS tagging
• Detects semantic types of words (e.g. location, person, organization)
• Extensible: we integrated a Maximum Entropy based Named Entity Tagger
Question AnalysisQuestion AnalysisWhat metal has the highest melting point?
metal, highest, melting, point
2. Keyword extraction
1. Parsing
3. Answer type detection
SUBSTANCE
4. Relation extraction
<SUBSTANCE, has, subj><point, has, obj><melting, point, lex-mod><highest, point, mod>
1. NL question is parsed
2. POS tags are used to select search keywords
3. Expected answer type is determined applying heuristic rules to the dependency tree
4. Additional relations are inferred and the answer entity is identified
What metal has the highest melting point?
subj lex-mod
obj
mod
Answer AnalysisAnswer Analysis
Tungsten is a very dense material and has the highest melting point of any metal.
1 Parsing
………….
2 Answer type check 3 Relation extraction
SUBSTANCE<tungsten, material, pred><tungsten, has, subj><point, has, obj>…
4 Matching Distance
Tungsten
6 Popularity Ranking
ANSWER
1. Parse retrieved paragraphs
2. Paragraphs not containing an entity of the expected type are discarded
3. Dependency relations are extracted from Minipar output
4. Matching distance between word relations in question and answer is computed
5. Too distant paragraphs are filtered out
6. Popularity rank used to weight distances
5 Distance Filtering
Match Distance between Question and AnswerMatch Distance between Question and Answer
Analyze relations between corresponding words considering:
number of matching words in question and in number of matching words in question and in answeranswer
distance between words. Ex: distance between words. Ex: moonmoon matching matching with with satellitesatellite
relation types. Ex: words in the question relation types. Ex: words in the question related by related by subjsubj while the matching words in while the matching words in the answer related by the answer related by predpred
More NLPMore NLP
NLP techniques largely NLP techniques largely unsuccessful at information retrievalunsuccessful at information retrieval– Document retrieval as primary measure
of information retrieval success• Document retrieval reduces the need for
NLP techniques– Discourse factors can be ignored– Query words perform word-sense
disambiguation
– Lack of robustness:• NLP techniques are typically not as robust
as word indexing
How these technologies help?How these technologies help?
Question AnalysisQuestion Analysis– The tag of the predicted category is added to
the query Named-Entity Detection:Named-Entity Detection:
– The NE categories found in text are included as tags in the index
What party is John Kerry in? (ORGANIZATION)
John Kerry defeated John Edwards in the primaries for the Democratic Party.
Tags: PERSON, ORGANIZATION
NLP TechnologiesNLP Technologies
Coreference Relations:Coreference Relations:– Interpretation of a paragraph may
depend on the context in which it occurs
Description Extraction:Description Extraction:– Appositive and predicate nominative
constructions provide descriptive terms about entities
Represented as annotations Represented as annotations associated to words, i.e. words in the associated to words, i.e. words in the same position as the referencesame position as the reference
Coreference RelationsCoreference Relations
How long was Margaret Thatcher the prime minister? (DURATION)
The truth, which has been added to over each of her 11 1/2 years in power, is that they don't make many like her anymore.Tags: DURATIONColocated: her, MARGARET THATCHER
Description ExtractionDescription Extraction
Identifies Identifies DESCRIPTIONDESCRIPTION category categoryAllows descriptive terms to be used Allows descriptive terms to be used
in term expansionin term expansion
Famed architect Frank Gary…
Tags: DESCRIPTION, PERSON, LOCATION
Buildings he designed include the Guggenheim Museum in Bilbao.
Colocation: he, FRANK GARY
Who is Frank Gary? (DESCRIPTION) What architect designed the Guggenheim Museum in Bilbao? (PERSON)
NLP TechnologiesNLP Technologies
Question Analysis:Question Analysis:– identify the semantic type of the
expected answer implicit in the queryNamed-Entity Detection:Named-Entity Detection:
– determine the semantic type of proper nouns and numeric amounts in text
Will it work?Will it work?
Will these semantic relations Will these semantic relations improve paragraph retrieval?improve paragraph retrieval?– Are the implementations robust enough
to see a benefit across large document collections and question sets?
– Are there enough questions where these relationships are required to find an answer?
Hopefully yes!Hopefully yes!
PreprocessingPreprocessing
Paragraph DetectionParagraph DetectionSentence DetectionSentence DetectionTokenizationTokenizationPOS TaggingPOS TaggingNP-ChunkingNP-Chunking
Queries to a NE enhanced indexQueries to a NE enhanced index
text matches bushtext matches bush
text matches PERSON:bushtext matches PERSON:bush
text matches LOCATION:* & PERSON: text matches LOCATION:* & PERSON: bin-ladenbin-laden
text matches DURATION:* text matches DURATION:* PERSON:margaret-thatcher prime-PERSON:margaret-thatcher prime-ministerminister
CoreferenceCoreference
Task:Task:– Determine space of entity extents:
• Basal noun phrases:– Named entities consisting of multiple basal
noun phrases are treated as a single entity
• Pre-nominal proper nouns• Possessive pronouns
– Determine which extents refer to the same entity in the world
Paragraph RetrievalParagraph Retrieval
Indexing:Indexing:– add NE tags for each NE category
present in the text– add coreference relationships– Use syntactically-based categorical
relations to create a DESCRIPTION category for term expansion
– Use IXE passage indexer
High ComposabilityHigh Composability
DocInfoDocInfo
PassageDocPassageDoc
Collection<DocInfo>Collection<DocInfo>
Collection<PassageDoc>Collection<PassageDoc>namedatesize
namedatesize
textboundaries
textboundaries QueryCursorQueryCursor
PassageQueryCursorPassageQueryCursor
next()next()
next()next()
CursorCursor
next()next()
Tagged DocumentsTagged Documents
QueryCursorQueryCursor
QueryCursorTaggedWordQueryCursorTaggedWord
QueryCursorWordQueryCursorWord
select documents where select documents where – text matches bush– text matches PERSON:bush– text matches osama & LOCATION:*
CombinationCombination
Searching passages on a collection Searching passages on a collection of tagged documentsof tagged documents
PassageQueryCursor<Collection<TaggedDoc>>PassageQueryCursor<Collection<TaggedDoc>>
QueryCursor<Collection>QueryCursor<Collection>
Paragraph RetrievalParagraph Retrieval
Retrieval:Retrieval:– Use question analysis component to
predict answer category and append it to the question
– Evaluate using TREC questions and answer patterns
• 500 questions
System OverviewSystem Overview
NE Recognizer
Coreference Resolution
Documents
IXE Search
Question Analysis
Question
Paragraphs
Description Extraction
Paragraphs+
Sent. Splitter
POS tagger
Paragr. Splitter
Tokenization
IXE indexer
Indexing Retrieval