QuALiM – Michael Kaisser The QuALiM Question Answering system Question Answering by Searching Large Corpora with Linguistic Methods

QuALiM – Michael Kaisser

The QuALiM Question Answering system

Question Answering by Searching Large Corpora

with Linguistic Methods


Talk Outline

• What does a QA system do?

• QuALiMs two answer strategies:– Fallback mechanism– Rephrasing algorithm

• TREC evaluation results

• Post TREC evaluation results


Question Answering - Definition

Definition from Wikipedia:

Question Answering (QA) is a type of information retrieval. Given a collection of documents (such as the World Wide Web) the system should be able to retrieve answers to questions posed in natural language. QA is regarded as requiring more complex natural language processing (NLP) techniques than other types of information retrieval such as document retrieval, and it is sometimes regarded as the next step beyond search engines.


Question Answering - Example

Start is MIT’s QA system: http://start.csail.mit.edu/







Better—however—would be:

“Albert Einstein was born on March 14th, 1879.”

The system should actually return a complete English sentence expressing the desired fact.


The Fallback Mechanism(exemplary for common answer finding techniques)


Fallback MechanismThe fallback mechanism creates queries based upon

keywords and key phrases from the question. Three

queries are send to Google:

• The first query contains all non-stop words from the question

• The second contains all NPs from the question (that contain at least one non-stop word)

• The third query contains all NPs and all non-stop words that do not occur in the NPs.


Fallback MechanismSo "When was Jim Inhofe first elected to the senate?”

becomes

• Jim Inhofe senate first elected• “Jim Inhofe” “the senate”• “Jim Inhofe” “the senate” first elected

Note: The results from the last query are weighted twice as high as the results form the first two queries.


Fallback Mechanism

72.0: "senator" 42.0: "senator jim inhofe" "senator jim" 41.25: "r" (abbreviation for republican)32.25: "oklahoma" 30.0: "r-okla" (abbreviation for republican-oklahoma)26.25: "1994" 25.0: "the leading conservative voices" "of the leading conservative voices“ "leading conservative voices" 24.0: "us senator" 23.25: "republican" 21.0: "okla" (abbreviation for oklahoma)

The result from the queries when placed in a Weighted Sequence Bag:


Fallback Mechanism

72.0: "senator" 42.0: "senator jim inhofe" "senator jim" 41.25: "r" (abbreviation for republican)32.25: "oklahoma" 30.0: "r-okla" (abbreviation for republican-oklahoma)26.25: “1994” 25.0: "the leading conservative voices" "of the leading conservative voices“ "leading conservative voices" 24.0: "us senator" 23.25: "republican" 21.0: "okla" (abbreviation for oklahoma)

But we know that we are looking for a date, so the answer is “1994”:


Definition QuestionsQuery: "Florence Nightingale“

20.0: "may 12, 1820" 16.0: "may 12" "nursing" 15.0: "august 13, 1910" 14.0: "1820-1910“13.0: "born" 12.0: "august 13" "museum" 11.0: "history" 10.0: "modern nursing" "lady with the lamp" "florence nightingale

museum" "the lady with the lamp" 9.0: "italy"8.0: "of modern nursing" "nurses" "london" 7.5: "on may 12, 1820" 7.0: "2 lambeth palace road london"


Definition Questions20.0: "may 12, 1820" 16.0: "may 12" "nursing" 15.0: "august 13, 1910" 14.0: "1820 1910“13.0: "born" 12.0: "august 13" "museum" 11.0: "history" 10.0: "modern nursing" "lady with the lamp" "florence nightingale museum" "the lady with

the lamp" 9.0: "italy"8.0: "of modern nursing" "nurses" "london" 7.5: "on may 12, 1820" 7.0: "2 lambeth palace road london“

Answer sentences in AQUAINT corpus:"on may 12, 1820, the founder of modern nursing, florence nightingale, was born in florence, italy.""on aug. 13, 1910, florence nightingale, the founder of modern nursing,

died in london.“


The Rephrasing Algorithm


Pattern Layout<pattern name="When+did+NP+Verb+NPorPP" level="5"> <sequence> <word id="1">When</word> <word id="2">did</word> <parse id="3">NP</parse> <morph id="4">V INF</morph> <parse id="5">NP|PP</parse> <final>?</final> </sequence> <target name="target1"> <ref>3</ref> <ref morph="V PAST">4</ref> <ref>5</ref> <word>in</word> <answer>NP</answer> </target> <target name="target2"> <word>in</word> <answer>NP</answer> <punctuation optional="true">,</punctuation> <ref>3</ref> <ref morph="V PAST">4</ref> <ref>5</ref> </target> ... more targets ... <answerType phrases="NP|PP"> <built-in weight="2">dateComplete</built-in> <namedEntity weight="4">Date</namedEntity> <built-in weight="3">year|in_year</built-in> <other ignore="true"/> </answerType> </pattern>

Sequences are matched against questions.

Targets describe (flat) syntactic structures of potential answer sentences.

AnswerTypes place restrictions on the expected answer type.


Sequences<sequence> <word id="1">When</word> <word id="2">did</word> <parse id="3">NP</parse> <morph id="4">V INF</morph> <parse id="5">NP|PP</parse> <final>?</final></sequence>

This sequence matches all questions

• beginning with “When”

• followed by “did”

• followed by an NP

• followed by a verb in its infinitive form

• followed by an NP or a PP

• followed by a question mark (which has to be the last element in the question)

question start word: When word: did phrase: NP POS: V INF phrase: NP or PP punctuation: ?question end


Sequences<sequence> <word id="1">When</word> <word id="2">did</word> <parse id="3">NP</parse> <morph id="4">V INF</morph> <parse id="5">NP|PP</parse> <final>?</final></sequence>

In the TREC 2005 question set this particular sequence matched 5 questions:

• “When did Floyd Patterson win the title?”

• “When did Amtrak begin operations?”

• “When did Jack Welch become chairman of General Electric?”

• “When did Jack Welch retire from GE?”

• “When did the Khmer Rouge come into power?”

question start word: When word: did phrase: NP POS: V INF phrase: NP or PP punctuation: ?question end


Targets<target name="target1"> <ref>3</ref> <ref morph="V PAST">4</ref> <ref>5</ref> <word>in</word> <answer>NP</answer></target>

<target name="target2"> <word>in</word> <answer>NP</answer> <punctuation optional="true">, </punctuation> <ref>3</ref> <ref morph="V PAST">4</ref> <ref>5</ref></target>

If a question matched a sequence, the targets are used to propose templates for potential answer sentences.

For the question “When did Amtrak begin operations”, these would be:

• ”Amtrak began operations in ANSWER[NP]”

• ”In ANSWER[NP] (,) Amtrak began operations”


Targetsanswer sentence start Amtrak began operations in answer (NP)answer sentence end

answer sentence start In answer (NP) (,)

Amtrak began operationsanswer sentence end

<target name="target1"> <ref>3</ref> <ref morph="V PAST">4</ref> <ref>5</ref> <word>in</word> <answer>NP</answer></target>



Targets<target name="target1"> <ref>3</ref> <ref morph="V PAST">4</ref> <ref>5</ref> <word>in</word> <answer>NP</answer></target>


The information from the targets can be used to create Google queries:

• ”Amtrak began operations in”

• ”In” “Amtrak began operations”


Snippet RetrievalFor the first query ”Amtrak began operations in” the first five sentences Google returns are:

• “Since Amtrak began operations in 1971, federal outlays for intercity rail passenger service have been about \$18 billion.”

• “Amtrak began operations in 1971.”

•“Amtrak of the obligation to operate the basic system of routes that was largely inherited from the private railroads when Amtrak began operations in 1971.”

•“Amtrak began operations in 1971, as authorized by the Rail Passenger Service Act of 1970.'‘

•“A comprehensive history of intercity passenger service in Indiana, from the mid-19th century through May 1, 1971, when Amtrak began operations in the state.”


Answer ExtractionThe sentences are parsed and tagged, and by matching then to the targets once more the exact position of the potential answer can be located:







QuALiM – Type Checking<answerType phrases="NP|PP">

<built-in weight="2"> dateComplete </built-in>

<namedEntity weight="4"> date </namedEntity>

<built-in weight="3"> year|in_year </built-in>

<other ignore="true"/>

</answerType>

The answerType element in the pattern tells us that we are looking for a date.

We’d like to have:

• a complete date in standard form, e.g. “May 1st, 1971”

• some form of a date, e.g. “5/1/1971”

•If we cannot have that, a year specification will also do. (E.g. “1971”)







</answerType>

An answerType may contain the following elements:

• NamedEntity

• WordNetCategory

• Built-in (date, year, percentage ect.)

• Measure (“15 meters”, “100 mph”)

• List (e.g. a list of movies)

• WebHypernym

• other


Excursus: WordNet


Excursus: WordNet


Excursus: WordNet


Excursus: Named Entity Recognition

The task: identify atomic elements of information in text

• person names• company/organization names• locations• dates&times• percentages• monetary amounts



Task of a NE System:

Delimit the named entities in a text and tag them with NE categores:

<ENAMEX TYPE=„LOCATION“>Italy</ENAMEX>‘s business world was rocked bythe announcement <TIMEX TYPE=„DATE“>last Thursday</TIMEX> that Mr.<ENAMEX TYPE=„PERSON“>Verdi</ENAMEX> would leave his job as vice-presidentof <ENAMEX TYPE=„ORGANIZATION“>Music Masters of Milan, Inc</ENAMEX> to become operations director of <ENAMEX TYPE=„ORGANIZATION“>Arthur Andersen</ENAMEX>.

•„Milan“ is part of organization name

•„Arthur Andersen“ is a company

•„Italy“ is sentence-initial => capitalization useless



Task of a NE System:

Delimit the named entities in a text and tag them with NE categores:

•„Milan“ is part of organization name

•„Arthur Andersen“ is a company

•„Italy“ is sentence-initial => capitalization useless

Italy‘s business world was rocked by last Thursday that Mr.Verdi would leave his job as vice-president of Music Masters of Milan, Inc to become operations director of Arthur Andersen.



How does it work?

Basically quite simple: The system accesses huge lists of:• First names• Last names• Cities• Countries• ...• And knows about special words/abbreviations like

Mr., Dr., Prof., Inc., Blvd. etc.• It knows the names of weekdays or months etc.



Some system use hand-written context-sensitive reduction rules:

1) title capitalized word => title person_namecompare „Mr. Jones“ vs. „Mr. Ten-Percent“=> no rule without exceptions

2) person_name, „the“ adj* „CEO of“ organization„Fred Smith, the young dynamic CEO of BlubbCo“=> ability to grasp non-local patterns

plus help from databases of known named entities







</answerType>

An answerType may contain the following elements:

• NamedEntity

• WordNetCategory

• Built-in (date, year, percentage ect.)

• Measure (“15 meters”, “100 mph”)

• List (e.g. a list of movies)

• WebHypernym

• other


QuALiM – Type CheckingWhen the answers are checked on their correct semantic type the first four sentences pass the test, the last one is ruled out:







TREC 2004 Results and Post-TREC Evaluation


TREC Results – factoid questions


TREC Results – combined score


Post TREC Evaluation

• Purpose: What is the performance and behavior of the different algorithms implemented?

• Performed with resolved questions.(“When was Franz Kafka born?” instead of “When was he born?”)

• No document localization, thus:– no NIL answers returned – no “unsupported” judgments


Post TREC Evaluation


Results ordered by their confidence value (correct answers)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1 11 21 31 41 51 61 71 81 91 101 111 121 131 141 151 161 171 181 191 201 211 221

number of questions

frac

tion

of c

orr

ect a

nsw

ers

strict

fuzzy

fallback

combined


Results ordered by confidence value (correct & inexact answers)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1 11 21 31 41 51 61 71 81 91 101 111 121 131 141 151 161 171 181 191 201 211 221

number of questions

frac

tion

of

corr

ect

an

d in

exa

ct a

ns

we

rs

strict

fuzzy

fallback

combined

Documents

QuALiM – Michael Kaisser The QuALiM Question Answering system Question Answering by Searching Large Corpora with Linguistic Methods