Multi-Perspective Question Answering Using the OpQA Corpus

Veselin Stoyanov

Claire Cardie

Janyce Wiebe

Cornell University University of PittsburghUniversity of Pittsburgh

10/08/05 HLT/EMNLP 2005. 2

Multi-Perspective Question Answering

• Fact-based question answering (QA):• When is the first day of spring?

• Do Lipton employees take coffee breaks?

• Vs Multi-perspective question answering (MPQA).

• How does the US regard the terrorist attacks in Iraq?

• Is Derek Jeter a bum?

10/08/05 HLT/EMNLP 2005. 3

• Properties of Opinion vs. Fact answers– OpQA corpus– Traditional fact-based QA systems– Different properties of opinion questions

• Using fine-grained opinion information for MPQA– Annotation framework and automatic classifiers– QA experiments

Talk Outline

10/08/05 HLT/EMNLP 2005. 4

Opinion Question & Answer (OpQA) Corpus

• 98 documents manually tagged for opinions (from the NRRC MPQA corpus [Wilson and Wiebe 2003])

• 30 questions – 15 fact – 15 opinion

[Stoyanov, Cardie, Litman, and Wiebe 2004]

10/08/05 HLT/EMNLP 2005. 5

OpQA corpus: Answer Annotations

• Two annotators• Include every text segment contributing to an

answer– Partial answers:

• When was the Kyoto protocol ratified?– … before May 2003 …

• Are the Japanese unanimous in their support of Koizumi?– … most Japanese support their prime minister …

• Minimum spans

10/08/05 HLT/EMNLP 2005. 6

Traditional Fact-based QA systems

IRsubsystem

1.Frag 3242.Frag 1113.Frag 4314.Frag 213

1.Frag 3242.Frag 1113.Frag 4314.Frag 213

Linguisticfilters

Guesses:

1.Frag 324

2.Frag 213

3. …..

Documents

(document fragments)

Questions

Syntactic filters

Semantic filters

10/08/05 HLT/EMNLP 2005. 7

Characteristics of Opinion vs. Fact Answers

• Answer length– Syntactic and semantic class– Additional processing difficulties

• Partial answers– Answer generator

Number of answers

Length (tokens) Number of partials

Fact 124 5.12 12 (9.68%)

Opinion 415 9.24 154 (37.11%)

10/08/05 HLT/EMNLP 2005. 8

Fine-grained Opinion Information for MPQA

• Recent interest in the area of automatic opinion information extraction.

– E.g. [Bethard, Yu, Thornton, Hatzivassiloglou, and Jurafsky 2004], [Pang and Lee 2004], [Riloff and Wiebe 2003], [Wiebe and Riloff 2005], [Wilson, Wiebe, and Hwa 2004], [Yu and Hatzivassiloglou 2003]

• In our evaluation:– Opinion annotation framework– Sentence-level automatic opinion classifiers– Subjectivity filters– Source filter

10/08/05 HLT/EMNLP 2005. 9

• Described in [Wiebe, Wilson, and Cardie 2002]• Accounts for both:

– Explicitly stated opinions • Joe believes that Sue dislikes the Red Sox.

– Indirectly expressed opinions• The aim of the report is to tarnish China’s image.

• Attributes include strength and source.• Manual sentence-level classification

– sentence subjective if it contains one or more opinions of strength >= medium

Opinion Annotation Framework

• Described in [Wiebe, Wilson, and Cardie 2002]• Accounts for both:

– Explicitly stated opinions • Joe believes that Sue dislikes the Red Sox.

– Indirectly expressed opinions• The aim of the report is to tarnish China’s image.

• Attributes include strength and source.• Manual sentence-level classification

– sentence subjective if it contains one or more opinions of strength >= medium

10/08/05 HLT/EMNLP 2005. 10

Automatic Opinion Classifiers

• Two sentence-level opinion classifiers from Wiebe and Riloff [2005] used for evaluation

• Both classifiers use unannotated data– Rulebased: Extraction patterns bootstrapped using

word lists – NaiveBayes: Trained on data obtained from

Rulebased

Precision Recall F

Rulebased 90.4 34.2 46.6

NaiveBayes 79.4 70.6 74.7

10/08/05 HLT/EMNLP 2005. 11

Subjectivity Filters

IRsubsystem

1.Sent 3242.Sent 1113.Sent 4314.Sent 213

1.Sent 324(o)2.Sent 111(f)3.Sent 431(f)4.Sent 213(o)

Subjectivityfilters

Document Sentences

Opinion Questions

Guesses

1.Sent 324

2.Sent 213

3. …..

Manual

Rulebased

NaiveBayes

Baseline

10/08/05 HLT/EMNLP 2005. 12

Subjectivity Filters Cont’d

• Look for the rank of the first guess containing an answer

• Compute:

1.Sent 3242.Sent 2133.Sent 007 (ans)4.Sent 212 5.Sent 211 (ans)6. …

– Mean Reciprocal Rank (MRR) across the top 5 answers

• MRR = meanall_questions(1/Rank_of_first_answer)

– Mean Rank of the First Answer• MRFA = meanall_questions(Rank_of_first_answer)

10/08/05 HLT/EMNLP 2005. 13

Subjectivity Filters Results

61.33

40.33 43.73

26.2

0

10

20

30

40

50

60

70

MRFA

0.4911

0.51890.5078

0.5856

0.44

0.46

0.48

0.5

0.52

0.54

0.56

0.58

0.6

MRR

Baseline

Manual

NaiveBayes

Rulebased0.4214

10/08/05 HLT/EMNLP 2005. 14

Source Filter

• Manually identify the sources in the opinion questions

• Does France approve of the war in Iraq?

• Retains only sentences that contain opinions with sources matching sources in the question

• France has voiced some concerns with the situation.

10/08/05 HLT/EMNLP 2005. 15

Source Filter Results

• Performs well on the hardest questions in the corpus

• All questions answered within the first 25 sentences with one exception.

MRR MRFA

Baseline 0.4911 61.33

Source 0.4633 11.27

10/08/05 HLT/EMNLP 2005. 16

Summary

• Properties of opinion vs. fact answers– Traditional architectures unlikely to be

effective

• Use of fine-grained opinion information for MPQA– MPQA can benefit from fine-grained

perspective information

10/08/05 HLT/EMNLP 2005. 17

Future Work

• Create summaries of all opinions in a document using fine-grained opinion information

• Methods used will be directly applicable to MPQA

10/08/05 HLT/EMNLP 2005. 18

Thank you.

Questions?

10/08/05 HLT/EMNLP 2005. 19

• Did something surprising happen when Chavez regained power in Venezuela after he was removed by a coup?

• What did South Africa want Mugabe to do after the 2002 elections?

• What’s Mugabe’s opinion about the West’s attitude and actions towards the 2002 Zimbabwe election?

10/08/05 HLT/EMNLP 2005. 20

Characteristics of Fact vs. Opinion Answers Cont’d

• Syntactic Constituent of the answersFact Opinion

Answers in best matching category

31% 16%

Syntactic type of best match

Verb Phrase

0 2

Noun Phrase

9 4

Clause 2 6

10/08/05 HLT/EMNLP 2005. 21

• All improvement significant using Wilcoxon Matched-Pairs Signed-Ranks Test (p<=0.05) except for source filter (p=0.81)