Upload
marquis-scammell
View
227
Download
0
Tags:
Embed Size (px)
Citation preview
1
Opinion Summarization Using En-tity Features and Probabilistic
Sentence Coherence Optimization
(UIUC at TAC 2008 Opinion Summarization Pilot)
Nov 19, 2008
Hyun Duk Kim, Dae Hoon Park, V.G.Vinod Vydiswaran, ChengXiang Zhai
Department of Computer Science
University of Illinois, Urbana-Champaign
Research Questions
1. Can we improve sentence re-trieval by assigning more weights to entity terms?
2. Can we optimize the coherence of a summary using a statistical coherence model?
2
Opinionated Relevant Sentences
SentenceFiltering
Step 2
General Approach
3
Target1001
Q1
Q2
…
OpinionSummary
SentenceOrganization
Step 3
Doc
Query 1
Query 2
Relevant SentencesSentence
Retrieval
Step 1
?
Step 1: Sentence Retrieval
Target1001
Question i
Doc
Indri Toolkit
Relevant Sentences
Named Entity: 10Noun phrase : 2Others : 1
Uniform Term Weighting
Non-Uniform Weighting
Step 2: Sentence Filtering
5
RelevantSentences
S1’S2’S3’S1’S2’S4’S6’S7’S4’
S1’S2’S3’S1’S2’S4’S6’S7’S4’
Keep only thesame polarity
S1’S2’S3’S1’S2’S4’S6’S7’S4’
Keep only Opinionated
Sentences
Remove redundancy
S1’S2’S3’S1’S2’S4’S6’S7’S4’
Step 3: Summary Organization (Method 1: Polarity Ordering)
– Paragraph structure by question and polarity
– Add guiding phrase
6
The first question is … Following are positive opinions…
Following are negative opinions…
The second question is … Following are mixed opinions… …
Step 3: Summary Organization (Method 2: Statistical Coherence Optimization)
7
S1
S2
Sn
…
S1’
S2’
Sn’
…
S3 S3’
c(S1’, S2’)
c(S2’, S3’)
c(Sn-1’, Sn’)+
…
• Coherence function: c(Si, Sj)
• Use a greedy algorithm to order sentences to maximize the total score
c(S1’, S2’)+c(S1’, S2’)+…+c(Sn-1’, Sn’)
Probabilistic Coherence Function(Idea similar to [Lapata 03])
8
0.1)()(
001.0)"("),(ˆ
||||
),(),(
,
vcountucount
sentencesadjacenttwoinvanducountvup
ss
vupssc
ji svsuji
ji
Sentence 1
Sentence 2
u
v v
v
vu
u
Yes YesNoNoNo
P(u,v) = (2+0.001)/(3*4+1.0)
Average coherence probability (Pointwise mutual information)
over all word combinations
Train with original document
Opinionated Relevant Sentences
SentenceFiltering
Step 2
General Approach
9
Target1001
Q1
Q2
…
OpinionSummary
SentenceOrganization
Step 3
Query 1
Doc
SentenceRetrieval
Relevant Sentences
Step 1
Query 2
?
Opinionated Relevant Sentences
SentenceFiltering
Step 2
Submissions: UIUC1, UIUC2
10
Target1001
Q1
Q2
…
OpinionSummary
SentenceOrganization
Step 3
Query 1
Doc
SentenceRetrieval
Relevant Sentences
Step 1
Query 2
?
UIUC1: non-uniform weightingUIUC2: uniform weighting
UIUC1: Aggressive polarity filteringUIUC2: Conservative filtering
UIUC1: Polarity orderingUIUC2: Statistical ordering
Evaluation
• Rank among runs without answer-snippet
11
F-Score Grammatical-ity
Non-redun-dancy
Structure/Coherence
Fluency/Readability
Responsiveness
UIUC1Polarity 6 15 15 5 6 8
UIUC2Coherence 3 9 10 15 16 4
(Total: 19 runs)
Evaluation
• Rank among runs without answer-snippet
12
F-Score Grammatical-ity
Non-redun-dancy
Structure/Coherence
Fluency/Readability
Responsiveness
UIUC1Polarity 6 15 15 5 6 8
UIUC2Coherence 3 9 10 15 16 4
Statistical orderingNothing
(Total: 19 runs)
NE/NP retrieval, Polarity filtering
Polarity ordering
Evaluation of Named Entity Weight-ing
Target1001
Question i
Doc
Indri Toolkit
Relevant Sentences
Named entity: 10Noun phrase : 2Others : 1
Uniform Term Weighting
Non-Uniform Weighting
Assume a sentence is relevant iff similarity(sentence, nugget description) > threshold
Polarity Module• Polarity module performance evaluation
on the sentiment corpus. [Hu&Liu 04, Hu&Liu 04b]
15
Classification result Positive Negative
NonOpinionated 1063 598
Positive 1363 371
Negative 383 412
Mixed 296 210
Total 3105 1591
Exact Match 1363/3105=0.44 412/1591=0.26
(1363+412)/(3105+1591)=0.38
Exact Opposite 383/3105=0.12 371/1591=0.23
(383+371)/(3105+1591)=0.16
(Unit: # of sentence)
Coherence optimization
• Evaluation methods– Basic assumption
• the sentence order of original document is coherent
– Among given target documents,use 70% as training set, 30% as test set.
– Measurement: strict pair matching• # of correct sentence pair / # of total adjacent sentence pair
16
Probabilistic Coherence Function
17
ji svsuji
ji ss
vupssc
, ||||
),(),( Average coherence probability
over all word combinations
2)(""
1)"("),(ˆ
sizedictionarypairswordofcounttotal
sentencesadjacenttwoinvanducountvup
0.1)()(
001.0)"("),(ˆ
vcountucount
sentencesadjacenttwoinvanducountvup
Point-wise Mutual information with smoothing
Strict joint probability
Probabilistic Coherence Function
18
))()(
),(log(),()
)()(
),(log(),(
))()(
),(log(),()
)()(
),(log(),(),(ˆ
vnotpunotp
vnotunotpvnotunotp
vnotpup
vnotupvnotup
vpunotp
vunotpvunotp
vpup
vupvupvup
Mutual information
where,
,1
25.0),(),(,
1
25.0),(),(
,1
25.0),(),(,
1
25.0),(),(
N
vnotunotcvnotunotp
N
vnotucvunotp
N
vunotcvunotp
N
vucvup
N = c(u,v)+c(not u, v)+c(u, not v)+c(not u, not v)For unseen pairs, p(u,v)=0.5*MIN(seen pairs in training)
Coherence optimization test
– Pointwise mutual information effectively penalize common words
19
Selection of training words Strict Joint Probability
MutualInformation
PointwiseMutual
Information
No Omission 0.022259 0.041651 0.056063
Omitted stopwords 0.031389 0.054554 0.057119
Omitted frequent words (counts > 33) (counts > 11) (counts > 6) (counts > 2)
0.0313890.0270130.0204480.019769
0.0514600.0457250.0322190.022259
0.0494980.0461030.0347850.021882
Omitted rare words (counts < 33) (counts < 14) (counts < 6)
(counts < 2)
0.0222590.0220330.0212030.022259
0.0328980.0328230.0370480.041802
0.0440650.0490450.0549310.056591
Coherence optimization test• Top ranked p(u,v) of strict joint probability
– A lot of stopwords are top-ranked.
20
u v p(u,v)
theto
thethetheof
anda
the…
thethetoof
andthethethea…
1.75E-0031.22E-0031.21E-0031.14E-0031.14E-0031.13E-0031.12E-0031.06E-0031.06E-003
…
Coherence optimization test
– Pointwise Mutual information was better than joint probability and normal mutual information.
– Eliminating common words, very rare words improved per-formance
21
Selection of training words Coherence score
Baseline: random order 0.01586
Strict Joint ProbabilityMutual Information
Pointwise Mutual Information (UIUC2)
0.043080.0416510.056063
Omitted stopwordsOmitted non-stopwords
0.0571190.020750
Omitted 95% least frequent words (counts < 33): Omitted 90% least frequent words (counts < 14): Omitted 80% least frequent words (counts < 6): Omitted 60% least frequent words (counts < 2):
0.0440650.0490450.0549310.056591
22
Conclusions
• Limited improvement in retrieval performance using named entity and noun phrase
• Need for a good polarity classification module
• Possibility on the improvement of statistical sentence ordering module with different co-herence function and word selection