1

Probabilistic Semantic Similarity Measurements

for Noisy Short Texts Using Wikipedia Entities

Masumi Shirakawa1, Kotaro Nakayama2, Takahiro Hara1, Shojiro Nishio1

1Osaka University, Osaka, Japan2University of Tokyo, Tokyo, Japan

Challenge in short text analysisStatistics are not always enough.

2

A year and a half after Google pulled its popular search engine out of mainland China

Baidu and Microsoft did not disclose terms of the agreement

Challenge in short text analysisStatistics are not always enough.

3

A year and a half after Google pulled its popular search engine out of mainland China

Baidu and Microsoft did not disclose terms of the agreement

Search enginesand China

They are talking about...

Challenge in short text analysisStatistics are not always enough.

4

A year and a half after Google pulled its popular search engine out of mainland China

Baidu and Microsoft did not disclose terms of the agreement

How do machines know that the two sentences mention about the similar

topic?

Search enginesand China

They are talking about...

Reasonable solutionUse external knowledge.

5

A year and a half after Google pulled its popular search engine out of mainland China

Baidu and Microsoft did not disclose terms of the agreement

Wikipedia Thesaurus [Nakayama06]

Related workESA: Explicit Semantic Analysis [Gabrilovich07]

Add Wikipedia articles (entities) to a text as its semantic representation.

1. Get search ranking of Wikipedia for each term (i.e. Wiki articles and scores).2. Simply sum up the scores for aggregation.

6

Apple Inc.

Apple Inc.

pricing

pearApple sells

a new product

Appleproduct

Key termextraction

Related entityfinding

AggregationiPhone

iPadiPhon

epearInput: T

t c Output: ranked list of c

sellsnew

business

Problems in real world noisy short texts“Noisy” means semantically noisy in this work. (We do not handle informal or casual surface forms, or misspells)

Term ambiguity• Apple (fruit) should not be related with Microsoft.

Fluctuation of term dominance• A term is not always important in texts.

7

We explore more effective aggregation method.

We propose Extended naïve Bayes to aggregate related entities

Apple Inc.

Apple Inc.

Probabilistic method

8

𝑃 (𝑡∈𝑇 ) 𝑃 (𝑒|𝑡 ) 𝑃 (𝑐|𝑒)∏𝑘𝑃 (𝑐|𝑡𝑘 )

𝑃 (𝑐 )𝐾−1

𝑃 (𝑐∨𝑇 )=∏𝑘=1

𝐾

(𝑃 (𝑡𝑘∈𝑇 )𝑃 (𝑐|𝑡𝑘)+(1−𝑃 (𝑡𝑘∈𝑇 ) )𝑃 (𝑐))𝑃 (𝑐 )𝐾− 1

From text T to related articles c

𝑃 (𝑐|𝑡 )

[Milne08]

[Mihalcea07] [Song1

1]

pricing

pearApple sells

a new product

Appleproduct

Key termextraction

Related entityfinding

AggregationiPhone

iPadiPhon

eiPadInput: T t c Output:

ranked list of c[Nakayama06]

When input is multiple termsApply naïve Bayes [Song11] to multiple terms to obtain related entity c using each probability P (c |tk ).

𝑃 (𝑐|𝑡1 ,…, 𝑡𝐾 )=𝑃 (𝑡 1 ,…, 𝑡𝐾|𝑐 )𝑃 (𝑐 )

𝑃 (𝑡1 ,…, 𝑡𝐾 )=𝑃 (𝑐 )∏

𝑘𝑃 ( 𝑡𝑘|𝑐 )

𝑃 (𝑡 1 ,… ,𝑡𝐾 )=∏𝑘𝑃 (𝑐|𝑡𝑘 )

𝑃 (𝑐 )𝐾−1

𝑡1𝑡 2

𝑡𝐾

…

Apple

product

new

iPhone

Compute for each related entity c

𝑐

𝑃 (𝑐∨𝑡1)

𝑃 (𝑐∨𝑡2)

𝑃 (𝑐∨𝑡𝐾)

“iPhone”

9

When input is multiple termsApply naïve Bayes [Song11] to multiple terms to obtain related entity c using each probability P (c |tk ).

𝑃 (𝑐|𝑡1 ,…, 𝑡𝐾 )=𝑃 (𝑡 1 ,…, 𝑡𝐾|𝑐 )𝑃 (𝑐 )

𝑃 (𝑡1 ,…, 𝑡𝐾 )=𝑃 (𝑐 )∏

𝑘𝑃 ( 𝑡𝑘|𝑐 )

𝑃 (𝑡 1 ,… ,𝑡𝐾 )=∏𝑘𝑃 (𝑐|𝑡𝑘 )

𝑃 (𝑐 )𝐾−1

𝑡1𝑡 2

𝑡𝐾

…

Apple

product

new

iPhone

Compute for each related entity c

𝑐

𝑃 (𝑐∨𝑡1)

𝑃 (𝑐∨𝑡2)

𝑃 (𝑐∨𝑡𝐾)

“iPhone”

10

By using naïve Bayes, entities that are related

to multiple terms can be boosted.

When input is textNot “multiple terms” but “text,” i.e., we don’t know which terms are key terms.

We developed extended naïve Bayes to solve this problem.Cannot observewhich are key

terms

𝑡1𝑡 2

𝑡𝐾

…

Apple

product

new

iPhone 𝑐

𝑃 (𝑐∨𝑡1)

𝑃 (𝑐∨𝑡2)

𝑃 (𝑐∨𝑡𝐾)

“iPhone”

11

Extended naïve Bayes𝑇 ′={𝑡1 }

…

𝑇 ′={𝑡1 , 𝑡2 }

𝑇 ′={𝑡1 ,⋯ ,𝑡𝐾 }

𝑇 …

Appleproduc

t

new

… Apple

Appleproduc

t

Appleproduc

t

new

Candidates of key

term

Apply naïve Bayesto each state T’

Probability that the set of key terms T is a

state T’ : …

12

Extended naïve Bayes𝑇 ′={𝑡1 }

…

𝑇 ′={𝑡1 , 𝑡2 }

𝑇 ′={𝑡1 ,⋯ ,𝑡𝐾 }

𝑇 …

Appleproduc

t

new

… Apple

Appleproduc

t

Appleproduc

t

new

Candidates of key

term

Apply naïve Bayesto each state T’

Probability that the set of key terms T is a

state T’ : ∑𝑇 ′

𝑃 (𝑐∨𝑇 ′)𝑃 (𝑇=𝑇 ′)=∏𝑘

(𝑃 ( 𝑡𝑘∈𝑇 )𝑃 (𝑐|𝑡𝑘 )+(1− 𝑃 ( 𝑡𝑘∈𝑇 ))𝑃 (𝑐 ))𝑃 (𝑐 )𝐾−1

…13

Extended naïve Bayes𝑇 ′={𝑡1 }

…

𝑇 ′={𝑡1 , 𝑡2 }

𝑇 ′={𝑡1 ,⋯ ,𝑡𝐾 }

𝑇 …

Appleproduc

t

new

… Apple

Appleproduc

t

Appleproduc

t

new

Candidates of key

term

Apply naïve Bayesto each state T’

Probability that the set of key terms T is a

state T’ : ∑𝑇 ′

𝑃 (𝑐∨𝑇 ′)𝑃 (𝑇=𝑇 ′)=∏𝑘

(𝑃 ( 𝑡𝑘∈𝑇 )𝑃 (𝑐|𝑡𝑘 )+(1− 𝑃 ( 𝑡𝑘∈𝑇 ))𝑃 (𝑐 ))𝑃 (𝑐 )𝐾−1

…14

Term dominance is incorporated into naïve Bayes

Experiments on short text sim datasets[Datasets] Four datasets derived from word similarity datasets using dictionary [Comparative methods] Original ESA [Gabrilovich07], ESA with 16 parameter settings[Metrics] Spearman’s rank correlation coefficient

15

ESA with well-adjusted parameter is superior to our method for “clean”

texts.

Tweet clusteringK-means clustering using the vector of related entities for

measuring distance[Dataset] 12,385 tweets including 13 topics

[Comparative methods] Bag-of-words (BOW), ESA with the same parameter, ESA with well-adjusted parameter

[Metric] Average of Normalized Mutual Information (NMI), 20 runs16

#MacBook (1,251) #Silverlight (221)#VMWare (890)#MySQL (1,241) #Ubuntu (988) #Chrome (1,018)#NFL (1,044) #NHL (1,045)#NBA (1,085)#MLB (752) #MLS (981) #UFC (991)#NASCAR (878)

Results

10 20 50 100 200 500 1000 200000.10.20.30.40.50.6

0.429 0.421 0.524

0.567

BOW ESA-sameESA-adjusted Our method

Number of related entities

NMI s

core p-value <

0.01

17

Results

10 20 50 100 200 500 1000 200000.10.20.30.40.50.6

0.429 0.421 0.524

0.567

BOW ESA-sameESA-adjusted Our method

Number of related entities

NMI s

core p-value <

0.01

18

Our method outperformed ESA withwell-adjusted parameter for noisy short

texts.

ConclusionWe proposed extended naïve Bayes to derive related Wikipediaentities given a real world noisy short text.

[Future work]Tackle multilingual short textsDevelop applications of the method

19