Interpretable and E�ective Opinion SpamDetection via Temporal Pa�ern Mining Across
Websites
Yuan Yuan, Sihong Xie, Chun-Ta Lu, Jie Tang and Philip S. Yu
Tsinghua University, Lehigh University and University of Illinois at Chicago
December 7, 2016
Online reviews & spam
Reviews and ratings influence our decisions
Spam reviews are misleading (the review below was filtered by Yelp)
Yuan et al. (BigData 2016) 2
Multiple review sites
One business may have information on multiple sites
What if we combine information on di�erent sites?
Yuan et al. (BigData 2016) 3
Basic idea: Bi-level framework
Yuan et al. (BigData 2016) 4
Main contributions
Proposed a novel spam detection framework using timeseries pa�erns defined over multiple data sources.
Performed in-depth studies to reveal a full picture of the de-fined pa�erns on two levels
Showed quantitative (prediction) and qualitative (casestudies) results demonstrate that the framework can preciselyidentify and explain a�acks that were not previously spo�ed
Yuan et al. (BigData 2016) 5
Single website time series construction
Useful single website Pa�ernsCount of Reviews, Average Rating, Five-star Ratio, Low-ratingRatio, Average Sentiment, Highly Positive Sentiment Ratio,Negative Positive Sentiment Ratio
e.g. Five-star Ratio: FRs(t) =∑
rs :time(rs )∈τt 1[rating(rs)=5]+αFRs
CRs(t)+α
Yuan et al. (BigData 2016) 6
Algorithm: Single site time series pa�ern detection
For each pair of segmentsCompute d = λ
(1/ |k1 |+1/ |k2 |)∆t+λ
Yuan et al. (BigData 2016) 7
Algorithm: Single site time series pa�ern detection
d = λ(1/ |k1 |+1/ |k2 |)∆t+λ > θ , and k1 > 0 and k2 < 0
a burst window is detected
Yuan et al. (BigData 2016) 8
Algorithm: Single site time series pa�ern detection
d = λ(1/ |k1 |+1/ |k2 |)∆t+λ > θ , and k1 > 0 and k2 < 0
a burst window is detected
Yuan et al. (BigData 2016) 9
Algorithm: Single site time series pa�ern detection
d = λ(1/ |k1 |+1/ |k2 |)∆t+λ > θ , and k1 < 0 and k2 > 0
a dive window is detected
Yuan et al. (BigData 2016) 10
Algorithm: Single site time series pa�ern detection
d = λ(1/ |k1 |+1/ |k2 |)∆t+λ > θ , and k1 < 0 and k2 > 0
a dive window is detected
Yuan et al. (BigData 2016) 11
Algorithm: Single site time series pa�ern detection
d = λ(1/ |k1 |+1/ |k2 |)∆t+λ > θ , and k1 < 0 and k2 > 0
a dive window is detectedtake the union of detected burst/dive windows
Yuan et al. (BigData 2016) 12
Algorithm: Single site time series pa�ern detection
each time window is classified into burst/dive/plateau
Yuan et al. (BigData 2016) 13
Cross-site time series pa�ern design and construction
detect single-site pa�erns in di�erent sites
combine the simultaneous pa�erns
assumption: di�erent cross-site pa�erns have di�erent spamratio (validate on dataset)
Yuan et al. (BigData 2016) 14
Data setup
Raw data
Foursquare: crawled 301,717 venues
Yelp: Yelp challenge dataset1
Matched by names and locations
95 businesses
Foursquare: 15,004 reviews, 12,147 reviewers
Yelp: 68,517 reviews, 31,092 reviewers
1http://www.yelp.com/dataset_challengeYuan et al. (BigData 2016) 15
Basic statistics of cross-site pa�erns
Table: Cross-Site pa�ern statistics
Pa�ern
(Y-F)
Yelp Foursquare
#bus
ines
s
#rev
iew
#rev
iew
er
#rel
ated
revi
ews
filte
red
rati
o
#bus
ines
s
#rev
iew
#rev
iew
er
BB 7 181 179 19133 27.07% 9 89 83BP 27 821 772 127427 26.31% 27 200 186BD 8 295 290 41713 18.98% 9 122 114PB 51 3795 3187 636679 13.68% 52 1154 1089PP 95 59830 23509 9364943 11.99% 95 12152 9491PD 33 3024 2589 548993 15.41% 34 1036 943DB 4 76 76 10321 21.05% 6 79 74DP 10 303 300 23822 48.18% 9 73 71DD 4 192 190 21059 28.13% 6 99 96
Yuan et al. (BigData 2016) 16
Human evaluation
Three human annotators independently label the sampled reviewsusing 3 levels of suspiciousness (1: not suspicious, 2: likely suspiciousand 3: very suspicious.)
Table: Human annotation results
Pa�erns # reviews Avg Scores Prec(> 1) Prec(> 2)B∗ 93 1.9785 0.9677 0.3871BB 18 1.9074 0.8889 0.4444BP 75 1.9956 0.9867 0.3733PB 68 2.0098 0.8971 0.3824PP 55 1.8606 0.9091 0.2909PD 14 1.7857 0.7857 0.2857
Yuan et al. (BigData 2016) 17
Microscopic classification - Behavioral Features
Table: Microscopic behavioral features of reviewers and reviews, and theircorrelations with the ground truths
Feature Corr. Description
DC +0.252 Proportion of days when a reviewer posts reviewson businesses in di�erent cities.
DS +0.230 Proportion of days when a reviewer posts reviewson businesses in di�erent states.
MP +0.183 Proportion of days when a reviewer posts 3 or morereviews.
LRR -0.148 Proportion of reviews with 1 or 2 stars posted by areviewer.
FRR +0.121 Proportion of reviews with 5 stars posted by a re-viewer.
RC +0.086 Sum of reviews posted by a reviewer.
Yuan et al. (BigData 2016) 18
Microscopic classification - Textual Features
Table: Microscopic textual features of reviewers and reviews, and theircorrelations with the ground truths
Feature Corr. Description
LC -0.010 Sum of le�ers in a review.
CWR +0.106 Proportion of ALL-CAPITAL words. (“I" excluded)
CLR +0.065 Proportion of capital le�ers.
1PP -0.034 Proportion of first person pronouns.
2PP +0.094 Proportion of second person pronouns.
EX +0.032 Proportion of exclamation.
Yuan et al. (BigData 2016) 19
Classification - Results
Prior methods [Rayana et al 2015]
0.0 0.2 0.4 0.6 0.8 1.0False Positive Rate
0.0
0.2
0.4
0.6
0.8
1.0
Tru
e P
osi
tive R
ate
B+T ROC (AUC = 0.65)
B ROC (AUC = 0.67)
T ROC (AUC = 0.55)
Random
0.0 0.2 0.4 0.6 0.8 1.0Recall
0.0
0.2
0.4
0.6
0.8
1.0
Pre
cisi
on
B+T Precision-Recall curve
B Precision-Recall curve
T Precision-Recall curve
Yuan et al. (BigData 2016) 20
Classification - Results
Linear regression
0.0 0.2 0.4 0.6 0.8 1.0False Positive Rate
0.0
0.2
0.4
0.6
0.8
1.0
Tru
e P
osi
tive R
ate
B+T ROC (AUC = 0.70)
B ROC (AUC = 0.68)
T ROC (AUC = 0.60)
Random
0.0 0.2 0.4 0.6 0.8 1.0Recall
0.0
0.2
0.4
0.6
0.8
1.0
Pre
cisi
on
B+T Precision-Recall curve
B Precision-Recall curve
T Precision-Recall curve
Yuan et al. (BigData 2016) 21
Case studies
Table: Case study: representative reviews (the codes under the site namesindicate detected pa�erns)
Representative reviews
Yelp
CR: P
AR: B
FR: B
LR: D
(5 stars)... really was awesome to be there. I don’t knowwhy people are complaining, ...
(5 stars) Ignore the negative reviews... that part was funin itself!(5 stars) ... I don’t know why people are complaining, theydon’t even have to have it opened, but they do. Enjoy it!
(5 stars) ... parking is FREE... they have items on displayfrom $100,000 and more to magnets of the cast for $8.00...
Yuan et al. (BigData 2016) 22
Case studies
Table: Case study: representative reviews (the codes under the site namesindicate detected pa�erns)
Representative reviews
Foursquare
CR: B
AS: D
HPSR: P
NSR: B
Waste of a trip!
They are way over priced on everything, including therefrancised items from the show.Extremely overpriced, they got famous on TV and nowscrew everyone with high prices!
An exhilirating experience. I find going to dumps andalmost ge�ing murdered exhilirating.
Waste of time‼!
Yuan et al. (BigData 2016) 23
Conclusion
MotivationCombine information across multiple sites
Proposed a bi-level frameworkMacroscopic to Microscopic
MacroscopicSingle-site pa�erns
Cross-site pa�erns
Human annotation
MicroscopicClassifications (Prior models and Linear Regressions)
Case studies
Yuan et al. (BigData 2016) 24