The Impact of Fake Reviews on Online Visibility: A ...cgi.di.uoa.gr/~gvalk/pubs/FakeReviewsImpact.pdf · Theodoros Lappas, Gaurav Sabnis, Georgios Valkanas Stevens Institute of Technology,

This article was downloaded by [155246152198] On 13 November 2016 At 1054Publisher Institute for Operations Research and the Management Sciences (INFORMS)INFORMS is located in Maryland USA

Information Systems Research

Publication details including instructions for authors and subscription informationhttppubsonlineinformsorg

The Impact of Fake Reviews on Online Visibility AVulnerability Assessment of the Hotel IndustryTheodoros Lappas Gaurav Sabnis Georgios Valkanas

To cite this articleTheodoros Lappas Gaurav Sabnis Georgios Valkanas (2016) The Impact of Fake Reviews on Online Visibility A VulnerabilityAssessment of the Hotel Industry Information Systems Research

Published online in Articles in Advance 09 Nov 2016

httpdxdoiorg101287isre20160674

Full terms and conditions of use httppubsonlineinformsorgpageterms-and-conditions

This article may be used only for the purposes of research teaching andor private study Commercial useor systematic downloading (by robots or other automatic processes) is prohibited without explicit Publisherapproval unless otherwise noted For more information contact permissionsinformsorg

The Publisher does not warrant or guarantee the articlersquos accuracy completeness merchantability fitnessfor a particular purpose or non-infringement Descriptions of or references to products or publications orinclusion of an advertisement in this article neither constitutes nor implies a guarantee endorsement orsupport of claims made of that product publication or service

Copyright copy 2016 INFORMS

Please scroll down for articlemdashit is on subsequent pages

INFORMS is the largest professional society in the world for professionals in the fields of operations research managementscience and analyticsFor more information on INFORMS its publications membership or meetings visit httpwwwinformsorg

Information Systems ResearchArticles in Advance pp 1ndash22ISSN 1047-7047 (print) ISSN 1526-5536 (online) httpsdoiorg101287isre20160674

copy 2016 INFORMS

The Impact of Fake Reviews on Online VisibilityA Vulnerability Assessment of the Hotel Industry

Theodoros Lappas Gaurav Sabnis Georgios ValkanasStevens Institute of Technology Hoboken New Jersey 07030

tlappasstevensedu gsabnisstevensedu gvalkanastevensedu

Extant research has focused on the detection of fake reviews on online review platforms motivated by thewell-documented impact of customer reviews on the usersrsquo purchase decisions The problem is typically

approached from the perspective of protecting the credibility of review platforms as well as the reputation andrevenue of the reviewed firms However there is little examination of the vulnerability of individual businessesto fake review attacks This study focuses on formalizing the visibility of a business to the customer base and onevaluating its vulnerability to fake review attacks We operationalize visibility as a function of the features thata business can cover and its position in the platformrsquos review-based ranking Using data from over 23 millionreviews of 4709 hotels from 17 cities we study how visibility can be impacted by different attack strategiesWe find that even limited injections of fake reviews can have a significant effect and explore the factors thatcontribute to this vulnerable state Specifically we find that in certain markets 50 fake reviews are sufficient foran attacker to surpass any of its competitors in terms of visibility We also compare the strategy of self-injectingpositive reviews with that of injecting competitors with negative reviews and find that each approach can be asmuch as 40 more effective than the other across different settings We empirically explore response strategiesfor an attacked hotel ranging from the enhancement of its own features to detecting and disputing fake reviewsIn general our measure of visibility and our modeling approach regarding attack and response strategies shedlight on how businesses that are targeted by fake reviews can detect and tackle such attacks

Keywords customer reviews fake reviews knowledge management vulnerability assessment decisionsupport systems

History Rob Fichman Ram Gopal Alok Gupta Sam Ransbotham Senior Editors Alok Gupta AssociateEditor This paper was received on March 2 2015 and was with the authors 8 months for 2 revisionsPublished online in Articles in Advance November 9 2016

1 Introduction

Mrs Richards When I pay for a room with a view Iexpect something more interesting than that

Basil That is Torquay madamMrs Richards Well itrsquos not good enoughBasil Well may I ask what you expected to see out

of a Torquay hotel bedroom window Sydney OperaHouse perhaps The Hanging Gardens of BabylonHerds of wildebeest sweeping majestically 0 0 0

Mrs Richards Donrsquot be silly I expect to be able tosee the sea

Basil You can see the sea Itrsquos over there between theland and the sky

Mrs Richards Irsquod need a telescope to see that

Fawlty TowersEpisode ldquoCommunication Problemsrdquo1979 British Broadcasting Corporation

This humorous exchange between proprietor BasilFawlty and dissatisfied customer Mrs Richards illus-trates an important phenomenon in the hospitalityindustry in which customers book hotels based pri-marily on features that are important to them ForMrs Richards the view from her window is impor-tant For someone else the top priority could be free

breakfast or air-conditioning Large travel websitessuch as TripAdvisorcom or Bookingcom allow usersto filter the hotels that they consider based on the fea-tures that they are interested in The popularity of suchplatforms is largely based on the availability of largevolumes of customer reviews which are considered tobe more credible than biased promotional campaigns(Bickart and Schindler 2001 Lu et al 2013) Relevantliterature has established the impact of reviews onpurchase decisions (Chatterjee 2001 Kwark et al 2014Ghose et al 2014 Duan et al 2008) and consequentlyon a firmrsquos sales and revenue (Ghose and Ipeiro-tis 2011 Forman et al 2008 Zhu and Zhang 2010)Nevertheless online reviews are also susceptible totampering from unscrupulous businesses who attemptto manipulate the available information by postingeither fake positive reviews about themselves or fakenegative reviews about their competitors resulting inreview fraud (Dellarocas 2006 Mayzlin et al 2012Lappas 2012 Luca and Zervas 2016) Although a con-siderable amount of research has focused on the iden-tification of fake reviews (Hu et al 2012 Lappas 2012Xie et al 2012 Jindal and Liu 2008 Mukherjee et al2013a) we still have a relatively limited understanding

1

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved

Lappas Sabnis and Valkanas The Impact of Fake Reviews on Online Visibility2 Information Systems Research Articles in Advance pp 1ndash22 copy 2016 INFORMS

of the nature and nuances of the vulnerability of thebusinesses themselves to review fraud To overcomesuch limitations our work studies the mechanisms bywhich fake reviews can mislead customers and affectthe targeted business Next we demonstrate the intu-ition behind our approach with a realistic example

Consider a businesswoman planning to arrive in anew city at 11 pm and spend the night preparing fora 10 am presentation the next day When she searchesfor hotels on an online platform she might choose freewi-fi as a filtering feature to have access to the Internetwhile working on her slides She might also ask for abusiness center to print documents for the presenta-tion When the same businesswoman travels with herfamily for vacation her preferences are likely to dif-fer For instance she might filter her results based onthe availability of a swimming pool that her childrencan enjoy or limit her search to hotels that offer fam-ily rooms In both scenarios the hotel she eventuallychooses will be among those that appear as a responseto her filtered search The website presents these com-petitive hotels in a ranked list in which the position ofeach competitor depends on its reviews (TripAdvisor2013 Holloway 2011a)

Previous work has repeatedly verified that the in-creased visibility of highly ranked items improves theirchances of being considered and selected by interestedusers (Pan 2015 Ghose et al 2014 Ifrach and Johari2014 Tucker and Zhang 2011) In this setting a hotelrsquosvisibility is determined by the set of features that it cancover the popularity of these features among potentialcustomers and the hotelrsquos position in the review-basedranking Even though the first two elements are imper-vious to manipulation the third visibility componentis clearly vulnerable to fake reviews Intuitively theinjection of a sufficient number of positive or nega-tive fake reviews could alter the ranking and have asignificant impact on the visibility of the ranked busi-nesses The pressing concern for such injection-basedattacks motivates us to contribute to the growing liter-ature on online reviews from a vulnerability perspec-tive by focusing on the following research objectives(i) formalizing the visibility of a business by defininga measure that considers its position in the review-based ranking (ii) understanding how visibility can beimpacted by review fraud and (iii) identifying effec-tive ways for businesses to protect their visibility fromfake review attacks

The cornerstone of our research methodology is thevisibility construct Our work identifies the compo-nents that contribute to an itemrsquos visibility within anonline platform and describes the operationalizationof each component First we use a large data set ofover 23 million reviews of 4709 hotels from 17 citiesto estimate the popularity of the different hotel fea-tures Our study reveals consistent user preferences

across cities and also provides specific guidelines onthe amount of review data that are sufficient to obtainaccurate estimates For the review-based ranking weevaluate the industry-standard average rating functionas well as TripAdvisorrsquos Popularity Index formulawhich considers the age quantity and quality of ahotelrsquos reviews (TripAdvisor 2013) Our frameworkalso accounts for the different ways in which users con-sider items in ranked lists We implement and evalu-ate three alternative consideration models motivatedby the significant body of relevant literature Finallywe simulate three different attack strategies to evalu-ate the vulnerability of visibility to fake reviews Ourevaluation reveals the factors that determine a hotelrsquosvulnerability and leads to findings with strong impli-cations about the ways in which platforms rank com-petitive hotels We address the vulnerabilities exposedby our study by proposing a fraud-resistant methodfor review-based ranking and suggesting differentresponse strategies for businesses that want to protecttheir visibility from fake reviews

The remainder of the paper is organized as followsFirst we discuss relevant work in Section 2 We de-scribe our TripAdvisor data set in Section 3 We thendescribe our visibility construct in Section 4 In Sec-tion 5 we describe attack strategies based on fakereviews Then in Section 6 we discuss response strate-gies that can be used to address the exposed vul-nerabilities Finally we conclude in Section 7 with adiscussion of our findings and their implications aswell as an overview of directions for future research

2 BackgroundFake reviews have been acknowledged as a criti-cal challenge by both the research community (Fenget al 2012) and the e-commerce industry (Sussin andThompson 2012 Breure 2013) Despite the commit-ment of review platforms to combat fraud the percent-age of fake reviews is estimated to be around 15ndash30(Sussin and Thompson 2012 Luca and Zervas 2016Belton 2015) In a recent formal complaint filed incourt by Amazoncom (Stempel 2015) the retail giantwrote ldquoWhile small in number these reviews threatento undermine the trust that customers and the vastmajority of sellers and manufacturers place in Ama-zon thereby tarnishing Amazonrsquos brandrdquo In October2015 the company filed a lawsuit against 1000 peo-ple who allegedly offered to hire themselves out asfake reviewers (Weise 2015) This lawsuit is a mile-stone in the ongoing fight against fake reviews as itwas the first legal action taken by a large corporationagainst individual users In the same year the Italianmagazine Italia a Tavola created a fictitious restauranton TripAdvisor The restaurant secured the highest

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved

Lappas Sabnis and Valkanas The Impact of Fake Reviews on Online VisibilityInformation Systems Research Articles in Advance pp 1ndash22 copy 2016 INFORMS 3

rank among all competitors in the region via the injec-tion of fake positive reviews (Fenton 2015) This con-troversial experiment came a year after TripAdvisorwas issued a $613000 fine by the Italian CompetitionAuthority for failing to adopt controls against falsereviews while promoting its content as ldquoauthentic andgenuinerdquo (Masoni 2014)

The prevalence of review fraud can be largely attrib-uted to the plethora of professional review-authoringcompanies that submit fake reviews on major reviewplatforms in exchange for a fee In fact these compa-nies have grown so confident in their ability to contin-uously adapt and avoid detection that they are oftenwilling to delay their compensation until after thefake reviews have penetrated the websitersquos defenses1

This phenomenon has prompted action by govern-ment bodies worldwide In 2015 the Attorney Gen-eral of the State of New York spearheaded ldquoOperationClean Turfrdquo an effort to identify and expose review-authoring firms (Schneiderman 2015) The operationresulted in the identification of 19 companies whichwere forced to cancel their services and pay over$350000 in fines Despite such ongoing efforts a num-ber of similar companies are still active and even will-ing to inject defamatory reviews about the competitorsof the business that hires them In the same year astudy by the UKrsquos Competition and Markets Author-ity (CMA) identified a growing number of compa-nies that offer review-authoring services (CMA 2015)Additional findings from the study include the devas-tating effects of fake negative reviews especially forsmall businesses as well as the customer practice ofusing defamatory reviews to blackmail businesses intoproviding some concession such as a price discountFinally the study provided valuable insight on the dis-tribution of fake reviews which we discuss in detail inSection 21

21 Studying the Distribution of Fake ReviewsOne of the main findings of the CMA study was thatfake positive reviews are more common than fake neg-ative reviews As a possible explanation the studyhypothesized that it is easier and less risky to affectoverall ratings by self-injecting positive reviews thanby posting a series of negative reviews about manybusiness rivals This is intuitive as culprits caughtsabotaging the reviews of competitors can face seri-ous repercussions in terms of lawsuits and long-termdamage to their brand Therefore a reasonable hypoth-esis would be that the practice of injecting negativereviews to competitors is more likely to be adoptedby new or small businesses which do not have anestablished brand to damage if they get caught How-ever high-profile cases that expose corporate giants

1 httprealtripadvisorreviewscom

as solicitors of fake negative reviews suggest a muchwider reach of this phenomenon For instance in 2013the Taiwan Fair Trade Commission fined Samsung$340000 for hiring two external companies to postnegative reviews about HTC one of Samsungrsquos maincompetitors The story which was extensively cov-ered by media outlets demonstrated that review fraudis a widespread practice that is not limited to smallbusinesses (Tibken 2013) Furthermore the relevantresearch literature on the distribution of fake reviewsis relatively limited and also provides some conflict-ing evidence For instance while positive fake reviewsare generally accepted as more prevalent and easierto generate (CMA 2015 Mayzlin et al 2012) evidencefrom multiple domains shows that reviews by nonver-ified buyers (which are more likely to be fraudulent)are significantly more negative than those by verifiedbuyers (Anderson and Simester 2014)

Given the ambiguity of previous findings we con-duct our own study on a data set collected fromYelpcom which applies a proprietary filter to identifyfraudulent and other low-quality reviews (Kamerer2014) Even though such reviews are not prominentlydisplayed on the page of the reviewed business theyremain online and can be viewed by the interested userPrevious work has verified these reviews as an accu-rate proxy for the set of fake reviews detected by theplatform (Luca and Zervas 2016) Our data set consistsof about 15000 hotel reviews Out of these about 56were positive (four or five stars) 29 were negative(one or two stars) and the rest did not have a clearpositive or negative rating (three stars) These percent-ages suggest that even though fake positive reviewsare indeed more prevalent fake negative reviews stillmake up a significant part (around one-third) of allreview injections In practice these percentages arelikely to differ across platforms and domains In addi-tion the estimation task is inherently problematic aswe cannot account for the undetected fake reviewsthat penetrated the platformrsquos defenses The attack-simulation framework that we present in Section 5 isthe first attempt toward addressing this limitation

22 Defense MechanismsThe growing concern over fake reviews has moti-vated review platforms to implement different types ofdefense mechanisms and review filters For instancereviews posted on TripAdvisor are subject to reviewby the websitersquos moderators (TripAdvisor 2015a) Sus-picious reviews can then be placed on hold pendingexamination and can even be eliminated if the web-sitersquos proprietary filtering process provides enoughevidence Businesses that are associated with fakereviews are penalized in the websitersquos rankings ex-cluded from press releases and top-10 lists and mayeven have a relevant banner placed on their page

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


While such penalties serve as deterrents they havealso been exposed as a creative way for an unscrupu-lous business to defame its competitors Specificallyby repeatedly injecting fake positive reviews about acompetitor a malicious business can manipulate theplatform into detecting an attack and penalizing theinjected competitor (TripAdvisor 2015a) In additiongiven the scarcity of ground truth data and the diffi-culty of detecting fake reviews filtering mechanismsare likely to lead to false positives (CMA 2015)

One of the most popular defense mechanismsagainst fake reviews is the ldquoVerified Buyerrdquo (VB)badge which only allows reviews by users that havepurchased the item or service Verified reviews havebeen adopted by multiple platforms and praised asa promising solution to review fraud (Mayzlin et al2012 May 2011) However as we discuss next theyalso introduce limitations and can even lead to unde-sirable consequences

First a motivated attacker or professional review-authoring firm can still buy a competitorrsquos productor service (especially in low-cost markets) earn theVB badge and proceed to inject fake negative reviewsThe task of self-injecting fake positive reviews wouldthen be even easier since the purchase cost wouldessentially return to the culprit Second only the per-son who used their information to make the purchasewould be able to submit a review In the hotels domainthis excludes friends and family members who stayedin the same room but would not have the chanceto share their insights Similarly verification excludescustomers who do not book through a website andprefer to pay cash or go through a travel agency Thislimits the reviewer demographic and could introducean unwanted bias to the ratings This is a legitimateconcern especially because only an estimated 43 ofall hotel bookings are done online (StatisticBrain 2015)In fact online payments are also not the norm for otherbusinesses in the service industry such as restaurantsmaking verification a challenging task Finally giventhat online reviews are public a website that allowsonly verified customers will unavoidably reveal infor-mation about its users This could raise privacy con-cerns and discourage reviewers who do not want toopenly share their purchase decisions While such con-cerns could be overcome by allowing reviewers to hidetheir identities this would also take away their abilityto build their reputation within the online communitywhich has been shown to be one of the principal moti-vations for review authoring (Wang 2010)

The above concerns about the VB badge have dis-couraged many platforms including TripAdvisor thelargest travel website in the world In a recent 2015statement the platform defended its choice to allownonverified reviews (Schaal 2015b) ldquoWe have con-sidered all of the verification options out there and

have elected to use our current model for one sim-ple reason2 The volume of opinions provides for themost in-depth coverage of consumer experience andit is the model that consumers preferrdquo A hybrid pol-icy that allows both verified and unverified reviewshas enabled TripAdvisor to accumulate over 250 mil-lion reviews a number that far surpasses that of itscompetitors (Schaal 2015a Olery 2015 TripAdvisor2015b) The website also boasts the largest marketshare among all travel websites in the hotel industryin terms of traffic more than twice its nearest com-petitor Bookingcom (Tnooz 2013) which only allowsverified reviews TripAdvisorrsquos unwillingness to sac-rifice volume for verification is consistent with rele-vant research which identifies the number of reviewsas the factor with the highest influence on the usersrsquotrust in the platform higher even than that of the qual-ity and detail of the reviews (Breure 2013) The hybridapproach has also been adopted by other major travelwebsites including Orbitzcom and InsiderPagescomas well as by market leaders in other industries suchas Amazoncom

In conclusion while buyer verification can supportthe effort against fake reviews it also has a numberof limitations and undesirable effects This has moti-vated large review platforms to opt for a hybrid pol-icy that allows both verified and unverified reviewerswhile bolstering their effort to identify and eliminatefake reviews Our goal is to make a decisive contri-bution to this effort by quantifying the vulnerabilityof review-based visibility to different types of reviewfraud and presenting informed response strategies forvigilant businesses and review platforms

3 The TripAdvisor Data SetWe use a large data set of reviews from Trip-Advisorcom one of the largest review platforms andtravel portals The data were collected during the lastweek of January 2015 and include over 23 millionreviews on 4709 hotels from 17 cities Each hotel ismapped to a space of 18 features for which we adoptthe representation presented by Ding et al (2008) intheir seminal work on lexicon-based opinion miningEach feature f isin F is represented via a finite set ofwords or phrases Wf which includes synonyms andother terms with a strong semantic connection to thefeature Table 1 shows the set of words for each featureFor example any occurrence of the words ldquoInternetrdquo orldquowifirdquo is mapped to the ldquoInternetrdquo feature In additionwe use the opinion-mining algorithm of Ding et al(2008) to analyze the sentiment (positivenegative)and the strength of the opinions expressed abouteach feature within a review The algorithm assignsa value in the 6minus11+17 range to each reviewed fea-ture with minus1 and +1 indicating a highly negative and

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


Table 1 Hotel Features and Their Respective Sets of Relevant Words

Feature f Set of relevant keywords Wf

1 Air-conditioning ac ac air condition(ing ed) roomtemperature

2 Airport transportation airport transportation airport transportairport shuttle airport ride airport bus

3 BarLounge bar lounge sitting area cafeteria cafe4 Business services business center business area business

service(s) conference room conferencearea

5 Concierge concierge doorman attendant6 Fitness fitness workout gym exercise athletics

center7 Breakfast breakfast morning meal8 Parking parking park (any word) car9 Internet Internet wi-fi wifi wireless network

ethernet10 Kitchenette kitchenette kitchen cooking area oven

stove11 Nonsmoking nonsmoking smoking smoker smoke

cigarette(s)12 Pets pet(s) cat(s) dog(s) bird(s) parrot(s)

pet friendly animal(s)13 Pool pool swimming pool14 Reduced mobility reduced mobility limited mobility disabled

disability disabilities handicappedwheelchair ramp

15 Restaurant restaurant buffet food16 Room service room service17 Spa spa sauna massage18 Suites suite suites

highly positive sentiment respectively The sentimentis then aggregated to the hotel level via averagingWe utilize the opinions mined from our data set for(i) the robustness check of our method for estimating

Figure 1 An Illustration of a Typical User Session Given the Features Required by the User the Review-Based Ranking Is Filtered to EliminateHotels That Cannot Cover the Requirements

Note The filtered ranking is then shown to the user who considers it prior to making a purchase decision

the importance of queries presented in Section 411and (ii) the feature enhancement strategy describedin Section 61 Additional details on the data are pro-vided in Table 1 of the online appendix (availableas supplemental material at httpsdoiorg101287isre20160674)

4 MeasuringVisibilityPrevious studies have focused on evaluating the effectsof customer reviews on booking intentions (Mauri andMinazzi 2013 Sparks and Browning 2011 Vermeulenand Seegers 2009) and sales (Ye et al 2009 2011) Whilethe consensus is that reviews can significantly affectuser decisions the precise mechanism by which thiseffect occurs is typically overlooked Instead the cor-pus of reviews about an item is simply represented bymeasures that capture aggregate valence such as theaverage star rating and the number of positive or neg-ative reviews In practice however the utilization ofreviews by online platforms goes beyond the compu-tation of such simple measures Specifically the set ofreviews about each item serves as the input to a rank-ing function The function then translates the reviewsinto a score that is used to rank the item among itscompetitors As we discuss in detail in Section 42ranking functions are typically proprietary and tendto differ across platforms However the underlyingtheme is that items with better reviews are rankedhigher and receive increased visibility within the plat-form Consider the illustration presented in Figure 1which demonstrates the typical session of the averageuser on a review platform such as TripAdvisorcom

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


or Bookingcom First the user specifies her require-ments with respect to different hotel features such asInternet access breakfast a business center and a poolThe platform then filters the ranking by eliminatinghotels that cannot cover the specified requirements andreturns a filtered list of qualified candidates The userthen browses through this ranked list and considersa subset of the ranked items by clicking on them toobtain further information and ultimately make a pur-chase decision We refer to this subset as the userrsquosconsideration set

As illustrated by this example a hotelrsquos visibility isdetermined by two factors

1 The hotelrsquos position in the review-based ranking Intu-itively hotels with a higher ranking are more likelyto be considered by the user This intuition has beenverified by numerous studies that have demonstratedthe significant impact of ranking on visibility In Sec-tion 43 we provide a detailed literature review anddescribe how the usersrsquo tendency to favor hotelswith higher rankings is taken into account by ourframework

2 The set of features that the hotel can cover By pro-viding more amenities a hotel is able to cover therequirements of a larger part of the customer baseThe popularity of the provided amenities is also cru-cial a hotel that cannot cover a popular requirement(eg breakfast) will be eliminated by a large numberof users On the other hand the lack of a less popularamenity (eg a kitchenette or a spa) is less detrimentalto a hotelrsquos visibility

Our work combines these two factors to delivera fully operationalized definition of visibility with aprobabilistic interpretation Specifically we define anitemrsquos visibility as the probability that it is included inthe consideration set of a random user Formally theuser specifies the query q of features that she requireswhere q is a subset of the universe of all possible fea-tures F A query q is sampled with probability p4q5from a categorical distribution with 2F possible out-comes such that

sum

qisin2F p4q5 = 1 As illustrated in Fig-ure 1 the website maintains a global review-basedranking GH of all of the hotels in the city Given theuserrsquos query q the platform returns a filtered rank-ing Gq

H including only the hotels that can cover all ofthe features in q A hotel h is included in the userrsquos con-sideration set with probability Pr4h q5 which dependson its rank in Gq

H Finally we define the visibility of anyhotel h isinH as follows

v4h5=sum

qisin2Fp4q5times Pr4h q50 (1)

The following three sections describe the compo-nents of our visibility framework in detail Section 41

describes two methods for estimating the probabil-ity p4q5 for every possible query of features q isin 2F Sec-tion 42 describes two alternative functions for comput-ing the global review-based ranking GH of a given setof competitive items H Finally Section 43 describesthree different simulation models for the considerationprobability Pr4h q5

41 Using Big Data to Estimate User PreferencesOur definition of visibility considers the probabilityp4q5 that a random customer is interested in a specificquery of features q for every possible q isin 2F Next wedescribe how we can estimate these probabilities fromreal data Ideally an algorithm with access to exten-sive logs of user queries could be used to learn therequired probabilities (Baeza-Yates et al 2005) In prac-tice however the sensitive and proprietary nature ofsuch information makes it very hard for firms to shareit publicly This is a typical limitation for research onsearch behavior (Korolova et al 2009) Cognizant ofthis limitation we present an estimation process basedon a resource that is already available for the evalua-tion of review-based visibility customer reviews

Extant research has validated the process of using re-views to estimate user preferences in multiple domains(Marrese-Taylor et al 2013 Ghose et al 2012 Deckerand Trusov 2010 Leung et al 2011) An intuitive ap-proach would be to estimate the demand for eachfeature separately and then aggregate the individualestimates at the query level However this approachassumes that the occurrence of a feature in a query qis not affected by the other features in q To avoid thisassumption and capture possible correlations we con-sider the set of features mentioned in each review asa single query We then compute the probability of aquery q by computing its frequency in our review cor-pus R and dividing it by the sum of all query frequen-cies Formally

p4q5=freq4q1R5

sum

qisin2F freq4qprime1R50 (2)

Ideally we would have access to the set of require-ments of all possible customers The maximum likeli-hood estimate of Equation (2) would then deliver thetrue probability of any query q While this type ofglobal access is unrealistic Equation (2) can still deliveraccurate estimates if the number of reviews in R islarge enough to accurately represent the customer pop-ulation The usefulness of the estimator is thus deter-mined by a simple question How many reviews do weneed to achieve accurate estimates As we demonstratenext the unprecedented availability of reviews allowsus to answer this question and validate the estimatedprobabilities

We conduct our study as follows first we merge thereviews from all of the hotels in a city into a single large

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


Figure 2 Evaluating the Convergence of Query Probability Estimates with Respect to the Number of Reviews

0 200 400 600 800 1000

Number of reviews

0

02

04

06

08

10

L1

dist

ance

Segment size = 25Segment size = 50Segment size = 100

(a) Los Angeles

0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance

(b) New York City (c) San Francisco

0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance

(d) Chicago (e) Philadelphia (f) London

Number of reviews Number of reviews

Number of reviews Number of reviews Number of reviews

0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10

set sort them by submission date and split the sortedsequence into fixed-size segments We then iterativelyappend segments to the review corpus R consideredby Equation (2) and recompute the probability of eachquery The vector of probabilities from the ith iterationis compared with that from the 4i minus 15th iteration viathe L1 distance the sum of the absolute differences ofcorresponding queries We repeat the process for seg-ments of 25 50 and 100 reviews Figure 2 shows theresults for 6 cities The results for the remaining citieswere nearly identical and are omitted for lack of space

First we observe near-identical trends for all citiesThis is an encouraging finding which suggests thatour conclusions will generalize beyond the citiesin our data set Second the figures clearly demon-strate the rapid convergence of the probabilities withthe reported L1 distance dropping to trivial levelsbelow 001 after the consideration of less than 1000reviews In fact 600 reviews were generally sufficientto achieve a distance of around 001 for all cities Thistrend is consistent for all three segment sizes As antic-ipated adding larger segments is more likely to delaythe convergence However even for segments of 100reviews the observed trend was the same in termsof both the rapid decay and the convergence to alow of around 001 after around 1000 reviews Therapid convergence of the probabilities is an especiallyencouraging finding that (i) reveals a stable categor-ical distribution for the preferences of the users overthe various feature queries and (ii) demonstrates that

1000 reviews are sufficient to converge to this distri-bution a number that is orders of magnitude smallerthan the hundreds of thousands of reviews available inour data set

The consistent underlying distribution motivates usto study its properties and consider possible impli-cations for our work We begin by plotting thecumulative distribution function (CDF) for each cityin Figure 3 in which the queries on the x-axis aresorted in descending order of their respective proba-bilities The results reveal that the 500 most popularqueries in each city cover about 90 of the distribu-tion This is an intriguing observation given that these500 queries account for only 002 of the 218 = 262144possible feature queries This finding has strong impli-cations for the efficiency of our computation ratherthan iterating over the exponential number of all pos-sible queries the results suggest that we can accuratelyapproximate visibility by focusing on a small subset ofthe most popular queries As we discuss in detail inSection 6 these queries can also help a business strate-gically select which features to improve to bolster itsvisibility Further investigation reveals a near-normaldistribution of the query size with around 70ndash80 ofthe queries including threendashsix features This findingcan inform methods for the estimation of query popu-larity as it constrains the space of sizes that have to beconsidered

411 Evaluating the Robustness of Our Estima-tion Method Our estimation method evaluates fea-tures in sets based on their occurrence in customer

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


Figure 3 The Cumulative Distribution Function of the Categorical Distribution Over the Queries

(a) Los Angeles

Perc

enta

ge c

over

ed

Queries sorted by probability

0

02

04

06

08

10

0 200 400 600 800 1000 1200

(b) New York City

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 100 200 300 400 500


(c) San Francisco

0 100 200 300 400 500 700


Perc

enta

ge c

over

ed

0

02

04

06

08

10

600

(d) Chicago

0 200 400 600 800 1000


Perc

enta

ge c

over

ed

0

02

04

06

08

10

(e) Philadelphia

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 200 400 600 800 1000


(f) London

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 50 100 150 200 250 300 350


reviews As a robustness check we study an alterna-tive approach that estimates the probabilities of indi-vidual features and then aggregates them at the querylevel For this study we adopt the utility model pro-posed by Li et al (2011) which uses regression analysisto estimate the sensitivity of a random user to eachfeature2 Formally given N hotels with K features letD be the N times1 vector of demands (ie hotel bookings)X be a N timesK feature matrix be a N times1 vector of coef-ficients and an iid random error term The modelcan then be written as follows

ln4D5=X+ 0 (3)

We take the logarithm of the dependent variable toaddress skewness in the demand Given that we donot have access to demand data we utilize the numberof reviews on each hotel as a proxy for D We pop-ulate the feature matrix X with the opinions that thereviewers of each business express on each feature asdescribed in Section 3 Then we estimate the coefficientvector via the standard ordinary least squares (OLS)approach Conceptually each coefficient gives us theimportance of the corresponding feature for a randomuser as captured by the demand proxy Given that ourgoal is to estimate query probabilities we normalizethe coefficients in into a pseudoprobability distribu-tion such that

sum

fisinF f = 1 Finally we need to definea formal way of computing the probability p4q5 of a

2 We thank one of the reviewers for suggesting this robustnesscheck

given query q based on the probabilities of the fea-tures that it includes A trivial approach would be tocompute the probability of a query q as p4q5=

prod

fisinq f However this would falsely penalize large queriesThus we adopt the following alternative formula

p4q5= q timesprod

fisinq

f 1 (4)

where q is the probability that a random user is inter-ested in exactly q features In other words is theprobability distribution of all of the possible subsetsizes (ie

sum

s s = 1) which we estimate via the dis-tribution of the number of features that appear in thereviews of our data set

Let v1 be the vector of probabilities of the top 500queries for a city as ranked by Equation (2) of thereview-based method described in the beginning ofthis section which considers all of the features in thereview as a single query Then let v2 be the correspond-ing vector computed via Equation (4) We compare thetwo vectors via Pearsonrsquos correlation coefficient (Pear-son 1895) a standard measure of the linear correlationbetween two variables The measure returns +1 for aperfect positive correlation 0 for no correlation andminus1 for a perfect negative correlation We focus on thetop 500 queries because as we discussed earlier in thesection they account for more than 90 of the entireprobability distribution We report the results in Fig-ure 4(b) For completeness we report the results of thesame process at the feature level in Figure 4(a) For thissecond test we compare the probabilities of individual

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


Figure 4 Comparing the Probabilities Learned by the Two Estimation Methods for Queries and Individual Features

0

02

04

06

08

10

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

NB

ER

PAR

BC

N

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

N

BE

RPA

RB

CN

Pear

sonrsquo

s co

eff

Cities

(a) Individual features

0

02

04

06

08

10

Pear

sonrsquo

s co

eff

Cities

(b) Queries

features (rather than queries) as estimated by the twomethods

The figures demonstrate a consistently high simi-larity between the results of the two methods Specif-ically the reported coefficients were around 008 forboth the query-level and feature-level comparisons forall 17 cities in our data set The correlation was evenhigher for the feature-level comparison with mostcities reporting near-perfect correlations Higher val-ues are anticipated for this comparison as the sizeof the compared vectors is considerably smaller (ie18 features versus 500 queries for the query-level com-parison) A secondary analysis that compares the twovectors via their L1 distance produced similar resultswith the reported values being consistently around 002and 0015 for the query-level and feature-level com-parisons respectively Overall the study validates theresults of our estimation method and demonstratesthat large review volumes are a valuable source formining user preferences This is an encouraging find-ing with positive implications for the ongoing researchefforts in this area which are often hindered by theabsence of query logs and similar data sets from largeplatforms Finally it is important to stress that our vis-ibility framework is compatible with any method thatcan assign a probability to a given query of features

42 Review-Based RankingIn this section we describe two alternative functionsfor the computation of the global review-based rank-ing GH of the set of hotels H in a given city The firstfunction is the standard average stars rating employedby a majority of online portals Formally let Rh be thecomplete set of reviews on a given hotel h The AverageRating formula is then defined as follows

gAV4h5=

sum

risinRhrating4r5Rh

0 (5)

The principal drawback of the gAV4 middot 5 function is thatit assigns an equal importance to all of the reviewsin Rh As a result it maintains the effect of old and pos-sibly outdated reviews In addition the function favors

older items that have more time to accumulate reviewsIn combination with the usersrsquo well-documented ten-dency to prefer items that are already popular (see Sec-tion 43 for a detailed literature review) this bias leadsto rich-get-richer effects and makes it harder for newhigh-quality items to rise in the rankings Motivatedby such shortcomings leading review platforms suchas Yelp and TripAdvisor employ proprietary rankingfunctions that consider the ldquofreshnessrdquo of the reviewsin addition to their quantity and quality A character-istic example is TripAdvisorrsquos Popularity Index whichconsiders the Quality Quantity and Age of the reviews(TripAdvisor 2013) To assess the practical implicationsof our work in realistic settings we use the large reviewdata set described in Section 3 to estimate TripAdvi-sorrsquos Popularity Index Next we describe the estima-tion process and verify the validity of the estimatedfunction via a prediction task

While we do not have access to the score assigned byTripAdvisorrsquos function to each hotel our data includethe official ranking of all of the hotels in each city ascomputed based on their scores Given the full set ofreviews Rh on a hotel h we first sort the reviews inRh by their date of submission from newest to oldestWe then split the sorted sequence into semesters anddefine Rt

h as the subset of reviews submitted during thet-th (most recent) semester The use of semesters deliv-ered the best results in our experimental evaluationwhich also considered months quarters and years Foreach semester Rt

h we consider the number of reviewsthat it includes as well as the corresponding averagerating Rt

h The PopularityIndex of an item h is thencomputed as follows

gPI4h5 = w1 middot 4R1h1 R2

h1 0 0 0 1 RTh 1 RgtT

h 5

+w2 middot 4R1h1 R

2h1 0 0 0 1 R

Th 1 R

gtTh 51 (6)

where RgtTh includes the reviews submitted before the

T -th semester Each interval is mapped to two coef-ficients one for the number of reviews and one forthe average rating This formulation allows us to con-trol the number of coefficients that need to be esti-mated by tuning the value of T The coefficients in

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


Table 2 Computed Coefficients for the gPI4 middot 5 Ranking Function

First semester Second semester Third semester Older reviews

R1h R1

h R2h R2

h R3h R3

h Rgt3h Rgt3

h

minus000003 minus20647 minus000004 minus00532 minus000004 minus00043 minus000002 minus00008

the w1 vector represent the importance of the num-ber of reviews submitted during each semester Simi-larly the coefficients in w2 represent the importance ofthe corresponding average rating From an optimiza-tion perspective w1 and w2 constitute a single vectorof coefficients that needs to be estimated to minimizethe difference between the ranking of our gPI4 middot 5 func-tion and the official TripAdvisor ranking Given twosets of rankings of hotels H in a city a pair of hotels4hi1hj5 is concordant if both rankings agree in the rela-tive order of hi and hj Given the corresponding hotelsets H11H21 0 0 0 1Hm from the m cities in our data setthe goal is to estimate a ranking function gPI4 middot 5 thatmaximizes the average percentage of concordant pairsper city As shown by Joachims (2002) this problemcan be formulated as a constrained optimization taskand solved by a linear support vector machine (SVM)We complete the optimization and obtain the coeffi-cients using SVM-rank3 the state of the art for trainingranking SVMs We set the number of semesters T via a10-fold cross validation On each of the 10 folds 90 ofthe cities in our data set were used to learn the coeffi-cients and predict the rankings for the remaining 10The best results were achieved for T = 3 for which 91of all possible hotel pairs were reported as concordantIncreasing the value of T led to near-zero values for theadditional coefficients and did not improve the resultsTable 2 holds the coefficients for T = 3

The high percentage of concordant pairs 4915 ver-ifies that our PopularityIndex function can accuratelyreproduce TripAdvisorrsquos ranking All of the com-puted coefficients were negative as anticipated be-cause lower rankings are desirable We also observe adeclining importance of the average rating for oldersemesters a finding that verifies that the functionfavors recent reviews Finally the coefficients of thenumber of reviews in each semester were consistentlyaround minus000003 which represents the marginal contri-bution of every additional review to the function

43 Computing the Consideration ProbabilitiesIn this section we describe three alternative models forthe consideration probability Pr4h q5 the probabilitythat a user with a query q includes a specific busi-ness h in her consideration set Our models are moti-vated by the extensive work on the impact of rankingon visibility In the hotels domain Ghose et al (2014)

3 httpwwwcscornelledupeopletjsvm_lightsvm_rankhtml

reported an average click-through rate (CTR) increaseof 10007 for a one-position improvement in rank Inthe Web apps domain Carare (2012) found that anapprsquos status as a best seller is a highly significant deter-minant of demand In the same domain Ifrach andJohari (2014) report that the top-ranked app received90 more downloads than the one ranked in the 20thposition Best sellers were also considered by Tuckerand Zhang (2011) who found that sales-based rank-ings have a significant impact on user behavior andlead to a rich-get-richer effect

Furthermore ranking effects have been extensivelystudied in the context of search engines Pan et al(2007) found that users strongly favor high-rankedresults even if they appear to be less relevant thanlower-ranked alternatives In a later study they foundthat the CTR for hotels follows a power-law distri-bution as a function of their rank on popular searchengines (Pan 2015) Brooks (2004) observed that anadrsquos click potential and conversion rate can drop sig-nificantly as its search engine rank deteriorates Hereported a nearly 90 drop between the first and 10thposition although the reductions between consecutivepositions did not follow a predictable pattern Hen-zinger (2007) found that the majority of search-engineusers do not look beyond the first three pages of theranked results In their eye-tracking studies Guan andCutrell (2007) found that users browse through rankeditems in a linear fashion from top to bottom They alsofound that users typically look at the first three to fouroptions even if they do not ultimately select one ofthem Similar findings were reported by Joachims et al(2005) and Lorigo et al (2008)

The extensive amount of relevant work reveals threeinteresting findings (i) there is a strong causal rela-tionship between an itemrsquos position in a ranked listand its visibility (ii) users tend to focus on a small setof top-ranked items that monopolize their attentionand (iii) the diversity in user behavior makes flexi-bility essential for modeling the connection betweenrank and visibility as it emerges in different settingsThese three observations inform our models for thecomputation of the consideration probabilities whichwe describe in the remainder of this section

Let GH be the review-based ranking of a set ofhotels H and let Gq

H be a filtered version of the rank-ing which includes only the hotels that cover all of thefeatures in a query q The user browses through thisranked list and considers a subset of the ranked itemsby clicking on them to obtain further information Theinclusion of a hotel into the userrsquos consideration set ismodeled via a random Bernoulli variable which takesthe value 1 with probability Pr4h q5 computed as afunction of the hotelrsquos rank in Gq

H We consider threealternative consideration models which we refer toas Linear Exponential and Stretched Exponential

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


Table 3 Alternative Models for the ConsiderationProbability as a Function of an ItemrsquosPosition x in a Ranked List with N Items

Title Formula

Linear Pr4x5= 1 minusx minus 1N + 1

Exponential Pr4x5= eminusx

Stretched Exponential Pr4x5= eminus424xminus15N5k

Table 3 includes the formula for the considerationprobability of an item ranked in the xth position of alist of N items for all three models Figure 5 then illus-trates the computed probabilities for a ranked list of50 items

The Linear model is a parameter-free model thatassigns a consideration probability proportional to theitemrsquos rank The Exponential model introduces thestandard rate parameter of the exponential distribu-tion and applies an exponential decay to the considera-tion probability as the rank increases (becomes worse)This is a strict model that greatly favors top-rankeditems This pattern has been repeatedly reported byprevious work as we described in detail earlier in thissection Figure 5 demonstrates that the strictness of themodel can be tuned by increasing the value of the rateparameter to expedite the decay Finally the StretchedExponential model follows a reverse sigmoid shapecontrolled by a parameter k As shown in the figureincreasing the value of k from 2 to 4 delays the decayand assigns a high probability to a larger number oftop-k items In practice the Stretched Exponentialmodel allows us to expand the bias that favors top-ranked items while maintaining an exponential decayfor the others This is a particularly useful propertybecause as discussed earlier in this section the exactnumber of top-ranked items that monopolize the usersrsquointerest has been shown to differ across studies anddomains

Figure 5 Consideration Probability for the Top 50 Positions for Eachof Our Three Models

Ranking

0

02

04

06

08

Con

side

ratio

n pr

obab

ility

LinearExponential = 025Exponential = 05Weibull k = 2Weibull k = 4

10 20 30 40 50

10

The Exponential and Stretched Exponential mod-els inherit the parameters of the distributions that theyare based on The interested practitioner can intuitivelytune these parameters to simulate different types ofconsideration patterns In practice however reviewplatforms have access to extensive user logs of search-and-click data that can be used to learn the most appro-priate consideration model for each context Previouswork has presented a number of effective algorithmsfor this task (Agichtein et al 2006 Joachims 2002Chapelle and Zhang 2009) It is important to stress thatour framework has no dependency on distribution-based models and their parameters Instead it is com-patible with any consideration model that assigns aprobability to each position in the ranking includingmodels that can be learned from real search data

Finally while higher rankings are typically con-nected with increased consideration it is impossible toexclude scenarios that are not significantly influencedby rankings For instance consider a market with avery small number of competitive items or a highlydemanding consumer whose multiple constraints limitthe consideration set to a trivial size In such cases theuser is likely to consider all available items regardlessof their rank While we acknowledge such possibilitiesour goal is to formalize the effect of ranking in sce-narios that include a nontrivial number of alternativesand do not allow for an exhaustive evaluation Theprevalence of such scenarios in modern markets hasbeen acknowledged by relevant literature as well asby virtually all major review platforms which developand utilize ranking systems to improve their usersrsquoexperience

5 Attack StrategiesBased on our definition of visibility hotels that areranked higher in the global review-based ranking havea higher probability of being added to the considera-tion set of a potential customer Therefore by injectingfake reviews and manipulating the ranking a competi-tor can directly affect a hotelrsquos visibility To achieve thisthe attacker can follow one of three strategies

bull Self-Pos self-inject positive reviews into its ownreview set

bull Target-Neg inject negative reviews into the re-view set of a competitor

bull Mixed a mixed strategy that combines both posi-tive and negative injections

The common goal of all three strategies is to increasethe attackerrsquos visibility at the expense of the visibilityof its competitors This raises the issue of measuring ahotelrsquos vulnerability to such attacks and motivates thefollowing question

Research Question Let H be a set of competitivehotels Then given any two hotels h1hprime isin H how vul-nerable is h to a review-injection attack by hprime

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


If a competitor could inject an infinite number ofreviews then a hotelrsquos visibility could be manipu-lated at will In practice however the probability ofbeing detected increases with the volume of injectedreviews This is because of the resulting traffic patterns(Mukherjee et al 2012 Xie et al 2012) as well as dueto the increasing complexity of creating fake reviewsthat are both realistic and dissimilar enough to avoiddetection (Jindal et al 2010 Lappas 2012) Thereforethe attacker would reasonably want to surpass a com-petitor with the minimum number of injections In thecontext of our research question we thus have to com-pute the smallest possible set of injections that enablesa hotel hprime to surpass a competitor h for each of the threestrategies An injection attack can be described in termsof the two alternative review-based ranking functionsdescribed in Section 42 the Average Rating functiongAV4 middot 5 and the PopularityIndex function gPI4 middot 5 For-mally let Rh be the set of reviews on a given hotel hand let I be a set of injected reviews Then consid-ering Equation (5) the updated Average Rating glowast

AV4h5can be computed as

glowast

AV4h5=

sum

risinRhcupIrating4r5

Rh + I0 (7)

With respect to the PopularityIndex function de-fined in Equation (6) the key observation is that allinjected reviews will be more recent than all preexist-ing reviews for the hotel Therefore all of the injectionswill be effectively applied to R1

h which is the first (mostrecent) semester of reviews considered by the func-tion Formally the updated glowast

PI4h5 can be computed asfollows

glowast

PI4h5 = w1 middot 4R1h + I1 R2

h1 0 0 0 1 RTh 1 RgtT

h 5

+w2 middot 4R1h cupI1 R2

h1 0 0 0 1 RTh 1 R

gtTh 50 (8)

Given Equations (7) and (8) an attacker hprime can opti-mally implement the Self-Pos strategy by greedilyself-injecting positive reviews until its visibility sur-passes that of the target h Similarly the practice ofgreedily injecting negative reviews to h until its visibil-ity falls below that of hprime is optimal for the Target-Negstrategy It can be trivially shown that the greedyapproach is not optimal for the Mixed strategy In theonline appendix we include an algorithm for comput-ing an optimal solution that is much faster than naivelyevaluating all possible combinations of positive andnegative injections We use this algorithm for the exper-iments of this section

We design our experiments as follows First werank the hotels in each city in descending order oftheir individual visibilities For every hotel hprime in thecity we compute the target set 8h2 v4h5 ge v4hprime59 of allhotels with an equal or higher visibility Then for each

attack strategy we compute the minimum number ofinjections that hprime needs to become more visible thaneach target We repeat the process for all cities allof the hotels in each city the two ranking functionsdescribed in Section 42 and the three considerationmodels described in Section 43 For the Exponentialand Stretched Exponential consideration models weset = 1 and k = 2 respectively We omit the resultsfor the Linear model as they were similar to those ofStretched Exponential and did not deliver additionalinsights Finally we focus our discussion on the resultsfrom New York City San Francisco and Los AngelesThe results with different parameter settings and theremaining cities were also similar and are omitted forlack of space

51 Measuring the Effectiveness of theThree Strategies

First we evaluate the percentage of all possible attacks(for all hotels and their respective targets) that canbe completed for increasing numbers of review injec-tions Figure 6 shows the results for the Average Ratingand PopularityIndex ranking functions under theExponential consideration model Figure 7 shows therespective results for Stretched Exponential

The figures reveal several findings First we observesimilar injection effects for all three cities within eachcombination of ranking functions and considerationmodels This demonstrates the influence of these twofactors and exposes a generalized vulnerability toattacks that manipulate an itemrsquos ranking and consid-eration probability Furthermore a series of notewor-thy findings and respective implications comes fromthe differences observed among the three attack strate-gies and between the two ranking functions Next wediscuss these findings in more detail

Are hotels more vulnerable to positive or negative re-views As anticipated the Mixed strategy consis-tently demonstrates its ability to successfully completean attack with less injections than the other twoapproaches A less anticipated observation is the rolereversal of the Self-Pos and Target-Neg strategieswith respect to the two consideration models For theExponential model the Target-Neg strategy fails tocomplete more than 40 of the attacks even after1000 injections On the other hand the percentageof Self-Pos increases consistently reaching a valuearound 70 for 1000 injections Surprisingly the rolesare reversed for the Stretched Exponential modelSelf-Pos converges after completing about 60 of theattacks while Target-Neg achieves percentages as highas 90 In addition the reversal is observed for boththe Average Rating and PopularityIndex ranking func-tions As we discuss next this finding is the result ofthe different way in which the two models distributeuser consideration

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


Figure 6 Percentage of Successful Injection Attacks for the Average Rating and PopularityIndex Ranking Functions Under the ExponentialConsideration Model with = 1

0 200

Number of injections

o

f su

cces

sful

atta

cks

400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

Mixed injectionsNegative injectionsPositive injections

(a) Los Angeles Average Rating (b) NYC Average Rating (c) S Francisco Average Rating

(d) Los Angeles PopularityIndex (e) NYC PopularityIndex (f) S Francisco PopularityIndex

The Exponential model limits the userrsquos considera-tion to a very small group of top-ranked hotels Theself-injection of positive reviews is effective in thissetting as it allows the hotel to eventually enter thetop listings and dramatically increase its considerationprobability On the other hand Target-Neg is onlyeffective in a minority of cases when both the attacker

Figure 7 Percentage of Successful Injection Attacks for the Average Rating and PopularityIndex Ranking Functions Under the StretchedExponential Consideration Model with k = 2

00

02

04

06

08

10

200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks



and the target are ranked in top positions with anontrivial consideration probability Conceptually anattacker cannot achieve significant visibility earningsby attacking a target that already has a zero ornear-zero visibility On the other hand the stretcheddecay of the Stretched Exponential model allocates anontrivial consideration probability to a much larger

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


number of hotels This favors negative injections asthey can benefit the attacker by both increasing its ownvisibility and reducing that of the target as long as thenumber of injections is enough to surpass the targetin the review-based ranking We revisit these findingsand their implications for businesses in Section 7

Which ranking function is more vulnerable to fake re-views Without loss of generality we focus our studyon the Mixed strategy which computes the optimal(smallest) number of injections required for each attackand thus allows us to objectively compare the vulnera-bility of the two functions First we examine the resultsfor the Exponentialmodel shown in Figure 6 For NewYork City 200 injections were enough to manipulatethe PopularityIndex ranking and complete 80 of allpossible attacks On the other hand for the AverageRating ranking the same number of injections was suf-ficient for only 60 of the attacks For Los Angelesjust 50 reviews were sufficient for 80 of the attacksfor the PopularityIndex On the other hand four timesas many reviews were required to achieve the samepercentage under the Average Rating

Furthermore for the PopularityIndex 450ndash500 in-jections were enough to eventually complete nearly100 of the attacks across cities The number waseven smaller for San Francisco and Los AngelesBy contrast about 700 injections were required toreach 100 in Los Angeles under the Average RatingIn fact this percentage was unattainable even after1000 injections for San Francisco and New York CityFinally as demonstrated by the steep curves for thePopularityIndex its vulnerability is even larger forsmall injection numbers (lt50) This is a critical obser-vation as small attacks are more likely to occur andavoid detection in realistic settings (Xie et al 2012Fei et al 2013) Figure 7 motivates similar obser-vations for the Stretched Exponential considerationmodel with the curves of the Mixed strategy exhibit-ing a steeper rise for the PopularityIndex than for theAverage Rating In addition less than 50 reviews wereenough to manipulate the PopularityIndex and com-plete nearly 100 of the attacks On the other handfour times as many reviews were required to achievesimilar results for the Average Rating

The results demonstrate that the PopularityIndexis consistently more vulnerable to fake reviews thanthe Average Rating This is a surprising finding asthe PopularityIndex was designed to address theshortcomings of the Average Rating as discussed inSection 42 However a careful examination of thefunction reveals the cause of its increased vulnerabilityBy definition the PopularityIndex assigns increasedimportance to the average rating of the most recentbatch of reviews In fact as shown in Table 2 theinfluence of each review declines with time Whilethis practice can eliminate outdated information italso favors fake reviews This bias has two causes

First fake reviews can only be injected in the presentand therefore have the highest possible contempo-rary influence among the reviews of the injected hotelSecond by reducing the importance of older reviewsthe PopularityIndex effectively reduces the number ofreviews that contribute to the final score and makes thescore more vulnerable to manipulation As an exampleconsider a hotel with an average rating of four starsand 1000 reviews To reduce the rating to three starswe would need to inject 500 one-star reviews Nowsuppose that we consider only the 100 most recenthotel reviews and the average of these 100 reviews isagain four stars In this setting just 50 one-star reviewsare sufficient to reduce the rating to three stars

52 Designing Fraud-Resistant Ranking FunctionsThe policy of favoring recent reviews in review-based rankings is adopted by industry leaders suchas TripAdvisor and Yelp (Kamerer 2014 Mukher-jee et al 2013b) Therefore the vulnerability of thePopularityIndex has major implications for these plat-forms and their users Even though the goal of thispolicy is to eliminate outdated information it isalso contrary to standard quality principles for user-generated content such as those found in large crowd-sourcing communities For instance on Wikipedianew contributions to a page are monitored and eval-uated by users who have placed the page on theirwatch list (Vieacutegas et al 2004) The community canthus promptly identify and eliminate false or mali-cious edits before they can be seen by a large numberof users

The watch list paradigm motivates us to propose theDelayIndex ranking function which maintains the ben-efits of the PopularityIndex while significantly reduc-ing its vulnerability to fake reviews As described indetail in Section 42 the PopularityIndex first aggre-gates a hotelrsquos reviews into groups (eg semesters)according to their date of submission It then assignsincreasingly high coefficients to more recent groupsThe DelayIndex function takes the same approachexcept that it swaps the coefficients of the most recentgroup with those of the second most recent Thisreduces the effect of freshly injected reviews by assign-ing them to the second level of importance In additionthis introduces a delay period before a newly injectedfake review reaches the semester with the highest coef-ficient This gives the platform time to identify andeliminate fake reviews and allows the attacked busi-ness to respond to such reviews before they can sig-nificantly impact its visibility We show the resultsof the DelayIndex function for the Exponential andStretched Exponential consideration models in Fig-ure 8 For lack of space we present the results for LosAngeles The results for the other cities were similarand did not lead to additional findings

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


Figure 8 Percentage of Attacks that can be Successfully Completed for Increasing Numbers of Review Injections for the Average RatingPopularityIndex and DelayIndex Ranking Functions Under the Exponential (andashc) and Stretched Exponential (dndashf) ConsiderationModels (LA Data)

o

f su

cces

sful

atta

cks

(b) PopularityIndex (Exp = 1)

0

02

04

06

08

10

0 200 400 600 800 1000


(c) DelayIndex (Exp = 1)

0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


(a) Average Rating (Exp = 1)

0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


(d) Average Rating (Str-Exp k = 2)

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

0 200 400 600 800 1000


(e) PopularityIndex (Str-Exp k = 2)

0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


(f) DelayIndex (Str-Exp k = 2)

The figures show that the DelayIndex is consistentlyless vulnerable than the other two functions for bothconsideration models In fact under this function theMixed strategy completed only 50 and 70 of thepossible attacks for the Exponential and StretchedExponential consideration models respectively evenafter 1000 injections These promising results provideinsights on how more robust ranking functions can bedesigned to protect visibility from fake review injec-tions We discuss possible research directions for fur-ther improvement in Section 7

53 Accounting for the Effects ofDefense Mechanisms

As we discussed in detail in Section 2 review plat-forms utilize different types of defenses against fakereviews The results that we presented in Section 52 donot account for the effects of such mechanisms whichcould prevent at least a percentage of the attemptedinjections Given an accurate estimate of this percent-age it would be trivial to adjust the expected numberof successful attacks in our experiments and updateour figures accordingly However as we discuss nextthis estimation task is ambiguous and far from trivialIn fact the correct percentage is likely to differ acrossplatforms Therefore rather than trying to find a per-centage that would only be meaningful in the contextof our data set we present an analysis of the estimationtask and the challenges that it presents as well as of thescenario where business managers personally disputefake reviews

In theory we can compute the percentage of iden-tified fake reviews as the True Positives(True Pos-itives + False Negatives) ratio However even in themost optimistic scenario we can only have access tothe set of reviews that have already been marked asfake by the platform Unfortunately the true positives(ie fake reviews marked by the platform) in such adata set would be mixed with false positives (ie realreviews that were falsely marked as fake) and couldnot be untangled In addition the set of false nega-tives is unattainable given that the platform cannotbe aware of fake reviews that were missed by its fil-ters The only way to obtain such information wouldbe to execute a strategically designed barrage of injec-tion attacks on the website However this would be adirect violation of the platformrsquos terms and is not anacceptable option

531 Simulating Defense Levels To address thislimitation we consider hypothetical scenarios involv-ing low medium and high injection success rates(ISRs) as a result of screening mechanisms used byplatforms4 Specifically we evaluate the cases whereonly 25 (low) 50 (medium) and 75 (high) of allinjections are completed while the rest are preventedWe then compute the percentage of successful attacksthat are possible given each ISR for different num-bers of attempted injections We present the results

4 We thank one of the reviewers for suggesting this experiment

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


Figure 9 Percentage of Successful Injection Attacks with Low Medium and High Injection Success Rates Under the Exponential ConsiderationModel (LA Data)

0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


(a) Average Rating (Exp = 1) (b) PopularityIndex (Exp = 1) (c) DelayIndex (Exp = 1)

Low success rateMedium success rateHigh success rate

for all three ranking functions in Figure 9 For lack ofspace we only include the results of the superior Mixedattack strategy under the Exponential considerationmodel on the Los Angeles data set The results for theother models and cities lead to similar findings consis-tent with the trends that we observed in our previousexperiments

The figure verifies the vulnerability of the Popular-ityIndex even for a low success rate 50 injectionattempts were enough to complete 40 of the attackswhile 100 injections led to a 60 completion rateThese rates were about two times those yielded bythe Average Rating function and almost three timesthose yielded by the DelayIndex for the same ISRand number of attempts In fact 400 attempts wereenough to complete more than 90 of all possibleattacks under the PopularityIndex regardless of theISR Another interesting finding is that while lowerISRs predictably reduce the effect of attacks it remainssignificant For instance at a 50 ISR 50 attempts wereenough to complete 50 and 30 of all possible attacksfor the PopularityIndex and Average Rating func-tions respectively This highlights the need for robustranking functions and vigilant policies against fakereviews A platformrsquos effort to strengthen its defensesand reduce the success rate of injection attacks is likelyto require valuable resources Therefore this type of

Figure 10 Information on the Types and Distribution of Responses Submitted by Businesses to Online Reviews

0

2 times 105

4 times 105

6 times 105

8 times 105

106

12 times 106

1 2 3 4 5

Num

ber

of r

evie

ws

Stars

Total reviewsResponses

35 41

40

36

40

(a) Responses per rating

0

01

02

03

04

05

06

07

08

AP APJ DSP OTH

Cov

ered

per

cent

age

Response type

Total reviews

(b) Response type percentages

simulation can be used to evaluate different optionsand conduct the cost-benefit analysis

532 Responding to Reviews Because of theinherent difficulties in estimating the percentage ofdetected fake reviews we focus on an alternative mea-sure of the platformrsquos vigilance On large review plat-forms such as Yelp or TripAdvisor businesses canrespond to reviews posted by their customers Rele-vant research has consistently verified the benefits ofthis practice (Cheng and Loi 2014 Ye et al 2008 Barskyand Frame 2009 Avant 2013 Xie et al 2014) Our Trip-Advisor data set includes 969469 such responses Fig-ure 10(a) shows the distribution of the reviews over thepossible star ratings as well as the number of reviewsin each rating that received a response We observethat the number of reviews that received a responseis consistently between 35 and 41 for all five rat-ings Given that we are interested in the vigilance ofbusinesses with respect to defamatory comments wefocus on the 101759 one-star and two-star reviews inour data Figure 10(a) suggests that around two-thirdsof all negative reviews do not receive a response Whilethis could be an alarming finding we cannot confi-dently attribute it to lack of vigilance as the true per-centage of fake reviews is unattainable To put this

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


finding into context and gain a deeper understandingof the types of responses we adopt a topic-modelingapproach based on the highly cited Latent DirichletAllocation (LDA) (Blei et al 2003) model LDA mod-els the generation of each term in a response d asfollows first the responder samples a topic i from adocument-specific distribution d A term is then sam-pled from a topic-specific distribution i The docu-ment is thus modeled as a mixture of topics whichin turn are modeled as term distributions Given thenumber of topics to be learned k the and distri-butions are estimated via Gibbs sampling (Blei et al2003) We set k via a 10-fold cross validation with theperplexity measure which has been consistently usedfor the evaluation of topic models (Blei et al 2003Asuncion et al 2009) After experimenting with k isin

8101201301401501601709 we found that k = 50 deliv-ered the lowest perplexity

The next step is to determine the context of eachof the 50 learned topics Given that each response ismodeled as a mixture of topics we achieve this byfocusing on pure responses that include a single dom-inant topic in their distribution Formally let di bethe probability of topic i in the mixture of response dThen i is the dominant topic of d if di gt 008 or equiv-alently if the response is 80 about topic T Then foreach topic i we sample 20 responses with di gt 008We choose 80 as the threshold as it is the highestvalue that delivers at least 20 responses for all 50 topicsAn examination of the data reveals at least 3 responsetypes (i) apologies without a justification posted sim-ply to placate the reviewer (ii) apologies with ajustification in which the responder offers an expla-nation for the customerrsquos unpleasant experience and(iii) unapologetic responses posted to challenge thecustomerrsquos opinions Table 4 includes an example ofeach type

To verify this composition and measure the preva-lence of each type in our corpus we asked five anno-tators to assign one of the following four labels to eachtopic based on its 20 associated responses AP apol-ogy without justification APJ apology with justification

Table 4 Examples of Response Types for Negative Reviews

Type Example response

AP Thank you for your feedback We were sorry for the impression your visit left you with your comments have been forwarded to management forimprovement We are confident that they will work diligently to prevent a future recurrence We truly appreciate your business and hope to seeyou on a visit in the future

APJ Upon check-in the guest was advised that housekeeping was finishing up with cleaning the rooms due to being sold out the previous night andwas welcomed to wait in the lobby and housekeeping would let them know as soon as the room was ready We do apologize about thisinconvenience

DSP After looking into these comments we can find no trace or evidence from staff or records kept that this is a genuine review It is also deeplysuspicious as all of our singles doubles and twins have brand new carpets and curtains throughout According to the vast majority of reviewswe are noted for our cleanliness and charm and that speaks volumes

DSP dispute or OTH other Note that the annotation taskis trivial due to the purity of the sampled responsesTherefore the number of disagreements was minimaland a label with at least four votes was reported forall 50 topics The process revealed 35 11 3 and 1 top-ics of type AP APJ DSP and OTH respectively Simplyreporting the number of topics for each type would beincomplete since some topics are more prevalent thanothers in the corpus Therefore we also compute thepercentage p4i5= 4

sum

d di54sum

d

sum

iprime diprime5 covered by eachtopic i and report the cumulative percentage for eachof the four response types in Figure 10(b) We observethat 75 of all responses are simply apologetic andmake no effort to challenge negative reviews An addi-tional 19 combined an apology with a justificationwhile only 5 disputed the reviews directly

Our study is inherently limited given that we cannotknow which reviews are fake and should have beendisputed Nevertheless the results reveal that evenbusinesses that are willing to monitor and respond totheir reviews are unlikely to be confrontational and dis-pute reviews We identify two alternative explanationsfor this finding First a confrontational response mightdemonstrate that the business is insensitive to thecomments of its customers and unwilling to acknowl-edge its faults thus damaging its brand Second giventhat reviews are inherently subjective it is easy foran attacker to write reviews that are hard to disputeas fake For instance a business can only respond togeneric statements such as ldquoI really didnrsquot like theirbreakfastrdquo or ldquoThe rooms were too small I wouldnever go back againrdquo with apologies justifications orpromises for improvement

In conclusion we find that (i) around two-thirdsof all negative reviews do not receive responses frombusinesses and (ii) only 5 of the responses are actu-ally disputes Thus despite the well-documented ben-efits of engaging the customers and responding tocriticism this practice is rarely used for disputing fakereviews As we discuss in Section 7 the detailed inter-pretation of these findings suggests a promising direc-tion for future research

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


6 Response StrategiesIn this section we focus on response strategies thataddress the following question How can a businessprotect its visibility from fake-review attacks We iden-tify two different types of strategies those based onenhancement and those based on confrontation Enhance-ment strategies focus on improving the attacked hotelrsquosqualities to increase its visibility and overcome thelosses resulting from an attack On the other hand con-frontational strategies focus on reversing the resultsof an attack by identifying and disputing the injectedreviews

61 Enhancement StrategiesAn enhancement strategy focuses on improving ahotel to increase its visibility and compensate for thelosses caused by an attack Such a strategy can also beapplied preemptively to safeguard the hotelrsquos visibil-ity or gain an advantage over its competitors Nextwe discuss how a hotel can increase its visibility by(i) improving its position in the review-based ranking(ii) covering additional features and (iii) improvingthe quality of existing features

611 Rising Through the Ranks To improve itsposition in the ranking a business needs to attract pos-itive reviews If the firm is confident in its own qual-ity then the need to attract positive reviews translatesinto a marketing effort In fact the connection betweenpromotional campaigns and the consequent arrival ofnew reviews has been verified by previous work (Byerset al 2012a b) By using our framework to simulatea positive-injection attack as described in Section 5 abusiness can estimate the number of positive reviewsrequired to surpass any competitor and set its market-ing goals accordingly We note that a firmrsquos rank wouldalso improve if its competitors receive enough nega-tive reviews to drop below the firm in the rankingHowever given that the firm cannot influence such adevelopment without violating ethical principles andthe terms and conditions of the review platform wedo not consider this approach

612 CoveringMore Features The features that ahotel can cover determine an upper bound for its visi-bility If a firm had infinite resources it could maximizeits coverage by simply enhancing the hotel to cover allpossible features In practice however the process ofimproving a hotel by adding new features requires sig-nificant financial and human capital In addition it isreasonable to anticipate a variance in the costs requiredto cover additional features For example adding freewi-fi is arguably much cheaper and easier than addinga pool The task of evaluating and incorporating newfeatures is a nontrivial process that needs to considermultiple factors in addition to cost such as the natureof the market and the firmrsquos reputation and pricing

policy (Nowlis and Simonson 1996 Tholke et al 2001)Our analysis in Section 41 revealed that 002 of allpossible queries make up 90 of the entire probabil-ity distribution By focusing on features that frequentlyoccur in such influential queries a firm can strategi-cally allocate its resources and improve its visibilityIn addition to adding well-known features an inno-vative firm can gain a pioneering advantage by intro-ducing a new feature that is not covered by any ofits competitors (Carpenter and Nakamoto 1989) As anexample consider the first hotel to offer free parkingin a densely populated metropolitan area with lim-ited parking options This will provide a first-moveradvantage (Kerin et al 1992) proportional to the num-ber of customers that include this new feature in theirrequirements Previous work has verified that suchmoves require careful consideration and high imple-mentation costs especially in mature markets (Tholkeet al 2001)

In conclusion after considering all contributing fac-tors a business may choose between differentiationwhich requires the introduction of a new feature andimitation which calls for the adoption of a knownfeature from its competitors (Narasimhan and Turut2013) Our framework can inform such efforts by com-puting visibility before and after the addition of eachcandidate feature and suggesting candidates that max-imize the expected visibility gain The fact that ouranalysis found similar user preferences and visibilitydistributions across cities is encouraging since it sug-gests that a hotelrsquos efforts can be informed by the suc-cess or failure of similar ventures in other cities

613 Improving the Quality of Existing FeaturesFeature coverage may go beyond simple availabilityFor instance while a hotel might have a pool it mightbe too small or poorly maintained leading to nega-tive comments To further inform enhancement strate-gies we study the correlation between visibility andcustomer sentiment on different features Given thatvisibility takes values in 60117 we perform a betaregression that is appropriate for continuous depen-dent variables with such values Formally the visibilityof yi of the ith business is modeled as a random vari-able that follows the beta distribution as parametrizedby Ferrari and Cribari-Neto (2004) The regressionmodel is then obtained by writing the mean i of yi asg4i5 =

sumKj=1 Xijj where is a K times 1 vector of coeffi-

cients and Xij is the aggregate sentiment of the review-ers of the ith business on the jth feature as describedin Section 3 Finally g4 middot 5 is a link function that maps60117 to After experimenting with different alter-natives (probit cloglog) we adopt the logit functionbased on log-likelihood The regression reveals multi-ple features with positive coefficients that are signifi-cant at a level of 0001 Business services (069) Kitchenette(025) Non-smoking (036) Pool (024) Restaurant (022)

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


and Room service (054) We provide the full regressionoutput in Table 2 of the online appendix This typeof analysis can help a business estimate the visibilitygains after improving specific features The interestedanalyst could extend our methodology and control forpossible confounding variables such as the price rangeor size of the hotel

62 Confrontational StrategiesA confrontational strategy focuses on locating andaddressing the injected fake reviews The importanceof responding to negative comments has been stressedby both relevant research and review platforms Forinstance TripAdvisor has published an article on han-dling consumer reviews (Barsky and Frame 2009)which highlights the importance of review-monitoringand encourages business managers to directly respondto reviews Most platforms allow firms to maintain averified business account through which they can con-nect with their customers and directly respond to crit-icism The importance of this communication mecha-nism has recently received increased attention from theresearch community For instance Avant (2013) foundthat the practice of systematically responding to neg-ative reviews improves a hotelrsquos image and enhancesthe customerrsquos intent to return Xie et al (2014) foundthat the number of responses is significantly correlatedwith the hotelrsquos performance formalized as revenueper available room Furthermore an increasing num-ber of research efforts (Xie et al 2011 Min et al 2015Sparks and Bradley 2014 Park and Allen 2013) arefocusing on designing effective response strategies tohelp firms manage their reputation If the responderis convinced that the review is fake or malicious thenshe can post a dispute or even report the review to theplatform for further examination In either case vig-ilance and a commitment to reputation managementare necessary for a business in a competitive market

For review platforms the effort to identify and elim-inate fake reviews has been established as a top prior-ity (Holloway 2011b TripAdvisor 2015c) In additionprevious work suggests that defensive mechanismsagainst review fraud can demonstrate the portalrsquos com-mitment to content quality and enhance its credibility(Lowe 2010) Our framework can be used to comple-ment such mechanisms by monitoring the visibility ofeach business and identifying bursts of reviews thatlead to significant changes These reviews can then becross-referenced with those reported by other traffic-based (Xie et al 2012 Fei et al 2013) or content-basedtechniques (Agichtein et al 2006) Previous work hasfocused on detecting fake reviews by looking for burstsin the number of reviews submitted for a single item orby a single reviewer (Xie et al 2012 Fei et al 2013) Byapplying similar techniques on the sequence of a firmrsquosvisibility scores we can identify influential activity that

cannot be detected by simply considering the rate ofsubmission such as a small set of strategically injectedreviews Finally as we showed in Section 5 a reviewplatform can use our framework to simulate differenttypes of injection attacks and compose a vulnerabilityreport for businesses that are interested in managingtheir reputation and protecting their visibility

7 Implications Limitations andDirections for Future Research

Our work studies the vulnerability of businesses tofake review attacks We propose a formal measureof the visibility of a business on a review platformbased on the probability that a random user choosesto include it in her consideration set Our operational-ization of visibility takes into account the popularityof the features that the business can cover its posi-tion in the platformrsquos review-based ranking and theway in which users consider rankings Our work intro-duces a number of artifacts that can support researchon online reviews including (i) a methodology forestimating feature popularity (ii) two review-basedranking functions (iii) three models that simulate alter-native behavioral patterns of how users process rank-ings and (iv) three different attack strategies that canbe used to estimate the vulnerability of a business indifferent settings

Even though our framework is designed to measurevulnerability to fake reviews it can also be applied asis to help a firm monitor its visibility and inform itsmarketing and enhancement strategies In addition areview platform with access to the reviews of all of theplayers in a market can use our framework to generatereports with valuable information such as the distri-bution of visibility in the market up-and-coming com-petitors that are threatening the firmrsquos visibility andthe features that the firm needs to cover or improve tomaximize its own visibility

Our analysis on a large data set of hotel reviewsfrom TripAdvisorcom revealed multiple findings withimplications for platforms businesses and relevantresearch

bull A mixed strategy of self-injecting fake positiveand injecting fake negative reviews about competitorsis the most effective way for attackers to overtake theircompetitors in visibility While the exact number ofinjections required to complete an attack varies acrossscenarios our study revealed that just 50 injectionswere enough to complete 80 of all possible attackswhen using TripAdvisorrsquos PopularityIndex rankingfunction

bull Positive injections have a stronger impact thannegative injections in markets where users focus onlyon a small set of top-ranked items On the other handfake injections become increasingly effective as user

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


consideration expands to a larger set of items In addi-tion to informing a platformrsquos detection efforts ouranalysis can inform a firm of its visibility status andpotential For instance a firm in an oligopolistic mar-ket can use our framework to determine the number ofpositive reviews that it needs to enter the ldquowinnerrsquos cir-clerdquo and enjoy vast visibility gains It can then calibrateits marketing and improvement efforts accordingly aswe describe in Section 61

bull While ranking functions that consider the ldquofresh-nessrdquo of the reviews reduce the effects of outdatedinformation they are also more vulnerable to the injec-tion of fake reviews than simple alternatives such asthe average star rating This finding has strong impli-cations for the vulnerability of leading platforms thathave adopted such functions such as TripAdvisor andYelp Despite the well-documented impact of review-based rankings findings on ways to improve rankingfunctions have been extremely limited We hope thatthe evidence that we provide in our work will moti-vate relevant research in this area We make our owncontribution in Section 52 where we introduce a newfunction that accounts for review fraud while elimi-nating outdated reviews

bull In the absence of extensive query logs that canbe used to directly learn user preferences over a setof features customer reviews can serve as an effec-tive proxy In Section 41 we describe a methodologyfor estimating the number of reviews required to con-verge to confident estimations For the hotels domain1000 reviews were sufficient for all of the cities in ourdata set

bull Our study on the TripAdvisor data set revealedthat 002 of all possible feature queries cover around90 of the entire distribution of query popularity Thisis a promising finding that can inform popularity-estimation efforts as well as help businesses identifyinfluential features with a high impact on their visibil-ity even beyond the context of fake reviews

The primary limitation of our work is the absenceof a large data set of injection attacks including thefake reviews that were successfully injected as wellas those that were blocked by the attacked platformrsquosdefenses This is a standard limitation for research onfake reviews As we described in Section 5 our solu-tion to this was to emulate the environment of a reviewplatform simulate different types of attacks and eval-uate their success ratio Additional limitations includethe lack of extensive search-and-click logs that can beused to learn user preferences as well as to estimate theway in which users process rankings These are typi-cal issues for research on online reviews caused by thesensitive and proprietary nature of the required infor-mation We describe the way in which we deal withthese limitations in Sections 41 and 43 respectively

Finally our work provides evidence that we hopewill motivate novel research on online reviews Ourfindings can support the development of defensemechanisms and fraud-resistant ranking functionsframeworks for attack simulation user considera-tion models and strategies that help businesses man-age their visibility We studied such a strategy inSection 53 which focused on the responses thatbusinesses submit to reviews Despite the well-docu-mented benefits of this practice we found strong evi-dence that this mechanism is rarely utilized to disputefake reviews While we provided our own interpre-tation of this finding after studying a large corpus ofresponses more extensive research is required to helpbusinesses optimize their response strategy The possi-bilities for further research are manifold and excitingWe hope that with this paper we further the bound-aries of our knowledge in this domain and help fosterrigorous and thoughtful research

Supplemental MaterialSupplemental material to this paper is available at httpsdoiorg101287isre20160674

ReferencesAgichtein E Brill E Dumais S (2006) Improving Web search ranking

by incorporating user behavior information Proc 29th AnnualACM SIGIR Conf (ACM New York) 19ndash26

Anderson ET Simester DI (2014) Reviews without a purchaseLow ratings loyal customers and deception J Marketing Res51(3)249ndash269

Asuncion A Welling M Smyth P Teh YW (2009) On smoothing andinference for topic models Proc 25th Conf Uncertainty ArtificialIntelligence (AUAI Press Corvallis OR) 27ndash34

Avant T (2013) Responding to TripAdvisor How hotel responsesto negative online reviews effect hotel image intent to stayand intent to return Unpublished Masterrsquos thesis Universityof South Carolina Columbia

Baeza-Yates R Hurtado C Mendoza M (2005) Query recommen-dation using query logs in search engines Proc 2004 InternatConf Current Trends Database Tech (Springer-Verlag Berlin Hei-delberg) 588ndash596

Barsky J Frame C (2009) Handling online reviews Best practicesHospitalitynet (August 28) httpwwwhospitalitynetorgnews4043169html

Belton P (2015) Navigating the potentially murky world of onlinereviews BBC (June 23) httpwwwbbccomnewsbusiness-33205905

Bickart B Schindler RM (2001) Internet forums as influential sourcesof consumer information J Interactive Marketing 15(3)31ndash40

Blei DM Ng AY Jordan MI (2003) Latent Dirichlet allocationJ Machine Learn Res 3993ndash1022

Breure E (2013) Infographic Hotel reviews can we trust themOlery (April 24) httpwwwolerycombloginfographic-hotel-reviews-can-we-trust-them

Brooks N (2004) The Atlas Rank report How search enginerank impacts traffic Insights Atlas Institute Digital Marketinghttpwwwinestingorgad2006adminsc1appmarketingtecnologicouploadsEstudosatlas20onepoint20-20how20search20engine20rank20impacts20trafficpdf

Byers JW Mitzenmacher M Zervas G (2012a) Daily deals Predic-tion social diffusion and reputational ramifications Proc FifthACM Internat Conf Web Search Data Mining (ACM New York)543ndash552

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


Byers JW Mitzenmacher M Zervas G (2012b) The Groupon effecton Yelp ratings A root cause analysis Proc 13th ACM ConfElectronic Commerce (ACM New York) 248ndash265

Carare O (2012) The impact of bestseller rank on demand Evidencefrom the app market Internat Econom Rev 53(3)717ndash742

Carpenter GS Nakamoto K (1989) Consumer preference formationand pioneering advantage J Marketing Res 26(3)285ndash298

Chapelle O Zhang Y (2009) A dynamic Bayesian network clickmodel for Web search ranking Proc 18th Internat Conf WorldWide Web (ACM New York) 1ndash10

Chatterjee P (2001) Online reviewsmdashDo consumers use them GillyMC Meyers-Levy J eds Advances in Consumer Research Vol 28(Association for Consumer Research Valdosta GA) 129ndash133

Cheng VT Loi MK (2014) Handling negative online customerreviews The effects of elaboration likelihood model and dis-tributive justice J Travel Tourism Marketing 31(1)1ndash15

CMA (2015) Online reviews and endorsements Competition andMarkets Authority (February 26) httpsgooglGxZ4J7

Decker R Trusov M (2010) Estimating aggregate consumer prefer-ences from online product reviews Internat J Res Marketing27(4)293ndash307

Dellarocas C (2006) Strategic manipulation of Internet opinionforums Implications for consumers and firms Management Sci52(10)1577ndash1593

Ding X Liu B Yu PS (2008) A holistic lexicon-based approachto opinion mining Proc 2008 Internat Conf Web Search DataMining (ACM New York) 231ndash240

Duan W Gu B Whinston AB (2008) Do online reviews matter Anempirical investigation of panel data Decision Support Systems45(4)1007ndash1016

Fei G Mukherjee A Bing L Hsu M Castellanos M Ghosh R (2013)Exploiting burstiness in reviews for review spammer detectionProc 7th Internat Conf Weblogs Social Media 175ndash184

Feng S Xing L Gogar A Choi Y (2012) Distributional footprintsof deceptive product reviews Proc 6th Internat Conf WeblogsSocial Media Vol 12 98ndash105

Fenton S (2015) TripAdvisor denies rating system is flawed afterfake restaurant tops rankings in Italy Independent (June 30)httpgooglNtSKpi

Ferrari S Cribari-Neto F (2004) Beta regression for modelling ratesand proportions J Appl Statist 31(7)799ndash815

Forman C Ghose A Wiesenfeld B (2008) Examining the relation-ship between reviews and sales The role of reviewer iden-tity disclosure in electronic markets Inform Systems Res 19(3)291ndash313

Ghose A Ipeirotis PG (2011) Estimating the helpfulness and eco-nomic impact of product reviews Mining text and reviewercharacteristics Knowledge Data Engrg IEEE Trans 23(10)1498ndash1512

Ghose A Ipeirotis PG Li B (2012) Designing ranking systems forhotels on travel search engines by mining user-generated andcrowdsourced content Marketing Sci 31(3)493ndash520

Ghose A Ipeirotis PG Li B (2014) Examining the impact of rankingon consumer behavior and search engine revenue ManagementSci 60(7)1632ndash1654

Guan Z Cutrell E (2007) An eye tracking study of the effect oftarget rank on Web search Proc SIGCHI Conf Human FactorsComput Systems (ACM New York) 417ndash420

Henzinger M (2007) Search technologies for the Internet Science317(5837)468ndash471

Holloway D (2011a) How are Yelprsquos search results ranked Yelp(June 21) httpsbizyelpcombloghow-are-yelps-search-results-ranked

Holloway D (2011b) Just another reason why we have a reviewfilter Yelp (October 3) httpofficialblogyelpcom201110just-another-reason-why-we-have-a-review-filterhtml

Hu N Bose I Koh NS Liu L (2012) Manipulation of online reviewsAn analysis of ratings readability and sentiments DecisionSupport Systems 52(3)674ndash684

Ifrach B Johari R (2014) The impact of visibility on demand in themarket for mobile apps Working paper Stanford UniversityStanford CA

Jindal N Liu B (2008) Opinion spam and analysis Proc 2008 Inter-nat Conf Web Search Data Mining (ACM New York) 219ndash230

Jindal N Liu B Lim E (2010) Finding unusual review patternsusing unexpected rules Proc 19th ACM Internat Conf InformKnowledge Management (ACM New York) 1549ndash1552

Joachims T (2002) Optimizing search engines using clickthroughdata Proc Eighth ACM SIGKDD Internat Conf Knowledge Dis-covery Data Mining (ACM New York) 133ndash142

Joachims T Granka L Pan B Hembrooke H Gay G (2005) Accu-rately interpreting clickthrough data as implicit feedback Proc28th Annual ACM SIGIR Conf (ACM New York) 154ndash161

Kamerer D (2014) Understanding the Yelp review filter An ex-ploratory study First Monday 19(9) httpfirstmondayorgarticleview54364111

Kerin RA Varadarajan PR Peterson RA (1992) First-mover advan-tage A synthesis conceptual framework and research propo-sitions J Marketing 56(4)33ndash52

Korolova A Kenthapadi K Mishra N Ntoulas A (2009) Releasingsearch queries and clicks privately Proc 18th Internat ConfWorld Wide Web (ACM New York) 171ndash180

Kwark Y Chen J Raghunathan S (2014) Online product reviewsImplications for retailers and competing manufacturers InformSystems Res 25(1)93ndash110

Lappas T (2012) Fake reviews The malicious perspective Bouma GIttoo A Metais E Wortmann H eds 17th Internat Conf Applica-tions Natural Language Processing Inform Systems Lecture NotesComput Sci Vol 7337 (Springer-Verlag Berlin Heidelberg)23ndash34

Leung CWK Chan SCF Chung FL Ngai G (2011) A probabilisticrating inference framework for mining user preferences fromreviews World Wide Web 14(2)187ndash215

Li B Ghose A Ipeirotis PG (2011) Towards a theory model forproduct search Proc 20th Internat Conf World Wide Web (ACMNew York) 327ndash336

Lorigo L Haridasan M Brynjarsdoacutettir H Xia L Joachims T GayG Granka L Pellacini F Pan B (2008) Eye tracking and onlinesearch Lessons learned and challenges ahead J Amer SocInform Sci Tech 59(7)1041ndash1052

Lowe L (2010) Yelprsquos review filter explained Yelp (March 18) httpofficialblogyelpcom201003yelp-review-filter-explainedhtml

Lu X Ba S Huang L Feng Y (2013) Promotional marketing or word-of-mouth Evidence from online restaurant reviews InformSystems Res 24(3)596ndash612

Luca M Zervas G (2016) Fake it till you make it Reputation com-petition and yelp review fraud Management Sci ePub aheadof print January 28 httpdxdoiorg101287mnsc20152304

Marrese-Taylor E Velaacutesquez JD Bravo-Marquez F Matsuo Y (2013)Identifying customer preferences about tourism products usingan aspect-based opinion mining approach Procedia Comput Sci22182ndash191

Masoni D (2014) TripAdvisor fined in Italy for misleading re-views Reuters (December 22) httpwwwreuterscomarticletripadvisor-italy-fine-idUSL6N0U62MW20141222

Mauri AG Minazzi R (2013) Web reviews influence on expecta-tions and purchasing intentions of hotel potential customersInternat J Hospitality Management 3499ndash107

May K (2011) Goodbye TripAdvisor welcome to verified reviewson Expedia tnooz (December 28) httpgooglA3dgpQ

Mayzlin D Dover Y Chevalier JA (2012) Promotional reviews Anempirical investigation of online review manipulation Techni-cal report National Bureau of Economic Research CambridgeMA

Min H Lim Y Magnini V (2015) Factors affecting customer satisfac-tion in responses to negative online hotel reviews The impactof empathy paraphrasing and speed Cornell Hospitality Quart56(2)223ndash231

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved


Mukherjee A Liu B Glance N (2012) Spotting fake reviewer groupsin consumer reviews Proc 21st Internat Conf World Wide Web(ACM New York) 191ndash200

Mukherjee A Venkataraman V Liu B Glance N (2013a) Fakereview detection Classification and analysis of real and pseudoreviews Technical report UIC-CS-2013-03 University of Illi-nois at Chicago Chicago

Mukherjee A Venkataraman V Liu B Glance NS (2013b) WhatYelp fake review filter might be doing Proc 7th Internat ConfWeblogs Social Media

Narasimhan C Turut Ouml (2013) Differentiate or imitate Therole of context-dependent preferences Marketing Sci 32(3)393ndash410

Nowlis SM Simonson I (1996) The effect of new product featureson brand choice J Marketing Res 33(1)36ndash46

Olery (2015) Hotel review sites wwwolerycomreviewsitesPan B (2015) The power of search engine ranking for tourist desti-

nations Tourism Management 4779ndash87Pan B Hembrooke H Joachims T Lorigo L Gay G Granka L (2007)

In Google we trust Usersrsquo decisions on rank position andrelevance J Comput-Mediated Comm 12(3)801ndash823

Park SY Allen JP (2013) Responding to online reviews problemsolving and engagement in hotels Cornell Hospitality Quart54(1)64ndash73

Pearson K (1895) Note on regression and inheritance in the case oftwo parents Proc Roy Soc London 58240ndash242

Schaal D (2015a) TripAdvisorrsquos hotel review collection is leavingrivals in the dust Skift (May 4) httpgooglvrOxbX

Schaal D (2015b) TripAdvisorrsquos reviews wonrsquot change even as itbecomes the next big booking site Skift (July 28) httpsskiftcom20150728tripadvisors-reviews-wont-change-even-as-it-becomes-the-next-big-booking-site

Schneiderman ET (2015) Operation clean turf httpgooglWZfq5L

Sparks BA Bradley GL (2014) A ldquotriple ardquo typology of respondingto negative consumer-generated online reviews J HospitalityTourism Res ePub ahead of print July 2 httpdxdoiorg1011771096348014538052

Sparks BA Browning V (2011) The impact of online reviews on hotelbooking intentions and perception of trust Tourism Management32(6)1310ndash1323

StatisticBrain (2015) Internet travel hotel booking statistics httpwwwstatisticbraincominternet-travel-hotel-booking-statistics

Stempel J (2015) Amazon sues to block alleged fake reviews on itswebsite Reuters (April 9) httpgoogloz13iZ

Sussin J Thompson E (2012) The consequences of fake fans likesand reviews on social networks Gartner Inc httpswwwgartnercomdoc2091515consequences-fake-fans-likes-reviews

Tholke JM Hultinka EJ Robben HSJ (2001) Launching new productfeatures A multiple case examination J Product InnovationManagement 18(1)3ndash14

Tibken S (2013) Taiwan fines Samsung $340000 for bashing HTCCNET (October 24) httpwwwcnetcomnewstaiwan-fines-samsung-340000-for-bashing-htc

Tnooz (2013) Top US travel websites httpgooglqJgOcuTripAdvisor (2013) TripAdvisor popularity ranking Key factors

and how to improve httpsgooglTzpHUSTripAdvisor (2015a) Review moderation and guidlines httpwww

tripadvisorcomvpagesreview_mod_fraud_detecthtmlTripAdvisor (2015b) TripAdvisor fact sheet httpwww

tripadvisorcomPressCenterc4Fact_SheethtmlTripAdvisor (2015c) We have zero tolerance for fake reviews

httpswwwtripadvisorcompagesfraudhtmlTucker C Zhang J (2011) How does popularity information affect

choices A field experiment Management Sci 57(5)828ndash842Vermeulen IE Seegers D (2009) Tried and tested The impact of

online hotel reviews on consumer consideration Tourism Man-agement 30(1)123ndash127

Vieacutegas FB Wattenberg M Dave K (2004) Studying cooperationand conflict between authors with history flow visualizationsProc SIGCHI Conf Human Factors Comput Systems (ACM NewYork) 575ndash582

Wang Z (2010) Anonymity social image and the competition forvolunteers A case study of the online market for reviews BEJ Econom Anal Policy 10(1)1ndash35

Weise E (2015) Amazon cracks down on fake reviews USA Today(October 19) httpwwwusatodaycomstorytech20151019amazon-cracks-down-fake-reviews74213892

Xie HJ Miao L Kuo P Lee B (2011) Consumers responses toambivalent online hotel reviews The role of perceived sourcecredibility and pre-decisional disposition Internat J HospitalityManagement 30(1)178ndash183

Xie KL Zhang Z Zhang Z (2014) The business value of onlineconsumer reviews and management response to hotel perfor-mance Internat J Hospitality Management 431ndash12

Xie S Wang G Lin S Yu PS (2012) Review spam detection viatemporal pattern discovery Proc 18th ACM SIGKDD InternatConf Knowledge Discovery Data Mining (ACM New York)

Ye Q Law R Gu B (2009) The impact of online user reviewson hotel room sales Internat J Hospitality Management 28(1)180ndash182

Ye Q Gu B Chen W Law R (2008) Measuring the value of man-agerial responses to online reviewsmdashA natural experiment oftwo online travel agencies ICIS 2008 Proc 115

Ye Q Law R Gu B Chen W (2011) The influence of user-generatedcontent on traveler behavior An empirical investigation on theeffects of e-word-of-mouth to hotel online bookings ComputHuman Behav 27(2)634ndash639

Zhu F Zhang X (2010) Impact of online consumer reviews on salesThe moderating role of product and consumer characteristicsJ Marketing 74(2)133ndash148

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved

Information Systems ResearchArticles in Advance pp 1ndash22ISSN 1047-7047 (print) ISSN 1526-5536 (online) httpsdoiorg101287isre20160674

copy 2016 INFORMS

The Impact of Fake Reviews on Online VisibilityA Vulnerability Assessment of the Hotel Industry

Theodoros Lappas Gaurav Sabnis Georgios ValkanasStevens Institute of Technology Hoboken New Jersey 07030

tlappasstevensedu gsabnisstevensedu gvalkanastevensedu

Extant research has focused on the detection of fake reviews on online review platforms motivated by thewell-documented impact of customer reviews on the usersrsquo purchase decisions The problem is typically

approached from the perspective of protecting the credibility of review platforms as well as the reputation andrevenue of the reviewed firms However there is little examination of the vulnerability of individual businessesto fake review attacks This study focuses on formalizing the visibility of a business to the customer base and onevaluating its vulnerability to fake review attacks We operationalize visibility as a function of the features thata business can cover and its position in the platformrsquos review-based ranking Using data from over 23 millionreviews of 4709 hotels from 17 cities we study how visibility can be impacted by different attack strategiesWe find that even limited injections of fake reviews can have a significant effect and explore the factors thatcontribute to this vulnerable state Specifically we find that in certain markets 50 fake reviews are sufficient foran attacker to surpass any of its competitors in terms of visibility We also compare the strategy of self-injectingpositive reviews with that of injecting competitors with negative reviews and find that each approach can be asmuch as 40 more effective than the other across different settings We empirically explore response strategiesfor an attacked hotel ranging from the enhancement of its own features to detecting and disputing fake reviewsIn general our measure of visibility and our modeling approach regarding attack and response strategies shedlight on how businesses that are targeted by fake reviews can detect and tackle such attacks

Keywords customer reviews fake reviews knowledge management vulnerability assessment decisionsupport systems

History Rob Fichman Ram Gopal Alok Gupta Sam Ransbotham Senior Editors Alok Gupta AssociateEditor This paper was received on March 2 2015 and was with the authors 8 months for 2 revisionsPublished online in Articles in Advance November 9 2016

1 Introduction

Mrs Richards When I pay for a room with a view Iexpect something more interesting than that

Basil That is Torquay madamMrs Richards Well itrsquos not good enoughBasil Well may I ask what you expected to see out

of a Torquay hotel bedroom window Sydney OperaHouse perhaps The Hanging Gardens of BabylonHerds of wildebeest sweeping majestically 0 0 0

Mrs Richards Donrsquot be silly I expect to be able tosee the sea

Basil You can see the sea Itrsquos over there between theland and the sky

Mrs Richards Irsquod need a telescope to see that

Fawlty TowersEpisode ldquoCommunication Problemsrdquo1979 British Broadcasting Corporation

This humorous exchange between proprietor BasilFawlty and dissatisfied customer Mrs Richards illus-trates an important phenomenon in the hospitalityindustry in which customers book hotels based pri-marily on features that are important to them ForMrs Richards the view from her window is impor-tant For someone else the top priority could be free

breakfast or air-conditioning Large travel websitessuch as TripAdvisorcom or Bookingcom allow usersto filter the hotels that they consider based on the fea-tures that they are interested in The popularity of suchplatforms is largely based on the availability of largevolumes of customer reviews which are considered tobe more credible than biased promotional campaigns(Bickart and Schindler 2001 Lu et al 2013) Relevantliterature has established the impact of reviews onpurchase decisions (Chatterjee 2001 Kwark et al 2014Ghose et al 2014 Duan et al 2008) and consequentlyon a firmrsquos sales and revenue (Ghose and Ipeiro-tis 2011 Forman et al 2008 Zhu and Zhang 2010)Nevertheless online reviews are also susceptible totampering from unscrupulous businesses who attemptto manipulate the available information by postingeither fake positive reviews about themselves or fakenegative reviews about their competitors resulting inreview fraud (Dellarocas 2006 Mayzlin et al 2012Lappas 2012 Luca and Zervas 2016) Although a con-siderable amount of research has focused on the iden-tification of fake reviews (Hu et al 2012 Lappas 2012Xie et al 2012 Jindal and Liu 2008 Mukherjee et al2013a) we still have a relatively limited understanding

1

Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved










Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved





















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







sum




v4h5=sum






p4q5=freq4q1R5

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200 400 600 800 1000

Number of reviews

0

02

04

06

08

10

L1

dist

ance


(a) Los Angeles

0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance


0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance




0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10






Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



(a) Los Angeles

Perc

enta

ge c

over

ed


0

02

04

06

08

10

0 200 400 600 800 1000 1200

(b) New York City

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 100 200 300 400 500


(c) San Francisco

0 100 200 300 400 500 700


Perc

enta

ge c

over

ed

0

02

04

06

08

10

600

(d) Chicago

0 200 400 600 800 1000


Perc

enta

ge c

over

ed

0

02

04

06

08

10

(e) Philadelphia

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 200 400 600 800 1000


(f) London

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 50 100 150 200 250 300 350



ln4D5=X+ 0 (3)


sum




prod


p4q5= q timesprod

fisinq

f 1 (4)


sum



Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

NB

ER

PAR

BC

N

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

N

BE

RPA

RB

CN

Pear

sonrsquo

s co

eff

Cities


0

02

04

06

08

10

Pear

sonrsquo

s co

eff

Cities

(b) Queries




gAV4h5=

sum

risinRhrating4r5Rh

0 (5)








h1 0 0 0 1 RTh 1 RgtT

h 5

+w2 middot 4R1h1 R

2h1 0 0 0 1 R

Th 1 R

gtTh 51 (6)



Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




R1h R1

h R2h R2

h R3h R3

h Rgt3h Rgt3

h












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



Title Formula







Ranking

0

02

04

06

08

Con

side

ratio

n pr

obab

ility


10 20 30 40 50

10









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




glowast

AV4h5=

sum


Rh + I0 (7)




glowast


h1 0 0 0 1 RTh 1 RgtT

h 5


h1 0 0 0 1 RTh 1 R

gtTh 50 (8)








Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200


o

f su

cces

sful

atta

cks

400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10






00

02

04

06

08

10

200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



o

f su

cces

sful

atta

cks

0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks







0

2 times 105

4 times 105

6 times 105

8 times 105

106

12 times 106

1 2 3 4 5

Num

ber

of r

evie

ws

Stars


35 41

40

36

40


0

01

02

03

04

05

06

07

08

AP APJ DSP OTH

Cov

ered

per

cent

age

Response type

Total reviews




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












sum

d di54sum

d

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved










Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved





















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







sum




v4h5=sum






p4q5=freq4q1R5

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200 400 600 800 1000

Number of reviews

0

02

04

06

08

10

L1

dist

ance


(a) Los Angeles

0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance


0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance




0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10






Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



(a) Los Angeles

Perc

enta

ge c

over

ed


0

02

04

06

08

10

0 200 400 600 800 1000 1200

(b) New York City

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 100 200 300 400 500


(c) San Francisco

0 100 200 300 400 500 700


Perc

enta

ge c

over

ed

0

02

04

06

08

10

600

(d) Chicago

0 200 400 600 800 1000


Perc

enta

ge c

over

ed

0

02

04

06

08

10

(e) Philadelphia

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 200 400 600 800 1000


(f) London

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 50 100 150 200 250 300 350



ln4D5=X+ 0 (3)


sum




prod


p4q5= q timesprod

fisinq

f 1 (4)


sum



Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

NB

ER

PAR

BC

N

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

N

BE

RPA

RB

CN

Pear

sonrsquo

s co

eff

Cities


0

02

04

06

08

10

Pear

sonrsquo

s co

eff

Cities

(b) Queries




gAV4h5=

sum

risinRhrating4r5Rh

0 (5)








h1 0 0 0 1 RTh 1 RgtT

h 5

+w2 middot 4R1h1 R

2h1 0 0 0 1 R

Th 1 R

gtTh 51 (6)



Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




R1h R1

h R2h R2

h R3h R3

h Rgt3h Rgt3

h












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



Title Formula







Ranking

0

02

04

06

08

Con

side

ratio

n pr

obab

ility


10 20 30 40 50

10









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




glowast

AV4h5=

sum


Rh + I0 (7)




glowast


h1 0 0 0 1 RTh 1 RgtT

h 5


h1 0 0 0 1 RTh 1 R

gtTh 50 (8)








Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200


o

f su

cces

sful

atta

cks

400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10






00

02

04

06

08

10

200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



o

f su

cces

sful

atta

cks

0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks







0

2 times 105

4 times 105

6 times 105

8 times 105

106

12 times 106

1 2 3 4 5

Num

ber

of r

evie

ws

Stars


35 41

40

36

40


0

01

02

03

04

05

06

07

08

AP APJ DSP OTH

Cov

ered

per

cent

age

Response type

Total reviews




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












sum

d di54sum

d

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved










Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved





















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







sum




v4h5=sum






p4q5=freq4q1R5

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200 400 600 800 1000

Number of reviews

0

02

04

06

08

10

L1

dist

ance


(a) Los Angeles

0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance


0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance




0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10






Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



(a) Los Angeles

Perc

enta

ge c

over

ed


0

02

04

06

08

10

0 200 400 600 800 1000 1200

(b) New York City

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 100 200 300 400 500


(c) San Francisco

0 100 200 300 400 500 700


Perc

enta

ge c

over

ed

0

02

04

06

08

10

600

(d) Chicago

0 200 400 600 800 1000


Perc

enta

ge c

over

ed

0

02

04

06

08

10

(e) Philadelphia

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 200 400 600 800 1000


(f) London

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 50 100 150 200 250 300 350



ln4D5=X+ 0 (3)


sum




prod


p4q5= q timesprod

fisinq

f 1 (4)


sum



Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

NB

ER

PAR

BC

N

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

N

BE

RPA

RB

CN

Pear

sonrsquo

s co

eff

Cities


0

02

04

06

08

10

Pear

sonrsquo

s co

eff

Cities

(b) Queries




gAV4h5=

sum

risinRhrating4r5Rh

0 (5)








h1 0 0 0 1 RTh 1 RgtT

h 5

+w2 middot 4R1h1 R

2h1 0 0 0 1 R

Th 1 R

gtTh 51 (6)



Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




R1h R1

h R2h R2

h R3h R3

h Rgt3h Rgt3

h












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



Title Formula







Ranking

0

02

04

06

08

Con

side

ratio

n pr

obab

ility


10 20 30 40 50

10









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




glowast

AV4h5=

sum


Rh + I0 (7)




glowast


h1 0 0 0 1 RTh 1 RgtT

h 5


h1 0 0 0 1 RTh 1 R

gtTh 50 (8)








Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200


o

f su

cces

sful

atta

cks

400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10






00

02

04

06

08

10

200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



o

f su

cces

sful

atta

cks

0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks







0

2 times 105

4 times 105

6 times 105

8 times 105

106

12 times 106

1 2 3 4 5

Num

ber

of r

evie

ws

Stars


35 41

40

36

40


0

01

02

03

04

05

06

07

08

AP APJ DSP OTH

Cov

ered

per

cent

age

Response type

Total reviews




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












sum

d di54sum

d

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved





















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







sum




v4h5=sum






p4q5=freq4q1R5

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200 400 600 800 1000

Number of reviews

0

02

04

06

08

10

L1

dist

ance


(a) Los Angeles

0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance


0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance




0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10






Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



(a) Los Angeles

Perc

enta

ge c

over

ed


0

02

04

06

08

10

0 200 400 600 800 1000 1200

(b) New York City

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 100 200 300 400 500


(c) San Francisco

0 100 200 300 400 500 700


Perc

enta

ge c

over

ed

0

02

04

06

08

10

600

(d) Chicago

0 200 400 600 800 1000


Perc

enta

ge c

over

ed

0

02

04

06

08

10

(e) Philadelphia

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 200 400 600 800 1000


(f) London

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 50 100 150 200 250 300 350



ln4D5=X+ 0 (3)


sum




prod


p4q5= q timesprod

fisinq

f 1 (4)


sum



Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

NB

ER

PAR

BC

N

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

N

BE

RPA

RB

CN

Pear

sonrsquo

s co

eff

Cities


0

02

04

06

08

10

Pear

sonrsquo

s co

eff

Cities

(b) Queries




gAV4h5=

sum

risinRhrating4r5Rh

0 (5)








h1 0 0 0 1 RTh 1 RgtT

h 5

+w2 middot 4R1h1 R

2h1 0 0 0 1 R

Th 1 R

gtTh 51 (6)



Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




R1h R1

h R2h R2

h R3h R3

h Rgt3h Rgt3

h












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



Title Formula







Ranking

0

02

04

06

08

Con

side

ratio

n pr

obab

ility


10 20 30 40 50

10









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




glowast

AV4h5=

sum


Rh + I0 (7)




glowast


h1 0 0 0 1 RTh 1 RgtT

h 5


h1 0 0 0 1 RTh 1 R

gtTh 50 (8)








Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200


o

f su

cces

sful

atta

cks

400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10






00

02

04

06

08

10

200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



o

f su

cces

sful

atta

cks

0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks







0

2 times 105

4 times 105

6 times 105

8 times 105

106

12 times 106

1 2 3 4 5

Num

ber

of r

evie

ws

Stars


35 41

40

36

40


0

01

02

03

04

05

06

07

08

AP APJ DSP OTH

Cov

ered

per

cent

age

Response type

Total reviews




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












sum

d di54sum

d

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved





















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







sum




v4h5=sum






p4q5=freq4q1R5

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200 400 600 800 1000

Number of reviews

0

02

04

06

08

10

L1

dist

ance


(a) Los Angeles

0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance


0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance




0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10






Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



(a) Los Angeles

Perc

enta

ge c

over

ed


0

02

04

06

08

10

0 200 400 600 800 1000 1200

(b) New York City

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 100 200 300 400 500


(c) San Francisco

0 100 200 300 400 500 700


Perc

enta

ge c

over

ed

0

02

04

06

08

10

600

(d) Chicago

0 200 400 600 800 1000


Perc

enta

ge c

over

ed

0

02

04

06

08

10

(e) Philadelphia

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 200 400 600 800 1000


(f) London

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 50 100 150 200 250 300 350



ln4D5=X+ 0 (3)


sum




prod


p4q5= q timesprod

fisinq

f 1 (4)


sum



Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

NB

ER

PAR

BC

N

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

N

BE

RPA

RB

CN

Pear

sonrsquo

s co

eff

Cities


0

02

04

06

08

10

Pear

sonrsquo

s co

eff

Cities

(b) Queries




gAV4h5=

sum

risinRhrating4r5Rh

0 (5)








h1 0 0 0 1 RTh 1 RgtT

h 5

+w2 middot 4R1h1 R

2h1 0 0 0 1 R

Th 1 R

gtTh 51 (6)



Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




R1h R1

h R2h R2

h R3h R3

h Rgt3h Rgt3

h












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



Title Formula







Ranking

0

02

04

06

08

Con

side

ratio

n pr

obab

ility


10 20 30 40 50

10









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




glowast

AV4h5=

sum


Rh + I0 (7)




glowast


h1 0 0 0 1 RTh 1 RgtT

h 5


h1 0 0 0 1 RTh 1 R

gtTh 50 (8)








Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200


o

f su

cces

sful

atta

cks

400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10






00

02

04

06

08

10

200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



o

f su

cces

sful

atta

cks

0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks







0

2 times 105

4 times 105

6 times 105

8 times 105

106

12 times 106

1 2 3 4 5

Num

ber

of r

evie

ws

Stars


35 41

40

36

40


0

01

02

03

04

05

06

07

08

AP APJ DSP OTH

Cov

ered

per

cent

age

Response type

Total reviews




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












sum

d di54sum

d

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







sum




v4h5=sum






p4q5=freq4q1R5

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200 400 600 800 1000

Number of reviews

0

02

04

06

08

10

L1

dist

ance


(a) Los Angeles

0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance


0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance




0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10






Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



(a) Los Angeles

Perc

enta

ge c

over

ed


0

02

04

06

08

10

0 200 400 600 800 1000 1200

(b) New York City

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 100 200 300 400 500


(c) San Francisco

0 100 200 300 400 500 700


Perc

enta

ge c

over

ed

0

02

04

06

08

10

600

(d) Chicago

0 200 400 600 800 1000


Perc

enta

ge c

over

ed

0

02

04

06

08

10

(e) Philadelphia

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 200 400 600 800 1000


(f) London

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 50 100 150 200 250 300 350



ln4D5=X+ 0 (3)


sum




prod


p4q5= q timesprod

fisinq

f 1 (4)


sum



Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

NB

ER

PAR

BC

N

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

N

BE

RPA

RB

CN

Pear

sonrsquo

s co

eff

Cities


0

02

04

06

08

10

Pear

sonrsquo

s co

eff

Cities

(b) Queries




gAV4h5=

sum

risinRhrating4r5Rh

0 (5)








h1 0 0 0 1 RTh 1 RgtT

h 5

+w2 middot 4R1h1 R

2h1 0 0 0 1 R

Th 1 R

gtTh 51 (6)



Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




R1h R1

h R2h R2

h R3h R3

h Rgt3h Rgt3

h












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



Title Formula







Ranking

0

02

04

06

08

Con

side

ratio

n pr

obab

ility


10 20 30 40 50

10









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




glowast

AV4h5=

sum


Rh + I0 (7)




glowast


h1 0 0 0 1 RTh 1 RgtT

h 5


h1 0 0 0 1 RTh 1 R

gtTh 50 (8)








Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200


o

f su

cces

sful

atta

cks

400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10






00

02

04

06

08

10

200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



o

f su

cces

sful

atta

cks

0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks







0

2 times 105

4 times 105

6 times 105

8 times 105

106

12 times 106

1 2 3 4 5

Num

ber

of r

evie

ws

Stars


35 41

40

36

40


0

01

02

03

04

05

06

07

08

AP APJ DSP OTH

Cov

ered

per

cent

age

Response type

Total reviews




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












sum

d di54sum

d

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200 400 600 800 1000

Number of reviews

0

02

04

06

08

10

L1

dist

ance


(a) Los Angeles

0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance


0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance

0 200 400 600 800 1000

L1

dist

ance




0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10

0

02

04

06

08

10






Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



(a) Los Angeles

Perc

enta

ge c

over

ed


0

02

04

06

08

10

0 200 400 600 800 1000 1200

(b) New York City

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 100 200 300 400 500


(c) San Francisco

0 100 200 300 400 500 700


Perc

enta

ge c

over

ed

0

02

04

06

08

10

600

(d) Chicago

0 200 400 600 800 1000


Perc

enta

ge c

over

ed

0

02

04

06

08

10

(e) Philadelphia

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 200 400 600 800 1000


(f) London

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 50 100 150 200 250 300 350



ln4D5=X+ 0 (3)


sum




prod


p4q5= q timesprod

fisinq

f 1 (4)


sum



Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

NB

ER

PAR

BC

N

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

N

BE

RPA

RB

CN

Pear

sonrsquo

s co

eff

Cities


0

02

04

06

08

10

Pear

sonrsquo

s co

eff

Cities

(b) Queries




gAV4h5=

sum

risinRhrating4r5Rh

0 (5)








h1 0 0 0 1 RTh 1 RgtT

h 5

+w2 middot 4R1h1 R

2h1 0 0 0 1 R

Th 1 R

gtTh 51 (6)



Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




R1h R1

h R2h R2

h R3h R3

h Rgt3h Rgt3

h












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



Title Formula







Ranking

0

02

04

06

08

Con

side

ratio

n pr

obab

ility


10 20 30 40 50

10









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




glowast

AV4h5=

sum


Rh + I0 (7)




glowast


h1 0 0 0 1 RTh 1 RgtT

h 5


h1 0 0 0 1 RTh 1 R

gtTh 50 (8)








Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200


o

f su

cces

sful

atta

cks

400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10






00

02

04

06

08

10

200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



o

f su

cces

sful

atta

cks

0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks







0

2 times 105

4 times 105

6 times 105

8 times 105

106

12 times 106

1 2 3 4 5

Num

ber

of r

evie

ws

Stars


35 41

40

36

40


0

01

02

03

04

05

06

07

08

AP APJ DSP OTH

Cov

ered

per

cent

age

Response type

Total reviews




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












sum

d di54sum

d

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



(a) Los Angeles

Perc

enta

ge c

over

ed


0

02

04

06

08

10

0 200 400 600 800 1000 1200

(b) New York City

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 100 200 300 400 500


(c) San Francisco

0 100 200 300 400 500 700


Perc

enta

ge c

over

ed

0

02

04

06

08

10

600

(d) Chicago

0 200 400 600 800 1000


Perc

enta

ge c

over

ed

0

02

04

06

08

10

(e) Philadelphia

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 200 400 600 800 1000


(f) London

Perc

enta

ge c

over

ed

0

02

04

06

08

10

0 50 100 150 200 250 300 350



ln4D5=X+ 0 (3)


sum




prod


p4q5= q timesprod

fisinq

f 1 (4)


sum



Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

NB

ER

PAR

BC

N

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

N

BE

RPA

RB

CN

Pear

sonrsquo

s co

eff

Cities


0

02

04

06

08

10

Pear

sonrsquo

s co

eff

Cities

(b) Queries




gAV4h5=

sum

risinRhrating4r5Rh

0 (5)








h1 0 0 0 1 RTh 1 RgtT

h 5

+w2 middot 4R1h1 R

2h1 0 0 0 1 R

Th 1 R

gtTh 51 (6)



Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




R1h R1

h R2h R2

h R3h R3

h Rgt3h Rgt3

h












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



Title Formula







Ranking

0

02

04

06

08

Con

side

ratio

n pr

obab

ility


10 20 30 40 50

10









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




glowast

AV4h5=

sum


Rh + I0 (7)




glowast


h1 0 0 0 1 RTh 1 RgtT

h 5


h1 0 0 0 1 RTh 1 R

gtTh 50 (8)








Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200


o

f su

cces

sful

atta

cks

400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10






00

02

04

06

08

10

200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



o

f su

cces

sful

atta

cks

0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks







0

2 times 105

4 times 105

6 times 105

8 times 105

106

12 times 106

1 2 3 4 5

Num

ber

of r

evie

ws

Stars


35 41

40

36

40


0

01

02

03

04

05

06

07

08

AP APJ DSP OTH

Cov

ered

per

cent

age

Response type

Total reviews




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












sum

d di54sum

d

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

NB

ER

PAR

BC

N

WA

SPH

XL

AO

RL

CH

IL

VD

AL

HO

U SFN

YC

PHI

AT

LSA

LO

N

BE

RPA

RB

CN

Pear

sonrsquo

s co

eff

Cities


0

02

04

06

08

10

Pear

sonrsquo

s co

eff

Cities

(b) Queries




gAV4h5=

sum

risinRhrating4r5Rh

0 (5)








h1 0 0 0 1 RTh 1 RgtT

h 5

+w2 middot 4R1h1 R

2h1 0 0 0 1 R

Th 1 R

gtTh 51 (6)



Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




R1h R1

h R2h R2

h R3h R3

h Rgt3h Rgt3

h












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



Title Formula







Ranking

0

02

04

06

08

Con

side

ratio

n pr

obab

ility


10 20 30 40 50

10









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




glowast

AV4h5=

sum


Rh + I0 (7)




glowast


h1 0 0 0 1 RTh 1 RgtT

h 5


h1 0 0 0 1 RTh 1 R

gtTh 50 (8)








Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200


o

f su

cces

sful

atta

cks

400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10






00

02

04

06

08

10

200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



o

f su

cces

sful

atta

cks

0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks







0

2 times 105

4 times 105

6 times 105

8 times 105

106

12 times 106

1 2 3 4 5

Num

ber

of r

evie

ws

Stars


35 41

40

36

40


0

01

02

03

04

05

06

07

08

AP APJ DSP OTH

Cov

ered

per

cent

age

Response type

Total reviews




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












sum

d di54sum

d

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




R1h R1

h R2h R2

h R3h R3

h Rgt3h Rgt3

h












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



Title Formula







Ranking

0

02

04

06

08

Con

side

ratio

n pr

obab

ility


10 20 30 40 50

10









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




glowast

AV4h5=

sum


Rh + I0 (7)




glowast


h1 0 0 0 1 RTh 1 RgtT

h 5


h1 0 0 0 1 RTh 1 R

gtTh 50 (8)








Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200


o

f su

cces

sful

atta

cks

400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10






00

02

04

06

08

10

200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



o

f su

cces

sful

atta

cks

0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks







0

2 times 105

4 times 105

6 times 105

8 times 105

106

12 times 106

1 2 3 4 5

Num

ber

of r

evie

ws

Stars


35 41

40

36

40


0

01

02

03

04

05

06

07

08

AP APJ DSP OTH

Cov

ered

per

cent

age

Response type

Total reviews




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












sum

d di54sum

d

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



Title Formula







Ranking

0

02

04

06

08

Con

side

ratio

n pr

obab

ility


10 20 30 40 50

10









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




glowast

AV4h5=

sum


Rh + I0 (7)




glowast


h1 0 0 0 1 RTh 1 RgtT

h 5


h1 0 0 0 1 RTh 1 R

gtTh 50 (8)








Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200


o

f su

cces

sful

atta

cks

400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10






00

02

04

06

08

10

200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



o

f su

cces

sful

atta

cks

0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks







0

2 times 105

4 times 105

6 times 105

8 times 105

106

12 times 106

1 2 3 4 5

Num

ber

of r

evie

ws

Stars


35 41

40

36

40


0

01

02

03

04

05

06

07

08

AP APJ DSP OTH

Cov

ered

per

cent

age

Response type

Total reviews




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












sum

d di54sum

d

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved




glowast

AV4h5=

sum


Rh + I0 (7)




glowast


h1 0 0 0 1 RTh 1 RgtT

h 5


h1 0 0 0 1 RTh 1 R

gtTh 50 (8)








Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200


o

f su

cces

sful

atta

cks

400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10






00

02

04

06

08

10

200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



o

f su

cces

sful

atta

cks

0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks







0

2 times 105

4 times 105

6 times 105

8 times 105

106

12 times 106

1 2 3 4 5

Num

ber

of r

evie

ws

Stars


35 41

40

36

40


0

01

02

03

04

05

06

07

08

AP APJ DSP OTH

Cov

ered

per

cent

age

Response type

Total reviews




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












sum

d di54sum

d

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0 200


o

f su

cces

sful

atta

cks

400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0 200


400 600 800 1000

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0

02

04

06

08

10






00

02

04

06

08

10

200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks

0 200 400 600 800 1000


0

02

04

06

08

10

o

f su

cces

sful

atta

cks




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



o

f su

cces

sful

atta

cks

0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks







0

2 times 105

4 times 105

6 times 105

8 times 105

106

12 times 106

1 2 3 4 5

Num

ber

of r

evie

ws

Stars


35 41

40

36

40


0

01

02

03

04

05

06

07

08

AP APJ DSP OTH

Cov

ered

per

cent

age

Response type

Total reviews




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












sum

d di54sum

d

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



o

f su

cces

sful

atta

cks

0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks







0

2 times 105

4 times 105

6 times 105

8 times 105

106

12 times 106

1 2 3 4 5

Num

ber

of r

evie

ws

Stars


35 41

40

36

40


0

01

02

03

04

05

06

07

08

AP APJ DSP OTH

Cov

ered

per

cent

age

Response type

Total reviews




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












sum

d di54sum

d

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks



o

f su

cces

sful

atta

cks

0

02

04

06

08

10

0 200 400 600 800 1000



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks









Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks







0

2 times 105

4 times 105

6 times 105

8 times 105

106

12 times 106

1 2 3 4 5

Num

ber

of r

evie

ws

Stars


35 41

40

36

40


0

01

02

03

04

05

06

07

08

AP APJ DSP OTH

Cov

ered

per

cent

age

Response type

Total reviews




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












sum

d di54sum

d

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved



0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks


0

02

04

06

08

10

0 200 400 600 800 1000

o

f su

cces

sful

atta

cks







0

2 times 105

4 times 105

6 times 105

8 times 105

106

12 times 106

1 2 3 4 5

Num

ber

of r

evie

ws

Stars


35 41

40

36

40


0

01

02

03

04

05

06

07

08

AP APJ DSP OTH

Cov

ered

per

cent

age

Response type

Total reviews




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












sum

d di54sum

d

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












sum

d di54sum

d

sum




Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved











Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved












Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved






















Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved
















































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved







































Dow

nloa

ded

from

info

rms

org

by [

155

246

152

198]

on

13 N

ovem

ber

2016

at 1

054

Fo

r pe

rson

al u

se o

nly

all

righ

ts r

eser

ved

Documents

The Impact of Fake Reviews on Online Visibility: A ...cgi.di.uoa.gr/~gvalk/pubs/FakeReviewsImpact.pdf · Theodoros Lappas, Gaurav Sabnis, Georgios Valkanas Stevens Institute of Technology,