Making Words Speak: Leveraging Consumer Insights from ... · Making Words Speak: Leveraging Consumer Insights from Online Review Text to Improve Service Quality Andrea Ordanini

Making Words Speak: Leveraging Consumer Insights from Online Review

Text to Improve Service Quality

Andrea Ordanini Bocconi University

Raji Srinivasan University of Texas at Austin

Anastasia NanniBocconi University

MSI Managerial Presentation

The Research Idea

2

How do managers make sense of online reviews?– Ratings are important but they cannot be highly diagnostic

(e.g., j-shaped distribution)

– Texts are potentially more informative but they are unstructured and thus challenging to exploit

Managers might benefit from textual information in structured form

Textual information in structured form should: – increase the managerial ability to change and improve the

offering,– especially when the manager has proper motivation to act,

and the organizational context provides adequate opportunity


Textual Data in Structured Form

3

Providing structure to textual data means:– Visual representation

• Thematic Maps

– Redundancy• Different categorization

– Dual Coding • Pictorial and Tabular Forms

Research Question - Does treating managers with textual data in structured form help to improve their results?


Research Approach

4

To avoid self-selection bias, unobserved heterogeneity issues, and given the time lag to detect the potential effect, we investigate our RQ through a Randomized Control Trial

Despite its challenges, RCT has been recently advocated in Marketing to increase the confidence in some strategic choices


Method

5

Jan 16 Mar 1st 16 Jun 1st 16 Aug 30th 16

Step 1Sampling

Step 2Start Treatment

Step 3Reinforcement

Step 4End Treatment

TREATMENT PERIOD

Perf. pre06/08 2015

Perf. post06/08 2016

Step 5Post- Hoc

Nov 16

n = 598 n = 203 n = 146 n = 135 n = 99

Primary Data

Stimulus andSecondary Data Stimulus Primary and

Secondary DataPrimary

Data

We focus on Italian hotel industry and the Tripadvisor context (for relevance)

We ensure collaboration from Federalberghi (for sampling) and a leading reputation mgmt company (for measures)


Step 1 Sampling

6

Hotel characteristics Participant Non-Participant Difference

Segment c2(2) =.42; p=.81

Budget 19% 21% Mid-price/Suite 57% 57%

Boutique 24% 22% Location c2

(1) =.38; p=.54 Urban 50% 47%

Touristic 50% 53% Size c2

(1) =.05; p=.82 <30 rooms 52% 51% >30 rooms 48% 49%

Respondent characteristics Participant Non-Participant Difference

Role c2(2) =2.56; p=.28

Owner 60% 64% Hotel Manager 32% 31%

Marketing Manager 8% 5%

Education c2(3) =3.87; p=.28

Primary 35% 29% Secondary (Hotel) 43% 44%

Secondary (General) 19% 20% Tertiary 3% 6%

Age 50.0 48.1 t=1.91; p=.06 Work experience (yrs) 3.2 3.5 t=1.80; p=.07

Perceptions on reviews (1-7) Participant Non-Participant Difference

Reviews Credibility 3.9 3.7 t=1.64; p=.10 Reviews are Informative (score) 4.0 3.8 t=1.60; p=.11 Reviews are Informative (text) 3.9 4.1 t=1.38; p=.17 Decision Confidence 4.1 4.0 t=0.39; p=.70 Reviews Usefulness 4.7 4.4 t=1.99; p=.05 Lack of Time for Reviews 3.3 3.6 t=1.49; p=.14 Lack of Staff for Reviews 3.5 3.6 t=0.90; p=.37


203 hotel managers participated to our study:

- 101 in the control condition

(report with text of the last year’s reviews

of their hotels and the ratings)

- 102 in the treatment condition

(wordcloud and sentiment analysis table from

last year’s reviews of their hotels, in addition to

what provided to managers in the control condition)

Steps 2: Stratified Randomization

7

Treatment

and

Typology

Location and Size

Urban Locations Touristic Locations*

Less than 30 rooms

More than 30 rooms Total

Less than 30 rooms

More than 30 rooms Total Total

Control

1-2 Stars 10 2 12 7 - 7

3 Stars 15 21 36 17 7 24

4-5 Stars 1 8 9 1 13 14

Total 26 31 57 25 20 45 102

Treatment

1-2 Stars 10 3 13 4 - 4

3 Stars 15 19 34 19 7 26

4-5 Stars 1 9 10 2 12 14

Total 26 31 57 25 19 44 101


Steps 2/3 – Example of Treatment (wordcloud + sentiment table)

8MSI Managerial Presentation

Steps 4 – Analysis

9

Our design reflects a Split-Plot model, where the effect of interest – the treatment effect - is the coefficient “Treatment x Year”


Online Ratings Coef. SE* P>|t|

Treatment (vs. Control) -.102 .114 0.37

Year (2016 vs. 2015) .007 .060 0.91 Treatment#Year

Treatment x Year .218 .095 0.03

Constant 4.186 .078 0.00

Random-effects

Parameters Estimate SE* [95% Conf. Interval]

Hotel: Id

Var (_cons) .294 .052 .207 .416

Var (Residual) .084 .014 .059 .117

Steps 4 – Analysis: Main Result

10

+5.4%


The managerial use of

online review text

analytics improves

service quality by

5.4% during the

treatment window,

comparing hotels in

the treatment vs.

control conditions

4.1

4.15

4.2

4.25

4.3

Aver

age

Perfo

rman

ce -

Rev

iew

s Sc

ores

2015 2016Year

Control Treatment

Average Treatment Effect

+5.4%

Heterogeneous Treatment Effect: Perceived accountability (Opportunity)


The positive effect

of review text

analytics

disappears when

managers do not

feel accountant-.10

.1.2

.3P

erfo

rman

ce -

Var.

(Pos

t - P

re)

4 5 6 7Perceived Accountability Pre-Treatment

Control Treatment

Average Treatment Effect * Perceived Accountability

22%ß à 78%

Heterogeneous Treatment Effect: Perceived Performance ‘Pre’ (Motivation)


The positive effect

of review text

analytics also

disappears when

managers are

satisfied with the

existing level of

performance

-.20

.2.4

.6Pe

rform

ance

- Va

r (Po

st-P

re)

1 2 3 4 5Perceived Performance Pre-Treatment

Control Treatment

Average Treatment Effect * Perceived Performance

69%ß à 31%

Main Implications

13

Textual data in structured form seems to provide a positive effect on online performances:– It increases the quality of the offering– It generates positive customers responses

The effect size is not trivial and is robust to several alternative assumptions

Decisional stimuli and organizational characteristics however act as important boundary conditions for such positive effect

The main benefit for managers from the use of textual data in structured form is to clearly and rapidly identify points of weakness in their offering


Documents

Making Words Speak: Leveraging Consumer Insights from ... · Making Words Speak: Leveraging Consumer Insights from Online Review Text to Improve Service Quality Andrea Ordanini