1

Using Customer Satisfaction to Model Loyalty

Bruce Ingraham

Ingraham Consulting

SF Data Mining

Predicting Consumer Behavior

June 19, 2012

San Francisco, CA

2

Overview

• Background

• Theoretical framework

• Method

• Results

• Discussion

• Questions

3

Background

• Financial services company “Golden Investments”

• Over 30,000,000 customers

• Client had a very good model to predict risk of defection based on customer attributes and transaction data

• Client also had monthly customer satisfaction surveys

• Does satisfaction data provide any additional information about defections?

4

Theoretical Framework

• Mittal, V., & Kamakura, W. A. (2001). Satisfaction, repurchase intent, and repurchase behavior: Investigating the moderating effect of customer characteristics. Journal of Marketing Research, 38, 131-142.

• Three components in model– Differential satisfaction thresholds

– Response bias

– Non-linear functional form

5

Theoretical Framework

• Differential satisfaction thresholds

– Customers have different “pain thresholds” or tolerance levels with respect to the decision to defect

– For example, new customers may be sensitive to customer service issues

– Satisfaction thresholds vary systematically with customer characteristics

6

ILLUSTRATION: Same Satisfaction Thresholds

If two groups of customers which differ in one characteristic, e.g. tenure, have the same thresholds, their response graphs have nearly identical lines, reflecting similar retention behavior for the same satisfaction level.

7

ILLUSTRATION: Differential Satisfaction Thresholds

If the groups have different thresholds, their response graphs have parallel lines, since given the same rating, customers with lower thresholds are more likely to remain customers.

8

Theoretical Framework

• Response bias

– Likert-scale satisfaction ratings are error-prone measures of unobservable (latent) true satisfaction

– Harsh-raters and easy-raters

– Response bias varies systematically with customer characteristics

9

ILLUSTRATION: Response Bias

If two groups of customers which differ in one characteristic have differential thresholds and the same response bias, then their response graphs look like the previous chart—parallel lines. However, if the response bias differs systematically between groups, the additional variation in retention appears as unequal slopes.

10

Theoretical Framework

• Non-linear functional form

– Likert-scale data is ordinal categorical at best

– The relationship between satisfaction and percent defecting is not interval data

– For example, the difference in percent defecting between a satisfaction rating of 1 and a rating of 2 may be different from that between a 4 and a 5

11

ILLUSTRATION: Nonlinearity

When using a Likert scale to measure satisfaction, the assumption is often made that the difference in response between rating levels is constant, i.e. the functional form is linear. When this assumption is true, the response graph is a straight line, and the ratings can be treated as interval data in modeling. When the assumption is false, the ratings need to be modeled as ordinal categorical data.

12

Theoretical Framework

• Binary probit model

• Main effects parameters γi capture the different thresholds for each customer group

• Interaction parameters δj capture the different response biases for each customer group by rating

13

BACKGROUND: Probit Models

Probit models are used to model a binary response variable, e.g. retained/defected, when it is assumed that the response depends an individual’s threshold being surpassed by an input variable, such as satisfaction. The threshold is assumed to vary among individuals, and to be normally distributed within the population.

14

BACKGROUND: Probit Models

Inthis model, the input variable is the satisfaction rating. Individual variation is modeled by characteristics, e.g.loyaltyscore, and interactions with the rating. The resulting threshold value gives the customer’s position in the normal distribution, e.g. z-score, and the CDF is used to find the corresponding probability of retention.

Threshold = 1

Probability of

retention = .84

15

Method

• Data

– Responses from 12 monthly surveys were combined to reach a sample of N=9,105

– The survey question of interest was worded “How likely are you to continue doing business with Golden Investments?”

– Ratings were on a 5-point Likert scale, where 1 indicated highly unlikely, and 5 indicated highly likely

16

Method

• Data

– Using the predictive defection model, each customer was assigned to a defection segment

– Risk % of population

• Very high 10%

• High 10%

• Moderate 15%

• Low 15%

• Very low 50%

17

Method

• The dependent variable was if the customer had defected within six months of the survey response

• Model estimation– Parameter estimation and hypothesis testing for

the probit model were carried out using SAS PROC PROBIT

– The reference levels• rating: 5

• risk: very low

18

Method

• Test for addition information about defection

– χ2 goodness-of-fit test to compare actual and predicted frequencies for the defection model and the probit model

19

Results

• Main effects parameters γi

– Intercept 2.9 (>.001)

– Rating γp

• 1 -1.3 >.001

• 2 -0.9 >.001

• 3 -0.05 >.001

• 4 -0.2 .066

• 5 reference

20

Results

• Main effects

– Risk γp

• VH -1.5 >.001

• H -1.1 >.001

• M -0.8 >.001

• L -0.4 .01

• VL reference

21

EXAMPLE: Find the retention rate forrisk=High, rating=2.

We start with the intercept, 2.9, which tells us the position of the reference group in the normal distribution. The reference group,risk= Very Low, rating = 5, has a retention rate of 99.8%.

2.9

99.8%

22

Next, the estimate forrisk= High is -1.11. Add this to 2.97 to get the new position, 1.86. This reduces the retention rate to 96.8%.

1.86

96.8%

23

Finally, the estimate for rating = 2 is -0.89. Add this to 1.86 to get the new position, 0.97. This reduces the retention rate to 83.1%, which is the answer!

If significant interactions between risk segment &rating had been found, they would be added in also.

0.97

83.1%

24

Start at 99.8

risk moves to 96.8

rating moves to 83.1

25

Results

• Interactions

– None of the interaction estimates were statistically significant

– Suspect Type II error: n too small in some cells relative to the variance, resulting in large s.e.

26

Results• χ2goodness-of-fit tests performed for the defection model

and for the probit model. The expected values for the risk model were obtained by averaging the risk scores for all of the survey customers in the segment, and then multiplying by the segment frequency. The expected values for the probit models were calculated similarly, using probabilities estimated by the model.

• The null hypotheses was rejected for the risk model (p=0.035). The large residuals which indicate lack of fit belonged to the Very High and Low segments.

• The null hypotheses was not rejected for the probit model (p=.99). The residuals were very small for all segments, and the fit was nearly perfect.

27

Discussion

• Differential satisfaction thresholds– Statistically significant main effects are evidence for

this claim

• Response bias– Statistically non-significant interaction terms do not

support this claim

– May be Type II error due to small n

• Non-linear functional form– Statistically non-significant estimate for rating of 4 is

evidence for this claim

28

Discussion

• Additional information about defection– χ2 tests are evidence that the survey data provides

additional information about defection

• Issues– Non-response bias and propensity to respond. Are

unhappy customers less likely to respond?

– Overfitting

– Predicted risk as independent variable

• Business application is to target retention programs to highest-risk segments

29

Questions?

30

Thanks for coming!