Upload
gur
View
52
Download
0
Tags:
Embed Size (px)
DESCRIPTION
TMLE ANALYSIS. The Causal effect of display advertising on conversion. NYC Predictive Analytics Ori Stitelman ( [email protected] ) Brian Dalessandro Claudia Perlich foster provost August 11, 2011. QUESTION OF INTEREST. WHAT IS THE EFFECT OF DISPLAY ADVERTISING - PowerPoint PPT Presentation
Citation preview
TMLE ANALYSIS
NYC PREDICTIVE ANALYTICSORI STITELMAN ([email protected])BRIAN DALESSANDROCLAUDIA PERLICHFOSTER PROVOSTAUGUST 11, 2011
THE CAUSAL EFFECT OF DISPLAY ADVERTISING ON CONVERSION
2
QUESTIONOF INTEREST
WHAT IS THE EFFECT OFDISPLAY ADVERTISINGON CUSTOMER CONVERSION? ?
3
WHAT ISTMLE?A SEMI-PARAMETRIC method
for estimating CAUSAL PARAMETERS
that DIRECTLY answer a
business question of interest?
4
SEMI-PARAMETRICRealistic assumptions
CAUSAL PARAMETERSEffect/Impact (NOT Coefficients)
DIRECTLYActionable
WHAT ISTMLE?A SEMI-PARAMETRIC methodfor estimating CAUSAL PARAMETERSthat DIRECTLY answer abusiness question of interest?
5
OUTLINE1. BACKGROUND: DISPLAY ADVERTISING &
2. PEEK AT RESULTS
3. A/B TESTING
4. ALTERNATIVE APPROACHES (NICE)
5. RESULTS
6. CONCLUSION
6
THEBROWSERPROCESS
2. Use observed data to build list of prospects
3. Subsequently observe same browser surfing the web the next day
4. Browser visits a site where a display ad spot exists and bid requests are made
5. Auction is held for display spot
6. If auction is won display the ad
7. Observe browsers actions after displaying the ad
1. Observe people taking actions and visiting content
7
RESULTSPREVIEW
RETARGETING
M6D
PROSPEC...
RETARGETING
M6D
PROSPEC...
RETARGETING
M6D
PROSPEC...
0%
2%
4%
6%
8%
10%
12%
14%
RELATIVE LIFT:EXPOSED VS. UNEXPOSED USERS
DID NOT SEE AD SAW ADCO
NVER
SIO
N RA
TE
1.05X
2.62X
1.11X
1.31X
0.92X2.26X
TELECOM COMPANY A
TELECOM COMPANY B
TELECOM COMPANY C
8
P
GENERAL APPROACH
?
Ψ(P)
1. State Question
3. Define Parameter
Ψ(Pn)4. Estimate Parameter
2. Define Causal Assumption/Likelihood
9
WHAT IS THE EFFECT OFDISPLAY ADVERTISING ON CUSTOMERCONVERSION?
?1. STATE QUESTION
DISPLAY ADVERTISINGShowing/Not showing a browser a display ad.
CUSTOMER CONVERSIONVisiting the advertisers website in the next 5 days.
10
P2. DEFINE CAUSAL ASSUMPTIONS/LIKELIHOOD
O = (W,A,Y) ~ P0
W – Baseline VariablesA – Binary Treatment (Ad)Y – Binary Outcome (Site Visit)
P0(O) = P(W)P(A|W)P(Y|A,W)QW
QY
gA
11
DATA STRUCTURE: OURVIEWERS
CHARACTERISTICS(W)
TREATMENT(A)
CONVERSION(Y)
Color Sex HeadShape
Ad No Ad
No Yes
12
Ψ(P)3. DEFINE PARAMETER
P0(O) = P(W)P(A|W)P(Y|A,W)
P0,a(O) = P(W)P(Y|A=a,W)
P0,a(O) = P(W)P(Ya|W)=
LIKELIHOOD
DISTRIBUTION UNDER INTERVENTION
13
Ψ(P)3. DEFINE PARAMETER
ΨAI(P)=E[YA=ad] – E[YA=no ad]
ΨRI(P)=E[YA=ad]/E[YA=no ad]
1. ADDITIVE IMPACT
2. RELATIVE IMPACT
P0,a(O) = P(W)P(Ya|W)
14
E[YA=ad] E[YA=no ad]
φn,ad/φn,no ad
1. PARAMETERS ARE COMBINATIONS OF TREATMENT SPECIFIC CONVERSION RATES
2. SO WE CAN COMBINE ESTIMATES OF THESE THESE RATES
Ψ(Pn)4. ESTIMATE PARAMETER
φn,ad-φn,no ad
15
Ψ(Pn)4. ESTIMATE PARAMETER
1. Optimal Experiment
2. A/B Testing
3. MLE Based Substitution Estimator (MLE)
4. Inverse Probability Estimators (IPTW)
5. Double Robust Estimating Equations (DR-IPW)
6. Targeted Maximum Likelihood Estimation (TMLE)
NICE
16
OPTIMALEXPERIMENT:
3.6 per 1,000
1.2 per 1,000
Compare conversion rates of seeing an ad to conversion rate without ad for SAME individuals.
17
OPTIMALEXPERIMENT:
BROWSER
SHOW AD?
OBSERVEOUTCOME
OR
OR
18
REALITY: OR
OR
OR
OR
19
OPTIMAL EXPERIMENT:Compare conversion rates of seeing an ad to conversion rate without ad for SAME individuals.
3.6 per 1,000
1.2 per 1,000
20
COMMON APPROACH:A/B TESTINGSince we can not both treat and not treat SAME individuals. Randomization is used to create “EQUIVALENT” groups to treat and not treat.
3.4 per 1,000
1.6 per 1,000
21
SIGNIFICANT COSTSASSOCIATED WITH DOING A/B TESTING WELL
1. Cost of displaying PSAs to the control(untreated group).
2. Overhead cost of implementing A/B test and ensuring that it is done CORRECTLY (Kohavi et al.)
3. Wait time necessary to evaluate the results.
22
NON INVASIVECAUSAL ESTIMATION (NICE)Estimate The Effects In The Natural Environment (Observed Data)
23
“WHAT IF”CAUSAL ANALYSIS ADJUSTING FOR CONFOUNDING
Need to adjust for the fact that the group that saw the advertisement and the group that didn’t may be very different.
24
TWO MACHINES:QY,n AND gA,n
gA
QY
Pn(A| )
Pn(Y| )Pn(Y| )
gA,n
QY,n
ESTIMATING QY AND gA
1. Many tools exist for estimating binary conditional distributions.
1. Logistic regression, SVM, GAM, Regression Trees, etc.2. Data adaptive estimation methods that use cross
validation.3. SuperLearner (R package, Eric Polley)
2. Causal analyses will benefit from the advances in parallelized routines for data adaptive estimation.
26
INVERSE PROBABILITY WEIGHTED ESTIMATORS (IPTW)
•Adjust For Confounding Through gn.
•Weight individuals that are unlikely to be shown an advertisement more than individuals that are likely to be shown an advertisement
STRATEGY
ESTIMATOR
Pn(A| )=1/2
=
1/10
Pn(A| )
QY
QY
MLE BASED SUBSTITUTION ESTIMATOR (MLE)
•Adjust For Confounding Through Q.•Predict how each observed browser will behave had they been shown an ad and had they not been shown an ad.
STRATEGY
ESTIMATOR
QY,n
QY,n27
28
DOUBLE ROBUST ESTIMATORSWhat if QY,n or gA,n are broken? The MLE based estimator and IPTW rely on consistent estimates of Q and g respectively.
gA
QY
P(A| )
P(Y| )P(Y| )
gA,n
QY,n
29
OR
STRATEGY
ESTIMATOR
AUGMENTED – IPTW (A-IPTW)
•Adjust For Confounding Through Q and g.
•Augments IPTW estimator with information from Q.
•Alternatively adjusts MLE with information from g.
Ψ(QY)
TARGETED MAXIMUM LIKELIHOOD ESTIMATOR (TMLE)
•Adjust For Confounding Through Q and g.
•Predict how each observed browser will behave had they been shown ad and had they not been shown ad.
•The new machine Q* is calibrated with concern for the parameter of interest.
•R package “tmle” at CRAN
STRATEGY
ESTIMATORQY
Ψ(QY)QY
Q*Y,n
Q*Y,n
30
CREATING Q*YN
•QYn is updated to Q*Yn using a clever covariate that is a function of g.
•Update is done through the use of a parametric submodel.
•Update is a univariate regression with the initial QYn as an offset.
QY
gA
Ψ(QY)QY
Q*Y,n
31
32
SAMPLING/ANALYSIS1. Select Prospects that we got a bid request for on day
0.
2. Observe if treated on day 1. For those treated A=1 and those not treated A=0. Collect W.
3. Create outcome window that is the next five days following treatment and observe if event occurs.
4. Estimate parameters using the methods previously described.
33
RESULTS
34
IPTW AND A-IPTW BLOW UP UNDER POSITIVITY VIOLATIONS
35
METHOD VALIDATION: A/B TEST VS TMLE
36
METHOD VALIDATION: NEGATIVE TEST
37
GROSSCONVERSIONRATESARE ONLY PARTOF THE STORY
RETARGETING
M6D
PROSPECTING
RETARGETING
M6D
PROSPECTING
RETARGETING
M6D
PROSPECTING
0.0%
0.1%
0.2%
0.3%
0.4%
0.5%
0.6%
0.7%
0.8%
0.9%
1.0%
0.5%
0.9%
0.1%
0.8%0.7%
ADDITIVE IMPACT: EXPOSED VS. UNEXPOSED USERS
ADDI
TIVE
IMPA
CT IN
CO
NVER
SIO
N RA
TE
-0.2%
TELECOM COMPANY A
TELECOM COMPANY B
TELECOM COMPANY C
38
EFFECTIVENESSVARIES BYMARKETER& CAMPAIGN
RETARGETING PROSPECTING RETARGETING PROSPECTING0%
2%
4%
6%
8%
10%
12%
14%
16%
RELATIVE IMPACTDID NOT SEE AD SAW AD
CONV
ERSI
ON
RATE
B2B COMPANY
A
B2B COMPANY
B
1.08X
1.08X4.23X
3.77X
39
RETARGETING
M6D P
ROSPECTING
RETARGETING
M6D P
ROSPECTING
RETARGETING
M6D P
ROSPECTING
RETARGETING
M6D P
ROSPECTING
0.0%
1.0%
2.0%
3.0%
4.0%
5.0%
6.0%
7.0%
8.0%
RELATIVE IMPACTDID NOT SEE AD SAW AD
CONV
ERSI
ON
RATE
1.71X1.34X
2.57X
1.17X
3.00X
2.33X
1.84X
1.11X
CAR RENTAL RESTAURANTHOTELAIRLINE
Effective creative and
targeting drive lift across
marketers and verticals
TEC
HN
OLO
GY
A
TRA
VE
L A
B2B
A
TRA
VE
L B
TELE
CO
M A
TRA
VE
L C
TELE
CO
M B
TRA
VE
L D
RE
TAIL
A
RE
TAIL
B
TRA
VE
L E
ED
UC
ATI
ON
A
TRA
VE
L F
RE
STA
UR
AN
TA
AU
TO A
RE
TAIL
C
RE
STA
UR
AN
T B
TRA
VE
L G
ED
UC
ATI
ON
B
TEC
HN
OLO
GY
B
RE
TAIL
D
TELE
CO
M C
RE
TAIL
E
AU
TO B
ED
UC
ATI
ON
C
FIN
AN
CE
A
TRA
VE
L H
B2B
B
RE
TAIL
F
TRA
VE
L I
-100%
0%
100%
200%
300%
400%
500%
600%
700%
800%
-100%
0%
100%
200%
300%
400%
500%
600%
700%
800%
M6D PROSPECTING LIFT RETARGETING LIFT
M6D
PRO
SPEC
TING
LIF
T
RETA
RGET
ING
LIF
T
For 25 out of 30 marketers,
relative lift was
higher for prospecting
candidates than for retargeting
candidates .
SUMMARY RESULTSAVERAGE RELATIVE LIFT OF 90% FOR M6D PROSPECTING
41
CONCLUSIONS1. Estimating causal effects allows one to directly estimate the
impact of the display advertisement.
2. Causal effects can be estimated in the observed data.
3. Display advertising works in varied ways. Causal analysis allows us to estimate effects on a case by case basis.
4. All methods for estimating effects are not equal.
5. These methods may be used for assessing other types of causal effects.
6. What is next?
42
VISIT OUR BLOGmedia6degrees.com/blog
43
OTHER RESOURCES FOR TMLE
Enter “Slide Show” view and click image to link to site.
44
ACKNOWLEDGEMENTS
Edward CaprioloBrian DalessandroRod HookBrian MayRyan OttobreClaudia PerlichTom Phillips
Mark van der LaanSusan GruberEric Polley
Foster Provost
45
REFERENCES1. O. Stitelman, B. Dalessandro, C. Perlich, and F. Provost. Estimating The Effect Of Online Display
Advertising On Browser Conversion. In Proceedings of KDD, Annual International Workshop on Data Mining and Audience Intelligence for Online Advertising, ADKDD ’11.
2. M. van der Laan and S. Rose. Targeted Learning: Causal Inference for Observational and Experimental Data. New York, NY: Springer Publishing Company, 2011. http://www.targetedlearningbook.com/
3. ‘tmle’ R Package http://cran.r-project.org/web/packages/tmle/index.html
4. R. Kohavi and R. Longbotham. Unexpected results in online controlled experiments. ACM SIGKDD Explorations Newsletter, 12(2):31–35, 2010.
5. R. Lewis and D. Reiley. Does retail advertising work: Measuring the effects of advertising on sales via a controlled experiment on yahoo. Technical report, Working paper, 2010.
6. D. Chan, R. Ge, O. Gershony, T. Hesterberg, and D. Lambert. Evaluating online ad campaigns in a pipeline: causal models at scale. In Proceedings of KDD, KDD ’10, pages 7–16, New York, NY, USA, 2010. ACM.