Bayesian AB Testing

Chris Campbell

Bayesian AB Testing

• Use prior information• Prevent peeking problem• Reduce impact of multiple comparisons• Probability of effect more interpretable than threshold

Section title

Search Item Click AB Test

5

Search Item Click

Section title

6

7

Search Item Clicks

Paired t-test

• Sample size

• Minimum detectable effect

power.t.test(n = 28, sd = 895, power = 0.9)

delta = 789.5388

9

Power Calculation

Bayesian Estimation Supersedes the t-Test

library(BEST)BESTmcmc(clk$A - clk$B,

priors = list(muM = 0, muSD = 1400))

11

BEST

BEST

Posterior Probability

= Prior

Probability +

New Evidence

Probability of Direction

13

82%

14

Probability of Direction

Highest Density Interval

Highest Density Interval

89% of densityCredible Interval

Region of Practical Equivalence

ROPE

Overlap of Credible Interval with ROPE

library(rstanarm)stan_glm(

clicks ~ group +weekday,

data = clk)

19

Bayesian GLM

Better control of additional explanatory variables

Bayesian GLM

Summary

• Large quantity of modelling methods available• Intuitive interpretation

21