Embed Size (px)
Citation preview
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 1/27
a/B TesTing
for markeTingopTimizaTion
An Intouction to Usin
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 2/27
Intoucto content is o aetes ho ae ne to the suject.
This content tpicall inclues step--step instuctions on ho
to et state ith this aspect o inoun aetin an lean its
unaentals. Ate eain it, ou ill e ale to eecute asic
aetin tactics elate to the topic.
inTroDUCTorY
is THis Book rigHT for me?
Not quite sue i this eoo is iht o ou? See the elo esciption to eteine i
ou level atches the content ou ae aout to ea.
Inteeiate content is o aetes ho ae ailia ith the
suject ut have onl asic epeience in eecutin stateies an
tactics on the topic. This content tpicall coves the unaentals
an oves on to eveal oe cople unctions an eaples.
Ate eain it, ou ill eel cootale leain pojects ith this
aspect o inoun aetin.
inTermeDiaTe
Avance content is o aetes ho ae, o ant to e, epets
on the suject. In it, e al ou thouh avance eatues o
this aspect o inoun aetin an help ou evelop coplete
aste o the suject. Ate eain it, ou ill eel ea not onl
to eecute stateies an tactics, ut also to teach othes ho to
e successul.
aDVanCeD
An introduction to A/b testingAn introduction to A/b testing2
www.HUbSPOT.COm www.
Share This Ebook! Share T
Th i s e b o k !
... ins ou hole aetin ol to-
ethe in one, poeul, inteate sste.
HUbSPOT’S ALL-IN-ONE
mArkETINg SOfTwArE
gt fud: Help prospects nd you online
Cvt: Nutue ou leas an ive convesions
ay: measue an ipove ou aetin
pu oe apps an inteations
LEAd
gENErATION
UbLOg
SOCIA
q
EmAIL &
AUTOmATIONMSE
OPTIms
mAr
ANA
YLEAd
mANAgEmENT
grequest A deo Vieo Ovevie
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 3/27
An introduction to A/b testingAn introduction to A/b testing4
www.HUbSPOT.COm www.
Share This Ebook! Share T
an inTroDUCTion To Using a/BTesTing for markeTing opTimizaTion
Eite maalena geoieva
maie geoieva is an inoun aetin
anae at HuSpot esponsile o ceatin
ne oes, incluin eoos an einas.
She has peviousl helpe ith HuSpot’s
eail aetin poa an the copan’s
lanin pae ceation an optiiation. maie
is a prolic marketing blogger and has also
itten o pulications lie bostInnovation.co
an The St. Petersburg Times. follow me on TwiTTer@mgieVa
ConTenTs
10 gUiDelines for effeCTiVe a/B TesTing /9
wHaT VariaBles YoU sHoUlD TesT /17
a/B TesTing Case sTUDies /31
rUnning & eValUaTing YoUr TesTs /39
ConClUsion & aDDiTional resoUrCes /50
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 4/27
www.HUbSPOT.COm
An introduction to A/b testingAn introduction to A/b testing6
www.HUbSPOT.COm www.
Share This Ebook! Share T
Why should
I conduct A/B testing?“
”
Such inceases in lea eneation can povie a hue copetitive avantae o ou co
act, chances ae ou copetitos aen’t oin A/b testin iht.
“In our ROI-driven world of digital marketing, testing without calculating statistical signi
the ata is not uch ieent than not testin at all,” ite maetinShepa in its epo
with such an eective lea eneation tool, ou can easil ceate oe oppotunities o
tea an et ahea o ou copetitos.
with the iht sotae, A/b testin costs net
to ipleent. An althouh soeties it’s the oe cople eleents that nee to e
soe o the ost aatic esults coe o quic an siple chanes. In othe os
not A/b testin, ou’e issin out on soe seious lea eneation oost.
A/b testin, also non as split testin, is a etho o testin thouh hich aetin vaiales
ae copae to each othe to ienti the one that ins a ette esponse ate. In this contet,
the eleent that is ein testin is calle “contol” an the eleent that is aue to ive a ette
esult is calle “teatent.”
Running A/B tests in your marketing initiatives is a great way to learn how to drive more trafc to
ou esite an eneate oe leas o the visits ou’e ettin. Just a e sall teas to
a landing page, email or call-to-action can signicantly affect the number of leads your company
attacts.
The insihts stein o split tests can asticall ipove the convesion ates o ou lanin
paes an the clicthouh ates o ou esite calls-to-action an eail capains. In act, A/b
testin o lanin paes can eneate up to 30-40% oe leas o b2b sites an 20-25% oe
leas o eCoece sites.
? ONLy 40% OfmArkETErS VALIdATETEST rESULTS.mArkETINgSHErPA LANdINg PAgE OPTImIzATIONbENCHmArk SUrVEy , 2011
40%
60
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 5/27
An introduction to A/b testingAn introduction to A/b testing8
www.HUbSPOT.COm www.
Share This Ebook! Share T
A/b TESTINg Or mULTIVArIATE TESTINg:wHAT’S THE dIffErENCE?
Althouh these to tes ae oth oten thon aoun in the aetin ol, ultivaiate anA/b testin ae not the sae thin.
A/b testin allos ou to onl test one vaiale at a tie. It’s use
to pic the ette out o to vaiales. fo instance, i ou ae
testin the acoun colo o ou lanin paes to eteine
hich one helps ou eneate the ost convesions, ou iht
test e vs. lue. you can also test on the pae level, hich
eans that ou ae looin at the entie pae as the vaiale.
multivaiate testin, on the othe han, enales ou to test an
vaiales siultaneousl. Theoeticall, ultivaiate testin
coines a seies o siultaneous A/b tests. In oe to conuct
statistically signicant multivariate testing, you need to have a lot
of trafc (the type of trafc youTue an goole et.)
This eoo coves A/b testin, ut an o the pactices entione in it can e applie to
ultivaiate testin as ell. Let’s ive into A/b testin an sta t ith a loo at ho to ceate an
effective A/B test and how to measure its signicance.
mUlTiVariaTe
TesTing
a/B
TesTing
CHapTer 1
10 gUiDelinesfor effeCTiVe
a/B TesTing
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 6/27
An introduction to A/b testingAn introduction to A/b testing10
www.HUbSPOT.COm www.
Share This Ebook! Share T
“”
A/b testin can quicl ive ise to a nue o
questions. what shoul ou stat testin? Ho
lon shoul the test last? Thins et conusin
ast.
Thee ae cetain uielines ou shoul eep
in in eoe ou ipleent ou tests. In
this section e ill cove soe est pactices
that ill ae it uch easie to easue ou
results and nd out which variation performed
better. Such knowledge will guide you in guring
out ho to est optiie ou lanin paes,
calls-to-action an eail.
Remember, a simple changecan drive big results.
onlY ConDUCT one TesT aT a
I ou test an eail capain that iects to a lanin pae at the sae tie that ou’etestin that lanin pae, ou esults can et ule pett easil. Ho oul ou no
chane cause the incease in leas?
As a eneal ule, an especiall i ou’e elativel ne to the A/b testin etho, on’t
ultiple tests at the sae tie.
#1
TesT one VariaBle aT a Time
In oe to evaluate ho eective an eleent on ou pae, call-to-action o eail cap
ou have to isolate that vaiale in ou A/b test. Onl test one eleent at a tie. Note t
tain the entire pae, eail o CTA as the vaiale, ou can achieve astic ipovee
#2
TesT minor CHanges, Too
Althouh it’s easonale to thin that i, seepin chanes can incease ou lea en
nues, the sall etails ae oten just as ipotant. while ceatin ou tests, ee
that even a siple chane, sa, sitchin the colo o ou call-to-action utton, can iv
ipoveents. In act, these sots o chanes ae usuall easie to easue than the i
#3
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 7/27
An introduction to A/b testingAn introduction to A/b testing12
www.HUbSPOT.COm www.
Share This Ebook! Share T
YoU Can a/B TesT
THe enTire elemenT
while ou can cetainl test a utton colo o a acoun shae, ou shoul also consie
ain ou entie lanin pae, call-to-action o eail a vaiale. Instea o testin sinle esin
eleents, such as healines an iaes, esin to copletel ieent paes an test the
aainst each othe. No ou’e oin on a hihe level. This tpe o testin iels the iest
ipoveents, so consie statin ith it eoe ou continue ou optiiation ith salle
teas.
#4
measUre as far Down THe
fUnnel as possiBle
Sue, ou A/b test iht have a positive ipact on ou lanin pae convesion ate, ut ho
about your sales numbers? A/B testing can have a signicant effect on your bottom line. You may
even see that a lanin pae that convete ee pospects pouce oe sales. As ou ceate
ou A/b test, consie ho it aects etics such as clicthouh ates, leads, trafc-to-lead
convesion ates, an eo equests.
#5
#5
mEASUrE ASfAr dOwNTHE fUNNELAS POSSIbLE
leaDs
CliCkTHroUgHraTe
DemoreqUesTs
ConVersionraTe
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 8/27
An introduction to A/b testingAn introduction to A/b testing14
www.HUbSPOT.COm www.
Share This Ebook! Share T
seT Up ConTrol & TreaTmenT
In an epeient, ou nee to eep a vesion o the oiinal eleent ou’e testin. when
conuctin A/b tests, set up ou unaltee vesion as ou “contol:” the lanin pae, call-to-
action o eail ou oul noall use. fo thee, uil vaiations, o “teatents:” paes, calls-
to-action o eail ou’ll test aainst ou contol.
fo eaple, i ou ae onein hethe incluin a testionial on a lanin pae oul ae a
ieence, set up your control page with no testimonials. Then create your variation(s).
• Variation A: Control (the unaltered, original version)
• Variation B: Treatment (the optimized version which you expect to perform better)
The sceenshot elo is an eaple o HuSpot’s avance lanin paes A/b testin tool. you
can easil clone the contol an oi it to ceate a teatent pae.
#6 DeCiDe wHaT YoU wanT To Te
As ou optiie ou lanin paes, calls-to-action an eail, thee ae a nue o vacan test. you on’t have to liit ousel to testin onl colo acoun o tet sie. Lo
vaious eleents on ou aetin esouces an thei possile altenatives o esin
laout.
In act, soe o the aeas ou can test iht not e instantl econiale. fo instance
test ieent taet auiences, tiin, alinent eteen an eail an a lanin pae
#7
spliT YoUr sample groUp ranDomlY
In oe to achieve conclusive esults, ou nee to test ith to o oe auiences that
fo instance, in eail A/b testin each o ou eail vaiations ust have as siila a
ecipients as possile. List souces, list tpe, an the lenth o tie a paticula nae h
a list ae all actos that a cause lae ieences in esponse ates. you test esults
conclusive o ou a a the on conclusions i ou o not split ou lists anol
I ou ant to copae the peoance o to o oe lists, eep all othe aspects o th
an tiin ientical so ou et clean esults ase on list an nothin else.
#8
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 9/27
An introduction to A/b testingAn introduction to A/b testing16
www.HUbSPOT.COm www.
Share This Ebook! Share T
TesT aT THe same Time
Timing plays a signicant role in your marketing campaign’s results – be it time of day, day of the
ee, o onth o the ea. I ou ee to un test A uin one onth an test b a onth late,
ou ouln’t no hethe the chane esponse ate as a esult o the ieent teplate o the
ieent onth.
A/b testin equies ou to un the to o oe vaiations at the sae tie. without siultaneous
testin, ou a e let secon-uessin ou esults.
#9
DeCiDe on neCessarY
signifiCanCe Before TesTing
Before you launch your test, think about how signicant your results should be in order for you
to ecie that the chane shoul e ae to ou esite o eail capain. Set the statistical
signicance goal for your winning variation before you start testing. 97-99% statistical signicance
is usuall a oo pecentae to ai o.
#10
CHapTer 2
wHaTVariaBles Yo
sHoUlD TesT
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 10/27
An introduction to A/b testingAn introduction to A/b testing18
www.HUbSPOT.COm www.
Share This Ebook! Share T
As ou uil lanin paes, call-to-action an eail capains,
ou’ve poal onee aout the eleents ou can test
an optiie to incease convesion ate. Shoul ou chane
the acoun colo o the lanin pae? Ho can ou oi
the lanuae on the call-to-action to attact the ost clics?
what i ou eove all iaes o ou eail capain?
Not all vaiales ae ceate equal, an soe a pove oe oth
o ou tie than othes. In act, thee ae soe vaiales ou shouln’t
othe testin at all ecause the on’t ipact clicthouh o convesion
ates all that uch.
you nee to econie the eleents that ou shoul spen ost tie
testin an optiiin. In this section e ill evie oth aeas that ae
citical an soe that ae less citical o lea eneation.
“”
What elements should I testand optimize?
eleents that ou can optiie
on a lanDing page:
CopY Ho shoul ou position ou oe? what essa
ill entice ou eae? Shoul ou a testionia
stenthen the visito’s incentive?
People looin o inoation online ill pa attention to the esciption o ou oe. C
ieent lanin pae cop that can help ou ive oe convesions. fo instance, ul
an ata-iven content have taitionall peoe ell o us, at HuSpot. Stat ith
test in hich ou copae a sho t, one-paaaph lon esciption to lone ut still val
of
You should start your optimization process by nding what types
o oes convet the ost visitos into leas an hich oes
help ou push leas on the sales unnel.
fo instance, at HuSpot e have oun that eoos peo ette than einas in co
visitos to leas, ut einas o ette than eoos in convetin leas to custoes. T
has le us to aintain a alance i o content. moe ile-o-the-unnel oes, suc
consultations an eos, ill ost liel e chaacteie hih custoe close ate.
You can also test different topics and nd out how they compare in driving business resu
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 11/27
An introduction to A/b testingAn introduction to A/b testing20
www.HUbSPOT.COm www.
Share This Ebook! Share T
wHole
pageAs e entione ealie, tain the entie pae as the
vaiale is the astest a to achieve astic esults an
pouce a lanin pae that ives a lot o convesions.
mae iteations to the hole pae that aect iae placeent, o lenth an its cop. Once
you have a statistically signicant result pointing to the variation that performed better, ou can
continue optiiin thouh salle teas.
form
fielDsShoul ou lea captue o onl equest an eail
aess o shoul it as o oe inoation?
Inoun aetes ivee in thei ecision hethe to place content ehin
a o o eep it o-ee o all the ol to see ith no echane o
inoation. Soe aue that os ceate ictions in the lea eneation
process, while others believe that forms are essential for qualifying trafc
an pioitiin o o the sales oaniation.
Lanin pae A/b testin enales ou to evaluate ho ou auience eacts to ieent questions--
what prospects are willing to answer and what information they would rather not share. Form elds
help you qualify leads and nur ture them. (With form-free content, you have to relinquish more lead
nutuin contol -- ou in’t as o thei inoation, so ho can ou et ac in touch ith the
unless the eithe ooa ou site?)
form
fielDsoffer
CopY image
l
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 12/27
An introduction to A/b testingAn introduction to A/b testing22
www.HUbSPOT.COm www.
Share This Ebook! Share T
Accoin to an Eetac III stu, the est placeent
o online ads is in the top an let position. That aes
sense: in the Enlish lanuae, e stat eain o
let to iht an o top to otto.
Hoeve, copite dean riec suests that once ou eaes et use to a paticula
placeent, the iht stat inoin the call-to-action o a. “So even the pie uppe le t aea
on’t o so ell i ou alas put as thee,” riec ites. The ost eective placeent
coul also epen on the contet o the est o the pae, so ae sue ou o soe testin. fo
instance, test the ieal top an let position vesus a placeent in hich the call-to-action is iht
eneath the oe esciption. run an A/b test to see i ou et oe clics an pic a inne.
plaCe-
menT
size
eleents that ou can optiie
in a Call-To-aCTion:
CopY The Eetac III stu also shoe that in a
visitos ea oe tet content than aphic
“People looin o inoation ae looin o tet, not pictues,” ites dean. mae su
wording is clear, specic, and action-oriented. Do some testing to gure out hat tet i
visitos oe liel to clic. I ou nee oe uiance on copitin o calls-to-action,
ou ee eoo, which includes some specic tips and examples.
Color
Thee has een a hue eate evolvin aoun usae
o the colo e o call-to-action esin. Soe aphic
esines aue that e can incease clicthouh
rates signicantly. Others explain that the context of
the e pae shoul eteine the colo.
when unnin A/b tests to eteine the hihest convetin colo, e inul o ou ov
website design. While you want to keep the colors contrasting, make sure all of them t
ou eneal esite colo schee an avoi usin pattens.
grapHiCwhile ou shoul ocus on ou call-to-action tet, on’t
oet that aphics can help conve eanin an
stenthen ou essae. Epeient ith vaious shapes
esies the stana utton-lie o an test ho the
ne loo aects ou clicthouh ate.
Sie is tihtl elate to the contet o the pae an the othe
chaacteistics o ou call-to-action. fo instance, the CTA
ill natuall e lae i it inclues a aphic o an iae that
stenthens the essae. Ceate an A/b test to see i a i call-
to-action that as value to the essae--e.. a customized lo
CTA--attacts oe clics than its contol.
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 13/27
An introduction to A/b testingAn introduction to A/b testing24
www.HUbSPOT.COm www.
Share This Ebook! Share T
sizegrapHiC
CopY Coloroffer
plaCe-
menT
eleents that ou can optiie
in YoUr email: you ust have notice that an o the eleents to test an
optiie e have iscusse so a ovelap acoss ieent
channels. Oes, cop an iae ae cetainl soe o the
vaiales ou ant to alas eep in in. The appl to eail
aetin as ell, so e on’t epeat the hee ut iscuss the
e ne ones e haven’t entione so a. Mwhat’s the est a to la out ou eail co
ive the ost enaeent? Shoul ou
stuctue as a neslette, iest o a eic
A/b testin can help ou eteine the iht oat o ou eail aetin capains
instance, neslettes can peo ell in speain the nes aout a e ieent piec
inoation, such as events, ne oes, iscounts an event pouct announceents. d
sens, on the othe han, can help ou ive the ost convesions to one call-to-action.
A/b test to eteine hich oat iels the esults ou see to achieve.
formaT
la
Siila to testin oats, laout is anothe eleent that ou
can test an optiie o on an onoin asis. Epeient
ith ieent iae an call-to-action placeent an atch
not onl ou clicthouh ates, ut also ou convesions to
eteine hich laout is ost eective.
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 14/27
An introduction to A/b testingAn introduction to A/b testing26
www.HUbSPOT.COm www.
Share This Ebook! Share T
Ou Science o Tiin eseach shos that the ost
clics on eail ae seen on the eeen.
So i ou have een taitionall aai o eailin on Satuas an Sunas, it’s tie to
econsie this pactice an epeient ith ou tiin.
Natuall, the optial tiin o senin aetin eails ill va inust an even copan.
Shoul ou pospect e sent ou net eail one hou late o one a late? Ienti the est tie
to convet pospects into leas unnin split tests.
Timing senDer
A/b eail aetin tests that e an at HuSpot in
2011 shoe that essaes coin o a pesonal
sene nae eceive hihe CTr ate than essaes
o a eneic sene nae.
So instea o senin eails o Marketing Team, e state ain the senes o o
aetin eails e the eal epets o ou tea. To einoce this pesonaliation an
consistenc, e have een ain the peson’s sinatue at the en o the essae.
Pesonaliation is a ell-non est pactice in eail aetin ut ou shoul still test
naes ith ou auience. Ensue that a chane lie this iels positive esults o ou
eoe ou ipleent it.
Eect o da-o-wee on Clicthouh rate
Suna mona Tuesa wenesa Thusa fia Satua
Suject lines ae the pat o eails that can a o
ecipients’ attention ieiatel an convince the
the eail an ea oe. So ou eall ant to et t
Inteestinl enouh, suject lines ae the pat aetes see to e testin the ost a
eail. In maetinShepa’s Eail maetin bencha suve o 2,735 poessionals
respondents said that they run A/B tests to determine a winning subject line. Yet only 3
tests eective.
I ou ecie on testin eail suject lines, t to euct a lesson o ou epeient
length of the subject line, mention of a discount or certain formatting (e.g., brackets or
ae the ieence?
sUBjeCT
lines
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 15/27
An introduction to A/b testingAn introduction to A/b testing28
www.HUbSPOT.COm www.
Share This Ebook! Share T
TsenDer
formaT laYoUT
sUBjeCT
72%
61%
50% 50%
46%
39%
34%32%
30%
26%
15%
35%
25% 26%
34%
21% 22%
32%
41% 42%
29%
22%
0%
10%
20%
30%
40%
50%
60%
70%
Usae Eecti veness
who ae ou senin eail to? Seentation can help ou
get high response rates because your recipients will nd the
essaes paticulal valuale.
That ill also incease the chances o the shain ou eail ith iens an cooes. In
act, maetinShepa’s suve shows that marketers nd testing target audiences to be a pretty
eective appoach.
TargeT
groUp
maetinShepa Eail maetin bencha SuveSuve methoolo: fiele Jul 2011, N=2,735
TargeT
groUp
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 16/27
www.HUbSPOT.COm
An introduction to A/b testingAn introduction to A/b testing30
www.HUbSPOT.COm www.
Share This Ebook! Share T
CHapTer 3
a/B TesTingCase sTUDies
So a e’ve iscusse soe vaiales that ou can test in oe
to optiie the peoance o ieent aetin activities.
Hoeve, A/b testin is tie-consuin an ou nee to e
sat aout the eleents ou ecie to optiie. Soe o the
on’t eseve ou tie.
mINOr TExT CHANgES
I ou’e oin to chane ou healine o suject line, chane it asticall. The
positionin an the content o ou cop atte, ut sitchin “The” to “A” isn’t oin to
ae an sot o ieence.
gENErAL ImPrOVEmENTS
I ou no that soethin is an ipoveent, lie a oile-ienl eail, thee is no
point in testin it to see ho it copaes to an eail that is not optiie o oile
viein. The sitch cetainl on’t hu t ou aetin!
eleents ou shouln’t
BoTHer TesTing:
d
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 17/27
An introduction to A/b testingAn introduction to A/b testing32
www.HUbSPOT.COm www.
Share This Ebook! Share T
“”
Learn from real-life A/Btesting examples below.
This test as pouce o the 2011 Lanin Pae Optiiation Suit hel mECLA
paent copan o maetinShepa & maetinEpeients. HuSpot as pat o a li
pae optiiation test in hich the auience as iven a contol pae an as ase to
teatent pae incopoatin chanes that oul positivel aect convesion ates.
Hee is a sceenshot o the contol pae:
an eaple o
a lanDing page TesT:
ConTrol
No that e have covee soe suestions on
eleents that ou can test an optiie, ou’ poal
lie to see ho these tests iht loo lie. what ae
soe eal-lie eaples o A/b testin in action?
In this section o the eoo e cove thee A/b tests acoss ieent aetin channels. we
eploe hat a split test o a lanin pae, call-to-action an eail loos lie. All o the eaples
that e iscuss hee have een conucte ees o the HuSpot aetin tea an each
test can teach you a lesson on A/B testing. More specically, our three case studies covered here
avise ou to:
OPTImIzE LANdINg PAgES THAT dON’T CONVErT AT A HIgH rATE
STArT yOUr OPTImIzATION PrOCESS wITH AN OffEr TEST
USE THE kNOwLEdgE yOU’VE gAINEd TO ImPrOVE ExISTINg PrOCESSES
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 18/27
An introduction to A/b testingAn introduction to A/b testing34
www.HUbSPOT.COm www.
Share This Ebook! Share T
The sceenshot elo is o a call-to-action A/b test that souht to copae to oe tpiae actuall illustates hat HuSpot’s hoepae use to loo lie in 2010!
Oiinall, HuSpot’s hoepae oee ou counit a seven-a ee tial. Hoeve,
cuious to see i oein a lone tial peio oul entice oe visitos to sin up. wou
signicant enough effect? In this case our control was a variation that offered the seven
tial an the teatent oee a 30-a ee tial.
Ou A/b test esults shoe that the contol convete at a ate o 47.91% an the teatent
convete at a ate o 48.24%, an asolute ieence o 0.3% hich is a pett neliile incease.
The conclusion e aive at is that the auience as hihl otivate the oe itsel. So hen
unnin A/b tests, stat ith paes that have a lo visito-to-lea convesion ate. It’s pett ha to
beat a page that converts 47% of the trafc.
The entie pae as taen as a vaiale an as evape. we chane the iae an its
placement and shortened the copy and the form. In an effort to drive a lot of trafc to this page, we
i a heav eail aetin push, lo posts an social eia pootion. The test as conucte
usin HuSpot’s lanin pae tools.
TreaTmenT
an eaple o
a Call-To-aCTion TesT:
Co
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 19/27
An introduction to A/b testingAn introduction to A/b testing36
www.HUbSPOT.COm www.
Share This Ebook! Share T
The takeaway emerging from this A/B test is that type of offer can exercise a tremendous inuence
over lead generation efforts. If you want to optimize your calls-to-action (and for that matter email
an lanin paes), copain ieent oes is a eat place to stat. Such epeients ill
povie ou ith a ette unestanin o hat popts ou visitos to convet into leas.
Results from the test showed that the 30-day free trial enticed more visitors and had a signicant
effect on conversion rates. The 30-Day free trial won with a 99.9% condence rate and ceate a
110% incease in HuSpot ee tials. The contol ha a 0.326% visito-to-ee-tial convesion ate,
hile the teatent ha a 0.709% visito-to-tial convesion ate.
TreaTmenT
“”
Type of offer can exercise a
tremendous infuence over lead generation efforts.
an eaple o
an email TesT:
Suject lines ae a citical eleent o eail aetin. The have the poe to a the
o ecipients an ipact clicthouh ates to a eat etent. That is h e alas an
sue that e ae usin the iht suject line hen e ae eailin ou suscies. we
conuct A/b tests to evaluate innin suject lines.
but esies a suject line, ecipients also see a sene nae in thei ino. who is the
coin o? This sene nae can ae a ieence on open an clicthouh ates
2011 e conucte a test to copae a eneic “HuSpot” sene nae to a pesonal
soeone on the aetin tea.
6.57%
0.73%
7.10%
0
0.00%
1.00%
2.00%
3.00%
4.00%
5.00%
6.00%
7.00%
Open rate Clic Thou h
fo "HuSpot" fo "maie geoieva, HuSpot"
CONTrOL - fo “HuSpot” sene n
TrEATmENT - fo “maie geoiev
HuSpot” sene nae
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 20/27
www.HUbSPOT.COm
An introduction to A/b testingAn introduction to A/b testing38
www.HUbSPOT.COm www.
Share This Ebook! Share T
CHapTer 4
rUnning &eValUaTing
YoUr TesTs
Our control had a 0.73% CTR and the treatment had a 0.96% CTR. With a condence of 99.9%
e ha a clea inne. Ou conclusion ate this A/b test as that eail sent a eal peson
ae oe liel to e clice on than eail sent o the copan. So ho oes CTr ipact ou
nue o leas eneate?
The teatent ot 292 oe clics than the contol. Since ou aveae visito-to-lea convesion
ate on lanin paes is 45%, that eans the teatent ot us 131 oe leas. That is eat
itsel ut thin aout ho valuale this lesson is hen applie to ou oae eail aetin
poa. we too this insiht an use it o the evapin o ou lea nu tuin capains,
ain a pesonal sene nae an sinatue to each o ou essaes. when ou A/b tests lea
to some signicant results, use the knowledge you’ve gained to improve existing processes.
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 21/27
An introduction to A/b testingAn introduction to A/b testing40
www.HUbSPOT.COm www.
Share This Ebook! Share T
“ ”Let’s look into how you can conductactual split tests.
No that e have covee the easons h ou shoul un A/b tests, hat vaiales to optiie
an ho these tests iht loo lie, let’s loo into the etails aoun thei actual ipleentation.
Ho o ou ceate ultiple
vesions o the sae
eleent ou ae testin?
How do you split trafc
anol? Ho o ou
no i ou esults ae
statistically signicant?
In this section e ill loo ehin the scenes o a e tools that ill enale ou to o A/b testin
an autoate the pocess so that ou on’t have to aintain a oen ecel speasheets. what’s
oe ipotant, e ill point ou to the actions ou can tae ate unnin a successul test.
ho to o conuct
a lanDing page TesT:
with lanin pae A/b testin ou have one UrL an to o oe vesions o the pae.
send trafc to that URL, visitors will be randomly sent to one of your variations. Standar
pae A/b testin tools eee hich pae the eae lane on an ill eep shoi
pae to the use. fo statistical valiit split tests nee to set a cooie on each visito to
the visito sees the sae vaiation each tie the o to the teste pae. This is ho Hu
avance lanin paes an goole’s wesite Optiie o.
HuSpot’s avance lanin paes enale ou to ceate A/b tests an tac a nue
to evaluate ho ou epeient is peoin. It eeps a eco o the nue o peop
viee each vaiation an nue o the people ho too the intene action. fo ea
it iht ino ou that each o ou lanin pae vaiations as viee 180 ties, it
top-performing one generating 20 clicks and the lowest performing one generating ve
while this inoation is ipotant, it’s not enouh to ae a ecision aout hethe o
results were signicant.
F F FA b
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 22/27
An introduction to A/b testingAn introduction to A/b testing42
www.HUbSPOT.COm www.
Share This Ebook! Share T
fo a hih-level vie, ou can poal tell i ou test
results are signicant or not. If the difference between
the tests is ve sall, it a e that the vaiale
you tested just doesn’t inuence the behavior of your
viees.
However, it’s important to test your results for statistical signicance from a mathematical
pespective. Thee ae a nue o online calculatos ou can use o this, incluin:
SPLIT TEST CALCULATOr
USEr EffECT’S SPLIT TEST CALCULATOr ANd dECISION TOOL
VISUAL wEbSITE OPTImIzEr
HuSpot’s avance lanin paes will tell you when the A/B test becomes statistically signicant
an hethe ou shoul continue the test o stop it. Once ou’e one ith the test, sipl isale
the vaiations that ae losin an un ith the innin pae.
DeTermine sTaTisTiCal signifiCanCe The scenshot elo is an eaple o a lanin pae test that e an at HuSpot usin
avance tool. In this eaple e ante to see i incluin testionials o one o ou
positivel aect convesions. fo the ata elo e ae ale to see ho an visitos
these vaiations ot an hat the coesponin nue o o suissions is.YThe tool tells us that the best choice is Variation D: it converted at 62.2% (1% better tha
E). However, we are also informed that this result is not statistically signicant and are e
to continue this test. This le us to the conclusion that the ieence eteen ou tests
sall to aect the ehavio o viees.
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 23/27
An introduction to A/b testingAn introduction to A/b testing44
www.HUbSPOT.COm www.
Share This Ebook! Share T
You may conduct an A/B test but not nd statistically signicant results. This doesn’t meou A/b test has aile. fiue out a ne iteation on ou net test. fo eaple, consi
the sae vaiale aain ith ieent vaiations, an see i that aes a ieence. I n
vaiale a have little eain on ou convesion ates, ut thee a e anothe ele
ou pae that ou can ajust to incease leas.
Eective A/b testin ocuses
upon continuous ipoveent.
reee that as lon as
ou eep oin toas
ipoveent, ou’e oin in
the iht iection.
if aT firsT YoU Don’T sUCCeeD, TrY againTesTing is aBoUT measUring oVer Time
you a e tepte to chec ou capain eveday to try to gure out the results of your A/B test.
This is oa, ut on’t a an conclusions o
this inoation just et. The eason ou can’t ae
ecisions et is that ou ouln’t e ull easuin
ou A/b test, onl onitoin it.
That’s lie eciin ou shoul ecoe a po aseall plae ase on a Little Leaue ae
where you hit a home run. Measurement looks at trends over time – wait a signicant amount of
tie eoe analin ou esults.
So how long should you wait? First, make sure you have sent enough trafc to the pages. (That
is oten a touh tas o sall usinesses.) fo a lanin pae A/b test, it’s a oo iea to ait
aout 15-30 as eoe evaluatin ou esults. This ill allo ou to see tens as ell as an
issues that a have poppe up. It shoul also ive ou enouh ata to coe up ith an accuate
conclusion.
If even after you’ve waited for 15-30 days and sent a lot of trafc to your test you haven’t seen a
statistically signicant result, then your test probably wouldn’t make a big impact on conversions.
don’t e aai to ove on to anothe, oe aical epeient.
tx“
Effective A/B testing focusesupon continuous improvement.
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 24/27
An introduction to A/b testingAn introduction to A/b testing46
www.HUbSPOT.COm www.
Share This Ebook! Share T
ho to o conuct
a Call-To-aCTion TesT:
Call-to-action split testin os pett uch the sae a as lanin pae split testin. you
ceate a e vaiations o ou CTA, place the on the sae pae an the shoul e isplae to
visitos anol. The oal hee is to eteine hich call-to-action attacts the ost clics.
Hoeve, e entione ealie in this eoo that it is ipotant to loo o esults uthe on
in the sales unnel. So it ill e ost useul to no the nue o convesions each o ou CTA
versions drove. This result is inuenced by the landing page and how well it is aligned with the
call-to-action. In the eaple elo e use HuSpot’s Call-to-Action oule to test the alinent
eteen the sae call-to-action an to ieent lanin paes:
HuSpot’s call-to-action oule (screenshot showed on the previous page) enables you
uil A/b tests an ill on to the ata that attes the ost to ou oaniation. fo
ou iht loo at the vies-to-clic ate in an eot to optiie the call-to-action, ut i
to-suission ate is supisinl lo, then the pole iht lie ith the lanin pae.
h, ultiatel, ou ant to e eepin an ee on ou vie-to-suission ate an t t
that.
reee that ou shoul e unnin onl one
A/b test at a tie, so on’t t to optiie oth the
call-to-action an the lanin pae siultaneousl.
mae chanes to one vaiale at a tie so that ou
unestan hich eleent tiee the esults ou
ae seein.
!
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 25/27
An introduction to A/b testingAn introduction to A/b testing48
www.HUbSPOT.COm www.
Share This Ebook! Share T
ho to o conuct
an email TesT:72%
61%
50% 50%46%
39%
34%32%
30%
35%
25% 26%
34%
21% 22%
32%
41% 42%
0%
10%
20%
30%
40%
50%
60%
70%
Usae Eecti veness
maetinShepa Eail maetin bencha SuSuve methoolo: fiele Jul 2011, N=2,735
most eail povies autoate the split testin
pocess an enale ou to copae ieent eleents
aout ou eail. The anoie the list o ecipients
into two or more groups (you need to ensure the
oups ae i enouh to ive ou a statisticall
signicant result) and associate each email variation
to each o the oups.
mailChip, for instance, splits your email campaign to help you nd out the best subject line and
tie o a to sen an eail. The eat thin aout a tool lie this is that it can sen the inne to
the eaine o ou oup. This is a antastic a to optiie ou list an elive the essae
that attacts the ost attention.
mailChip an ost stana eail povies enale ou to pic a inne ase on eithe open
ate o clicthouh ate. Hoeve, ou also ant to see hich eail is inin in the ost
convesions. In othe os, ienti hich vaiation, coine ith the iht lanin pae,
elives the est esults. fo this tpe o epotin ou nee to inteate ou eail aetin to
aetin analtics.
I e evisit the 2011 suve o maetinShepa, e ill see that the vaiales hich
the est esults hen optiie ae lanin pae an taet auience. So even hen it
to ipovin ou eail aetin poa, stat testin lanin paes an put ep
seentation. Such tests ill suest optiiation stateies that can help ou achieve
eectiveness out o ou aetin eails.
M$
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 26/27
www.HUbSPOT.COm
An introduction to A/b testingAn introduction to A/b testing50
www.HUbSPOT.COm www.
Share This Ebook! Share T
ConClUsion& aDDiTional
resoUrCes
A/b testin ives ou the oppotunit to
aiie convesion ates, solve poles, an
challene ou assuptions. moe ipotantl,
A/b testin ill signicantly incease the
nolee ase o ou copan alloin
ou to eteine a set o optiie pactices to
oost ou oveall rOI.
In this eoo e covee the ounations o A/b testin, alin ou thouh pocesses
ipove the peoance o ou lanin paes, calls-to-action an eail capains. y
no ho to conuct split testin an hat vaiales to isolate. you also no hee to
hat to spen the least tie on. As ou ein split testin, eep in in that all these o
eots ae iectl tie to incease lea eneation an custoe acquisition.
“”
These optimization efforts are diretied to increased lead generation customer acquisition.
0rOI
8/2/2019 Introduction to Ab Testing for Marketing Optimization
http://slidepdf.com/reader/full/introduction-to-ab-testing-for-marketing-optimization 27/27
www.HUbSPOT.COm
An introduction to A/b testingAn introduction to A/b testing52
www.HUbSPOT.COm www.
Share This Ebook! Share T
aDDiTional resoUrCes
Chec out soe aitional HuSpot esouces on A/b
testin i ou ae inteeste in leanin oe:
T
HuSpot’s Avance Lanin Paes
HuSpot’s Call-to-Action moule
B pt
wh maetes A/b Testin Shouln’t be Liite To Sall Chanes
9 A/b Tests mee maetin motals Can Use to Incease Leas
5 best Pactices o wesite A/b Testin
10 do’s an don’ts o A/b Testin
How to Determine If Your A/B Test Results Are Signicant
eb d wb
Top 7 Inspiational Eail Testswhat’s woth A/b Testin?
rUn powerfUl
a/B TesTsSin up o HuSpot’s 30-a tial an stat unnin eas-tuse lanin pae an call-to-action A/b tests.
.Hubst.c/f
![AB Testing [INFOGRAPHIC]](https://img.dokumen.tips/doc/110x75/5400aeb18d7f7289408b492f/ab-testing-infographic.jpg)
