AB Testing Blue Paper

4imprint.com

A/B Test ing

© 2014 4imprint, Inc. All rights reserved

Learn to read: the ABCs of A/B test ing

Remember when you learned your ABCs? It seemed complicated at first, but as

you got the hang of it you realized the letters held the secret to the English

language. A/B testing, which is a method to compare performance of two

variables at the same time, can have the same effect and transform the way you

build a website or enhance online marketing. If you do it correctly, it takes the

guesswork out of website improvements and enables data-driven decisions that

shift conversations from “we think” to “we know.” For many companies, A/B

testing is attractive because it can improve sales, revenue and lead generation.

In fact, research shows that companies that use A/B testing can generate up to

30-40 percent more leads for business-to-business (B2B) sites, and 20-25 percent

more leads for ecommerce sites.1

A/B testing is exactly what it sounds like: You have two versions of an element (A

and B) to evaluate user behavior, and subject both versions to experimentation

simultaneously to improve website optimization. With A/B testing, you can

confidently conclude that if version B is preferred over version A, then version B is

the design to implement. Do it correctly and it’s almost as simple as the ABCs.

It’s important to note that A/B testing is not a one-and-done activity but an

ongoing process. It should occur on a regular basis and become ingrained into the

fabric of your organization with defined processes. Companies should test early

and often to enable continuous improvement.

Yet wide adoption still isn’t the case with A/B testing. Why isn’t it if it’s so

effective? In many cases, organizations either don’t know about it or don’t know

how to do it. Some marketers only have a basic understanding of A/B testing

and haven’t quite figured out how to take it to the next level. Yet it’s not as

complicated as it sounds and almost any company can master the art and science

behind it.

This Blue Paper® will teach you the ABCs of A/B testing, and uncover some of the

benefits it can yield for your organization. It will also walk you through the steps

you need to implement successful testing. But like anything, A/B testing has its

challenges and limitations. This Blue Paper® identifies the limits of A/B testing and

provides tips on how to address any obstacles. So grab a notebook and get ready

to go back to school; class is in session.

1 Soskey, Ginny. “11 Common A/B Testing Myths BUSTED.” Hubspot. N.p., 12 Mar. 2013. Web. 29 Apr. 2014. <http://blog.hubspot.com/blog/tabid/6307/bid/34254/11-Common-A-B-Testing-Myths-BUSTED.aspx>.


Start with the letter A: The bas ics A/B test ing

A/B testing is also referred to as randomized controlled experiments, online

controlled experiments and split testing. It is considered to be a subset of

the more complicated multivariate testing and is commonly used for Web

development, landing page optimization and online marketing. A/B testing is

particularly effective on ecommerce websites, news and media websites.

So why would anyone want to bother with testing anyway? Think of it this way—

every business website has an objective once a user lands on a page. Whether you

want users to purchase products or sign up for subscriptions, the goal is to convert

visitor behaviors that support the objectives. Perhaps you want to increase leads

or attract new customers—what you want users to do varies depending on your

business model. But the rate at which a website is able to get users to behave as

intended is called a “conversion rate.” A/B testing measures what variations (A or

B) are most likely to convert visitors to goal achievers.

As mentioned, either A or B variants are presented unknowingly to selected

visitors to determine which option has the highest performance rates. It enables

companies to see if changes will increase conversion rates through comparison of

A and B. Almost anything online that a user interacts with can be tested. Perhaps

you want to see if changes to headlines make a difference or whether or not

certain links and images lead to higher conversion rates. You might even conduct

advanced tests related to pricing structures, sales promotions, free trial lengths

and more. Figure 1. below visually outlines the concept of A/B testing in the

simplest form.

Figure 1. A/B testing in action2

2 Chopra, Paras. “Mulitvariant Testing in Action.” Smashing Magazine. N.p., 24 Nov. 2010. Web. 05 May 2014.


To get your creative juices flowing, some of the elements that are well suited for

A/B testing include (but are not limited to):

•headline or product descriptions;

•form length and types of fields;

•layout and style of websites;

•product pricing and promotional offers;

•images on landing and product pages, and

•amount of text on pages (short vs. long).

In reality, you can run an unlimited amount of tests and even run multiple

tests at the same time. The list above is just a sample of the more commonly

tested elements. In addition to landing pages and webpages, many marketers

conduct A/B tests on emails and pay per click (PPC) campaigns. Email testing

variables might include the subject line, personalization features, sender name,

and others. If you are testing PPC campaigns, you can A/B test the headline,

body text, link text or keywords. A/B testing is also frequently used with

call-to-action (CTA) options. You might evaluate the impact of altering the text

on a CTA or its shape, color or placement. The options are limitless—to gain a

better understanding of what you might test, HubSpot® published a list of 28

different A/B tests to consider.

Again, what to test depends on the conversion goals. For example, if your goal is

to increase the number of sign-ups for product promotion, then you might test

the length of the sign-up form or the types of fields in the form. Basically, if the

site doesn’t do a good job of convincing users to sign up you’ll want to explore

what might be changed to improve conversions. Will a shorter form lead to more

conversions? If you address privacy concerns will it lead to greater conversion

rates? There are unlimited opportunities to test—if you do it correctly, it can be

an easy and effective way to take the guesswork out of user preferences and help

identify what works best.

B is for benef i t : The benef i ts of A/B test ing

If you’re not sure why you should bother with A/B testing in the first place,

consider some of the benefits. Perhaps the most compelling reason to use A/B

testing is that it works—leading to improvements in sales, conversation rates and

revenue. As noted, companies that use A/B testing generate up to 30-40 percent

more leads with B2B sites, and 20-25 percent more leads for ecommerce sites.3

3 Soskey, Ginny. “11 Common A/B Testing Myths BUSTED.” Hubspot. N.p., 12 Mar. 2013. Web. 29 Apr. 2014. <http://blog.hubspot.com/blog/tabid/6307/bid/34254/11-Common-A-B-Testing-Myths-BUSTED.aspx>.


If that doesn’t grab your attention, there are plenty of other reasons to explore

A/B testing. Among other things, A/B testing is beneficial because it is:

•affordable,

•easy to maintain,

•able to measure small performance differences, and

•valuable for fact-based, decision-making with quantifiable results.4

First, A/B testing can be an affordable and relatively easy way to measure what

works best. There’s no need for expensive usability specialists to monitor user

behavior or analyze complicated interaction design questions. And after the

upfront investment, there’s relatively low cost to conduct another test.

After you identified what to test, it’s only a matter of execution. There are a

number of free or low cost resources available at your fingertips. For example,

Google® Analytics Content Experiments is a free tool that can help you develop

an A/B test. And that’s just the tip of the iceberg, as indicated later in the paper,

there’s an abundance of similar tools to implement A/B testing.

Another advantage is that A/B testing can be easy to maintain in the long-term

and replicate. Organizations can quickly move from one test to the next without

much effort, and the cost and resources for maintenance are low.

Also, A/B testing can measure the smallest of changes that may have a big impact

on conversion rates and the bottom line. Although it may sound implausible,

small variations, like the color or size of a button, can improve results significantly.

For example, what if you wanted to see if small modifications to the button users

click to make a purchase will improve conversion rates? You would administer two

test versions with different sizes and wait until a significant population tested

each variation. You might discover that the site with the bigger button had a

higher conversion rate that led to a one percent increase in sales. Although a one

percent increase doesn’t sound like much, the truth is that it can have a huge

impact to sales and profits. If you are a big company, like Amazon® a one percent

sales increase would amount to roughly $69 billion annually. Even for companies

with smaller sales, like eBags.com which generates $40 million annually, a one

percent translates into an extra $400,000.5

Still not convinced that small changes can have a measureable impact? Perry

Marshall, marketing expert and author of The Ultimate Guide to Google AdWords

4 Nielsen, Jakob. “Putting A/B Testing in Its Place.” Neilsen Norman Group. N.p., n.d. Web. 30 Apr. 2014. <http://www.nngroup.com/articles/putting-ab-testing-in-its-place/>.

5 Nielsen, Jakob. “Measuring 1% Increase in Sales Through A/B Testing.” Nielsen Norman Group, n.d. Web. 30 Apr. 2014. <http://www.nngroup.com/articles/measuring-sales-increase-a-b-testing/>.


recalled an A/B test that evaluated the click-through rate (CTR) of two ads. The

only difference between sites was the inclusion of a single comma. Despite this

seemingly irrelevant detail, the variant that featured the comma had a CTR of

4.40 percent which represented an improvement of .28 percentage points over

the control.6

A/B testing also helps resolve key business decisions with quantifiable results—it

replaces gut instinct to make key decisions. In other words, because it measures

the actual behavior of customers under real-world conditions companies can

eliminate the guesswork when determining what and how to improve. An

e-commerce site, for example, might debate the use of coupons on a website and

whether they are effective to increase sales. Rather than relying on marketing

or other business units to make an educated guess, A/B testing can uncover how

users behave with or without coupons and special offers. In some cases, you

might find that coupons actually lower sales and conversion rates. In fact, some

companies that tried A/B testing with and without coupons reported that overall

sales typically increased by 20-50 percent when they did not use coupons.7 Every

company is different, but with A/B testing the numbers don’t lie, so you’ll know

what’s working with quantifiable metrics.

Why the ABCs aren’t as easy as 123

So if A/B testing is so beneficial, why aren’t all companies doing it? For some

companies, A/B testing seems too complex and time consuming, even though it

doesn’t have to be. Not everyone gives A/B testing high marks because of some of

its limitations that include:

•it can only be used for projects that have one, clear goal;

•it focuses on short-term goals and improvements; and

•it does not capture behavioral insights.

First, A/B testing can only be used for projects that have one goal and a single key

performance indicator (KPI). This goal must be measurable by counting simple

actions. Examples of measurable actions include:

•sales for an e-commerce site;

•users subscribing to an email newsletter;

•users opening an online banking account; and

• users downloading a white paper, asking for a salesperson to call, or

otherwise explicitly moving ahead in the sales pipeline.

6 Shewan, Dan. “A/B Testing Best Practices Can Save You Time, Money and Effort.” CTR - Google Search. N.p., 26 Feb. 2014. Web. 28 Apr. 2014. <https://www.google.com/search?q=CTR&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:official&client=firefox-a&channel=sb>.

7 Nielsen, Jakob. “Putting A/B Testing in Its Place.” Neilsen Norman Group. N.p., n.d. Web. 30 Apr. 2014. <http://www.nngroup.com/articles/putting-ab-testing-in-its-place/>.


Yet it is rare that the actions above are the only goal of a site. If you are in the

e-commerce industry, for example, the amount of revenue collected through sales

is critical to capture. Yet sites that don’t close sales online can’t really say that a

single desired user action is the most important goal. This limits the ability of A/B

testing to truly measure overall impact.

Also with B2B sites, the objective is to make sure that users leave the site feeling

better about your product or reputation. But these are immeasurable goals that

can’t be captured by simple user actions. In other words, goals like improving

brand reputation or supporting the company’s public relations efforts aren’t

measured by whether or not users click a specific button. Similarly, while you

can easily measure how many users sign up for your email newsletter, you can’t

assess the equally important issue of the value they attach to your content and

whether it has a positive impact on the customer.

Another weakness of A/B testing is that it focuses on short-term goals, which

is not always optimal. What if you need to determine if you should add a

promotion to your homepage or product pages? A/B testing won’t necessarily

help with that decision or give direction on long-term planning. Since most

companies can’t afford to wait years before making a decision, A/B testing is a

quick and easy way to get immediate feedback. Long-term planning, however,

often falls to the wayside and that can have future implications.

Perhaps the biggest disadvantage to A/B testing is that it doesn’t provide

behavioral insights. You don’t know why certain options are selected because you

are not observing users or privy to their thought process. All that is revealed is

that statistically more people performed a certain action with design A or B.8 You

have no idea why a variant was chosen, yet this information would be helpful to

make improvements to other aspects of your site.

Sing the ABC song: Implementing A/B test ing

Now that you might be sold on the idea of A/B testing, where should you start?

It’s like the first time you sing the ABC song, you memorize the letters and start

at the beginning. At a minimum, before you even break out testing tools, you

need to:

•develop concrete goals for testing;

•develop hypotheses on potential outcomes; and

•define what to test.

8 “Putting A/B Testing in Its Place.” Nielsen Norman Group. N.p., n.d. Web. 05 May 2014. <http://www.nngroup.com/articles/putting-ab-testing-in-its-place/>.


The first step is to define the goals of A/B testing. Keep in mind that A/B testing

most commonly fails because the test itself has unclear goals, so you’ve got

to know what you’re testing and why. Companies run into trouble with A/B

testing when theories are too vague or test two entirely different designs with

multiple variants. In addition, a lack of clear direction may lead to additional

and potentially unnecessary testing. It can also lead to missed opportunities

that might have been identified had the test been performed with a specific

objective in mind.

Even before you get started, clearly identify what you are trying to achieve. Are

you trying to increase sales or memberships? What are the conversion rates you

want to achieve and what changes might improve them? You might use A/B

testing to test a theory—perhaps adding a picture to a landing page will increase

conversions. Do you know if people are more likely to click a red button or a blue

button? What if the headlines are changed to stress the time limit of the offer?

These are all changes that can be easily quantified and measured with A/B testing.

You should also develop a hypothesis regarding test outcomes. When you

learned the ABCs, for example, even if it wasn’t explicitly articulated, there was a

hypothesis: Learning the alphabet will help you learn to read. Similarly, with A/B

testing, a hypothesis is imperative so you know the objective and expectations.

Think about what you expect to see or what you think will happen from the test.

Do you think a red versus green button will lead to more conversions? What is

the probability that a call-to-action button will be more effective if it is placed

differently? Formulating a hypothesis doesn’t have to be complicated, but it will

support the goals of the test and help identify what will be tested.

Lastly, but most importantly, it’s time to determine exactly what to test, keeping

goals and objectives in mind. What elements do you think will have the greatest

impact if changed? Once identified, it’s time for the fun part: Create and

administer an A/B test, keeping the hypotheses and goals in mind. This will

require the selection of the right tool to administer the test and capture results;

some of these tools are highlighted in subsequent sections.

If you need help getting started with the basics, the Visual Website Optimizer®

outlines a similar process that encourages users to follow a scientific method

when adopting A/B testing. Figure 2. visually depicts the scientific method

that should be used to develop A/B testing, highlighting the steps required

to ensure success.


Figure 2. The scientific method and process for A/B testing.9

This method (Figure 2.) also includes a series of steps to follow for A/B testing.

You can access a step-by-step process on the Visual Website Optimizer. Briefly,

there seven steps to the scientific method.10 These steps include:

1. Start by asking a question. Start with a question. For example, will the

location of the search button make a difference to sales or bounce rates?

2. Collect background research. Do the homework to fully understand visitors’

behaviors. You can use Google Analytics or other analytic tools available to

understand how customers behave on your site.

3. Develop a hypothesis. A potential hypothesis, for example, might be that

when the search button is on the left customers stay on the site longer.

4. Outline the number of visitors or days required to achieve meaningful

results. Set benchmarks to determine the testing duration or number of

visitors needed to obtain statistically significant results. You might want to

use the A/B Test Duration Calculator provided on Visual Website Optimizer.

5. Test the hypothesis. Create an A/B test that will capture variations between

versions A and B. Again, if you are evaluating the location of the search

button, versions A and B will be located in two different spots to test the

hypothesis.

6. Analyze data and draw conclusions. After you collect the results, analyze

9 “12 Game-Changing A/B Testing Tips for 2014.” Visual Website Optimizer Blog. N.p., 7 Jan. 2014. Web. 15 May 2014. <http://visualwebsiteoptimizer.com/split-testing-blog/ab-testing-tips-for-2014/>.

10 The Complete Guide to A/B Testing.” A/B Testing: The Complete Guide N.p., n.d. Web. 15 May 2014. <https://visualwebsiteoptimizer.com/ab-testing/>.


the option that achieved the desired affect and draw conclusions on what

makes a difference. For example, if the search button placement improves

bounce rates, you might want to consider whether or not relocating other

features might have an impact.

7. Report results. Data is only valuable if it is shared. Inform other functions

like marketing and information technology of the results to assist with

future planning.

Need a tutor? Tools to implement A/B test ing

A/B testing requires some type of technology to implement testing elements and

analyze data. Basically, the more sophisticated the testing the more sophisticated

the tool that is required. If you are intimidated with this aspect of A/B testing,

fear not, there are plenty of tutors that can do the work for you or teach you

how to implement it. Some tools are available at low or no cost, and can walk you

through the steps to develop and implement an A/B test. If you want to see an

A/B test in action, visit WhichTestWon. Once on the site, it’s a good idea to take

an actual test and see how it works to gain a better understanding of the process.

There are many tools that help with A/B testing—there are probably enough

to fill a classroom. Some of the free basic tools include the Google® Analytics

Content Experiments and Optimizely. If you want an alternative tool with

extra features consider the Visual Website Optimizer or Unbounce. If you are a

small business, you might want to reference the article on 10 A/B Testing Tools

for Small Businesses published by PracticalECommerce to explore some of the

technology options.

Whatever tool that is selected will need to make sense of the numbers and draw

comparisons between A and B results. Most A/B testing tools have statistical

analysis capabilities built into the programs but there are also a number of free

calculators and tools that can be used to draw comparisons. Two of these tools

are the usereffect.com and HubSpot’s free A/B testing calculator. For even more

resources to help compute statistical significance visit the Occam’s Razor blog by

Avinash Kaushik. Kaushik walks you through the basic elements you’ll want to

evaluate in order to draw conclusions on the impact to conversion rates.

Of course, you can fully outsource A/B testing to another organization. This is a

viable option if your organization has limited internal resources or is new to the

concept. Some find this option produces faster results, since the agency provides

all necessary testing roles and expertise. It might be more expensive to outsource

initially, but some companies find the payback justifies the investment.


However, aside from cost, there are a few downsides to consider if you outsource

A/B testing. For one, you might have less control over testing if the outsourced

partner takes the lead in determining what to test. In addition, a partner might

not fully understand the complexities of your business and goals as well as an

in-house team might, and the partner might lack detailed industry experience.

On the flip side, outsourcing A/B testing might be a great way to learn about

testing tools and best practices which can later be transferred to an in-house

testing team.

Conducting testing in-house is another option. Some companies choose to

create an internal team to handle all aspects of A/B testing. This method is

often selected because it has lower initial startup costs compared to a testing

agency. Some companies also prefer having more control over the testing

process—an internal team chooses what and when to test. This is appealing

to some because internal employees might have more knowledge of business

functions and goals, and in turn, develop more meaningful tests. According

to some data, companies that are most successful with A/B testing eventually

develop and administer A/B testing in-house.11

That’s not to say you won’t encounter some challenges if you conduct A/B testing

internally. It can take a long time to build a team and develop the expertise

required to initiate A/B testing. Companies must find, hire and train resources.

Plus you’ll need to fulfill a variety of roles depending on the scope of your testing,

including a testing manager, testing lead, test developers and test designers. It

usually takes longer with an in-house team to start uncovering results, since the

learning curve is substantial.

Perhaps you might consider doing a little of both—outsourcing some of the

testing while building internal skills simultaneously. WhichTestWon is an excellent

resource for this option. If desired, it will fly an expert instructor to your location

for your choice of a full- or half–day of testing and conversion optimization

training. The trainer will help companies determine what to test and how to

implement new tests. It will also help develop an action plan for your website,

landing pages or email campaigns, and help your team leverage best practices.

Frequent ly asked quest ions of A/B test ing

How long should you conduct A/B testing? The amount of time required varies

significantly depending on the amount of website traffic and overall conversion

11 “ Should You Outsource Your A/B Testing or Build an In-house Team?” Visual Website Optimizer Blog. N.p., 6 Mar. 2013. Web. 05 May 2014. <http://visualwebsiteoptimizer.com/split-testing-blog/outsourcing-your-conversion-rate-optimization/>.


rates. If you select the right tool, it can tell you when you have enough data to

draw reliable conclusions. You might also establish internal testing benchmarks.

For instance, you might decide to end the test when there is a 60 percent

confidence preference for A or B or if there is a clear preference early in the test.

If you have questions regarding testing length, check out a duration calculator

provided by the Visual Website Optimizer®. Keep in mind that ideally you should

only run the experiment as long as necessary to achieve a high confidence that

the test is accurate.

What should you do after a test is completed? The first thing to do is remove

all testing elements and update the site with preferred variations as quickly as

possible. After that, you should move on to the next test. Remember, A/B testing

isn’t a one-and-done process. In reality, A/B testing should be continuous by

exploring incremental changes that might impact performance.

What are the potential outcomes of an A/B test? Overall, an A/B test can have

four outcomes: no result, a negative result, a positive result or an inconclusive

result.12 You might find there is no discernable difference between variants and

that users chose A or B an equal amount of times. You could also have a positive

or negative result from the test and find that one variation is either harmful or

helpful to your site. An inconclusive result is one that produces inconsistent data;

it can also mean the test fails to present any result whatsoever.

After you’ve mastered A/B testing you might want to move on to multivariate

testing (MVT). While they ultimately share the same goal—to make sure a website

is achieving the desired effect and supporting business objectives—MVT lets you

test multiple elements at the same time and can be highly effective for complete

redesigns. It will also evaluate how elements work together on a page. However,

MVT is far more complex and requires greater knowledge for its execution. So,

ensuring you’ve mastered A/B testing first is a good idea.

Master ing your ABCs: A/B test ing success stor ies

How, exactly, have companies used A/B testing to make improvements? There

are plenty of textbook examples that prove the value of testing. The company

37Signals®, for example, found that a subtle headline change generated 30

percent more sign-ups than its original headline. Changing the headline to “30-

Day Free Trial on All Accounts” from “Start a Highrise Account” led to more

conversions overall.13

12 Chopra, Paras. “The Ultimate Guide to A/B Testing.” Smashing Magazine. N.p., 24 June 2010. Web. 30 Apr. 2014. <http://www.smashingmagazine.com/2010/06/24/the-ultimate-guide-to-a-b-testing/>.

13 Chopra, Paras. “The Ultimate Guide to A/B Testing.” Smashing Magazine. N.p., 24 June 2010. Web. 30 Apr. 2014. <http://www.smashingmagazine.com/2010/06/24/the-ultimate-guide-to-a-b-testing/>.


4imprint serves more than 100,000 businesses with innovative promotional items throughout the United States,

Canada, United Kingdom and Ireland. Its product offerings include giveaways, business gifts, personalized gifts,

embroidered apparel, promotional pens, travel mugs, tote bags, water bottles, Post-it Notes, custom calendars,

and many other promotional items. For additional information, log on to www.4imprint.com.

How would you like to increase leads by 201 percent? That’s exactly what

happened when Majesticwines® revamped its category page design in order to

increase online inquiries for wedding services.14 In another example, Daily Burn®,

a diet and exercise tracking site, used A/B testing to improve conversion rates

by 20 percent.15 Using the Google Website Optimizer, the company tested a

simple variation that gave visitors fewer options to choose from and found that

simplifying the site increased conversion rates significantly.

Still not a believer? In yet another example, the company CareLogger realized

a 34 percent conversion rate increase simply by changing the color of a CTA

button from green to red.16 The company continued to test different aspects

of headlines and texts, eventually realizing a whopping 72 percent increase in

conversion rates overall. There are many examples that demonstrate the impact

of A/B testing—these and other case studies can be accessed on the Omnicore®

website. WhichTestWon also has 453 case studies to reference.

Do you know the ABCs of A/B Test ing?

By now you might be ready to receive a completion certificate for the ABCs of A/B

testing. After all, it can be a valuable activity for almost any company that wants

to improve conversion rates. It’s relatively easy, quick, affordable and effective

if you follow proper protocol. Most importantly, it works—companies report

significant savings and revenue from testing.

Yet it’s important to do your homework and proceed with a clear direction—

learning the ABCs takes practice, dedication and patience. To be truly successful,

become a lifelong learner and constantly test new ways to optimize your website

and improve conversation rates. Armed with A/B testing knowledge, there’s no

reason you can’t learn the ABCs and become an expert reader.

14 “The Complete Guide to A/B Testing.” A/B Testing: The Complete Guide. N.p., n.d. Web. 05 May 2014. <https://visualwebsiteoptimizer.com/ab-testing/>.

15 Ferriss, Tim. “Google Website Optimizer Case Study: Daily Burn, 20%+ Improvement.” The Blog of Author Tim Ferriss. N.p., n.d. Web. 01 May 2014.

16 “How We Improved Our Conversion Rate by 72%.” Dmix:. N.p., n.d. Web. 01 May 2014. <http://dmix.ca/2010/05/how-we-increased-our-conversion-rate-by-72/>.

Technology

AB Testing Blue Paper