AB Testing for ECommerce LemonStand v2

Francis Teo, Sid Bharath

A/B TESTING FOR eCOMMERCE

The Ultimate Guide to A/B Testing Your eCommerce Store

1A/B Testing for eCommerce https://lemonstand.com

Version 2

Table of ContentsINTRODUCTION

What is the book about? Who is this book for? How should I use this book? What is LemonStand?

THE BASICSFirst Things First: What is Your eCommerce Conversion Rate?What is Conversion Rate Optimization?Why Invest in eCommerce Conversion Rate Optimization? Get More Visitors at the Same Budget

A Higher Conversion Translates into Better PPC Results

What is A/B Testing?The Scientific Method 101 Step 1: Collecting data

Step2:CreatingaScientificHypothesis

Step3:TestingtheHypothesis

Step 4: Concluding the Test

Step 5: Repeat

STEP 1: COLLECTING DATAQuantitative Research GA Overview

Setting Goals

Audience Demographics

Interests

Geographic Information

Behavior

Technology

Mobile Visitors


Version 2

Aquisition

Traffic Channels

Campaigns

Keywords

Behavior

Behavior Flow

Content

Site Search

Conversions

Goals

eCommerce Tracking

Multi-Channel Funnels and Attribution

Qualitative Research Surveying Existing Customers

Surveying Website Visitors

Customer Support

Usability Tests

Heat Maps

STEP 2: CREATING A HYPOTHESISInterpreting Quantitative Data Funnel Visualization

Site Content

Browser and Device Conversions

eCommerce Reports

TrafficSourcesandReferrals Mix it up

Interpreting Qualitative Data Interpreting Customer Surveys

Interpreting Website Surveys

Interpreting Customer Support

Interpreting Usability Tests

Interpreting Heat Maps


Version 2

The Hypothesis

STEP 3: TESTING THE HYPOTHESISPrioritizing your testsChoosing a softwareCreating the test

STEP 4: CONCLUDING THE TESTHow long should you run the test?Interpreting the resultsDeclaring the winner

TEST IDEAS: WHAT CAN YOU TEST?Context is Everything in Conversion OptimizationTesting your stores value propositionTesting Your eCommerce Checkout eCommerce Checkout Test Challenges Smaller Sample Size

Technical Challenges with Implementation

Test Guest vs. Login Checkouts

Test Coupon Code Fields

Test Trust and Security Seals

Testing your product page Layout Product Description

Social Sharing

Testing your category page Layout

Filters

Testing your homepage

WHERE TO GO FROM HERE


Version 2

INTRODUCTIONWHAT IS THE BOOK ABOUT?

This book will teach you everything you need to know about optimizing your eCom-merce conversion rates with A/B testing. Well show you how to collect relevant data about your conversion rates, create hypotheses from the data, run A/B tests and inter-pret the results.

WHO IS THIS BOOK FOR?

This guide is for eCommerce stores that are focused on growth and looking for new waystoincreasesales.Yourbusinessdoes6figuresandmoreinsaleseachyearandyou take an analytical approach to running your business.

HOW SHOULD YOU USE THIS BOOK?

The entire book outlines a step by step approach to A/B testing. As you read each section, follow the highlighted Action Items and implement them on your own. Then come back and do the next section. A/B testing never ends so well include case stud-ies and examples that you can keep looking at for inspiration and ideas.

WHAT IS LEMONSTAND?

LemonStand is an eCommerce platform for professional online retailers. The eCom-mercestoresthatuseourplatformaresuccessfulbusinessesdoingnorthofsixfiguresayear.Oursoftwareoffersthempowerandflexibilitytheyneedtogrowtheirbusinesses.

https://lemonstand.com


Version 2

THE BASICSFIRST THINGS FIRST: WHAT IS YOUR ECOMMERCE CONVERSION RATE?

BeforewestarttalkingaboutConversionRateOptimization,weneedtodefinewhatConversion Rate actually is.

ForeCommercestores,yourConversionRateistypicallydefinedasthepercentageofunique visitors that make a purchase.

To illustrate, its probably easier to use some numbers. Lets say your eCommerce store receives 10,000 unique visitors each month, and 200 of these visitors makes a pur-chase. That means 2 out of every 100 people who visit your site go on to buy, making your conversion rate 2%.

WHAT IS CONVERSION RATE OPTIMIZATION?

Conversion Rate Optimization (CRO) for eCommerce is the science and art of getting moreofyouronlinestoresvisitorstopurchase,withoutgettingmoretraffic.So,contin-uing with our hypothetical scenario, if you were to optimize conversions on your store, youdstillhave10,000uniquevisitorsbutmorethan200(hopefullysignificantlymore)would buy something.

Now lets make it more interesting by adding monetary values to the example. Lets say your average order is $100. At 200 purchases a month, you make $20,000 in revenues.

Now you decide to make some changes to your site. You think that adding a reviews boxtoyourproductpagewillconvincemorevisitorstobuy.Youfindaproductreviewsoftware and add a line of code to your site that manifests as a reviews box. A simple change.

Suddenly more people are buying as they see how others have rated your products so well. Instead of 200 orders a month you now have 300 orders a month. Your conversion rate has increased 50% from 2% to 3%, and youre now making $30,000 every month!Thats the power of conversion rate optimization and examples like this arent uncom-mon. In fact, this example isnt entirely hypothetical. Express Watches made the very same change on their online store. They added a reviews widget on their product pages and saw their conversion rates jump up 58%. Well explore this in greater detail later.


Version 2

WHY INVEST IN ECOMMERCE CONVERSION RATE OPTIMIZATION?

Asyouvealreadyseen,thefirstobviousbenefitofCROisthatitwillallowyoutosellmoreonyourstorewithoutgettingmoretraffic.Thisdirectlytranslatesintomorerev-enueandalsousuallytranslatesdirectlyintoprofits(sinceyourenotspendinganythingextra to sell to these customers).

Secondly, a higher conversion rate means that youre now making more per web-site visitor. This means that you can now spend more to acquire each visitor. If youre spendingmoneytogetcustomersthroughpaidtrafficchannelsyoucangetalargervolume of visitors to visit your site.

Finally, being able to spend more to get each visitor means that you can bid a higher amount per click.

This means that you can possibly outbid your competitors to get a higher placement in thepaidlistings,bringingyouevenmorepaidtraffic.

WHAT IS A/B TESTING?

Aswementionedearlier,CROisascientificprocess.ExpressWatchesdidntrandomlydecidetoaddareviewswidgettotheirsite.Theytesteditfirsttoseeifitwouldwork,before rolling out the new feature to all visitors.v

Thats where A/B testing comes into the picture. It is at the core of a properly executed Conversion Rate Optimization strategy. Basically, instead of just guessing what changes tomake,youscientificallytestyourwebsiteforhigherconversions.


Version 2

Heres the gist of it. With an A/B test, you have two versions of the same page - Version A and Version B. Version A is your control page, the one you already have. For Express Watches it was the product page without the reviews widget, the one they had before they started optimizing for conversions.

VERSION A


Version 2

Version B is the page with the change you want to make. For Express Watches it was the same product page except it now had the reviews widget.

VERSION B


Version 2

Soyouhavetwoidenticalpageswithonedifference.Astrafficcomesintoyoursite,you randomly split it up, sending one set to Version A and one to Version B. You then track which version ends up with higher conversion rates and thats your winning ver-sion.

By seeing how the two versions perform side by side, youre ensuring that all variables remain the same. Any external forces that affect conversion rates on Version A will also affect Version B. Any difference in conversion rates can be attributed to the change you made on the page.

Simultaneous testing also lets you revert back to your original page should the new versionprovetobeafailure.Itsascientificprocessthatminimizesyourriskandusesdata rather than guesswork to systematically increase your conversion rates.

THE SCIENTIFIC METHOD 101

Sowhatdoesthisscientificprocessentail?Itallstartswithrigorousquantitativeandqualitative research into your websites data and customers. This data will help you identify areas on your site that have poor conversion rates. You propose a hypothesis that will improve conversion rates and then run a test to prove the hypothesis. If the test is successful, you implement the solution and repeat the process.

Step 1: Collecting data

Conversionrateoptimizationisallaboutdata.Youfirstneedtocollectdataonyourcustomers to understand what brought them to your store, which pages they visited, what they were looking for and why they bought something or didnt buy anything. After you analyze the data you can identify areas on your site that you can improve to increase conversion rates. Then, you create a hypothesis.

Step 2: Creating a Scientific Hypothesis

Now,whatisahypothesis?

Bydefinition,ahypothesisisaproposedstatementmadeonthebasisoflimitedevi-dence that can be proved or disproved and is used as a starting point for further inves-tigation.

In the case of eCommerce stores, you propose a change that might cause conversion rates to improve. For example, a hypothesis could be that adding a security seal to the cart page will increase the percentage of people checking out.

Creating a good hypothesis is very important in Conversion Optimization, otherwise youll be testing random stuff blindly, and are unlikely to get good results.

10

A/B Testing for eCommerce https://lemonstand.com

Version 2

Step 3: Testing the Hypothesis

Ratherthanimplementingtheproposedchangeinyourhypothesis,youfirstneedtorun an A/B test to see if your hypothesis was actually correct.

Inatypical50/50splitA/Btest,yousendhalfthetraffictotheoriginalwebsite(calledthecontrol),andhalfthetraffictothenewproposeddesign(calledthetreatment).After some time, there will be enough visitors going through the test to draw a conclu-sion about the outcome of the test.

Step 4: Concluding the Test

Once the test is over, and if the test shows that the treatment performed better, its timetoimplementitonyoureCommercesiteandreapthebenefitsofahigherconver-sion rate.

Dont be quick to dismiss a test if it shows that the treatment performed worse than the control. Its still a valid result as you can build upon the knowledge of what did not work to come up with a different hypothesis.

Step 5: Repeat

Thats right. Do it all over again. You neednt repeat step 1 if you collected enough data. Go back to step 2 armed with new insights from your recent test and create an-other hypothesis to test for.

Conversion rate optimization is a continuous process. Its a learning cycle consisting of collecting data, creating tests and making changes. Even the smallest improvement in conversion rates is important because they all add up.

Basically, never stop testing.

11


Version 2

STEP 1: COLLECTING DATAA/B testing starts with data. Without data, you cant construct a hypothesis.

Researching your website and customers can be time-consuming but its the most important part of the process. If you are thorough, youll have enough data to come up with new tests for a long time. Do it right so that you dont have to do it of-ten.

There are two types of data - quantitative and qualitative. Quantitative data consists ofhardnumbersliketrafficstats,pagevisits,bounceratesandsoon.Qualitativedataconsists of subjective information, like opinions, suggestions and feedback from cus-tomers.

In short, quantitative research tells you what is wrong and qualitative research tells you why

QUANTITATIVE RESEARCH

Most of your quantitative data can be collected from analytics software. Were going to use Google Analytics as an example because its free. There are other, more powerful, softwares which well come to later.

Google Analytics Overview

Setting up Google Analytics is pretty simple. You can access it at http://www.google.com/analytics/ and sign in with your Google account. Follow the steps to add theGAtrackingcodetoyourstoreandGooglewillstarttrackingtraffic.Dontforgettoenable eCommerce and Enhanced eCommerce settings in the Admin tab.

After everything is set up correctly, youll start seeing data in your reporting tab. This is where all the research happens so lets get familiar with it.

Besides information on the Audience of your websites visitors, Google Analytics uses the ABC model to organize your Analytics data. This stands for Acquisition, Behavior and Conversions, which describes the customer journey as people move through your eCommercestoreandmakepurchases.Youllfindthesesectionsontheleft-handside-bar in the Reporting tab.

12


Version 2

Beforewegetintohowtouseeachreport,weneedtosetupgoalsfirst.

Setting Goals

Goals in Google Analytics are how you measure certain objectives on your site. For your eCommercve store, the primary goal should be making a purchase. Every time a visitor makes a purchase, it counts as a completed goal and shows up in your conver-sion reports.

13


Version 2

To start, go to the Admin tab, click on the Goals section in the left-hand sidebar and create a new goal. You can either create one from scratch by clicking the big red but-ton, or you can import from a Gallery of goals submitted by other users.

If you create a goal from scratch, youll see the setup screen. For the purposes of this guide, were only going to focus on creating a goal to track when a purchase occurs.

However, you can create other goals, up to 20, if there are more actions youd like to track on your store.

14


Version 2

Thefirststepistoselectatemplate.TotrackproductpurchasesonyoureCommercestore, select Make a payment.

Step2isdescribingyourgoal.Giveitanidentificationnameandchoosetheactionthat triggers the goal. The thank you page after a customer makes a purchase is perfect for eCommerce stores.

15


Version 2

ThefinalstepistodefinetheURLofthethankyoupage.Thevaluesettingaddsamonetary value to each conversion. Ignore this because your eCommerce tracking will take care of it.The Funnel setting is extremely important and will show you exactly where visitors are converting, or not converting, on your site. Well look at the setup now and come to the application later.

To create the funnel, you need to manually crawl through your site. Identify the pages and URLs that each visitor would go through, from the homepage to the category and product pages, to the cart, checkout and thank you page.Now you can only create one funnel per goal and 20 goals in total. This means you cant create funnels for each product if you have more than 20. That isnt a problem because the eCommerce settings in GA track product conversion rates anyway.

The best thing to do is focus only on steps that customers take after they hit the cart. Regardless of the product or products they buy, they will all have to go through the cart and beyond to complete the purchase. Creating a funnel like this will throw up insights into your cart abandonment rates and help you optimize your checkout.

The screenshot is just an example but you need to make sure you have every single step in there so that your data is not incomplete. Some stores have extra steps for account logins and shipping addresses. After setting up the goal, click on the Create Goal button and youll start seeing conversion data for your reports.

16


Version 2

Audience

The Audience section of Google Analytics provides a breakdown of the type of visitors that are coming to your online store. This can include geographic and demographic data, as well as the type of browser and device they used.

DemographicsTocollectdemographicdata,youllneedtodosomeadditionalconfigurationonyourwebsite, and also disclose that youre tracking this information to your visitors. Once you start tracking demographics youll have a better understanding of who your cus-tomers are, with information on their age and gender. Use this information to dictate the tone and voice of your website and product copy.

InterestsGoogle can also tell you about your visitors interests with the Interest report. Using cookies, they track the other websites your visitors surf and makes inferences on what theyre interested in. Again, this will help you determine what changes visitors are most likely to be interested in.

Geographic InformationBy knowing where your visitors are coming from, youll be able to customize and local-ize content and information for each geographic segment. Most A/B testing software have geographic targeting features that you can use to target only visitors from certain locations. Well come to this later.

BehaviorThe behavior report under the Audience section shows you how many new visitors you have vs. repeat visitors. You can also see the frequency with which customers visit your site and their engagement in terms of visit duration.

TechnologyThe most important reports in this group are the browser and operating system reports. You eCommerce site may render differently for each browser so its important to see whether any of the browsers are impacting your conversion rates in any way.

Mobile VisitorsIf a large proportion of your customers are browsing your online store on mobile devic-es, and maybe even making purchases, you might want to invest in creating a custom-ized mobile experience for these visitors.

Ideally, your eCommerce store should be fully responsive so that you dont have to build a mobile store from scratch. LemonStand provides a seamless mobile responsive design out of the box so any changes you make on the web page are automatically translated to the mobile screen.

17


Version 2

Acquisition

The Acquisition part of Google Analytics can provide very useful information on how visitors are coming to your site.

Traffic ChannelsThisisabreakdownofwhereyouronlinestoreisgettingtrafficfrom.Fromhere,youllknow the proportion of your sites visitors are coming from paid search, organic search, social media, email marketing or some other marketing campaign.

Thisreportalsoprovidesinsightsintothemindsetofyoursitesvisitorswhentheyfirstcome to your site. For example, somebody who comes to your online store from paid search is more likely to buy on that visit, compared to somebody who comes to your online store through a social media channel.

Youcanalsozoomintoeachsourceoftrafficandlookatthedomainnamessendingtraffictoyoursite.Findoutwhichreferralsitesaremostvaluablebylookingatthecon-version rates of visitors from those sites.

CampaignsThe Campaigns report is particularly useful if you create regular email marketing cam-paigns. It tracks the number of customers clicking through from your emails and shows you whether they go on to convert by purchasing something. This is great feedback for how well the campaign is performing and what you need to change to optimize for conversions.

18


Version 2

CampaignsKeyword Analysis has been a mainstay of any online marketing campaign ever since searchenginesbecameaviablecommercialtrafficchannel.Avisitorcomingtoyouronline store from a search engine by typing in certain buyer keywords might be more motivated and should be treated differently from a visitor who typed in informational keywords.

For example, a buyer keyword might be buy neckties versus an information keyword such as how to tie a necktie. You can then analyze how these search terms are con-tributing to your conversion rates.

Behavior

The Behavior part of Google Analytics shows what visitors do and the pages they visit when they reach your online store. By analyzing the data here, youll be able to get a better idea of how to best serve your visitors and improve their website browsing expe-rience.

Behavior FlowThe Behavior Flow report shows the paths that your visitors are taking through your online store. This can provide insights into the conversion thought sequence of your visitors. For example, are they browsing directly from your homepage to your special offerspage?Thismightindicatethattheyareverypricesensitive.

Theflowalsoshowsyouwherevisitorsaredroppingoutofyoursite.Theremightbesome pages causing more drop-offs than others. Figuring out why will help improve conversion rates.

ContentThe Content report shows the various pages viewed by visitors on your online store. Usethistofindoutwhichareyourmosttraffickedpages,andwherepeopleusuallyenter and exit your site from.

Youllalsofindbounceratesandexitratesforeachpagehere.Thebouncerateistheproportion of people who left your site after landing directly on that page from an outside source. The exit rate is how many people left the site after reaching that page from another page in your site.

This data will help you prioritize pages to optimize. Well come to prioritizing tests later, but for now read up on Page Value to understand how each page contributes to your revenue.

Site SearchThis shows you how visitors are using the search function on your store. Product search isanimportantfeatureofanyeCommercestoreandifcustomerscantfindproductsquickly theyll leave your site. See what products customers are searching for and which pages theyre searching from.

19


Version 2

Conversions

The Conversions part of Google Analytics provides information about your Conversion Goals. For an eCommerce store, this means purchases by your sites visitors.

GoalsThis section gives you an overview of your goal performance. Within this, there are two important reports you should familiarize yourself with - funnel visualization and goal flow.

Funnel visualization breaks down the number of visits for each step of the funnel you haddefinedwhensettingupyourgoal.Itshowsyouhowmanypeoplecomeintothestep, how many exit and how many move on to the next step.

The example on the left is a standard checkout funnel for an eCommerce store. You can see that 428 people entered the shopping cart but only 195 moved on to the shipping info stage. From there on, there are no visitors entering the steps directly (from the left). They can only enter from the previous step, which is how any shopping cart works.

As you can see, the funnel visualization report is one-directional and focusses on conversions between steps. To see the actual paths that visi-tors take, you need the Goal Flow report. This willshowyouwhichsourcesbroughttraffictothe start of your funnel and the path that visitors take, including loops from going back and forth between two steps.

20


Version 2

eCommerce TrackingThe eCommerce reports show you how your products are doing. What products are your customers buying, in what quantity, and how much money are these products making you. You can also see how many sessions it takes before visitors are ready to make a purchase.

Google also recently released Enhance eCommerce reports. These are incredibly pow-erful because they contain a lot more information about your products and customers shopping behavior on your site. However, its pretty complex to set up so its out of the scope for this guide. We suggest you go through the implementation guide from Google to understand how it works.

TherearetwoimportantmetricswithintheeCommercetrackingreport.Thefirstisyourconversion rate, and the second is your Average Order Value (AOV). Now while we do want to improve conversion rates, the aim is to increase the value of each unique visitor ofyoursite,sothatwiththesameamountoftrafficyouhavemorerevenue.Thus the ultimate metric to measure how well our tests are doing is the Revenue Per Visitor (RPV).Mathematically, RPV = Conversion Rate X Average Order Value. Lets say you do a test that increase your conversion rate by 10%. If, during this period, the AOV decreased by 25% then your RPV actually went down 15%. Not a good result after all!

Multi-Channel Funnels and AttributionFormostonlinestores,thetypicalcustomerjourneyfromfindingoutaboutyourprod-uct to making an actual purchase is not a one-channel or one-step process. The pro-spective customer probably interacted with your website and brand a few times before making a purchase.

The Multi-Channel Funnels and Attributionreportscanhelpyoufindoutwhatcom-bination of channels your visitors touched before they made a purchase. It doesnt make sense to attribute your conversions to just last touch (the default for Google Analytics)orjustfirsttouch(thedefaultforGoogleAdwords).

Forexample,ifyoufindoutthatyourbestcustomerscameinfromsocialmediatwicebefore visiting your site directly (by typing in your URL into their browser), then its likely that increased visibility in that social media channel would provide an increase in moti-vated customers to your site.Fortheadvanceduser,youcanevenconfigureyourowncustomAttribution Model to allocatecertainvaluestoeachtrafficchannel.

21


Version 2

ACTION ITEM Ifyouhaventalreadydoneit,setupGoogleAnalyticsonyoureCommercestore.Dontforgettoen-ableeCommercetrackingaswell.

Makesureyoufamiliarizeyourselfwitheachreportandthatyouhavegoalsandfunnelsforyourcheckoutsandpurchases.Playaroundwithallthereports,mixandmatchsegments,andtrytoun-derstandwhatshappeningonyourstore.

Alternative Analytics Software

While Google Analytics is a fantastic tool, there are other paid softwares that are more powerful and provide deeper insights into customer behavior. Here are two of the more popular ones -

Thefirstis Kissmetrics. While Google Analytics tells you what is happening on your site, Kissmetrics tells you who is doing it. The software ties on-site actions back to the peo-ple doing it, so you can track each visitor as they proceed through the buying process onyoursite.UseKissmetricstogetherwithGoogleAnalyticstofindthebiggestconver-sion roadblocks in your funnel.

The second is RJMetrics, an all-in-one analytics and business intelligence solution. It can give you an end-to-end overview of how you acquire, convert and engage custom-ers,thusreplacingyourothertools.Useittofindnewwaystoimproveyourconversionrates and set daily conversion goals.

22


Version 2

QUALITATIVE RESEARCH

Besides taking a look at Analytics, there are several other methods to do more qualita-tive research on your customers. Its important to have both qualitative and quantitative research in order to get a better picture of what your customers are thinking.

Surveying Existing Customers

Surveying your existing customers is a great way to gain some insight into what your customers are thinking and the possible value proposition of your store.

This is easily done using tools like PopSurvey, Fluid Surveys, or Survey Monkey.

Your existing customers can tell you why they like your store and what they like about your products. They can also tell you what they didnt like and anything that might have been a deal breaker for them. With this information you can double down on what worksandfixwhatdoesnt.

The best thing about emailing surveys to customers is that they can answer them at leisure. Its a convenient method and because of the format it allows you to collect lots of feedback with one mail.

Ofcourse,youdontwanttoaskthemtofilloutpagesofinformation.Toolongandthey wont bother spending time on it. Keep the number of questions below 10 and include some multiple choice questions to make it easier for them.

Here are some things you can ask in your customer surveys:

Whatdidyoulikeaboutourproduct?

Whatconvincedyoutopurchasetheproduct?

Howwouldyoudescribeustoafriendifyouweretoreferus?

Wasthereanythingthatmadeyouhesitatewhenyouwerebuying?

Arethereanyunansweredquestionsyouhaveaboutourproducts?

Toincentivizecustomerstofilloutthesurveyyoucanofferthemadiscountontheirnext purchase. You get your questions answered and they save money.

23


Version 2

Surveying Website Visitors

On-site surveys are another great way to gain insight into what your sites visitors are thinking. Typically, the survey pops up from the corner of your page like a little chat box. Tools like Qualaroo allow you to ask any type of question you want based on trig-ger rules like which page the visitor is on.

The on-site survey allows you to get a good understand of what the visitor is thinking whilenavigatingyoursite.Thisishugelybeneficialbecauseiftheyarespendingtimeonyoursiteandthenleavingwithoutbuying,youcanfindoutwhybeforetheyleave.

Perhapstheyarefrustratedwithyoursitebecausetheycannotfindwhattheywant?Perhapstheyfindyourproductstooexpensive?Ormaybetheydontthinkyouretrust-worthy. Whatever it is, an on-site survey takes the guesswork out of it.

When combined with quantitative data, you can pinpoint whats causing visitors to drop of from your site. For example, if you see high drop off rates on a certain product page, you can trigger the survey to run on that page and ask visitors why they are dropping off.

Now, because the survey box is small, youll have to limit the number of questions you want to ask, and you have to be very precise. You can ask targeted questions related to each page like:

Didyoufindwhatyouwerelookingforonthispage?

Whatsstoppingyoufrompurchasing[currentproduct]?

Whatfurtherinformationabouttheproductdoyouneed?

Whatwillconvinceyoutobuytheproduct? Whyarentyoucompletingyourpurchase?

Try them out one at a time and allow visitors to respond in a sentence or two.

Customer Support

Talking to your customer support staff is also an important place to get useful insights into the mind of your customers. These are the people who talk to your customers day in and day out, and are an important resource.

This isnt restricted to just phone or email support. You can also analyze live chat logs to get some information about how your customers think about your brand and prod-ucts.

24


Version 2

Youcanfindoutfromthemwhatthecommoncustomerqueries,concernsandcom-plaints are. From there, you can further build your knowledge about your customer and create hypotheses to test based on this insight.

Its also helpful to train your customer support staff to be more aware and systemati-cally record feedback which may be helpful in formulating tests.

Usability Tests

Actions speak louder than words. While customers might be able to tell you what they like or dislike about your store, they may not be able to explain, in writing, how they feel about the usability.

Usability tests solves this by recording customers as they navigate your site. You can set up steps that they need to complete and ask them to explain their thought process as they use your site. You might be surprised about the things you missed out on your site that are frustrating customers. Sometimes, even the little things matter; and your cus-tomers can point these out during the tests.

Onewaytorunausabilitytestistobringparticipantstoyourofficeorastudioandask them to use your site while recording them. If youre looking for a more automated method, try UserTesting. Youll have to pay but they make it easier by letting you create and run tests online.

Heat Maps

Heat mapping and mouse tracking technologies allow you to see where your visitors are looking on each page. Find out which areas of your webpages are most looked at orclickedon.Dopeoplescrollbelowthefold?Dotheylookattheleftcolumnfirstortheright?Heatmapscananswerallthesequestionsandmore.

Crazy Egg is a popular heat mapping tool. You can track mouse movements and clicks as well.

ACTION ITEM Thefirstthingyouneedtodoiscreateasurveythatyoucanemailtocustomers.Itmighttakeadayortwobeforeyougetresponses,sostartnow.

Formoreimmediateresults,setupasurveyonyoursitewithQualarooandtrackheatmapswithCrazyEgg.

Finally,talktoyourcustomersupportteamandcollectdataontheconcerns,complaintsandfeed-backthatcustomershaveforyou.

25


Version 2

STEP 2: CREATING A HYPOTHESISOk, so youve collected all this data in the previous step. However, its just raw data at the moment. Before you try and create a hypothesis, you need to interpret the data to understand what its trying to tell you.

INTERPRETING QUANTITATIVE DATA

If you have only recently set up Google Analytics, you might not have enough data just yet to glean useful insights from. Youll have to wait a few days till you start seeing clear patterns.

Now, there are so many combinations and permutations of reports in Google that it would be impossible to go through them all in this guide. Its up to you to look at all of themandfindwhereyourwebsiteisleakingmoney.Togetyoustarted,wellcoverafew reports here and show you how to make sense of them.

Funnel Visualization

Weve already looked at what the funnel visualization report is and how to set up a funnel for your cart and checkout process. Around 68% of your customers abandon the shopping cart, so optimizing this area offers a huge potential for improving conversion rates.

Lets look at the example funnel from before again.

26


Version 2

You can see that people are dropping off at every step of the funnel. Only 46% of cus-tomers move from shopping cart to shipping info. Of that, only half give their shipping info and move to payment info. Finally, 62% of people who reach this stage actually make a payment.

Overall, the conversion rate of this store is 15.19% from the moment a person adds a product to the cart. That means 85% are abandoning the cart, 17% more than the industry average.

Each step of the funnel can be improved but the shopping cart seems to be the weakest link. The report tells us that less than half move on to the next step but it doesnt say why. To get the full picture, youll need to check your qualitative research.

27


Version 2

Site Content

The Site Content report under the Behavior section can show you which pages are let-ting you down. Most of your visitors might not even reach the cart stage, so its impor-tant to see where they fall off.

StartbylookingatAllPagestofindpageswhichseethemosttraffic.Thedefaultviewshows you how many times the pages were viewed, and the corresponding bounce and exit rates.

A better view is the comparison view which you can toggle in the top right corner as seen in the next image.

28


Version 2

This shows you a graphical comparison of the performance of each page against the site average. Here, were comparing bounce rates but you can also compare any of the other metrics. The red bars are the ones performing poorly so investigate these pages tofindoutwhytheyarecausingpeopletoleaveyoursite.

Set up page values for each page, then arrange them in descending order of value in-steadofviews.Pageswithhighervaluecontributemoretoyourrevenuesofiguringoutwhy these pages are doing poorly will have a greater impact on conversion rates.

Browser and Device Conversions

The Audience section has two reports - Technology and Mobile. Technology shows you dataforeachbrowserandoperatingsystemthatvisitorsusewhensurfingyoursite.The Mobile report shows you a breakdown of Desktop vs. Tablet vs. Mobile, plus the names of each device if you click through.

Many people ignore this because they arent aware of how different browsers or devic-es affect their stores. Maybe the navigation looks poor on the iPhone. Maybe the Add to Cart button doesnt work on a Samsung tablet. Maybe your product videos dont appear on Internet Explorer.

The fact is, your conversion rates are different for each technology. Look through each reporttofindoutifthereareanyglaringdifferences.Youcanignoretheoutdatedtechnologiesthatsendahandfulofcustomerstoyoursite,butifyouseelotsoftrafficcoming from certain browsers or devices, take them very seriously.

eCommerce retailers are starting to catch on to the growing usage of smaller devices. WalmartsCanadianwebsiteanalyzedtrafficsourcebydeviceandfoundthatmostoftheirtrafficcamefromtabletsyettheconversionswerepoor.Theyweresofocusedondesktop only that they neglected tablets till as late as 2013.

Because they use a custom eCommerce solution they had to rebuild the entire site with a responsive design. It cost them a lot of time and money but it eventually increased their conversion rates by 20%. If you dont have Walmarts budget, you might want to use a cloud solution with built-in responsivity.

You can also mix and match to see how conversion rates change for each browser on each device. This allows you to simultaneously compare across browsers for the same device, and across devices for the same browser.

29


Version 2

eCommerce Reports

The basic eCommerce reports are pretty straightforward. You can view how each product fares in terms of conversion rates. The comparison feature will show you their performance against your site average. If some products are not converting as well as others,youllwanttofigureoutwhy.

For deeper insights, add segments to your report. For example, you can add a seg-ments for new visitors and returning visitors. Your report will then show you overall con-version rates for a product as well as conversion rates depending on if the visitor is new or returning. If the conversion rates for new visitors are low for a particular product, you need to investigate why.

Traffic Sources and Referrals

ThetrafficsourcereportsunderAcquisitioncanshowyouwhichsourcesareconvert-ingandwhicharent.Yourtrafficmightbecomingfromemails,socialmedia,organicsearches, paid searches or other websites. To get the most bang from your buck, you wanttodoubledownonthebetterperformingsourcesandfigureoutwhytheothersources arent doing well.

Mix it up

As we said, there are so many reports, and so many ways you can slice and dice them. It will take a few more books to comb through everything. The best way to learn is to keep playing around. Remember to always keep an eye out for discrepancies in conver-sion rates.

Quantitative data is just one part of the picture. For example, you might see high abandonment rates at checkout but does the problem really lie in the checkout page, ordoesitlieintheproductpage?Theanswertothatwillcomefromyourqualitativedata.

30


Version 2

ACTION ITEM Startlookingateachreportwhilenotingdownareasthatseemtohaveproblems.AgoodideaistomaintainanExcelsheetwhereyoucancollectallyourobservations.Fornow,justuseonecolumntostorequantitativedatalikeProductXhaslowconversionratesforfirst-timevisitorsor10%conver-sionrateforcustomerswhoarrivedfromemailcampaignsontheirmobiledevices.

Aswegothroughthisguide,welladdmorecolumns.

INTERPRETING QUALITATIVE DATA

Interpreting Customer Surveys

Depending on the size of your survey and the number of customer who answered, you may have a lot of data to sift through. Its best to go question by question and try to findpatternsintheanswers.

Look for certain words or phrases that customers use to get an idea of their language. Try to spot patterns in the way they describe your site or products.

When AppSumo launched their Monthly1k product, they sent out an email to 300,000 people. The results were terrible with only 30 people buying it at a conversion rate of 0.1%. Curious about why the conversion rate was so low, they decided to send out a survey to people who had clicked on the mail but not bought the product asking them why they didnt buy.

They received many responses but after going through them they found four main reasons why people didnt buy. Now customers might convey their answers differently, but AppSumo distilled each answer down to the gist and tabulated the frequencies. In decreasing order of frequency, these were the results -

1. Im Afraid

2. Will this work in my country.

3. How will I know that this will work

4. Ive failed before, whats different this time

31


Version 2

Interpreting Website Surveys

When creating your survey, you asked one or two targeted questions to each visitor. To interpret the responses, youll need to look at all the answers to a particular question andtrytofindcommonthemes.

For example, lets say you had a question pop up on your product page asking visitors whats stopping them from buying the product. Lets assume you get a bunch of results like this -

1. I dont know if this product will actually work like it says

2. Im not sure if this is the right product for me.

3. Im just browsing around.

4. I dont know if this product is authentic.

5. I need to know if this product will last a long time.

Thereareonlyfiveresultsforthishypotheticalscenariobutinrealityyoullreceivealot more. The point is, its fairly easy to spot trends like AppSumo did with their email survey.

In this example you can see that answers 1, 2 and 5 relate to trust. Customers are unsure about the products authenticity, functionality and durability. They need more reassurance that its a quality product that works perfectly.

This is what Express Watches found out when they did their surveys. They realized visi-tors didnt trust them, which is why they installed the Trust Pilot widget.

Interpreting Customer Support

Your customer support logs will contain a wealth of information about the burning questions that customers have about your store and products. Again, look for themes andpatternsinthelogstofindrecurringissues.

For example, you might have a number of people calling or mailing about whether they can return the product and if theyll need to pay to return. Of course, they would have got the right answers, but to save other visitors the trouble you should make it more obvious on your home and product pages.

32


Version 2

Interpreting Usability Tests

Your usability tests will show you the little things that you might have ignored. Lets look back at the Walmart example. In their initial design, on their product listings and category pages, they had one of two CTAs for each product. If the product was in stock, they had a green Add to Cart button. If the product was out of stock, they had a greyed out button.

When conducting usability tests, they found that visitors were still clicking on the greyed out button. The fact that there was a button there lead them to think that they could click on it. While it seemed obvious to the website developers, it wasnt to the customers and it ended up frustrating them. Assuming the site didnt work, these peo-ple would leave.

Interpreting Heat Maps

Figuring out where people are looking and clicking on each page of your site will help you understand whats attracting their attention and what isnt. Use them in conjunction with what you learnt from your quantitative research.

For example, lets say Product X has a very poor conversion rate yet the customers who boughtitsaytheylovedit.Whatgives?Checkingtheheatmapmightexplainit.Itspossible that the Add to Cart button on the product page is below the fold and many customers arent scrolling all the way down.

ACTION ITEM Asyoufindtrendsandpatternsinyourqualitativedata,addthemtoyourExcelsheet.Rememberthatquantitativeandqualitativedataaretwosidesofthecoin,socreateanothercolumnandmatchyourqualitativeinsightswithquantitativeobservations.

Forexample,oneofthequantitativeobservationsyounotedinColumn1mightbethatProductXhasaverylowconversionrate.AnyqualitativeinsightsaboutwhythisishappeningshouldgointoColumn2alongthesamerow.

33


Version 2

THE HYPOTHESIS

Now that you understand what is happening on your site and why, you can come up with a hypothesis for a test. Lets continue with the Express Watches example. The quantitative data showed that their product pages had poor conversions, while the qualitative research revealed that customers didnt trust them. One hypothesis would be that adding a reviews widget will build trust and lead to more purchases.

AnotherhypothesisisthataddingaCertifiedDealerbadgetoyourheaderandfooterwill build trust and increase conversions. You cant test it out at the same time as the reviews widget so youll have to save it for later.

To keep track of your hypotheses, add them to your Excel sheet. Create a third column and for each row of quantitative and qualitative data, add a corresponding hypothesis or hypotheses.

Well talk more about prioritizing hypotheses and conducting the actual test in the next section.

ACTION ITEM TrytocomeupwithatleastonehypothesisforeachrowofdatayouhaveintheExcelsheet.Remem-berthatnohypothesisistoosillyortooobvious.Therehavebeencaseswherechangingacoupleofwordsorthecolorofabuttonhaveresultedindramaticincreasesinconversionrate.Keepanopenmindandlettheactualtestdecidewhetheritwasrightorwrong.

34


Version 2

STEP 3: TESTING THE HYPOTHESISPRIORITIZING YOUR TESTS

If youve been following the Action Items, you should have multiple hypotheses to test. If youre still struggling for ideas at this stage, look at the section on testing ideas for some inspiration.

You want to prioritize tests by their impact. The ones that have the potential to make thebiggestimpactaretheonesyouwanttotestfirst.

Letscontinuewiththetrustexample.Youhavetwotestideas-addingcertificationbadgesandaddingproductreviews.Youcantdefinitivelysayonewillbemoreimpact-ful than the other but an educated guess is that the reviews widget would be a good testtotryfirst.Reviewsdontjustbuildtrust,theyarealsosocialproof,apowerfulwaytoinfluencepeople. Put all your testing ideas in a list in order of priority. Its time to start testing.

CHOOSING A SOFTWARE

Before you start testing youll need to get an A/B testing software.

There are some great tools out there that dont require any downloads or complicated installations. We suggest using Optimizely. Set up is extremely easy and it comes with a visual editor that allows you to make changes on your site and create tests without touching code.

An alternative is Visual Web Optimizer. They have similar features so check them out andpicktheonethatfitsyoubest.

35


Version 2

CREATING THE TEST

The actual steps you need to take to create a test depends on which software you use. Both above softwares have guides and tutorials to teach you the ropes. It doesnt take too long to get the hang of it.

For simple tests you can use the visual editor to make changes on your page. Lets say your hypothesis is that making the Add to Cart button more prominent on your prod-uct pages will lead to more purchases. You can use the editor to change the size, color, placement and copy of the button.

You can also go advanced and make changes to the code if need be. Either way, youll be able to create variations of your product page that have different cart buttons. Then set the software to track the number of purchases of the product as the conversion rate.

Afterthatthesoftwarewilltakecareoftherest.Astrafficcomestoyoursite,itwillautomatically and randomly show one of the variations to the visitor and then track whether that visitor goes on to purchase the product.

To illustrate, lets do a quick walkthrough using Optimizely. Well use Zippo, one of our customer stores, as an example and try to optimize their category page. This is just a demonstration and not an actual A/B test. Any statements, observations and assump-tions we make are purely for the purposes of the demonstration.

SoherestheDashboardviewinOptimizely.Thisiswhereyoucanfindyourexperi-ments. To create a new experiment click on the green Create Experiment button. Just to be clear, experiment refers to A/B test.

36


Version 2

When you create a new experiment, you need to enter a name to identify it and a URL that you want to run the test on. Were using the URL of Zippos category page for lighters. Youll be taken to the URL with the Optimizely graphical editor. On the top left youll see tabs for the original page and variations that you want to test.

Asyoumoveyourmousearound,youllfindeachelementonthepagehighlightedby a blue box. You can see the blue box around the second lighter. This is Optimizely identifying each element and making it editable.

Lets say we found, in Google Analytics, that the percentage of people moving from this category page to the product page for one of the lighters was very low. Thats disturbing because it means that people arent even reaching the product page of our most successful products.

After surveying customers, lets say we found that they werent sure which lighter to click on. We could hypothesize that adding a brief description to each lighter will im-prove click-through rates.

So,forourfirstvariation,letsaddsometexttoeachproduct.Allyouhavetodoisclickon the product and the editor will show up.

37


Version 2

Choose an option, like Insert Text, and start typing in a short description. Click Done and youll see the change on the page. Its that simple.

Youve successfully set up the experiment, but before you can run it you need to tell Optimizely what to track. Click the Goals button next to the green Start Experiment to bring up the goals screen. You can track multiple goals by either creating a new one or adding a saved one. By default, Optimizely tracks Engagements. The other goal types are clicks, pageviews and custom.

38


Version 2

With a click goal, you can select an element on the page and track clicks on that ele-ment. For this example, we could create a click goal to see if people clicked on the lighter. If we ran the test, our results will show us how this variation performed against the original page in terms of clicks on that lighter.

Withapageviewgoal,wecansetaparticularpageasthefinaldestinationwewantvisitors to reach after arriving on this page. To see how the test affects our conversion rates, we could set the Thank You page after checkouts as our goal.

In the screenshot above, youll notice a revenue goal. This will show you how your design changes lead to a change in revenue. Remember, the ultimate goal is to make more money, so always keep an eye on that revenue goal.

One thing to note, revenue tracking will not work for eCommerce platforms like Shop-ify. This is because the checkout on Shopify is not hosted on your URL, its hosted on Shopifys URL. Youll either have to upgrade to the Shopify Plus plan, or switch to an eCommerce platform with a customizable checkout.

Now, youre ready ready to run the experiment! Youll have to paste a small code gen-erated by Optimizely in your site for it to run. By default, the experiment will be shown to all site visitors.

If you want to go advanced, you can specify custom audiences that you want to test on, like only people using Chrome, or only people from USA. You can also integrate Google Analytics and Crazy Egg for additional data on the performance of your test.

ACTION ITEM StartbyprioritizingyourhypothesesintheExcelSheet.Oncedone,pickthehighestpriorityhypoth-esisandcreateatestinOptimizely.

Addanewcolumninthesheetandenterthetestyoujustcreatedonthesamerowasthecorre-spondinghypothesis.Specifywhatgoalsyouweretracking,whichaudiencesyoutargetedandwhentheteststarted.

39


Version 2

STEP 4: CONCLUDING THE TESTYouvecreatedyourfirsttestandyoureeagertoseehowitperformed.Dontgettooexcitedjustyetoryoumightfinishprematurely!

HOW LONG SHOULD YOU RUN THE TEST?

A big mistake many people make is to run the test for a short period of time. If you dont collect enough data you wont get the right results.

Lets take a very basic example. Lets say 5 people have visited a product page and 2 of them bought the product. Thats a 40% conversion rate. Now lets say another person visits and buys the product, making it 3 buyers out of 6. The conversion rate is not 50%, a big jump up with just one visitor.

Now heres what it looks like with large numbers. 100 people have visited your page and 40 have bought. Conversion rate is still at 40%. Now one more person visits and buys the product. Thats 41 buyers out of 101 making the conversion rate 40.6%. See thedifference?

Atlowernumbersyourconversionratesfluctuatewildlysoyoudontgettherealpic-ture. Only if you let the experiment run for a long time and wait till the conversion rates normalizecanyoureachstatisticalsignificance.

Dont run the test for less than a week. This should give you enough time to collect a lot of data and to take into account the effects of each day of the week. Your software willcollectdatainrealtimeanditwillcalculatestatisticalsignificanceforyou.Onceitsabove 95%, you can stop the test.

INTERPRETING THE RESULTS

A95%statisticalsignificancemeansthatyoucanbe95%surethatonevariationisbet-terthantheotherifthatswhatthenumberssay.Thehigherthesignificance,thesureryou can be.

Lets look at the Express Watches case study again. Youve already seen the original product page (the control) and the variation with the reviews widget (the treatment). Express Watches ran the experiment for over 2 months and tested it on thousands of visitors.

40


Version 2

The treatment converted at a rate that was 58% higher than the control. Because they rantheexperimentforsolongtheyreachedastatisticalsignificanceof99%.

If you used Optimizely to create the test, you can view results by clicking on the experi-ment in the dashboard, and clicking again on View Results. You will be taken to the results page where you can see a high level summary of the experiment and results for each goal you created.

You can expand the results for each goal to view the full chart with how all your varia-tions performed over time. The chart will also show you visitors and conversion rates for each variation, the improvement against the baseline (original page) and the statisti-calsignificance.

DECLARING THE WINNER

At the end of the experiment Express Watches could safely say that adding the reviews widget would increase conversion rates on their site. They could roll it out to all pages and make it a permanent feature of their online store.

As we mentioned earlier, conversion rate optimization is an ongoing process. You keep testing new changes to your site. Each test you run can give you further ideas about new tests to run. Express Watches learnt that adding product reviews that highlighted their credibility helped buyers trust them. They could take that concept further and test more ideas related to trust and credibility.

ACTION ITEM Oncethetestiscomplete,addittotheExcelsheet.Createanewcolumnforresultsandentertheconversionrateimprovementnexttothecorrespondingtest.

Nowmoveontothenexthypothesisinthesheetandrepeat!

41


Version 2

TEST IDEAS: WHAT CAN YOU TEST?Hopefully youve been following the action items and youve already started testing. If youre stuck for ideas this section will help you out.

CONTEXT IS EVERYTHING IN CONVERSION OPTIMIZATION

Before you start copying the ideas here, bear in mind that eCommerce conversion rate optimization is highly contextual in nature. That means that one change or best prac-tice that works well on one store, might not really work well on your store. It all de-pends on what is going on in the mind of the customer.

Very often store owners will use the large stores such as Amazon as an example of the best thing to do on their store. While Amazon does extensive A/B testing and their website usability very good, you have to remember that Amazons customers are not your customers.

The other important thing you should know is that testing is a process, and not a one-off affair. Very often store owners new to testing get very disappointed when their test doesnotincreaseconversions.Butyoucandefinitelylearnalotfromteststhatdidnotincreaseconversionrates,ordidnotreachstatisticalsignificance.

That being said, you can always learn from and take inspiration from tests conducted on other sites. In this section well talk about testing various key sections of your eCom-merce store - your value proposition, your checkout, your product page, your category page and your homepage.

42


Version 2

TESTING YOUR STORES VALUE PROPOSITION

Your value proposition is the primary reason why a customer would buy from your store and not from a competitor.

Many eCommerce stores have no idea what their value proposition is. For those that do have some idea, they often have not tested this value proposition. Without testing, youre just guessing at what would appeal to your customers the best.

Smileycookiewantedtofindoutwhichvaluepropositioninfluencedvisitorsmost.Theycame up with 5 different value propositions -

Order Today -> Ships Next Business Day

Wanttosave$5OFFyournextpurchase?SIGNUPNOW->

$6.99 Ground Shipping For Your Entire Order

FREE SHIPPING on any order over $40

Cookies Made Fresh & Hand Iced For You!

You can see how each one focuses on something different. Fast shipping, discounts, flat-rateshipping,freeshippingandfreshness.Insteadofassumingwhichonewouldmake the most impact, they decided to test it.

They found that all visitors looked at the top left corner of the home page, just above the banner image. A prominent position like this would be the best place to try out an eye-catching value proposition. A heat-mapping tool like Crazy Egg will help you iden-tify spots like this.

Before we look at the results of the test, try guessing which one won.

43


Version 2

Free shipping seems like a good guess, but the one about fresh cookies sounds deli-cious. However, if youve learnt anything from this book, its that guesses get you no-where.

Thewinnerwasthefirstone,ordertodayandgetyourordershippedthenextday.Conversion rates increased by 41% and all it took was adding a few words to the homepage banner.

TESTING YOUR ECOMMERCE CHECKOUT

The checkout and cart portions of your eCommerce shop offer the largest potential forincreasingconversionratesthananyotherpage.Thisisthefinalstepinthebuyerjourney, the point of no return. Its nice to look at products and add them to shopping carts,butwhenthefinalbillcomesupmanypeoplegetcoldfeet.

In fact, the average cart abandonment rate across various eCommerce verticals is 67.91%! Thats 2/3rd of all people dropping out at checkout and not buying. By op-timizingthispartoftheconversionfunnel,youhavetheopportunitytoinfluencethisgroup and turn cart abandoners into buyers.

44


Version 2

eCommerce Checkout Test Challenges

There are several important things to note about running an A/B test on the cart and checkout parts of your eCommerce site.

Smaller Sample SizeSinceonlyarelativelysmallnumberofpeople(comparedtothetotaltrafficofyourwebsite) ever reach the add to cart or checkout stages of your eCommerce store, you have a smaller sample size for the test.

This means that it will likely take much longer for you to validate results for cart tests.

Technical Challenges with ImplementationFor a lot of hosted carts, its technically challenging to run an A/B test on the checkout or cart portion, due to the highly dynamic nature of how the cart and checkout is being presented. Changes in the checkout design (for example accordion style vs. one page checkout)usuallycannotbetestedaswell,asitsconfiguredsimplybyatoggleontheadministrative interface. As we mentioned before, some platforms dont let you host the checkout on your site, so it becomes impossible to test it.

Using a more technicallyflexibleplatform can help reduce the overhead and frustra-tions for A/B testing your checkout process.

If you can overcome the technical challenges, testing such a critical part of your conver-sionfunnelwilldefinitelyreapyouhugerewardsinthelongrun.

Test Multi-page vs. Single page Checkouts

TherearesomeformfieldsthatareunavoidableineCommercecheckoutpages.Thequestion is, how do you manage them. Do you create one long checkout page, or do youbreakitupintomultipleshortsteps?Moreimportantly,whichmethodisleastlikelytoscarecustomersaway?

As always, the answer is to test. The Vancouver Olympic store had a four step checkout process. Each step was short and they had a progress bar on top to show customers how far in they were.

45


Version 2

They decided to test a longer single-page layout and immediately found that it was a winner.Theyplayeditsafeandranthetesttilltheyreachedstatisticalsignificance,butwith a conversion lift of 21.8% the new layout was always going to be a winner.

46


Version 2

47


Version 2

Test Guest vs. Login Checkouts

Depending on whether you have more returning customers or new visitors to your website, you might want to emphasize one option over another to see if that will in-crease overall conversions. A more advanced test would be to emphasize different types of checkouts for different kinds of visitors.

In the Vancouver Olympic store case study, youll notice they initially had a registering optionasthefirststepintheirmulti-pagecheckoutflow.Aftertheymovedtothesin-glepageflow,customerssawtheoptiononlyaftertheyhadcompletedtheirpurchase.Blivakker, a Norwegian cosmetics retailer, asked customers to login with Facebook be-fore completing the purchase. When they looked at their site performance, they found that the Facebook login was adding extra complexities so they decided to test if it was even worth having it.

They tested the above page against one that didnt have the Facebook section on the right. Their conversion rate actually went up 3% without the Fb login, resulting in an extra $10,000 in sales per week.

48


Version 2

Test Coupon Code Fields

Anotherthingyoucantestisthewayyoupresentyourcouponcodefieldswhencheck-ing out. Some ideas include whether to display coupon codes on the checkout or hide/de-emphasizethefield.Youcouldeventesthavingasmalllinkthat,whenclicked,displaysthecouponfield.

Its important to remember that attention spans are pretty short on the internet. If peo-pleseeacouponcodesfieldandtheygooffinsearchofthecoupon,theymightgetdistracted and never return.

If you never give out coupon codes then it would make sense to not have a coupon codefield.Ontheotherhand,ifyouarehighlydependentoncouponcodepromo-tions,hidingordeemphasizingthecouponcodefieldmightactuallyreduceconver-sions.

Come up with a hypothesis based around these ideas to test the way you present your couponcodefieldstoyoursitesvisitors.

Test Trust and Security Seals

Trust and security seals decrease anxiety for the visitor by reassuring them that their pri-vacy and security are protected when shopping. There are numerous seals online, some that are more trusted than others. Which ones convert the most, or do they increase conversionsatall?Thatssomethingyouneedtotest.

Sometimes, they can even decrease conversions. One possible reason for this decrease is that an improperly placed trust or security seal might remind the visitor that they are giving away their private information to a 3rd party, and increase their anxiety when making their purchase.

The placement of the security and trust seals also matter a great deal. They should generally be placed at very prominent and visible locations.

Buyakilt.comtestedtwosealsontheircheckoutpage.ThefirstwastheNortonbyVeriSign badge which they added to the top right corner of the page. They ran the testforfiveweeksandfoundthatitincreasedconversionsby17%withaconfidenceof99.29%.

49


Version 2

Thatsanamazingresultforsomethingthatcanbeaddedtoyoursiteinfiveminutes!However, in true CRO fashion, buyakilt.com decide to see if any other badges could do better. They swapped out the Norton badge for a Comodo seal. This time conversions increased even further by 12.49%!

Combining the two results, their overall increase in conversion rates was 31.6% which translatedtoasixfigureincreaseinrevenue.

50


Version 2

TESTING YOUR PRODUCT PAGE

The product page is your make or break page. This is where the sale happens. Visitors look at your product pictures, read the descriptions and look at reviews before they decide to buy. If the page doesnt convince visitors they need your product, youve lost the sale.

The main elements to test are your product images, descriptions and CTAs, but even the little things could make a difference. Weve already written a lot about creating product pages that convert well so well show you some tests that defy conventional thinking to highlight the importance of testing.

Layout

Common product page convention is to place the product price under the title and above the description. The CTA usually comes after the description.TaylorGiftsdecidedtotestadifferentlayout.Theyfirsthadastandardlayout.

In the variation, they clubbed the price and CTA together in a box on the right, Ama-zon style. Their hypothesis was that placing all the important information closer to the buy button would make it easier for customers to purchase.

51


Version 2

The test proved their hypothesis correct. Their conversion rates increased by 10% and shows why there is no such thing as best practice in the online world.

Product Description

FreestyleXtreme had product pages that didnt contain any product descriptions. They focused on large images but, after research, found that customers wanted some prod-uct descriptions.

52


Version 2

They spent some time creating interesting product descriptions and added them to the site.TotesttheperformancetheyrananA/Btest.Canyouguesswhichversionwon?

If you thought adding product descriptions increased conversions, youre wrong. The conversion rate was actually less than the original rate by 31.38%!

Obviously this is a very surprising result so its worth trying to analyze whats going on. One explanation is that the product descriptions pushed the images and Add to Bas-ket button further down the page, meaning fewer people saw them. Another is that the white on black copy was tough to read and turned people off.

You can come up with a number of other reasons but you cant say anything for certain unless you test them out. This examples highlights how one simple test can lead to further ideas.

53


Version 2

Social Sharing

Heres another interesting case study. Many eCommerce sites have social sharing widg-ets on their product pages but Taloon.com actually decided to test how effective they were. Their original page had sharing buttons under the product images.

You know where were going with this. They removed the sharing options and saw an 11.9% increase in conversion rates.

Whatgives?Itspossiblethatthelackofshares,asseenbythezeroes,mighthavegiven visitors a negative impression of the product.

Doesthatmeanproductswithhighershareswillconvertbetter?Itssomethingworthtesting.

54


Version 2

TESTING YOUR CATEGORY PAGE

People look at category pages when they are searching for something. They have a general idea of what product they need so they go to the appropriate category and search for the one that they like best. Here are some ideas to test -

Layout

How you display your products on the category page will determine whether people click on them to read more. You want to portray them in the best light so that they evoke curiosity in visitors.

Mall.cz, a large online retailer from the Czech Republic, wanted to make their category pages more compelling. Initially, they displayed products with a small image, an ex-cerpt, a price and the Add to cart button. This layout allowed visitors to scan through alltheproducts,readabitaboutthem,andthenfigureoutwhichonetheywantedtolook at in more detail.

55


Version 2

Mall.cz wasnt happy with the click-through rates though, so they decided to test a couple of other layouts. They hypothesized that placing more emphasis on the images wouldenticevisitorstoclick.Buthowmuchmoreemphasis?Theanswertothatcanonly be found after testing.

Theirfirstvariationhadalargerimagebutkeptthetextaswell.

56


Version 2

Their second variation was pretty much the same except they displayed the text only on mouse-over.

Theytestedthetwovariationssimultaneously,whichmeantthattrafficwasdividedbe-tween these two and the control layout.

The winner turned out to be the second variation. The large images without text made for a cleaner layout with fewer distractions. Customers could look at the product and price and, if anything interested them, they could click through and learn more. Sales increased by 9.46% with this one test, a big result for something that doesnt take too long to code in.

57


Version 2

Filters

ForeCommercestoreswithalargenumberofproducts,afilterisaveryusefultooltohelp visitors narrow down search results on category pages. No one has time to sift throughmultiplepageslookingfortherightproduct,soyouneedtohelpthemfinditfaster.

Of course, as weve mentioned so many times already, you cant take anything for granted.Doyoureallyneedafilter,orisitjustadistraction?Ifyouneedit,howdoyoudesignit?Doesitgointheside-barorontop?Doyouusecheckboxesorsliders?Itallneeds to be tested.

Weve already seen how Buyakilt tested security seals on their checkout page. They have a policy of testing everything because theyve seen how optimizing conversion rates can generate huge returns with small investments.

Theyevendecidedtotestwhetherafilterontheircategorypagewouldhelpbeforerollingitoutpermanently.Atfirsttheircategorypagelookedlikethis-

58


Version 2

Theycreatedavariationwithafilterintheleftside-barandasortingmechanismabovethe rows of products.

The test worked like a charm, boosting conversions by 26%. What was more impressive was that revenues increased 76.1% meaning that people were spending more on the sitebecausetheycouldfindtherightproductsfaster.

59


Version 2

Nowherestheflipside.UkToolCenteralreadyhadaprettygoodfilterontheircatego-ry pages.

Howevertheywerenthappywiththeirconversionsandthoughtthatthebigfilterwasa distraction. They tested removing it so that the products would come up higher in the page. Surprise, surprise, they found that visitors spent much more time on their catego-ry pages, browsing around and clicking on products.

60


Version 2

TESTING YOUR HOMEPAGEYour homepage is typically not a landing page for most new visitors. If someone is searching for a product online theyll most likely land directly on your product page through organic search results or PPC ads. They might then click over to the homepage just to learn more about your store.

Of course, others might come directly to your homepage. Maybe they saw your brand somewhere or were referred by friends. Maybe they clicked through from a blog men-tion or typed your address directly because theyve seen your site before.

That makes the homepage a great place for displaying important messages. Weve al-ready talked about your USP and the homepage is perfect for that. You should also test promotions and other incentives that visitors will see value in.

Ecofoil, an online retailer of home insulations, realized their customers needed more help choosing the right products. They already had a little help section on their home-page but it was below the fold, so many customers didnt know about it. They would land on the homepage looking for help and then get confused seeing the products.

So Ecofoil decided to test the placement of their help section. They brought it higher and placed it in the left sidebar.

61


Version 2

Now, the moment visitors landed on the homepage they would see the help section, which would pick the right products for them. This simple movement increased their sales by a massive 86%!

Another important section to test on the homepage is the navigation. This is what visi-tors will use to reach the category pages and product pages. Optimize your navigation tocreateabetteruserexperienceandhelpvisitorsfindproductsfaster.

Like most other eCommerce store, Body Ecology used a drop-down in their naviga-tion bar. However, their products were pretty complicated but the drop-down only had enough space to display category and product names. This confused visitors and Body Ecology realized they needed more information before they could pick an option.

They hypothesized that it might make more sense for the Products link in their naviga-tion to lead to a page instead of showing a drop-down. The page would have enough space for an excerpt about each category.

62


Version 2

The test was a huge success. Conversions were up by 56.43% and this resulted in an extra $230,880 in revenue per year for them.

63


Version 2

A/B TESTING TIPS FROM EXPERTS

FINDING NEW TESTING OPPORTUNITIES USING GOOGLE ANALYTICS

PEEP LAJA CONVERSIONXL

COMBINING QUANTITATIVE DATA WITH QUALITATIVE DATA

STEPHANIE LIURJMETRICS

10 NOT-SO-OBVIOUS A/B TEST IDEAS FOR YOUR ECOMMERCE STORE

CARA HARSHMANOPTIMIZELY

HOW TO PRIORITISE TESTS USING THE PIE FRAMEWORK

CHRIS GOWARDWIDERFUNNEL

POST-TEST ANALYSIS

BRIAN MASSEYCONVERSIONSCIENTIST

64


Version 2

Userflowscanbeagoldmine.Butitmightnotbeobviouswhattolookfor.Thedataisthere,itsuptoyoutofindinsightswithinthedata.

So I was working on a website where you could either search for stuff on the home page, or browse for stuff (via course directory). User flowshowedmethis:

It looks like users cant make up their mind. They went to the course directory, and back to the home page. And again. And again.

I found this behavior to be puzzling, so I set up 2 advanced segments: a) people who visit course directory pages, and b) people who dont. Then I looked at the conversion rate data for both:

What do you know! People who wandered into course directory converted almost 2x less! Ouch. Key learning: course directory pages leak money! So now Id either have to fixthem,orstoppeoplefromgoingthere.Illbettertestbothapproaches!

Sokeepdigginginyourflowreports,andseewhatyoucanfind.

FINDING NEW TESTING OPPORTUNITIES USING GOOGLE ANALYTICS

PEEP LAJA CONVERSIONXL

65


Version 2

Over the past four months I have improved RJMetrics landing page conversion rates by as much as 74.9% and increased click through rates on our homepage by 88.2%.

One of my tasks involved testing a new pricing page. Here is what we were working with originally:

COMBINING QUANTITATIVE DATA WITH QUALITATIVE DATA

STEPHANIE LIU RJMETRICS

66


Version 2

We wanted to test a completely new variation that included an input section where the user could type in the number of customers their company has and our website would return the appropriate pricing option for them:

While the new design got a substantial amount of people using the input section, it was getting fewer people to actually click the sign up button as compared to the origi-nalpricingpagenotgood.Ineededtofindawaytogetpeopletonotonlyinputtheir options, but to also convert.

67


Version 2

I had integrated the initial test with CrazyEgg. This made it easy to go back to our scroll maps of the experiment to analyze what possibly went wrong. Heres what I saw:

68


Version 2

Scroll maps work in such a way that the hotter zones are the sections where users spend the most time while the cooler areas are the parts of the page users scroll quicklypast.Onournewdesign,thecall-to-action(CTA)buttoncoulddefinitelyusesome warming up.

My hypothesis was that moving the button into the white hot scroll map area would cause the design to have a higher conversion rate as compared to the original pricing page. More people would pay attention to the button simply because their eyes would be lingering there longer.

I was expecting this to have a slight impact, but nothing major. I know that when I visit a websites pricing page, I am usually very thorough with reading all the content on the page before I commit to clicking a sign up button. I didnt think that placing a button in the midst of all the details would spur me to want to click it any more than placing it right beneath the important content, but I was willing to give it a try.

Heres the variation I tested:

I was doubtful of how effective it would be to tweak a design that had been previously tested, and failed. Imagine my surprise when the the results come back with a 310% increase in conversion rate.

Lesson learned: always test your assumptions.

69


Version 2

I speak at marketing conferences around the world, and one of the most common questions Im asked about conversion optimization is, How do I know where on my website to testfirst?

Its a good question to ask. You cant test all pages at once. With limited time and resources and, most importantly, limitedtraffictoallocatetoeachtest,testprioritizationisanimportant part of your conversion rate optimization plan.

Those are all aspects to consider, but they dont give the full picture.

Optimizing the optimization process is often just as important as the tests themselves. Pri-oritizing where you invest energy will give you better returns by emphasizing pages that are more important to the business.

Any effort spent on low-impact areas of your website is wasted effort.At WiderFunnel, we use a framework for prioritizing tests, called the PIE Framework. Here, Ill show you the basic elements of PIE that you can use to prioritize your own optimiza-tion plan.

HOW TO PRIORITISE TESTS USING THE PIE FRAMEWORK

CHRIS GOWARD WIDERFUNNEL

The Prioritization Criteria

Youneedtoconsiderthreeaspectsofyourpagestofindthehighestopportunityareas.They are Potential, Importance and Ease.

Now, keep in mind that pages or target areas or templates are all options to consider in your test plan. A test area might be a standalone page like a landing page, a site-wide template like your product details page, or a multi-step path like your checkout or lead-gen form.

PotentialHowmuchimprovementcanbemadeonthesepages?AlthoughIveyettofindapage without some potential for improvement, you cant test everywhere at once and you should prioritize your worst performers. This should take into account your web analytics data, customer data and expert heuristic analysis of user scenarios.

ImportanceHowvaluableisthetraffictothesepages?Yourmostimportantpagesaretheoneswiththehighestvolumeandthecostliesttraffic.Youmayhaveidentifiedpagesthatperformterribly,butiftheydonthavesignificantvolumeofcostlytraffic,theyarenttesting priorities.

70


Version 2

EaseHowcomplicatedwillthetestbetoimplementonthepageortemplate?Thefinalconsiderationisthedegreeofdifficultyatestwilltaketogetrunningonapage,whichincludes technical implementation and organizational or political barriers. The less time and resources you need to invest for the same return, the better. This includes both technical and political ease. A page that would be technically easy may have many stakeholders or vested interests that can cause barriers. Im looking at you, home page.

You can quantify each of your potential opportunities based on these criteria to create your test priority list.

Then, you can insert each of your pages into the PIE Framework table to turn all of the data inputs into an objective number ranking.

Your Prioritization is Unique

Remember that there are no standard rules for which pages are best to prioritize. Dont just test somewhere because someone said it was most important. Thats misguided.

Your website lives in a unique target market, including factors like your competition, seasonality, and internal cultural environment; all of these affect how your site is used and should be optimized. The priority rating you give each of your potential test pages will depend on this unique business environment.

By using the PIE Framework, youll remove gut feeling from the decision and focus your team on the most important activities.

71


Version 2

Add Tweet Button to the Order Success PageAnythingthatcaninfluencesomeonesdecisiontomakeapur-chase is worth trying. While social media is typically not a direct sales channel, it can do wonders for promoting brand awareness, increasing customer loyalty and help customers overcome reluc-tance to purchase.

Addingatweetbuttontoyourpurchaseconfirmationmessagingand experimenting with how its displayed is a worthwhile test.

A/B test everything about your call to action buttonsThe location, size, quantity, color and wording on call to action buttons are all prime elements to experiment with. Here is a list of test ideas for those buttons with screen shot examples from around the web.

Test different product image themesThe way a product is displayed can be one of the pivotal elements that factors into a consumersdecisiontopurchaseornotpurchase.Insteadoftestingspecificimagesagainst one another, test different image themes, such as models vs. no models, or static images vs. product video.

Experiment with how you engage your social media followingWhile doing some research into the social media presence of brands who topped the online shopping charts on Cyber Monday 2013, I observed an interesting fact: most of them have hundreds of thousands of Twitter followers but are only following a couple thousand or even less. What would happen to to sales if they started following more of theirfollowers?

Run A/B tests on your mobile websiteIn 2013, shoppers used their smart phones and tablets to make purchases at an incred-iblerate55%moreoftenthanin2012.Highermobilewebsitetrafficpresentsagreatopportunity and incentive to optimize the mobile experience of your e-commerce store.

10 NOT-SO-OBVIOUS A/B TEST IDEAS FOR YOUR ECOMMERCE STORE

CARA HARSHMAN OPTIMIZELY

72


Version 2

Test emphasizing return policy on product pagesYourtestinggoalistofindthebestwaytopromoteeasyreturnswithoutdistractingshoppers from completing their purchases. Finding this balance will drive higher con-version rates by addressing the central-most fears present in users minds as they gift-shop online.

Test holiday hero messagingHero messages garner a lot of attentioneven more so during December, as more shoppers head to your site for last minute gifts and deals. Hero messages are a great place to run quick and simple tests to see which ones increase sales.

Try adding a countdown clock to increase conversionsIf you are running a promotion with a deadlinefree 2-day shipping for Christmas, for exampletry adding a countdown clock that urges visitors to purchase before time is up.

Gauge interest in a new product with a dummy button and clicksPlanninganewseasonalorpermanentofferingforyourbusinessrequiressignificantpreparatory research to determine the market opportunity, costs, and the projected return on investment (ROI).

Instead of sending a customer survey, or soliciting feedback on your social media chan-nels(allgoodmethods),whynotrunananonymoussurveyatscaleusingyourwebsite?You can run a simple test using a homepage or product category page on your website to track interest in the form of clicks from your audience.

Test the impact of badges on product thumbnailsWhile browsing the sea of products on your site, customers can get lost in all of the op-tions. Test adding Best Seller or Featured badges to your product thumbnails on the category or browse pages to help these products stand out. In many cases, less is more for e-commerce retailers because options overload could lead to visitor fatigue resulting in lost reve

Documents

AB Testing for ECommerce LemonStand v2