Upload
lneosr
View
11
Download
0
Embed Size (px)
DESCRIPTION
An ebook by Optimisely on building the data culture.
Citation preview
How to identify and use the right metrics to delight your customers, engage your audience, and optimize your business.
Building your Company’s Data DNA
Building Your Company’s Data DNA 2
Table of Contents: Section 1: Introducing Data DNASection 2: Finding your Company Metrics Understanding and Measuring Goals
Metrics by Business Type
Section 3: How to Take Action on your Metric Quantitative Data (with contributions from KISSmetrics)
Qualitative Data (with contributions from Qualaroo)
Section 4: Building Data into the Optimization Flywheel Six Steps of Optimization
Craft your Experiment Hypothesis
Assembling your Data DNA
Section 1: Introduction
Building Your Company’s Data DNA 4
There’s no question: data is here to stay. The promise
of data, analytics, and web-based tools make the task
of building a business online accessible and afford-
able. Access has created a new challenge: volume.
Leveraging the entirety of a business’ data on an
ongoing basis is tremendously valuable, but requires
equally tremendous resources, beyond the reach or
scope of many businesses’ technology investments.
Given these challenges, how can you make intelligent
decisions, informed by data, to grow faster?
You can incorporate data to inform decisions you
make about growing your presence online. At each
juncture, you have choices to make—how should I
describe my product to my customers? How can I col-
lect more email addresses to grow my audience? How
should I roll out a new feature and collect feedback?
A clear understanding of metrics and the right type
of data at every level of an organization are essential
for focus. Data is also key to uncovering customer
insights and running a more effective optimization
strategy, but should be properly aligned from the
top-down in order to achieve success.
“The best data-driven companies don’t just passively store and analyze data, they actively generate actionable data by running experiments. The secret to getting value from data is testing, and if you’re looking to grow your online business, implementing well-executed, consistent A/B testing is a necessity.”
—Wyatt Jenkins, Shutterstock*
Access to Data Increases Expectations
In a recent survey, 78% of all marketers
report feeling pressure to become more
data-driven.* The challenge is to choose
which measurements of online perfor-
mance are essential to a data-driven
strategy.
*Teradata Data-Driven Marketing Survey 2013
*”A/B Testing and the Benefits of an Experimentation Culture,” Harvard Business Review.
Building Your Company’s Data DNA 5
How do I know if this guide is for me?
In this guide, we’ll outline an approach to transform-
ing data into action at every level of your organiza-
tion.
You’ll benefit from this guide if:
• You’re already capturing and reporting on data regularly internally
• You have a team or company culture that is open to experi-mentation
• You’re looking for methods to turn your company data into action
• You’re looking to get more meaningful ‘wins’ from your optimization strategy
Here’s what you’ll learn:
• Why becoming data-driven as an organization should start with your optimization programs
• How to align your organization around a failproof guiding light metric
• How to prioritize your optimization strategy against your shared company goals
• How to move past random, ad-hoc A/B tests to an optimi-zation process that produces wins (and if not winning tests, valuable insights derived from a hypothesis)
Being Data-Driven is Tied to Performance
Companies that rate themselves sub-
stantially ahead of their peers in their
use of data are three times more likely to
rate themselves as substantially ahead
in financial performance, according to
findings from the Economist Intelligence
Unit.*
*Tableau Software & Economist Intelligence Report:
“Fostering a Data-Driven Culture.”
“Low hanging fruit tastes great, but the rest is worth the work, too.” —Patrick McKenzie, Kalzemeus Software
Building Your Company’s Data DNA 6
In this guide, we’ll cover many different types of
data. The important point to remember is that
the ideal of being ‘Data-Driven’ is actually to be
‘Data-Informed.’ Data points are an important
set of inputs, but they are not a replacement for
human intuition and judgement. This guide will
provide a list of inputs that are strong positive
indicators for how to improve your company goals
and optimization program. What data points
cannot provide, however, is input on what your
business metrics should be, or what initiatives you
want to prioritize.
What is A/B Testing?
A/B testing is a simple way to test
changes to your page against the cur-
rent design and determine which ones
produce positive results. It is a method to
validate that any new design or change to
an element on your web pages or mo-
bile apps is improving your conversion
rate. It is the assumption that you have
never reached the ‘best’ version of your
website, mobile app, or product, and that
the best method of learning about your
customers is through iterative testing and
experimentation.
What is Optimization?
Optimization begins with broad, sweep-
ing changes and tests that encompass all
of your website traffic, but can become
more granular, by running A/B tests to
understand how changes affect different
types of visitors, and then adapting site
content to eventually deliver a more per-
sonalized experience for each individual.
Section 2: Finding your Company Metrics
Building Your Company’s Data DNA 8
You’ve probably wondered at some point: “What’s
the fastest way for me to create business impact?”
We could frame this question with two others:
1. “What’s the fastest way to make a measurable business impact?”
2. “How do I define success?”
To reach the goal of becoming a data-driven organization
or a data-informed individual, we recommend striving
for one metric that defines the goals of your business.
When you choose your ‘guiding light’ metric, ask
these questions to determine whether it will stand the
test of time, and generate strong optimization results:
• Is it quantifiable? Your metric should be measurable and understandable at-a-glance.
• Is it visible and clearly communicated? Your metric should serve as a reference point in conversations.
• Is it understandable? It should require very little explanation.
• Is it comparative? You should be able to compare the metric to benchmarks, like your competitors’ performance or other points in time.
• Is it a rate or ratio? Showing a rate of change is more illustrative than an absolute number.
• Does it change your behavior? If the metric you choose doesn’t enact change, you need a different one.*
• Is it attributed to revenue? For many people in organiza-tions, this measurement is what communicates above all else. If you have difficulty attributing your efforts to value for the business, you may struggle proving ROI.
Many analytics experts have discussed
the value of identifying a single metric,
which should be carefully chosen and
then rigorously tested. We recommend
Lean Analytics by Ben Yoskovitz and
Alistair Croll.
Use these questions to test and strengthen your metric statement.
Before you take any actions to improve
your metric, ensure that you have tested
it internally to ensure that it will with-
stand scrutiny and testing over time.
*Questions 3-6 reference Ben Yoskovitz and Alistair Croll, Author of Lean Analytics
Building Your Company’s Data DNA 9
Having a singular goal is helpful not only for orga-
nizational focus, but it will be invaluable for more
specific tasks, like your optimization efforts. In order
to run more intentional, ROI-positive A/B tests and
experiments, you need a guiding metric that you can
brainstorm around and prioritize against. We’ll cover
the data inputs that can inform these types of tests in
Section 3.
“What gets measured, gets managed.” —Peter Drucker, Business Management Thought Leader
Building Your Company’s Data DNA 10
Understanding and Measuring Your Goals
What do you do on a day-to- day basis to grow your business?
Use this framework as a starting point for determining what your priority should be for your business, and by association, your online marketing and other customer touch-
points. What parts of your website directly serve the metric you are working to improve? Think of these areas as prime candidates for optimization.
What does this activity help you accomplish?
What is the priority for each of these activities?
What metrics do you currently report on on a regular basis?
If you were to pick one activity and corresponding goal to work on for the next 1-3 months, what would it be?
Which of the metrics from Ques-tion #4 help you to understand the progress you are making on the answer to Question #5?
Which of the metrics from Ques-tion #4 could you live without?
Essential
Important
Nice-to-have
1.
2.
3.
4.
5.
6.
7.
Building Your Company’s Data DNA 11
If you: You might use your website to: And could be measured as:
Sell products or services
(E-Commerce, Platforms)
Collect orders for your product
Distribute and engage your content
Capture, engage, and retain users
Time on siteArticles viewed
Votes or comments generatedSocial traffic
Signed up usersPaying customers
Up-sold customersChurned customers
Number of orders over timeAverage order valueRepeat customers Conversion Rate
Publish content(Media, user-generated content
like Reddit and Wikipedia)
Develop a web-based tool
(SaaS or Mobile app company)
Metrics by Business Type
Good candidates for top metric statements:
“I measure conversion rates from free trials to customers because my most important metric is acquiring new customers.”
“I measure average donation value, because my most important metric is dollars raised during our fundraising campaigns.”
“I measure conversion rate on my homep-age, because that is where the majority of my new leads are acquired.”
Not as strong metric statements:
“I measure pageviews to product pages on my website.”
“I measure social shares.”
“I measure the amount of traffic to my pricing page.”
These statements reflect strong metric statements sup-
ported by a clear rationale. Strive to consolidate and
strengthen your overarching metrics so that everyone can
understand and tailor their priorities to match them.
These metrics are not supported by a strong rationale, and
are more likely to be vanity metrics. They are not enough (on
their own) to justify prioritizing for optimization.
Section 3: How to take Action to Improve your Metric
Building Your Company’s Data DNA 13
You may be wondering: how does eliminating metrics
help me use data? The answer lies in the touchpoints
you have with your customers. Once you have identi-
fied your overarching goal, you will be in a position to
identify touchpoints that can be improved in service
of the corresponding metric.
In order to take action that will improve the outcome
of your metric, it is important to effectively identify
high-value areas of your website, product, or other
customer touchpoint. We will explore the best ways
to gather information about these high-value areas
later in this section.
Each business metric can be broken
down to apply to different levels
and functions within your orga-
nization. A team should plan
to adopt a secondary metric
that supports your top-level
company goal. These levels
may resemble the diagram
to the right:
At the most granular level,
your website and product
goals can be supported by
running experiments in an
attempt to improve the con-
versions that increase perfor-
mance, which in turn affect your
highest-level goals.
Hypothesi ze Tests to Increase AOV
Building Your Company’s Data DNA 14
How do you measure the success of these facets of
your online business? Your efforts to present your
business online and capture interest, engagement, or
revenue require conversions.
Take stock of the conversions that support your
top-priority metric. These areas are prime candidates
for optimization, where you’ll run experiments and
A/B tests in an effort to improve conversions.
Getting started with optimization is not the most
difficult step to success; the challenge will present
itself in the struggle to keep continuously A/B test-
ing to find improvements, uncovering winning tests
consistently.
In the following section, we will explore how to use
a combined qualitative and quantitative approach to
build your Data DNA into your website and app per-
formance. Then, we’ll discuss how to funnel these
data points into experiments that uncover improve-
ments and build your data-driven approach to opti-
mization in Section 4.
Using Quantitative Data to Determine which Areas of your Site to Test
Your website analytics are an indispensable tool for
determining the strengths and weaknesses of your
website. Leverage your website data in order to effec-
tively prioritize and brainstorm the best experiments
to run on your website.
What is a conversion? A desired action
taking place on your at a customer
touchpoint—a click, or registration, or
product action, for instance.
Your conversion rate is the current pace
at which website visitors take action at a
given touchpoint.
Conversion rate optimization and A/B testing describe the practice of experi-
menting with elements to try to improve
a conversion rate.
Mapping Your Company’s Data DNA 15
KISSmetrics Contributed Section:
Before you can find ways to accelerate your acquisition and growth, you’ll need to start tracking your core funnel.
What’s your Core Funnel?
It’s slightly different for each business but it measures how many people move through the major steps of your acquisition. Don’t worry about tracking every little click, page, or action that people take to become a customer. Focus on the core steps.
Let’s look at a few examples.
SaaS (subscription as a service) businesses drive traffic, convert that traffic into free trials, get those free trials to use their prod-uct, and convince people to purchase an ongoing subscription to the software. So a SaaS funnel is built from these steps:
Using Funnels to Find Big Wins and Accelerate Growth
SaaS Funnel
1. Visited site
2. Signed up for free trial
3. Used product
4. Purchased subscription
Mapping Your Company’s Data DNA 16
An ecommerce funnel looks pretty similar:
For whatever business model you have, break your acquisition
down into 4-5 core steps that everyone goes through.
Tracking your Funnel
Ideally, you’ll have a customer analytics tool that can track people as they move through the different steps of your funnel. When you get to the point where you’re doing a lot of optimiza-tions on your funnel, have plenty of traffic, and acquire custom-ers consistently, start looking at customer analytics tools that will help you track the entire funnel.
Even if you’re not there yet, you still want to get a general sense for how your funnel performs. Google Analytics and some inter-nal tracking will get you pretty far.
E-Commerce Funnel
1. Visited site
2. Visited product detail page
3. Added product to cart
4. Entered checkout
5. Finished checkout
Mapping Your Company’s Data DNA 17
Ecommerce companies have it easy by setting up ecommerce tracking in Google Analytics. This tells you how many people purchase and their conversion rates. You’ll also want to config-ure custom events to see how many times products get added to a cart or visitors enter your checkout.
For SaaS, set up Google Analytics and trigger a goal when people start a trial. You’ll also want to build out some internal tracking that tells you how many new accounts you acquired, how many of them started using the product, and how many of them purchased. Add this data to your internal customer data-base and query it every week or so to see how you’re doing.
Don’t worry about having perfect tracking when you get start-ed. Your main goal is to have a general idea for how your funnel performs.
Optimizing your Funnel
As soon as someone’s ready to start optimizing a funnel, they usually pick a random element on a random step. It might be a button color on a call to action, a snippet of copy, an image, or a layout that just feels “off”. Then they’ll launch an A/B test if they have enough traffic or maybe they’ll just launch the change and hope for the best.
After a few rounds of this, you’ll find that your conversions are exactly the same as where they are now. They won’t budge an inch.
Why? Because it’s incredibly difficult to find tests that really make a difference. Most online marketing “best practices” give 2%-3% wins. Not only is it incredibly hard to detect small wins like this (you need a ton of data to test on), small wins aren’t useful to a business. They only matter if you can find a lot of them in a short time period. Which, again, requires large amounts of data.
Mapping Your Company’s Data DNA 18
So how do we find bigger wins consistently?
Start with qualitative data on the step of your funnel that you struggle the most with. Qualitative data is feedback from users or customers that isn’t a metric. Surveys, customer interviews, feedback forms, usability studies, and heatmaps are all qualita-tive data.
It’s common to have one step that is a major roadblock to acquisition. Focus on collecting feedback from your customers at that step of the process. Ask them why they don’t want to purchase, why they didn’t use the trial, or why they didn’t finish the checkout.
Asking for targeted feedback on an underperforming step of your funnel will give you plenty of ideas on what to test. There’s never a 100% guarantee of finding a big win but you’ll definitely find them more frequently than if you test random elements across your funnels.
Mapping Your Company’s Data DNA 19
By examining your website analytics, you will be able
to pinpoint valuable areas of your site that are in need
of improvement. If you are successful in running A/B
tests and experiments to improve these key areas of
your website, you’ll be one or more steps closer to
moving that ‘guiding light’ metric, and shifting your
company towards a more data-driven approach to
solving problems.
Website analytics have a tremendous number of ben-
efits for structuring your optimization strategy:
• Quantifiable, high-value traffic segments
• Common goals that sync across your website analytics to your website optimization platform
• Ability to scale and analyze website data across segments, cohorts, and more
Quantifiable analysis of your website is only one part
of the data equation. To complete the picture of how
your visitors are converting and engaging (and more
importantly, why they aren’t), you’ll need to supple-
ment your quantitative data with qualitative insights
to enhance your data DNA.
Using Qualitative Insights to Determine which Areas of your Site to Test
Qualitative, anecdotal feedback is an effective way to
incorporate data into your business at any stage. This
data is essential because it comes directly from your
visitors and users.
Mapping Your Company’s Data DNA 20
This information is essential to determining why
certain events occur on your website. When quantita-
tive data provides insight into what happens (or isn’t
happening), these qualitative data points can illus-
trate why. Why did that customer buy? What brought
them to your website in the first place? What do your
visitors expect from your product or offering?
Qualitative insights come in many forms, including,
but not limited to:
• User testing: For a fee, a person unfamiliar with your website will com-plete a task on your website while providing their thoughts and feedback. There is a bias inherent in this qualitative data, since the person knows they are being tested.
• On-page surveys: Use these surveys to collect ‘in the moment’ feedback from visitors to a website. The more questions you ask, however, the lower your completion rate is likely to be.
• Heat mapping: These tools will show which areas of a given web page attract the most ‘heat,’ or attention, from clicks and scroll-ing on your site.
• Long-form surveys: Often delivered via email, these surveys can be used to collect in-depth feedback on your company’s positioning, perception from customers, and some of their most valued offerings or products from your business.
“A solid test hypothesis is an informed solution to a real prob-lem – not an arbitrary guess. The more research and data you have to base your hypothesis on, the better it will be.”
—Michael Aargaard, Conversion Rate Optimization Expert
Mapping Your Company’s Data DNA 21
• Feedback from cancellations, returns and complaints: Take your customers’ constructive feedback, and decide what visible changes to your website could help to alleviate pain points and frustrations they might be having.
• Live chat feedback: Uncovering pain points and visitor intent will be key to generating winning experiment ideas.
• Customer feedback: Anecdotal feedback from engagements with customers on calls, at events, and in other formats.
You can leverage these tools (many of which have
affordable and easy-to-use plans for businesses)
to collect feedback directly from the visitors to your
website or customers. You must hone in on motivat-
ing factors, however, in order to collect meaningful
feedback.
Building Your Company’s Data DNA 22
To ask questions of your website visitors and collect more data points about what you should optimize, practice asking the right questions.
Here are two types of questions you should ask:
1. Intent - What did your website visitors come to your website looking to find, or do?
2. Frustration - Could you find what you were look-ing for? What was broken?
Consider the types of questions you could ask to understand key parts of your website funnel:
On your homepage:• What did you come to our website today hoping to accom-
plish?
Before a key conversion:• Do you have any additional questions about this product
or service? Determine what information your visitors are looking for before they purchase.
• Is there anything stopping you from completing this order? Perhaps the visitor needs more information to make their decision. Maybe an unexpected bug with your checkout flow is preventing them from converting.
After a key conversion:• What made you purchase/sign up/donate to us today?
On your content site:• What topics would you like to see us cover more? You could
make this multiple choice or leave as a free response
• Would you like to see more recommended content on this topic? Determine the best way to design recommended content modules for your content.
AAA: Ask your Audience Anything
A few tips to get even better data from your qualitative research:
Be polite: You’re human - so are your
website users. Ask the questions in an
approachable, visitor-friendly manner.
Phrase questions with a “What do you
think of your website? Kindly tell us:”
approach.
Ask for honesty: Your website visitors
are also nice people. It’s sometimes dif-
ficult to provide constructive feedback.
“We’d appreciate your brutally honest
feedback as we work on improving our
website experience for you.”
Keep questions open: Try to avoid ask-
ing questions like, “What was frustrating
about your experience today?” or, “What
do you think of our [product or service]?”
These questions predicate a certain type
of response, and will bias your visitors’
responses.
Ask visitors to self-select: You can
also learn more about your audience by
asking them about their interests, their
business type, or products and services
that they like to use. This will help you to
segment your web traffic and experiment
with running more targeted tests.
Building Your Company’s Data DNA 23
Now, it’s your turn: What are the steps (or clicks) in your website funnel, and what questions would you ask at each step to reduce friction and improve your
visitors’ experience?
?
?
?
?
Mapping Your Company’s Data DNA 24
What do you think is the most common mis-conception about Conversion Rate Optimi-zation and A/B testing?
The number one misconception is that A/B testing is simply about running a test here and there and hoping for improved results. Without an optimization process that focuses on continual improvement A/B testing often fails to live up to it’s promise. Without organizational rigor to make A/B testing a priority, companies give up failing to see early wins, which ultimately costs them the long-term gains that come from a systematic approach to optimization.
Do you think that most companies online are effectively communicating with their customers? Why or why not?
Most companies lack a process for regularly collecting and then taking action on user feedback. Users provide feedback in numerous ways, from bouncing off of web pages, to taking surveys, leaving reviews, filling out customer support tickets and posting on social media. With all of this feedback com-ing in, you’d expect companies to be constantly processing it and using it to improve the visitor experience, and ultimately, their business. But more often, feedback is triaged to manage customer complaints rather than used for true learning and
Online Success and Optimization: Why Quantitative Data isn’t Enough
Q:
Q:
A:
A:
Interview with Startup Marketer and Qualaroo CEO Sean Ellis
Mapping Your Company’s Data DNA 25
business improvement. The most successful companies have a process for collecting, parsing and using customer feedback to improve their business.
What is the best way to get actionable, qualitative data from a website survey?
The single best way to get actionable qualitative data from a website survey is to use them to try to understand specific user behavior. For example, using a website survey on pages that have high bounce rates, or pages within your conversion funnel have high drop-off rates, can give you insights right from the customer that help you understand why they’re leaving. When you understand why a behavior is happening, it’s much easier to take action on the feedback and try to change the behavior.
Do you have any tips on how to choose which feedback should be incorporated into your testing pipeline?
Feedback from qualified people is the most important. The people that are potential customers who aren’t converting are the ones you want to focus on. Ignore the people who aren’t qualified or interested in what you’re offering from the begin-ning. For example asking people who converted a question such as “What almost stopped you from signing up?” or “What made you decide to sign up?” helps you understand the needs of qualified visitors. This qualified feedback will help you sort through the data from exit surveys that include both qualified and unqualified responses.
What would you tell someone who is look-ing for help creating strong hypotheses for their tests?
Q:A:
Q:
A:
Q:
Mapping Your Company’s Data DNA 26
There are two ways to create a hypothesis. In the first case you can look at the data and then spend hours or days with your team trying to interpret what the data means and what you should test next. You can create hypotheses from your interpre-tation and from your team’s opinions. Or you can ask visitors to that page what the actual problem that they’re encountering is. No need to interpret data, just ask visitors and get immediate feedback. I believe asking visitors what issues they’re encoun-tering on your site, conducting user research, is the best way to formulate strong hypotheses that make for valuable tests.
How do you think companies should ap-proach staying focused when it comes to optimization? How do you measure the progress of your CRO program?
Having the organizational rigor to stay focused on conversion optimization is the hardest part of this process. A/B testing is not a one-off project—rather it is a continuous process of improve-ment that needs to be in motion at all times. Econsultancy reports that 87% of companies doing A/B testing run between 1 and 5 tests each month. The best companies run many times that number.
Companies need to commit to A/B testing as a core part of their digital marketing program and invest in it accordingly. Stick to the process and eventually it will become an addictive habit, with the organization constantly trying to outdo its previous test. Systematic, ongoing A/B testing is going from a compet-itive advantage to a competitive necessity. It’s a key require-ment for online success today.
A:
Q:
A:
Building Your Company’s Data DNA 27
At Clearlink, the team asked the question: “How can
we see inside the minds of our customers and be sure
we are offering what they need?” Despite following
testing best practices, the team has frequently en-
countered unusual or contradictory A/B test results.
Utilizing qualitative data gives them insights into
these results and helps them better understand and
explain when met with this challenge.
They hypothesized that using a qualitative insights
tool in conjunction with their optimization software
would enable them to better serve their customer
needs through well-researched and well-executed
tests, resulting in the ability to integrate and better
understand tests’ success and failure, and ultimately
leading the to understand our market better.
While many times quantitative metrics might be interesting, they can-
not tell the entire story when it comes to the customer’s path. In short,
we know the ‘what’, but not the ‘why.’ For these issues we have to turn
to qualitative research and customer insights to make sense of the
customer needs and dive in deeper. This is how we innovate; this is
how we iterate; this is how we make sense of the larger picture.
—Rachel Johnson, Consumer Insights Research Manager, Clearlink
How Clearlink Built Qualitative Insights into their Data DNA Content and conversion services provider, Clearlink,
uses survey software* to foster its data-driven ap-
proach to engaging with its audience and collecting
qualitative feedback from its online audience.
*Clearlink implemented Qualaroo survey software for the purpose of these experiments.
Building Your Company’s Data DNA 28
By using on-page surveys to offer time-sensitive
promotions, the team was better able to serve more
relevant promotions to their website visitors in a
more time-effective manner. This led to overall
higher conversion rates for their sites:
• Stand-Alone vs. Bundle Pricing: The team learned from
survey results that stand-alone pricing was what Clearlink’s customer was looking for. After testing stand-alone pricing messaging against bundled pricing options, stand-alone pricing won with a conversion rate improvement of +5.4%, contributing to a testing win of $41,500 in annualized revenue impact.
• Lead Capture: The team hypothesized that by using the survey tool, they would be able to send the customer directly to their conversion funnel (the sales floor) by engaging them more quickly on the initial landing page. In addition to the phone number, they added promotional messaging (similar to display ads) to the messages to further drive conversion. Using the survey tool bypassed the normal lead time required for a custom overlay to be designed, developed, and pushed to live site, significantly increasing Marketing’s agility in responding to changing customer tastes, preferences, and demands.
Using quantitative and qualitative research meth-
ods together led to several “wow” moments for the
Clearlink team. By combining the two data streams,
qualitative and quantitative, they created an en-
hanced testing cycle, making the most of testing
wins, successes, and failures. Qualitative findings
sped up the testing cycle, and the team gained im-
pactful insights more often. Clearlink’s prevailing
sentiment is that “The more often you fail, the more
often you’ll win; so speed up the frequency at which
you fail.”
Clearlink’s mantra is: “The more often
you fail, the more often you’ll win; so
speed up the frequency at which you
fail.”
Building Your Company’s Data DNA 29
Integrating quantitative and qualitative data into the
optimization cycle creates a more effective testing
program for attaining website goals. Although tak-
ing the time to add consumer research and the cus-
tomer’s point of view can seem like the longer path,
for teams like Clearlink it ultimately improved the
team’s feedback loops and led to insights that previ-
ously might have gone undiscovered.
At this point, you have used data to inform your
strategy from your top-level business goals down to
your website and product-level goals. Now that you
have a sense of what could stand to be improved, let’s
turn to brainstorming and prioritizing your new ex-
periment ideas to complete your Data DNA sequence.
Section 4: Build Data into the Optimization Flywheel
Building Your Company’s Data DNA 31
You’re well on your way to running high-impact,
ROI-positive experiments to improve your customer
touchpoints. The next step is to distill the data you’ve
collected and begin to take action. In this process,
you will funnel your quantitative and qualitative data
points into concepts for A/B tests that can be run as
experiments on your website to improve key conver-
sions and support your business goals. We call this
six-step framework the “Optimization Flywheel.”
In order to see success with this approach, you
should:
• ●Build a rigorous practice of consistently collecting quantita-tive and qualitative data about your website funnels.
• Maintain visibility of your organization’s overarching metric. If your focus as a company changes, your website goals and individual experiment objectives should change accordingly.
• Maintain a sense of experimentation as broadly as possible across your company. Any visible changes to your website help to understand what works best for your customers and visitors. The pursuit of this understanding should support the goals of the company as well, as discussed in Section 2.
Building a Data-Driven Optimization Strategy 32
What is the Optimization Flywheel?
Optimization is a continuous cycle. In order to ensure that you
are responsive to the intent and frustrations experienced by your
audience, you can leverage optimization to run experiments and
discover improvements to your website.
There are six steps of the Optimization Flywheel:
Building Your Company’s Data DNA 33
Step 1: Define Goals
As defined in Section 2, it is important that you begin
your optimization process clearly aligns to your busi-
ness goals. A clear understanding of the metrics you
are optimizing for will help with prioritization and
enables continual iteration and learning from exper-
iments.
Step 2: Determine Optimization Points
Identify a step in your funnel that is a prime candi-
date for optimization. It might be your homepage call
to action (CTA), your campaign landing pages, your
checkout flow, or your recommended content.
Make sure to choose an area for optimization that has
a direct correlation to your business goals. The quan-
titative funnel analysis in Section 3 will provide con-
text for which areas of your website can be improved.
Will optimizing this step of your website experience
create a measurable change to the metric you identi-
fied in Section 2?
Step 3: Hypothesize Improvements
A strong hypothesis about how your experiment will
perform is core to running a winning A/B test. Your
qualitative data collected in Section 3 will provide
clear indicators of what should be tested to better
match your visitors’ intent and solve their frustra-
tions with your product or website. We recommend
you take the time to collaborate on hypotheses with
*Reference “Craft your own Experiment Hypothesis” for more detailed instructions on hypothesis creation.
Building Your Company’s Data DNA 34
your team, and properly document them along with
the data that you used to inform them.
A hypothesis is the following type of statement:
Hypothesize what types of changes to your website
would produce a positive change in conversions.
Step 4: Create Variations
When setting up an A/B or multivariate test, you
must develop a variation of your website feature that
you would like to test against the current version.
Use your hypothesis and your understanding of your
analytics and qualitative data to isolate a variable that
can be changed. The fewer the variables, the more
straightforward your test will be. Changing multiple
variables at once is possible with multivariate testing,
but complicates the test and confounds the hypothe-
sis, making it difficult to simply prove or disprove.
If ________________________ , then _________________ will occur.*
[I make this change to my website]
[the following outcome]
“It’s about coming up with your hypothesis first and then develop-ing a test from there. Rather than saying, ‘I want a test button cre-ated,’ or ‘I want to try two different layouts,’ you create a hypothe-sis first, for example, ‘I think green buttons attract more attention and therefore will have a higher clickthrough rate.”
—Michael Burk, Senior Online Product Manager, Electronic Arts
Building Your Company’s Data DNA 35
Once you have isolated your variable, create varia-
tions of your website based on the modification of
that variable.
Step 5: Run Tests
Prepare an experiment using your A/B testing soft-
ware of choice. For more detailed instructions on how
to set up and run an experiment using Optimizely,
visit our Knowledge Base.
Step 6: Measure the Impact
Allow your experiment to run until it has reached a
statistically significant result. Did your variation win,
lose, or draw even with the current variation?
If the test is a win, congratulations! Your hypothesis
was correct, and you can continue to build upon that
test by applying the data-informed learning to other
areas of your site, or testing another more advanced
hypothesis.
At this point, make sure to extrapolate the value of
your improvement to the website across all of your
traffic over an extended period of time. What will
your lift translate to in sign-ups, orders, et cetera
over the course of a year?
If your test was a draw or your variation lost, investi-
gate why that might be. It could be the case that your
hypothesis needed additional research, or that you
didn’t account for a behavior or event that skewed
your test results. In the event of a losing test or a
Building Your Company’s Data DNA 36
draw, it is still possible to gain additional insights
that you had not anticipated through further analysis
and discussion with your testing team. At this point,
discuss what the data couldn’t account for as you
planned the experiment. What would you do differ-
ently in your next hypothesis and experiment?
Putting it all Together
Here’s an example of Data DNA in action:
• Take an e-commerce company that wants to improve the value of each customer they acquire. This is the top-level metric that they have identified is an excellent indicator of the success of their business.
• To improve this metric, the team uses website analytics to identify the top traffic pages for a high-value customer segment: returning traffic. They identify conversion points for returning traffic that differ from other traffic types.
• To collect qualitative data, they survey their customers to understand why they may not be converting at this point.
• Based on this data, the team hypothesizes a change that could help to improve the conversion rate for these visitors. They run an experiment and to determine whether their hypothesis was correct.
Mapping Your Company’s Data DNA 37
What does a winning test hypothesis look like? It’s a represen-tation of your Data DNA on a hyper-focused level. Your qualita-tive and quantitative data collection will help you formulate a strong, testable prediction.
Hypotheses are statements, not open-ended questions. They address a question with a proposed solution. Crafting a hypoth-esis to address an open question or problem on your website enforces a well-rationalized, thoughtful proposal for how to address that problem.
To take your hypothesis even further, consider what you would learn if your prediction was proven correct or incorrect in an experiment. What would you learn in each scenario?
Craft your own Experiment Hypothesis
“IF _____________ , THEN ___________ DUE TO ____________.”[Variable] [Result] [Rationale]
The Variable: A website element that can be modified, added, or taken away to pro-duce a desired outcome.
Result: The predicted outcome. (More email sign-ups, clicks on a call to ac-tion, or another type of behavior.)
Rationale: Demonstrate that you have informed your hy-pothesis with research: what do you know about your visitors from your qualitative and quantita-tive research that indi-cates your hypothesis is correct?
Building Your Company’s Data DNA 38
Assembling your Data DNA Now, it’s time for you to take action:
• Assess your top metrics with your team and company at large.
• Take stock of your web analytics, and bring together your sources of qualitative data from your customers.
• Most importantly, take action by applying an experimental mentality to your online presence with an intentional, mea-surable approach to continuously optimizing interactions with your audience.
By incorporating data into your metrics and goals at
every level of your business, you’ll move your
organization towards a data-driven process for
making decisions and instrumenting change. More
importantly, your data will become actionable at
the customer touchpoint level, where you’ll better
engage and convert your audience into valuable cus-
tomers, repeat visitors, and users.
Building Your Company’s Data DNA 39
ABOUT THIS GUIDE Building your Company’s Data DNA Written By:Shana RusonisContent Marketing Specialist, Optimizely@srusonis
Designed By:Jon SaquingCommunication Designer, Optimizely@JSaq
Thank you to: Ural Cebeci, KISSmetrics, Sean Ellis and Morgan Brown.
SOURCES• “Teradata Data-Driven Marketing Survey,” Teradata.
• “Fostering a Data-Driven Culture,” Tableau Software &
Economist Intelligence Report.
• “A/B Testing and the Benefits of an Experimentation
Culture,” Harvard Business Review.
• Lean Analytics by Alistair Croll and Benjamin Yoskovitz.
• For more detail from Hiten Shah, Co-Founder of KISS-
metrics: “Increase your Testing Success by Combining
Quantitative and Qualitative Data,” OptiCon 2014.
ABOUT OPTIMIZELY
Optimizely is the world’s leading
optimization platform, providing
A/B testing, multivariate testing,
and personalization for websites
and iOS applications. The platform’s
ease of use empowers organizations
to conceive of and run experiments
that help them make better data-
driven decisions. With targeting
and segmentation using powerful
real-time data, Optimizely meets the
diverse needs of any business looking
to deliver unique experiences to their
visitors.
Amsterdam OfficeNes 76
1012 KE Amsterdam
The Netherlands
San Francisco Office631 Howard Street, Suite 100
San Francisco, CA 94105
OPTIMIZELY.COM/DEMO
To learn more about Optimizely, schedule a live demo today at
To learn more about best practices for people, process, and technology for a
winning optimization strategy, download a copy of our Roadmap to Building a
Data-Driven Optimization Team.
Optimization can be instrumental in improving the performance of your paid
search funnels. Learn how to maximize your marketing ROI by reading
Optimizing your Conversion Engine: Search Engine Marketing.