AB/MV TestingA Guide to Passion Optimization

By Josh Stanley

Guide Outline:• Why Testing Matters

• OKR Framework

• Objectives

• Recommendations

Why AB/MVT testing matters?

• AB/MVT is a tool for connecting people to their passion. Testing allows us to truly create specialty retail sites that best serve the customer.

• There’s big money in small changes. We saw $x million in incremental revenue last year from testing, and we only ran xx major tests.

• Pain points exist. We know pain points exist for our customers and we can identify the right way to address them.

• We don’t know what really works. On average, 20% of experiments are winners. So we’re right 1 out 5 times

• Competitive advantage is out there. The opportunity to understand our customer better and refine our strategy around what they perceive and care about is huge!

Objectives KPIs Risks

“Improve what matters”Get more people to the products they want, adding more gear to carts, and checking out more often.

• Incremental revenue gains of $XX million.

• Bounce rates – , Exit pages –• Add to cart +, AOV +• Cart Abandonment –,

Conversion +

• Late start • Lack of dedicated resources• Time for coordination of

testing with other efforts

“Iterate on lessons”Seek to understand behavior behind test results with variations of the same test.

• Increase the # of concurrent campaigns by X%

• Increase reused campaigns and versioning by X%

• Lack of a repository • Time to coordinate test

retrospectives and promote follow-up tests

“Test early and often”Create simple and effective processes to test everything, everywhere, and all the time.

• Increase the # of tests run by X%

• Reduce time to test by X%• Full portfolio testing

• Late start• Time to focus on process

improvement• Cost effectiveness of scaling

our tool

“Institutionalize testing”Make AB/MVT testing part of the culture. Bring AB/MVT testing into the daily workflow of key stakeholders.

• Increase user handling of testing by X%

• Increase backlog by X%

• Time to campaign, educate, and propagate testing.

“Improve what matters”

Idea

Competitive AnalysisData MiningBest PracticesRisk Mitigating

Hypothesis

Prediction

Req

uire

men

ts &

KP

Is

Test

Result

Home Page

PLP

PDP

Cart Page

Checkout Page

What about exit pages?

What about Customer pain

points?

Where’s the focus and strategy alignment?

Success or failure?“Decide and move on”

Current Process

Is this in strategic

alignment?

“Improve what matters”

Strategic Objective (Focus)

Hypothesis

Prediction Test

Analyze & Retest

Focus on a strategic objectiveDefine problem statementsGet “ideas” focusedWork from a backlog

HypothesisFrame for tests at the start for re-testing

PredictionAlign expectations to the objective(s)

TestBuilding tests to support variations

Analyze & RetestAnalysis with a retrospectiveRetest with new variations, rules, or thresholds to refine learning.

Ideal Process

“Iterate on lessons”

Performance Drivers (Quantitative):• What reduces bounce rates?• What drives conversion?• What increases AOV?

Behavioral Knowledge (Qualitative):• Do we understand our customer?• How do they make decisions?• Does reality match perception?

Guaranteed Value

Hidden Value

Single pass AB/MVT testing does not prove a hypothesis. It only supports the hypothesis.

So, why do we stop learning after we identify a “winner”?Why do we assume we tested the hypothesis correctly when we see a “loser”?

THE REAL VALUE PROPOSITION OF TESTING IS THE “HIDDEN VALUE”!

“Iterate on lessons”

If our goal is to get to the core of our customer’s behavior, why isn’t behavioral study apart of what

we do?Best Practices Competitiv

e AnalysisData

MiningTesting Cycles

Behavior

From the “Consumer Decisions” class I taught - based on an employee survey.

Even Backcountry employees are susceptible to cognitive biases.

“Test early and often”

Our Test TypesOptimizationChanging elements of the site to achieve “local maximum”ConceptualFeature improvements testing that’s informative, or supports “better design” hypothesisFunctionalSite elements that should be dynamic with functional ownership (i.e. targeting)

“Test early and often”

Process Principals for Velocity:• Follow agile philosophies where we can: backlog, sprints,

retrospectives, etc.• Incorporate the 3rd party into the agile process.• Test “small” when it makes sense. Get down to the essence

of the hypothesis to ensure we’re not over-scoping the test.• Make the process simple and transparent.

“Institutionalize testing”

• Functional Testing– Identifying the opportunities

to create stakeholder owned functional tests.

• Educational Campaign– Tech Series Speakers – Wisdom Wednesdays

• Testing Campaign– Workshops, Trainings– Infographics, Blogs, etc.

• Engineer the Process– Build testing into the dev

process

Online Merchandising Functional Test Concept

Allowing stakeholder to A/B/n test any messaging

in slots at any time!

“Institutionalize testing”

Areas to focus on…• Test coordination

– Understanding and resolving conflicts– Transparency into testing priorities and progress– Communicating campaigns, impacts, and follow-up

• Maintaining the backlog and functional tests– Prioritizing the backlog based on strategy and predictive value– Maintaining functional tests and resulting value– Team BL Monetate BL Creative BL BI BL SPI BL Test

• Knowledge management– Building the knowledge repository/reference guide– Coordinating knowledge share activities (workshops, retrospectives,

etc.)– Tracking and reporting testing value.

• Continuous Improvement/ Roadmap

We need a dedicated “Testing Manager”.

– Product/Project-like role– Backlog(s) Management– Department Coordination– Vendor Management– Internal Champion– Future Planning– Reporting

Mktg

VisualMerch

BI

Corp Initiatives UX

Portfolio Testing BacklogCreative, BI, and SPI Requests

BikeBacklog

OutdoorBacklog

ASRBacklog

TestingStories(CSS,

Javascript)

Content

TestingStories(CSS,

Javascript)

TestingStories(CSS,

Javascript)

Pro

duct

Man

agem

ent

Product Manager, A/B/n Testing

Recommendation