Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)

Video A/B Testing Deconstructed: What? How? Where?

(with REAL Data & Examples)

#WistiaFest / @acapland


Hey, I’m Andrew

• Lead the Growth team at Wistia

• Huge marketing & data nerd

• Surfer & snowboarder

• 90’s hip-hop fan


I like testing things


I testing things



We test things, like CTAs


And page layouts


And value props


And self-guided tours


Then we redesign and do it all again


testing worksBecause marketers are crazy


WHY aren’t marketers testing

with video?


Video is more compelling than static content


SCREEN SHOT OF MY TALK


That’s my post!

ads

WHY?



Video is a powerful tool that humanizes your brand, and compels

your viewers to take action

Marketers don’t understand the conversion power of video


Action Increase signups from website

Drive product activation Accelerate sales process

Decrease churn

Emotional Storytelling

Humanize brand Establish connections

Let people get to know you


Action Increase signups from website

Drive product activation Accelerate sales process

Decrease churn

You can easily test and measure these!


You need to be testing your videos

Two main ways to a/b test with video


Video content Video placement

#WistiaFest / @twitter-handle

1. Why should I test my videos?

2. Where should I test my video?

3. Video testing lessons from Wistia

Let’s dig in more:


1. Why should I test my videos?


This is the Wistia creative team

Danielle

Dan

Liat

Trevor

Chris


They make awesome things

Danielle

Dan

Liat

Trevor

Chris


Hey we’re redesigning our About Us page,

should it have a video?

Of course, let’s make it weird. Everyone

loves weird!

Oh yeah! That would be sick.

Let’s put it on the top of the page.

Let’s film it with our new lens. It’ll

look awesome. Oh, I‘m jacked up!


About Us page


80’s themed pizza party to celebrate!

#WistiaFest / @acaplandJAN FEB MAR APR MAY JUNE JULY

Conversion)Rate

Ruh roh

Conversion Rate



Was it this?

#WistiaFest / @acaplandJAN FEB MAR APR MAY JUNE JULY

Conversion)Rate

Too late

Test to avoid unexpected results


JAN FEB MAR APR MAY JUNE JULY

Conversion)Rate

More please

Test to understand what works


LEARN:

WHAT WORKSWHAT DOESN’T WORK


Don’t embed it and forget itSee Margot de Cunah’s talk

Don’t publish videos and just

hope for the best


We get 30k monthly visits to our page, but few signups. I wonder

how we could get more?

And what video might get us the most signups?

Hmmn. I wonder if a video would help?

We could film a few different versions? Oh, I am jacked up!


1. Set clear goals 2. Brainstorm 3. Rank 4. Execute (quickly) 5. Iterate


Sometimes your most boring ideas are the most effective


You need to be testing your videos

Learn wicked fast


2. Where should I test my videos?


Hint: not everywhere


Test in actionable places where you can learn


Also, somewhere that gets a decent amount of traffic


Product page(s) Email

About us page Pricing page

Onboarding flow Homepage


Optimizely Sample size calculator - wi.st/sizecalc

A/B confidence calculator - wi.st/statsig

How to run an a/b test


3. Lessons we learned from testing video at Wistia

ReminderThere are two main ways you can a/b test with video


Video content Video placement


1. Set clear goals 2. Brainstorm 3. Rank 4. Execute (quickly) 5. Iterate

Remember the system


Test 1: Welcome to Wistia page


Test 1: Is this the best video for our goal?


Test 1: Version A


Test 1: Welcome to Wistia page

Hypothesis: Teaching people about

the tool will increase account activations.



Test 1: What do you think?

8% increase in activations


Well, like duh!


Learnings

Straightforward educational content is more helpful

(for new users)


Test 1: Educational follow-up

Hypothesis: Highlighting product

differentiation will increase activations


Test 1: Follow-up



3% decrease in activations


Learnings

Teaching how to use tool more helpful than highlighting features

(for new users).


Test 2: Wistia product page


Test 2: Should this page have a video?


Test 2: Automated Emails


Test 2: Product Page

Hypothesis: Static content above the fold may be more

effective.



9% increase in signups!


Test 2: Learnings

New visitors may want to browse quickly on the

product page


Test 3: Onboarding flow


Test 3: Is this the best video?


Test 3: Onboarding flow

Hypothesis: Sharing how to use all

the features will increase action with

new users.




30% increase in usage!


Test 3: Learnings

Reinforced our “how to” learning. Showing better than telling.

Context is king.


You should be testing your videos

Everyone wins!


RECAP

1. Set clear goals 2. Test to learn faster & understand impact 3. Test where it’s actionable (not everywhere) 4. Test your way to success


Use technology to find hidden gems

Collect data (Google Analytics to ID high traffic pages) Watch users (FullStory, Inspectlet)

Survey visitors (Qualaroo, Intercom)


Optimizely Sample size calculator - wi.st/sizecalc

A/B confidence calculator - wi.st/statsig

A/B Testing Resources


Thank you

Ask me questions anytime:

[email protected] @acapland

Marketing

Video A/B Testing Deconstructed: What? How? Where? (with real data and examples)