Million Dollar Impact Through Metrics, Analytics, & A/B Testing

Hello!I am Azhar Bande-Ali

Software Engineer turned PM with a passion for people and data

You can find me at:@AzharBA

In this talk

◎ Discuss web products at scale◎ Narrow approach due to time constraints◎ Principles can be adapted to your world

Feel free to ask questions

MetricsDefine your measurement criteria

Top Down Metrics

Measure what your business cares about.

Everything else is vanity.

Place your screenshot here

E-commerce WebsiteImprove yearly revenue from online sales

Metrics for growing revenue

Conversion Rate

Metrics for growing revenue

Drop-off Rate Return Rate

Conversion Rate

% of unique sessions with a submitted order

Metrics for growing revenue

Drop-off Rate

For every page in the ordering flow: % of sessions that don’t graduate to next step

Return Rate

% of new users who repeat order

Compare each metric to your target/goal and industry average!

AnalyticsMeasure your metrics

OR

Getting started with Analytics

◎ Needs code change - development cost◎ Measure:

○ Conversion rate of funnels and ○ Drop off rates on pages

◎ Event tracking for more intricate analysis

Marry analytics to business goals

Conversion Rate

Drop off Rate

Return Rate

Measure

Frameworks

◎ Funnels

◎ AIDA○ Awareness○ Interest○ Desire○ Action

Measure

Frameworks

◎ Funnels

◎ AIDA○ Attention○ Interest○ Desire○ Action

Events

Drop-offs

ROI

How to get buy in from your boss and partners

$1,085,136How to get buy in from your boss and partners

A/B TestValidate your hypothesis

HypothesisMaking “Continue” button more prominent will improve the checkout conversion rate.

Variant 1Remove “Cancel Order” Button

Variant 2Remove “Cancel Order” button and rename “Continue” to “Checkout”

And the winner is..

And the winner is....neither!

Truth about A/B testing

Diminishing ReturnEven a successful A/B test variant will eventually start failing for a variety of reasons.

No WinnersIt is possible that none of your variants cause a big enough change to be declared winner

Success RateAccording to KissMetrics, only 1 out of 8 A/B tests have a valid winner

SignificanceSample being tested on should be big enough to represent population

Control GroupAlways show a ‘no change’ state to a control group to compare understand what would happen if you did nothing

RepresentationThe test should be conducted on the same group of users that are a sample of the actual user set

Next Steps

Org SupportEnsure that your organization recognizes the value of the opportunity enough to prioritize it

IterateBe ready to come up with several hypothesis about which features create value for users

FailureEven the best teams have a 12.5% chance of success. Know that it isn’t easy and you’ll fail a lot

Thanks!Any questions?

You can find me at:@AzharBApcampatl@azharb.com