Embed Size (px)
Citation preview
Million Dollar Impact Through Metrics, Analytics, & A/B Testing
Hello!I am Azhar Bande-Ali
Software Engineer turned PM with a passion for people and data
You can find me at:@AzharBA
In this talk
◎ Discuss web products at scale◎ Narrow approach due to time constraints◎ Principles can be adapted to your world
Feel free to ask questions
MetricsDefine your measurement criteria
Top Down Metrics
Measure what your business cares about.
Everything else is vanity.
Place your screenshot here
E-commerce WebsiteImprove yearly revenue from online sales
Metrics for growing revenue
Conversion Rate
Metrics for growing revenue
Drop-off Rate Return Rate
Conversion Rate
% of unique sessions with a submitted order
Metrics for growing revenue
Drop-off Rate
For every page in the ordering flow: % of sessions that don’t graduate to next step
Return Rate
% of new users who repeat order
Compare each metric to your target/goal and industry average!
AnalyticsMeasure your metrics
OR
Getting started with Analytics
◎ Needs code change - development cost◎ Measure:
○ Conversion rate of funnels and ○ Drop off rates on pages
◎ Event tracking for more intricate analysis
Marry analytics to business goals
Conversion Rate
Drop off Rate
Return Rate
Measure
Frameworks
◎ Funnels
◎ AIDA○ Awareness○ Interest○ Desire○ Action
Measure
Frameworks
◎ Funnels
◎ AIDA○ Attention○ Interest○ Desire○ Action
Events
Drop-offs
ROI
How to get buy in from your boss and partners
$1,085,136How to get buy in from your boss and partners
A/B TestValidate your hypothesis
HypothesisMaking “Continue” button more prominent will improve the checkout conversion rate.
Variant 1Remove “Cancel Order” Button
Variant 2Remove “Cancel Order” button and rename “Continue” to “Checkout”
And the winner is..
And the winner is....neither!
Truth about A/B testing
Diminishing ReturnEven a successful A/B test variant will eventually start failing for a variety of reasons.
No WinnersIt is possible that none of your variants cause a big enough change to be declared winner
Success RateAccording to KissMetrics, only 1 out of 8 A/B tests have a valid winner
SignificanceSample being tested on should be big enough to represent population
Control GroupAlways show a ‘no change’ state to a control group to compare understand what would happen if you did nothing
RepresentationThe test should be conducted on the same group of users that are a sample of the actual user set
Next Steps
Org SupportEnsure that your organization recognizes the value of the opportunity enough to prioritize it
IterateBe ready to come up with several hypothesis about which features create value for users
FailureEven the best teams have a 12.5% chance of success. Know that it isn’t easy and you’ll fail a lot
