Analytics & Testing for PMZakka Fauzan

- prepared exclusively for PDC’17 -

Brief Information

´ All presented here is related to (quantitative) data.

´ However, analytics can be either qualitative or quantitative data (or both).

Analytics

´ Why, as PM, do we need to do analysis?´ Because analytics is one of MANY skills needed by a PM.

´ Here I will just explain two experiments to be analyze:´ Results of feature developed

´ Results of feature enhanced

Principle of Doing Analytics

´ Always test it, more about this later.

´ MVP! You can measure better with MVP.´ Beware of MVP, it should be Minimum and Viable both.

Principle of Doing Analytics

Easy way to understand MVP, what it should be.

Principle of Doing Analytics

´ Put the tracker everywhere so you have better analytics.´ When it’s written everywhere, it does mean everywhere.

´ “Easy” case is when seeing the whole funnel, not acquisition vs activation only (e.g., sessions vs transactions), but also sessions for each page.

´ Difficult case, when seeing a form, not only see “how many visiting the form” vs “how many finished filling the form”, but also see the drop/exit rate for each field.

Principle of Doing Analytics

´ Be skeptical

´ Be aware of correlation trap, a.k.a. cum hoc ergo propter hoc.´ When we find any correlation, make sure that it’s causation (or not).

´ Case in point, Bukalapak in 2014, #pelapak vs #trx.

´ When having a causation, there are 3 possible cases.

Principle of Doing Analytics

´ Remember of the pirate metrics of a startup´ AARRR – Acquisition, Activation, Retention, Revenue, Referral

´ There are two approaches on this:´ If you know exactly where you want to fix a thing, get focused on it

´ If you are not sure or don’t know, check every part of the metrics, fix the one having lots of low-hanging-fruit enhancement.

´ At one time (define it by yourself, be it week, 2 weeks, sprint, month, or whatever) just get focused on one or two out of those five metrics.

´ Too much focuses = no focus at all.

Tools for Analytics

´ Google Analytics

´ Google Tag Manager

´ Google Search Console (GWT) especially if your products involve search engine a lot

´ Mixpanel

´ Some analytics tool for qualitative data can as well be used if you’re executing qualitative analytics

Testing

´ Same questions for analytics skill applies here as well, PM does need testing skill as one of a lot of skills he needs to master.

´ There are other advantages:´ Being more independent from data people.

´ Getting respected. One of the easiest ways to gain respect from people around is by speaking their language. Hence, understanding the (general) language of data is important.

Types of Testing

´ Historical Testing

´ On-off Testing´ Quite similar to historical testing

´ AB testing / Multivariate testing´ Multivariate testing is just AB testing with more variants (more than two)

´ When not to use AB testing?

Some tips for doing A/B Testing´ Make sure that your A/B testing is 100%

random.

´ A/B testing should be done at least one week, because each day is unique.

´ Wait for enough confidence level (usually 90/95/99%) before deciding.

´ If it’s possible, always run the test AFTER the winner is decided.

´ Measure MORE than one metric.

Some extra notes regarding A/B Testing

´ To be able to do A/B testing properly, make sure these four property supports:´ Resource – time, money, etc

´ Tool - next slide

´ Focus – because sometimes doing A/B testing will make tasks executed a bit slower

´ People – it’s not enough to have data-mindset on only data, product or even engineering people, it needs to be spread widely on every part of the company, even to the C-levels.

Best Resources for AB Testing (in my opinion)

Tools for Testing

´ Visual Web Optimizer

´ Optimizely

´ Mixpanel

´ Growthhackers.com

´ Build it yourself!

Thank you