Getting Started with Product Analytics - A 101 Implementation Guide for Beginner and Aspiring PMs

Getting Started with Product AnalyticsA Workshop for Beginner & Aspiring Product Managers

March 19, 2017 (Sunday)

UpGrad Xchange, Bangalore

What is all the fuss about?Making a case for product analytics & feedback loops

“In God we trust. Everyone else must bring

along data.”Proctor & Gamble operates on this

“You cannot improve upon something you can’t measure.”

Pretty obvious no?

“Plan - Design - Spec - Build - Launch - Measure - Iterate”Typical product cycle for PMs

“Without data, you’re just

another person with an opinion.”You don’t want to fight on opinions!

PMs and data (hence, analytics) are interrelated

Data is all pervasive!

Let’s take a PM walk through the how products are developed to understand why you cannot fly blind when building a product or it features.

Associated Costs with Product Development• Time to build & grow• Money to build & grow• Resources to build & grow• Opportunity Cost (!)

often ignored

Meet the “Empathy Debt”A gauge of whether you understand your users and how they use your product

Let customers do the talking for you

Customer research, interviews, ethnography(External Feedback Loop)

Let numbers do the talking for you

Product & Marketing Analytics, Deductions(Internal Feedback Loop)

Why is it absolutely critical to get your product analytics implementation right?

“Data is the new oil. It’s only useful

when it’s refined.Jess Greenwood, Contagious

Getting it right

Measurement vs Analytics

How analytics data is being used to drive product & business decisionsReal-World Examples

Let’s all get on the same pageSome essentials before we begin…

• What will we NOT discuss about?• Data Analytics Techniques• Data Mining & Manipulation• Data Analytics Algorithms• and so on…

• What will we discuss about?• Product Analytics• Product Metrics

• User-level vs aggregate• Marketing & Campaign Metrics• Implementing product analytics

• What should you keep in mind?• Datapoints - Identifier, Attributes,

Derived Metrics, Dimensions• Definition – Sessions, Events, User• Paradigms – Segments, Cohorts,

Funnels, Experiments• Reporting and Visualization• Analysis, Hypothesis Verification• Analytics Tools• Correlation between Metrics• Misleading Metrics• Vanity vs Real/Actionable metrics

Let’s put on our thinking caps!Quiz Time

• # of Downloads• App Install Base• Total Revenue• # of Daily App Opens• Session Interval• Session Length• # of Daily Users

What do you think of these? Are these vanity or insightful metrics?

Startups (or SMBs) vs Established EnterprisesImplementing Product Analytics

STARTUP / SMB

• Starved of capital, analysts and engineering resources, hence prefer an out-of-box 3P solution

• Focus is on integration to hit the ground running

• Often different analytics tools each plugged into different properties

ENTERPRISES

• Abundance of capital and resources, hence prefer in-house, custom built solutions also owing to IP protection & tight data-sharing/privacy policies

• Often uniform analytics platform powering different product lines

Learning through a Group ActivityImplementing Product Analytics

>

Implementation Pipeline

Plan Design

>Spec

>Execute

>Rollout


Our Case Companions

• Industry: Ecommerce, Shopping• Product: Physical (Groceries)• #1 Business Goal: Purchase

• Industry: Real-time Communication• Product: Virtual, Software• #1 Business Goal: Usage (+Conversion)


Some pointers towards what to measure?

• Acquisition: Downloads, Clicks, Visits etc.• Activation: Onboarding, Login/Register• Engagement: category & product views, size select• Retention: Loyalty/stickiness, repeat visit/purchase• Referral: Viral growth, Share, Invitation• Monetization: Conversion, Subscription

• Acquisition: Downloads, Clicks etc.• Activation: Onboarding, Register• Engagement: Opens, # of chats, calls etc.• Retention: Frequency of users coming back, duration etc.• Referral: Viral growth, Share, Invitation• Monetization: Subscription, In-app purchases

Commandment #1Planning Your Product Analytics

What business decisions and actions should your analytics implementation aid?

“ “How does analytics tie up with larger business goals?

Commandment #1 (contd.)Planning Your Product Analytics

Examples of Business Goals / Decisions

• Product Usage• Effectiveness of product discovery properties (CTRs etc.)• User flow between screens

• Efficacy of Traffic & Marketing Channels• Traffic Sources – effectiveness and quality• Attribution of a sale to a traffic source

• Merchandise Performance• Product impressions, views/clicks, conversions• Product categories performance

• Communication & Campaign Management• Single tool vs different tools for marketing campaigns• Real-time performance tracking of marketing campaigns

What business decisions and actions should your analytics implementation aid?

“ “


Why does it matter?

• Product requirements• Delivery timelines• Selection of analytics tools• Infrastructure• Program management• Engineering responsibilities• Single vs multiple code bases• Sync between platform-specific PMs• Sync in terms of data points, definitions, terminology, event attributes,

reporting dashboards etc.• Unified Campaign Management• Adoption costs by stakeholders

Concept of “Earned Differences”

Does your analytics implementation span across platforms or is limited only to one?

“ “You’d most likely not be revisiting this fundamental consideration.


Become a “Business PM” at this stage - Each stakeholder will use your implementation differently as per their needs.

• Product team• Engineering team• Business Analysts• Data Scientists• Business / Revenue team• Marketing team• Digital and Growth team• Operations team • Category / Merchandise team• Customer Support team• Management & Executive Leadership team

Gather business requirements NOT data points to empower stakeholders.

PM needs to understand needs of each function to the determine:

• Data points to be collected to fulfil requirements• Aggregation, reporting and visualization to be done thereafter

Double down on gathering business requirements from

different functions or stakeholders.

“ “Listing down their requirements should come naturally to you.


Build vs Buy: Choose the analytics tool and

infrastructure best suited to satisfy your

requirements.

“ “What should you keep in mind?

• Native Platform Support• Designed to play equally well across platforms?• Support for unifying data across platforms?

• Ease of Instrumentation• How easy is it to integrate in your codebase? • Is basic out-of-box reporting possible with minimal effort? • Is the implementation scalable as your platform expands?

• Will the tool support your present & future scale? • Tier-pricing • Processing scale

Huge decision - once implemented, you’ll stick with the tool for at least a few years!


• Does it address your data processing needs? • Events and attributes are hygiene • Does it support user-level profiles? • How many default and custom dimensions? • Does it unify data signals and profiles across platforms?

• Reporting & Visualization Support• How easy is it to setup reporting in the tool? • What kind of reports are supported? • What technical expertise is required to operate the tool? • How sharply can you segment your reports? • What level of pivoting and aggregation is supported?• Does it support private/public reports/dashboards? • Does it support “smart” dashboards (eg: derived metrics?)• Does it support exporting data for offline consumption?



requirements.

“ “


• Action Orientation• Can you configure marketing interventions, campaigns,

communication (eg: emails, push notifications etc.)?• What behavioural and profile conditions can be used?• Can the action be taken in real-time?• Are there nuances to the data on which you’ll take action?

• Support for Ingesting External Data• Does the tool allow you to upload data sitting elsewhere? • Particularly useful if you’ve historic data• Also if you’re enriching campaigns from outside info

• User Management• Support for user management & different access rights? • You would not want your employees and vendors both to

have access to the same view of data!



requirements.

“ “


• Automation• Does the tool support putting conditional alerts?• Can it send an automatic summary of metrics & reports?

• Programmability and API Support• Does it support APIs to access data & run campaigns?

• Raw Data Access• Does it let you consume full raw data eg: clickstreams? • Is it provided in real-time, near real-time or deferred?

• Integration and Technical Support• Will your provider handhold you during integration?• How responsive will support be? Is there team in India?• What level of support if free vs paid?



requirements.

“ “


Back it up!

At least 2 tools (maybe one free and another paid) so that:

• You can compare and cross-check data• You can troubleshoot any discrepancies• Tools stay available to fight downtime/outages• You can compensate for missing features in each tool (for eg:

visualization capabilities vs real-time data collection capabilities)

Always back up your primary analytics

implementation with a secondary one!

“ “Never ever put all your eggs in only a single basket.

Commandment #6Design, Spec & Execute the Implementation

I can guarantee that reading documentation, user guides is worth a PM’s time to avoid getting caught unaware down the road.

Know the tools you’ve chosen inside out. Do not assume or leave

any doubt about how they work.

“ “Yes, it means reading the dev docs and user guides as each tool internally behaves little different.

Commandment #6 (contd.)Design, Spec & Execute the Implementation

It’s worth investing one’s time to understand the following:

• Definitions of Common Terms• Tools always promise a very brief integration period• It’s essential to understand (and not assume!) what the

metrics mean in the context of your tool• Sessions, Users, Browsers, Visitors, Bounce• Attribution Window, Lookback Window• Funnel definition

• Reporting, Dashboard’ing and Visualization Capabilities:• How many and what pivots do you need?• Which pivots and how many are supported in dashboard?• Can your engineers tactfully get more out of the tool?



“ “


• Attribution• A change in attribution model may result in dramatically

different RoI splits between traffic sources. • What schema(s) of attribution is supported?• Degree of configurability

• Multi-Platform Behaviour• Same tool may have different definitions of common terms

and attribution across platforms• Design and spec needs to account for data points and user

profiles that will be shared across each of your properties• How you implement can make or break such use cases



“ “

Commandment #7Design, Spec & Execute the Implementation

Some pointers to approach this activity:

• Spreadsheet it out• The tabular format lends a natural structure • No compromise on detailed spec’ing• Track engg. status against each scenario/data point

Design and spec your requirements like a Yoda. Do not leave

even minute details.

“ “Being brief or assuming here doesn’t count, but being crystal clear does.


• Single living document across platforms• Something I picked up from my stint at Microsoft• Living spec also a handy source of future reference

• Scalability for instrumentation• Extensive instrumentation of your codebase is needed• Does it make sense to introduce a standard middleware?• Decide basis the following:

• Scalability - “instrument once, use everywhere” • Ease of Troubleshooting• Ease of Correction – correct at only a single place• Extensibility – add new logic quickly & at low risk



“ “


• Translate all known use-cases into data points• Which events will you trigger?• How many & which dimensions to instrument for? • What information is required for every individual event?• Which events will be interaction vs non-interaction? • What will be the micro-funnel steps be? • Which data points will you capture per user profile? • In which context should information be captured in?

• Decide how you'd ingest external data• Which data points sitting in your own systems do you wish

to ingest into the analytics tool you’ve chosen? • How do you upload these data points to your tool and in

which format?• How frequently will you need to do this?• Do you need tool’s API integration with internal systems?



“ “

Commandment #8 Design, Spec & Execute the Implementation

Your stakeholders are your “internal customers”

• Essential to share your design - even boring, technical nuances• Call out limitations (if any) of the tool• Capabilities of reporting dashboards • Get stakeholders to sign-off before engineering picks up your

spec for implementation

Do not start your implementation

without reviewing it with stakeholders and

getting sign-offs.

“ “PMs need to behave as business manager and change managers in their organization.

Commandment #9 Design, Spec & Execute the Implementation

Become a “Technical PM” at this stage

• Doesn’t matter how much overview and briefing you provide• Your developers will almost always have lesser context than you

• Typical queries you may expect:• Source of data to be sent to the analytics tool• Exact trigger of data event• Format required for instrumenting a data-point

Engineering implementation of

your analytics spec is high-availability time

for you as a PM.

“ “Always be around for your developers to clarify ambiguity.

Commandment #10Finalizing and Deploying Your Implementation

The undisputed onus lies on the PM. The buck stops at you!

While QA engineers are generally more aware of user scenarios than developers, they will still not have as much context as a PM.Testing & validating

your analytics implementation is the

PM's job too!

“ “Is the analytics implementation ready to power reports, visualizations & campaigns?


Dependency alignment with stakeholders is an often neglected but a critical step. It’s time to ensure data inputs are clean!

Be obsessed about not polluting your

brand new analytics implementation.

“ “Align stakeholders to ensure correct data is passed on to your analytics infrastructure.


Empower all consumers of

analytics data and reports through

thorough training.

“ “A new implementation will never be 100% at par with what your stakeholders were used to. Handhold first and then let self-serve follow!


A successful rollout represents the beginning of a culture change.

• It’s unlikely you will get it 100% right in the first go

• Improving your analytics is a continuous process.

• With time and new features getting added to your product, with new business requirements and use cases, you’ll discover:

• Plenty of missing scenarios over time (eg: filters)• Edge cases not taken care of• Data corruptions you will find only at scale• Logic errors• Reporting/visualization needs that will demand design

changes in the way you’ve implemented the analytics• New requirements

Remember, enriching and fine-tuning

analytics is a continuous process.

“ “

Here’s your brownie!Time to celebrate your first implementation

Here’s an elaborate version of what we covered in 2 hours!Want to read more about the 13 commandments?

• Part 1

5 Things I Learnt While Planning Out My Product

Analytics Implementation

• Part 2

Rolling Up Your Sleeves to Design, Spec &

Execute Your First Product Analytics

• Part 3

This is how I made my 1st product analytics

implementation truly effective

https://www.linkedin.com/pulse/5-things-i-learnt-while-planning-out-my-product-analytics-shukla

https://www.linkedin.com/pulse/rolling-up-your-sleeves-design-spec-execute-first-product-shukla

https://www.linkedin.com/pulse/truly-effective-analytics-implementation-depends-how-you-shukla

Vishrut Shukla

in.linkedin.com/in/vishrutshukla

Thank You! Now get started.Feedback, comments and suggestions welcome.

Technology

Getting Started with Product Analytics - A 101 Implementation Guide for Beginner and Aspiring PMs