Upload
ilias-flaounas
View
647
Download
1
Embed Size (px)
Citation preview
The story of the Product Growth team at Atlassian
Ilias Flaounas Sr. Data Scientist
!!!!
Web Analytics Meetup, Sydney 14 October 2015
Atlassian - Trivia• Atlassian is an Australian enterprise software
company specialised in products that support project management and collaboration.
• ~13 years old
• >1.5K employees
• Headquarters in Sydney
• Best place to work in Australia (2014 & 2015)(The Australian Financial Review)
Key Requirements:!
• Support from upper management!
• Some minimum critical mass of customers!
• Serious investmentin infrastructure, people,time
Product Growth Team• 18 members (from initially 6 members
on 2013)
• Main mission: Increase Active Users!
• How? * Discover value in behavioural data * Experiment* Suggest improvements!
• Champion data-driven decision making
Reed Johnson
Ilias Flaounas
Stephen Lee
Shaun Clowes
Graeme Smith
Tim Garnsey
Fardin Sarker
Theo Voronov
Herman Chow
Leo Balan
Rob Sangster
Andrew Wakeling
Martin Smyrk
Houda Chehab
Lakshan Perera
Isabelle Flores
Antony Russel
Benjamin Humphrey
Where do we fit in the org? A kind of stretched analogy…
Patient User
Illness Bad User Experience
Medical doctor Product manager
Diagnosis Interviews, surveys…
Treatment UX modification
Medical researchers Growth teamObservations Data mining
Medical trials A/B tests
Ethics Privacy PolicyEvidence-based
medicineData-driven
development
The Challenges• Build infrastructure!
- break data silos - replace existing “solutions” - improve data quality - add instrumentation
We analyse the behavioural dataTimestamp Instance User Event Attributes
1356958862 abcd2134abcd AB12FF89 Login1356968863 abcd2134abcd DA99A11 Login1356978862 abcd2134abcd AB12FF89 dashboard view1356988862 abcd2134abcd DA99A11 dashboard view1357884442 abcd2134abcd FF12FF89 view issue {issueID = 1234}1357956862 abcd2134abcd AB12FF89 create issue {issueID = 1000}1357957862 abcd2134abcd DA99A11 view issue {issueID = 1000}1357958862 abcd2134abcd DA99A11 comment issue {issueID = 1000}1357959862 abcd2134abcd AB12FF89 create issue {issueID = 1001}
… … … … …
Analytics Infrastructure (v.2)On 2014:!
~3-4TB overall (highly compressed)!
800GB per day produced (uncompressed)
• Build infrastructure!- break data silos - replace existing “solutions” - add instrumentation - improve data quality
• Cultural change & Education!- help people ask the correct
questions!- challenge gut feelings!- “analytics is counting”
The Challenges
Help people ask the correct questions…
Help people ask the correct questions…
Report 1 Report 2 Report 3 Report 4 Report 5 Report 6 Report 7 Report 8 Report 9
No usage
No usage Low usage
No usage Low usage Medium usage High usage
Deal with gut feelings & common sense…
How Common Sense Fails Duncan Watts, 2011
κοινὴ αἲσθησις Aristotle, 384–322 BCE
Correlation & Causality
• Source: P. Chopra, The ultimate guide to A/B testing, http://www.smashingmagazine.com, 2010
Voting time: Raise your Left or Right hand for the version that worked best; No hand if you believe there’s no difference
Subject: %%first_name%%, assemble the
bitbucketeers!
Subject: Grab the Bitbucketeers...4, 3, 2, 1
LAUNCH!
Voting time: Raise your Left or Right hand for the version that worked best; No hand if you believe there’s no difference
Voting time: Raise your Right hand for success; Left hand for failure; No hand if you believe there’s no difference
Voting time: Raise your Right for success; Left hand for failure; No hand if you believe there’s no difference
Voting time: Raise your Right hand for success; Left hand for failure; No hand if you believe there’s no difference
Deal with “Analytics is counting”…
X. Fu & H. Asorey, “Data driven product Innovation”, Tutorials, KDD 2015
We are completing a transition…
• Phase A: Data scientists as Owners • Phase B: Data scientists as a Service • Phase C: Data scientists as Partners
Take away message
• Becoming data-driven is hard
• Cultural change is the key challenge - More difficult than the engineering part
• The transition is a marathon not a sprint
• There’s no common sense, thus potential is high
Thank you!