View
170
Download
0
Category
Preview:
Citation preview
Data in developmentOscar Carlsson Data Scientist @ Spotify
Move fast!
No decision making bottlenecks
Every squad should be autonomous
No dependencies
Product Owner
QA
Client dev(s)
Backend dev(s)
Designer
Data Analyst
Playlist Your Music
Audio playback
Göteborg
Traditional development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release it
Traditional development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release it
“This seems
reasonable to do”
“I would really
like this feature”
Traditional development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release it
What if the idea
isn’t useful to users?
Nobody uses this.
Who came up
with this idea?
Are we representative of our users?
totally fictional graph to illustrate that employees who use the product is probably not representative of the normal users!
Ratio
of u
sers
0
0,225
0,45
0,675
0,9
Engagement score
0 1 2 3 4 5 6 7
Normal users Employees
Data to the rescue!
1. Analyse the data to get insights on what is
useful for real users
A little more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release itInsights
A little more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release itInsights
Probably something that is useful
A little more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release itInsights
Probably something that is useful This particular solution works.
But is it the best?
A little more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Code it Release itInsights
Are we sure what we are
working on solving the problem we saw?
A little more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Release itInsights Test TestPrototype
One platform only?
Think it Code it Release itInsights
Code it
How to test this? A/B testing!
A/B testing
Statistical hypothesis testing
Simple randomized experiment with >= 2 variants (A, B)
A/B testing
Control A B
“The shuffle button test”
Hypothesis: From insights X we believe redesigning the shuffle button will increase number of users who click on it
A/B testing Reach
Shar
e of
use
rs w
ho p
ress
sh
uffle
(%)
0
10
20
30
40
50
Control A B
Control A B
Fictional numbers!
A/B testing Reach
Shar
e of
use
rs w
ho p
ress
sh
uffle
(%)
0
10
20
30
40
50
Control A B
Control A B
Depth
Clic
ks p
er u
ser/
day
0
1
2
3
4
5
Control A B
Fictional numbers!
A/B testing Reach
Shar
e of
use
rs w
ho p
ress
sh
uffle
(%)
0
10
20
30
40
50
Control A B
Control A B
Depth
Clic
ks p
er u
ser/
day
0
1
2
3
4
5
Control A B
Retention
Ratio
of u
sers
that
clic
k ag
ain
0
3,75
7,5
11,25
15
Control A B
Fictional numbers!
2. Measure and test if what you are doing is
solving what you hoped to
Even more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Release itInsights Test TestPrototype Code it
Maybe do something else
Even more data-driven development
illustration courtesy of Dan McKinley, Etsy
Think it Release itInsights Test TestPrototype Code it
Data Data Data
All of this requires collection of data
and users to test on
Data Measures from the clients and services
User surveys
User interviews
Measures from the clients and services > 30 TB per day
Event logs
2000 machine Hadoop cluster (moving to GCP)
Playlist and Your Music squad
More than
2 billion playlists
Less users
create playlist
on iOS
Fictional insights that could be presented to the Playlist squad
Some users don’t
invest time in
creating playlists
Revisit
playlist flow
on iOS
Pre curated
playlists
Audio playback squad
30 million songs to
75 million users in
59 countries
Swedish data plans are
enormous compared to
countries X, Y,Z
Fictional insights that could be presented to the Audio playback squad
Some users only
have internet at home
but want to
stream everywhere
Offline, caching?
Improve
Offline
Experience
Summary• Incorporate data in all steps of development and in organisation
• Cross functional squads - feature teams
• Let Data/Insights drive ideas • A/B tests to evaluate solutions
Thanks for listening!Oscar Carlsson lad@spotify.com
@Oscarlsson
Oscarlsson1
Recommended