Data in development @ Spotify

Data in developmentOscar Carlsson Data Scientist @ Spotify

Move fast!

No decision making bottlenecks

Every squad should be autonomous

No dependencies

Product Owner

QA

Client dev(s)

Backend dev(s)

Designer

Data Analyst

Playlist Your Music

Audio playback

Göteborg

Traditional development

illustration courtesy of Dan McKinley, Etsy

Think it Code it Release it




“This seems

reasonable to do”

“I would really

like this feature”




What if the idea

isn’t useful to users?

Nobody uses this.

Who came up

with this idea?

Are we representative of our users?

totally fictional graph to illustrate that employees who use the product is probably not representative of the normal users!

Ratio

of u

sers

0

0,225

0,45

0,675

0,9

Engagement score

0 1 2 3 4 5 6 7

Normal users Employees

Data to the rescue!

1. Analyse the data to get insights on what is

useful for real users

A little more data-driven development


Think it Code it Release itInsights




Probably something that is useful




Probably something that is useful This particular solution works.

But is it the best?




Are we sure what we are

working on solving the problem we saw?



Think it Release itInsights Test TestPrototype

One platform only?


Code it

How to test this? A/B testing!

A/B testing

Statistical hypothesis testing

Simple randomized experiment with >= 2 variants (A, B)

A/B testing

Control A B

“The shuffle button test”

Hypothesis: From insights X we believe redesigning the shuffle button will increase number of users who click on it

A/B testing Reach

Shar

e of

use

rs w

ho p

ress

sh

uffle

(%)

0

10

20

30

40

50

Control A B

Control A B

Fictional numbers!

A/B testing Reach

Shar

e of

use

rs w

ho p

ress

sh

uffle

(%)

0

10

20

30

40

50

Control A B

Control A B

Depth

Clic

ks p

er u

ser/

day

0

1

2

3

4

5

Control A B

Fictional numbers!

A/B testing Reach

Shar

e of

use

rs w

ho p

ress

sh

uffle

(%)

0

10

20

30

40

50

Control A B

Control A B

Depth

Clic

ks p

er u

ser/

day

0

1

2

3

4

5

Control A B

Retention

Ratio

of u

sers

that

clic

k ag

ain

0

3,75

7,5

11,25

15

Control A B

Fictional numbers!

2. Measure and test if what you are doing is

solving what you hoped to

Even more data-driven development


Think it Release itInsights Test TestPrototype Code it

Maybe do something else

Even more data-driven development


Think it Release itInsights Test TestPrototype Code it

Data Data Data

All of this requires collection of data

and users to test on

Data Measures from the clients and services

User surveys

User interviews

Measures from the clients and services > 30 TB per day

Event logs

2000 machine Hadoop cluster (moving to GCP)

Playlist and Your Music squad

More than

2 billion playlists

Less users

create playlist

on iOS

Fictional insights that could be presented to the Playlist squad

Some users don’t

invest time in

creating playlists

Revisit

playlist flow

on iOS

Pre curated

playlists

Audio playback squad

30 million songs to

75 million users in

59 countries

Swedish data plans are

enormous compared to

countries X, Y,Z

Fictional insights that could be presented to the Audio playback squad

Some users only

have internet at home

but want to

stream everywhere

Offline, caching?

Improve

Offline

Experience

Summary• Incorporate data in all steps of development and in organisation

• Cross functional squads - feature teams

• Let Data/Insights drive ideas • A/B tests to evaluate solutions

Thanks for listening!Oscar Carlsson [email protected]

@Oscarlsson

Oscarlsson1

mailto:[email protected]?subject=

Data & Analytics

Data in development @ Spotify