Music Recommendation and Discovery

recsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma

Music Recommendation and Discovery Remastered

Tutorial

@recsys, 2011


@plamere @ocelma


INTRO CHORUS VERSE BRIDGE OUTRO



How many songs fit in my pocket?

10 Songs1979

1,000 Songs2001

10,000,000 Songs2011

Music Recommendation is importantrecsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma

What's so special about music?● Huge item space● Very low cost per item● Many item types● Low consumption time● Very high per-item reuse● Highly passionate users ● Highly contextual usage● Consumed in sequences● Large personal collections● Doesn't require our full attention● Highly Social


Music recommendation is broken ...If you like Britney Spears you might like...

...Report on Pre-War Intelligence

Let's look at some of the issues ....


What makes a good music recommendation?

● Relevance● Novelty / Serendipity● Transparency / Trust● Reach● Context





Relevancerecsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma

Relevance – cold start new or unpopular items

If you like Gregorian Chants you might like Green Day


Cold Start – New User - Enrollmentrecsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma


New User – Implicit taste dataThe Audioscrobbler


Relevance – Metadata Mismatches


Relevance – Metadata Mismatches

Why?


Relevance - The grey sheep problem


Relevance – Cultural Mismatches




Novelty and Serendipityrecsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma

Popularity Bias - The Harry Potter Effect


...also known as the Coldplay effectrecsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma

Novelty / Serendipity – the enemy

High stakes competitions focused on relevance can reduce novelty and serendipity





“If you like NiN you might like Johnny Cash” The Opacity Problem

Why???



Is this a good recommendation?

If you like Norah Jones ...

You might like Ravi Shankar


Is this a good recommendation?

If you like Norah Jones ...

You might like her father, Ravi Shankar

Photo cc by Mithrandir3

???????


Brutal Death Metal Quiz

Brutal Death Metal Quiz


Hacking the recommenderrecsys 2011 | music recommendation and discovery tutorial | paul lamere & oscar celma




The limited reach of music recommendationHelp! I’m stuck in the head

Popu

lari

ty

Sales Rank83 Artists 6,659 Artists 239,798 Artists

0% ofrecommendations

48% of recommendations

Study by Dr. Oscar Celma - MTG UPF

52% of recommendations


Personal discovery a challenge tooMusic Discovery Challenge

Listener StudyListeners 5,000

Average Songs Per User 3,500

Percent of songs never listened to

65%






Music Recommendation is not just shopping

● It is not just for shopping, but...● Discovery● Exploration● Play ● Organization● Playlisting● Recommendation for groups● Devices

● Doesn't have to look like a spreadsheet!


Context: Tools for exploration

http://techno.org/electronic-music-guide/

Ishkur's Guide to Electronic Dance Music






Ingestion process

● Input data source● Own data, Customer, Labels, UGC, ...

● Protocol● Ingestion format

– TSV, XML, DDEX, XLS!, …● Method

– FTP, API, ...● Frequency

– Offline processing: Daily / weekly?– Data freshness!

● Documentation


Ingestion process

● Post-processing● Data cleaning: Duplicates, normalization● Allow customer to use its own Ids!

● Add links to external sources● Rosetta Stone


Ingestion process

● Considerations● Allow customer to use its own IDs when using the

rec. system.● How long does it take to process the whole

collection?● Incremental updates




music recommendation approaches

● Expert-based● Collaborative filtering● Social-based● Content-based● Hybrid (combination)



● Expert-based

“X similar to (or influenced by) Y”

Editorial metadata (Genre, Decades, Location, …)

Music Genome● Collaborative filtering● Social-based● Content-based● Hybrid (combination)



● Expert-based● Collaborative filtering

“people who listen to X also listen to Y”● Social-based● Content-based● Hybrid (combination)









Raw plays:





Raw plays:

Normalize to [5..1]





Probability distribution:0.32 , 0.24 , 0.18 , 0.1 , 0.07 , 0.04, 0.03 , 0.02

Normalize to [5..1]

Raw plays:





Probability distribution:0.32 , 0.24 , 0.18 , 0.1 , 0.07 , 0.04, 0.03 , 0.02

Binary:1 0 0 1 0 0 0 0 0 1 0 0 0 0 0 0 1 1 0 0 0 0 0 1

Raw plays:

Normalize to [5..1]




Matrix Factorization. E.g: SVD, NMF, ...● Social-based● Content-based● Hybrid (combination)



● Expert-based● Collaborative filtering● Social-based

● WebMIR [Schedl, 2008]

● Content-based● Hybrid (combination)

Content Reviews Lyrics Blogs Social Tags Bios Playlists



● Expert-based● Collaborative filtering● Social-based● Content-based

“X and Y sound similar”● Hybrid




Audio features

– Bag-of-frames (MFCC) [Aucouturier, 2004], Rhythm [Gouyon, 2005], Harmony [Gomez, 2006], ...

Similarity

– KL-divergence: GMM [Aucouturier, 2002]– EMD [Logan, 2001]– Euclidean: PCA [Cano, 2005]– Cosine: mean/var (feature vectors)– Ad-hoc




Audio features

– Bag-of-frames (MFCC) [Aucouturier, 2004], Rhythm [Gouyon, 2005], Harmony [Gomez, 2006], ...

Similarity

– KL-divergence: GMM [Aucouturier, 2002]– EMD [Logan, 2001]– Euclidean: PCA [Cano, 2005]– Cosine: mean/var (feature vectors)– Ad-hoc

http://xkcd.com/26/



● Expert-based● Collaborative filtering● Social-based● Content-based● Hybrid

Weighted (linear combination)– E.g CF * 0.2 + CT * 0.4 + CB * 0.4

Cascade– E.g 1st apply CF, then reorder by CT or CB

Switching

...




Search

● Metadata search

Bruce*

● Using filters: “Popular Irish bands from the 80s”

popularity:[8.0 TO 10.0] AND

iso_country:IE AND decade:1980

● Audio search (and similarity)● Query by example


Similarity


Similarity

Using Last.fm-360K dataset

? ? ?


Similarity



Similarity' (include feedback)



Beyond similarity


Recommendation

● “If Paul likes Radiohead he might also like X”

vs.● “If Oscar likes Radiohead he might also like Y”


Recommendation

● “If Paul likes Radiohead he might also like X”

vs.● “If Oscar likes Radiohead he might also like Y”

SIMILARITY != RECOMMENDATION


Recommendation

● To whom are we recommending? Phoenix-2 (UK, 2006)


lamere @ last.fm


mini-lamere's @ last.fm

● Clustering (k-means) lamere top-50 artists


mini-lamere's @ last.fm

● Clustering (k-means) lamere top-50 artists


@lamere - Radiohead


@lamere - Radiohead

Vs. Radiohead similar artists...


@lamere - Radiohead



@lamere - Radiohead



@lamere - Radiohead



Personalization (Itemization?)

● ...but also which Radiohead era?


Analytics

● Big data processing● capture, storage, search, share, analysis and

visualization● (local) Trend detection● Tastemakers● ...




Real-world Music Recommendation


Real-world Music Recommendation


Contextual Web Crawl


Audio Processing


Hybrid Recommendation


● 100 million registered users ● 37 million active monthly users● More than 900,000 songs in catalog● More than 90,000 artists in catalog● More than 11 billion thumbs● More than 1.9 billion stations● 95% of the collection was played in July 2011


Curation and Analysis


Weighting vectors


For unknown artists


For popular artists




Country: UK

Record Labels: Acid Jazz, Sony BMG, Columbia

Genres: Funk, Disco, Acid Jazz, Jazz Fusion, Pop-rock

Years active: 1992 - present

Associated acts: Brand New Heavies, Guru, Julian Perretta


Country: UK

Record Labels: Acid Jazz, Sony BMG, Columbia

Genres: Funk, Disco, Acid Jazz, Jazz Fusion, Pop-rock

Years active: 1992 - present

Associated acts: Brand New Heavies, Guru, Julian Perretta

Mood: upbeat, energetic

Rhythm: 120bpm, no rubato, high percusiveness

Key: Dm

Tags: acid jazz funk dance

Sounds like: Sereia (Tiefschwarz Radio Edit) by Mundo Azul


“I want some upbeat songs from unknown US bands, similar to Radiohead“

http://ella.bmat.ws/collections/bmat/artists/radiohead/similar/tracks ?filter=mood:happy +speed:fast +iso_country:US +popularity:[0.0+TO+4.0]




Evaluation

"The key utility measure is user happiness. It seems reasonable to assume that relevance of the results is the most important factor: blindingly fast, useless answers do not make a user happy."

– "Introduction to Information Retrieval" (Manning, Raghavan, and Schutze, 2008)


RMSE


RMSE?


NO RMSE


NO RMSE (in music)


Evaluation

● Limitations of current metrics (RMSE, P/R, ROC, Spearman Rho, Kendall Tau, etc.)

● skewness– performed on test data that users chose to rate

● do not take into account– usefulness– novelty / serendipity– topology of the (item or user) similarity graph– ...


Evaluation

If no RMSE then...?● Predictive Accuracy vs. Perceived Quality● Does the recommendation help the user? (user

satisfaction)● Familiarity vs. Novelty

● Does the recommendation help the system?● $$$● Catalog exposure


NEXT SONG?


NEXT SONG?

?

Mean Reciprocal Rank+

User feedback


??



Novelty & Relevance


Novelty & Relevance


Novelty & Relevance


Novelty & Relevance


Novelty & Relevance

WTF?


Novelty & Relevance

Emitt Rhodes


Novelty & Relevance

WHY as important as WHAT


Novelty & Relevance



Novelty & Relevance



Novelty & Relevance



WTF


Other evaluation techniques

How can I evaluate a 3rd party recommender: objective measures:

coverage, reach

subject measures:Focus on precision

Measure irrelevant results: The WTF test


The WTF Test

Why the Freakomendation?


Evaluation

● Research Datasets● Million Song Dataset (CB, Social, Lyrics, Tags and

more)

http://labrosa.ee.columbia.edu/millionsong/

● Last.fm (CF)

http://ocelma.net/MusicRecommendationDataset/ – Last.fm 360K users <user, artist, total plays>– Last.fm 1K users <user, timestamp, artist, song>




Monitoring

● Do not monitor (or test) only the Algorithm, but the WHOLE recommender system: KPIs

● Catalog● % matches against full catalog?● Ingestion time?● Availability?

● Data & Algorithms● Time computing (e.g. Matrix factorization)?● Matrix size (e.g. ~10M x ~1M) in memory?

– 10M vectors with 300 floats per vector → ~11Gb

● Time computing vector similarity O(n)?


Monitoring

USAGE● Search

assert_equal(ID(search('The The')), ID('The The'))

● Similarity

assert(similarity(U2, REM) > 0.8)

assert(similarity(AC/DC, Rebecca Black) < 0.3)

● Recommendation

0) create_profile(@ocelma)

1) assert(similarity(@ocelma, U2) >= 0.8)

2) dislike(@ocelma, track(U2,Lemon))

3) assert(similarity(@ocelma, album(U2,Zooropa)) < 0.8)


Monitoring

● (web) API● Measure query response

– Jmeter, Apache Benchmark● Process real logs

– Fake (repeated) queries → fast because using cache?


Monitoring

● (web) API● Measure query response

– Jmeter, Apache Benchmark● Process real logs

– Fake (repeated) queries → fast because using cache?




Conclusions

● Music Recsys is multidisciplinary● search and filtering, musicology, data mining,

machine learning, personalization, social networks, text processing, complex networks, user interaction, information visualization, and signal processing (among others!)

● Music Recsys is important● These technologies will be integral in helping the next

generation of music listeners find that next favorite song

● Strong industry impact● Music Recsys is special


Further research

● How well do music recommenders work?● lack of standardized data sets and objective

evaluation methods

● How to recognize and incorporate context into recommendations? ● listener’s context (exercising, exploring, working,

driving, relaxing, and so on)


Further research

● How to make recommendations for all music? ● consider all music including new, unknown, and

unpopular content.

● What effect will automatic music recommenders have on the collective music taste?


Music Recommendation and Discovery Remastered

Tutorial

@recsys, 2011

Technology

Music Recommendation and Discovery