Emotion-Based Music Recommendation By Assciation Discovery from Film Music

Fang-Fei Kuo1, Meng-Fen Chiang2, Man-Kwan Shan2 and Suh-Yin Lee1

Department of Computer Science and Information Engineering, National Chiao-Tung University Hsinchu, Taiwan {ffkuo, sylee}@csie.nctu.edu.tw

Department of Computer Science, National Cheng-Chi University Taipei, Taiwan {g9309, mkshan}@cs.nccu.edu.tw

13th ACM international conference on Multimedia MM '05 Publisher

outline

• Introduction

• Model

• Emotion Detection

• Music Feature Extraction• Association Discovery and

Recommendation

• Performance evaluation

• Conclusions

introduction

• Two major approaches for the personalized music recommendation

(1) Content-based filtering

(2) collaborative filtering

• Recommend music based on the emotion

Introduction(con)

• Potential application

production of home video , shopping mall to stimulate sales , music therapy etc.

Introduction(con)• Recommend music based on emotions 1. recommend music by the rules (psychological research) 2. learn the rules by training from music labeled with emotion types

• Recommendation model to recommend music by association discovery from film music

Model

Emotion Detection

Music Feature Extraction

• Music elements which affect the emotion include melody , rhythm , tempo , mode , key , harmony , dynamics and tone-color

Melody Extracting

• Melody Extracting (1) All-mono (2) Entropy-channel (3) Entropy-part (4) Top-channel

• Improved the All-mono to get more precise melody sequence

instrument and volume

Melody Extracting

Improved the All-mono

Three steps:

step1.Remove channels of instruments which are unlikely for melody performing

step2.For each measure , select the channel of the largest volume

step3.Keep the highest note

• Chord assignment algorithm 1.We chose 60 common chords as the candidate

s 2.count score of each candidateStage1:Determine the chord sampling unitStage2: step:A step:B step:CStage3: step:A step:B step:C

Melody Extracting

Stage1: Determine the chord sampling unit

find the prevailing note , which has the

longest total performance length half note = four measures

quarter note = two measures

eighth note = one measure

Melody Extracting

Melody Extracting

Stage2:

• 1.chord of the sampling unit which

sounds first (10)

• 2.each kind of note (1)

• 3.longest pitch duration>half (2) else(1)

• 4.no sharps or flats—discard the chord

expect the first chord

Melody Extracting• Stage3: 1. root is descending fifth ,descending third or ascending second (2) 2. if I7 , add two points to IV chord if II7 , add two points to V chord if III7 , add two points to VI chord if IV7 , add two points to V chord if V7 , add two points to I and VI chord , one point to V chord if VI7 , add two points to II chord if VII7 , add two points to III chord 3. add two points to the chords whose root is the lowest pitch

• Rhythm is the music feature that describes the timing information of music

• Extracting the beat sequence base on percussion instrument

• quarter note = 1000 basic unit is set to sixteenth note long

• Highest Repeating pattern

Rhythm Extracting

Tempo calculated

• Tempo=NB/NS

• NB is number of beat

• NS is the length of the rhythmic pattern

Association Discovery and Recommendation

• Mixed Media Graph

• Music Affinity Graph

Mixed Media Graph

• Represent all the objects , as well as their attributes as nodes in a graph

• Nearest neighbor links (NN-links)

• Object-attribute-value links (OAV-links)

• Similarity function si(*,*) to computing the k nearest neighboprs

Mixed Media Graph (con)K=1

Mixed Media Graph (con)

• Correlation discovery by random walk with restarts

uA(B) denoted the steady-state probability

Music Affinity Graph

• For each music object vertex , four types of attribute vertices – emotion , chord , rhythm , and temp vertices

Music Affinity Graph (con)

• The edge between chord vertices is constructed based on the k-nearest neighboring

• The edge between emotion (rhythm) vertices is constructed only the same emotion (rhythm) vertices

K=1

Music Affinity Graph (con)

• steady-state probability

• Not necessary to say that the music feature value with high affinity is highly corrlative to the query emotions

• Complement affinity graph G’

• Final affinity equal G-G’

Review

Performance evaluation

• K=7 Scorei= |Ei∩Eq| / (√|Ei|x|Eq|)

Conclusions

• Recommend music based on emotion

• Construct the recommend model from film music

• Experimental result shows that the top-one result’s average score achieves 85%

End