Noise Resilient Piano Note & Chord Recognitioncnx.org/.../noise-resilient-piano-note-chord-recognition-4.1.pdf · Noise Resilient Piano Note & Chord Recognition By ... which is the

Noise Resilient Piano Note & ChordRecognition

By:Eric KangJohn Yan

Abhipray SahooChenxi Liu

Noise Resilient Piano Note & ChordRecognition

By:Eric KangJohn Yan

Abhipray SahooChenxi Liu

Online:< http://cnx.org/content/col11603/1.4/ >

C O N N E X I O N S

Rice University, Houston, Texas

This selection and arrangement of content as a collection is copyrighted by Eric Kang, John Yan, Abhipray Sahoo,

Chenxi Liu. It is licensed under the Creative Commons Attribution License 3.0 (http://creativecommons.org/licenses/by/3.0/).

Collection structure revised: December 24, 2013

PDF generated: December 24, 2013

For copyright and attribution information for the modules contained in this collection, see p. 31.

Table of Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Onset Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Fourier Analysis of Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 Harmonic Pitch Class Pro�le (HPCP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 Sources of Error & General Improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 Poster . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 Results & Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 Source Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30Attributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

iv

Available for free at Connexions <http://cnx.org/content/col11603/1.4>

Chapter 1

Introduction1

Automatic Music Transcription: An IntroductionMusic Transcription is the process of transcribing a piece of music, usually onto sheet music . This processinvolves identi�cation of notes, rhythm, key signature, etc. Automatic Music Transcription has been ahot topic of research in signal processing for decades now. In terms of signal processing, automatic musictranscription can be broken down into two large problems: locating the notes and identifying what pitchthey are. While these problems have been solved for monophony (single line of melody), they are stilllargely unsolved for polyphony (multiple lines of independent melodies). Our project aims at accurate androbust detection of homophony (single melody line with a chord line). The di�erences between the threecan better be understood by listening to them:

Baa Baa Black Sheep: A Monophony [Media Object]2

Saints Go Marching On: A Homophony [Media Object]3

Stars And Stripes Forever: A Polyphony [Media Object]4

1This content is available online at <http://cnx.org/content/m48363/1.4/>.2This media object is an audio �le. Please view or download it at

<BaaBaaBlackSheep.wav>3This media object is an audio �le. Please view or download it at

<SaintsGoMarchingOn.wav>4This media object is an audio �le. Please view or download it at

<StarsStripes.wav>


1

2 CHAPTER 1. INTRODUCTION

Figure 1.1: A simple overview of Automatic Music Transcription

Our Motivation And ApproachMotivation We aim to explore the inner workings of automated music transcription. In doing so, we hope torealize our goal of making an improved and more user accessible note detection and transcription algorithm.We also aim to streamline the process between the playing and writing of music. Composers will be ableto play their work and have it transcribed by the end of it. People who want the sheet music to a songthey like who don't have perfect pitch can run a recording through our algorithm and get the notes themselves.

Our project's main constraints were:1. Single Instrument (Grand Piano)2. Frequency of piano key presses of less than 3 notes/sec3. No reverberation

Our team started o� with an independent implementation before exploring several existing imple-mentations. We augmented our approach with the use of Harmonic Pitch Class Pro�le (HPCP) thatextended our algorithm by giving us likelihoods of a note being classi�ed as one of 12 piano notes in anoctave.


3

Figure 1.2: General overview of our approach to automatic piano note and chord recognition


4 CHAPTER 1. INTRODUCTION


Chapter 2

Onset Detection1

Onset DetectionOnset detection is the detection of attacks, or the start of each note. When a song is plotted over time, theeasily identi�able spikes represent the attack of a new note. We exploit this property to divide the song intoindividual notes in sequence. (It is important to note that there are songs where all the notes do not movein sequence, e.g. a chord is sustained while the melody moves through many di�erent notes. This problemis discussed later.)

Example 2.1: Input Signal "Baa Baa Black Sheep"

1This content is available online at <http://cnx.org/content/m48375/1.3/>.


5

6 CHAPTER 2. ONSET DETECTION

At �rst sight, we can see that note attacks are clearly de�ned by large spikes. Though these spikes arevisually easy to identify, when we take a closer look at the signal, we see that it is �uctuating rapidly,meaning that there is a lot of high frequency content in each note. Thus, to identify the peaks of each note,we can't simply look at all the high points or the points where the value of the data rises exponentially.We'll have to do some pre-processing and �ltering to identify the edges computationally.

Example 2.2: Zooming in on one note

We decided to try out a low-pass �lter to smooth out the sharp changes in the song so all that's left aresmooth increases and decreases, where each rise and fall corresponds to a single note. First we decided thatthe length of the �lter should be proportional to the sampling frequency, which �xes its length in time. Thenwe decided to try out a simple boxcar �lter and convolve that with the square (i.e. the power) of our signal.It turned out that we decided to convolve it with a boxcar 3 times over, which is the same as convolve itonce with 3 boxcars convolved together. Visually, this was our �lter:


7

Example 2.3: Thrice Convolved Boxcar

And here's our smoothed signal superimposed on our original �le:



Example 2.4: Original Signal with Smoothed Curve

We proceed to use MATLAB's function ��ndpeaks.� We twiddled with parameters like the minimum heightof the peaks listed and the minimum distance between peaks to ensure that small ripples in some attacksdid not get counted twice. The algorithm then takes the distance between the peaks as the lengths of thenotes.

There is a second part of the algorithm that calculates the beats per minute and attempts to characterizeeach note in terms of its beats, i.e. a half-note, quarter-note, eight-note, etc. This is done by �nding thesmallest interval, corresponding to the shortest note, and assumes all the note durations are multiples of thislength. We assume the bpm of our song ranges from 100 bpm to 200 bpm � if we don't, then we could saya quarter note at 80 bpm is the same as a half note at 160 bpm. There is a degree of tolerance (that can bevaried) to determine whether two note durations correspond to the same note, as the lengths of each notewe get are not perfect multiples of each other. This mapping works very well for synthesized music, but isnot practical for live recordings of music or overtly rubato music, as the length of each beat is not �xed.

Below is a sample output of our note divider function where data1 is a synthesized audio �le of Hot CrossBuns, fs is the sampling frequency of 44.1kHz, bpm is the beats per minute, notes is the estimated length ofeach note, and notetimes has is the length of each note in samples:

[bpm, notes, notetimes] = notedivider(data1, fs)


9

bpm notes note times

120.8288 1.000 22754

1.000 21846

2.000 43401

1.000 22925

1.000 21847

2.000 43819

0.500 10987

0.500 11054

0.500 10563

0.500 11136

0.500 10793

0.500 11321

0.500 10803

0.500 10938

1.000 23082

1.000 21847

3.000 61401

Table 2.1

Gain CompressionIn order to make our detection algorithm robust, we performed gain compression on the signal before pro-cessing it. Compression �attens large spikes and ampli�es rest of the signal. It therefore allows us to "cleanup" the signal especially when the pianist plays certain notes softly and other notes loudly.

Below is our algorithm. x[n] is our input signal, y[n] is the compressed signal, lambda is a parametergreater than 1. Increasing it equalizes the note heights more and more. For our implementation we used alambda value of 2.

(2.1)



Example 2.5

After compression, we smooth the signal using a thrice convolved boxcar �lter. This will produce a singlecontinuous line that outlines the edge of the signal. This makes detecting the onset of a note easier byallowing us to locate the peak of a note with more ease.


Chapter 3

Fourier Analysis of Notes1

Fourier Analysis Of NotesOnce we have the start times for each note, we know the duration of each note as well as their start and endtimes. We create a hamming window of length 1.25 times the length of the note of interest. The window isthen used to extract out the note. The following �gure illustrates this:

Example 3.1

note: We decided to use a hamming window because it mitigates errors in the onset detection algo-rithm. Moreover, it doesn't allow for as many spurious oscillations in the spectrum as a rectangularwindow does. Changes in the spectrum are better captured.

Now that we have the signal of interest, we take its fourier transform using the �t algorithm. We use thefrequency domain because the pitch of a note is determined by what frequencies it has power in, while thedistribution of power among harmonics determines the sound of the instrument. E.g. 440 Hz corresponds toan "A". For an A on a piano, there will be power at 440, 880, 1320 Hz,etc, and the shape of this distributiondistinguishes it from an A on a �ute or guitar. We detect the frequencies with the highest power in the�t along with the corresponding power. The HPCP algorithm explained in the next module analyzes thesepeaks to �nd the probabilities of the note being anyone of the 12 pitches.



11

12 CHAPTER 3. FOURIER ANALYSIS OF NOTES

Peak DetectionPeak detection refers to the detection of the peaks of the Fourier transform of each note. These correspond tothe harmonics of the fundamental frequency of whatever key is being pressed. We used a MATLAB functioncalled "�ndpeaks" to help us. ... A di�cult part of peak detection is determining what constitutes a "peak".For our purposes, we normalized our Fourier transform plot to have a max of 1, and considered all pointsabove a threshold to be a peak. We chose our threshold after carefully looking at the Fourier transforms ofmany signals.Hamming WindowIn the extraction of notes, you may ask yourself, why did we choose to using a hamming window? Why nota boxcar �lter? The answer to this question is simple. Imagine the boxcar �lter in the frequency domain.What do we see? We basically see a low pass �lter (what we want) with lots of ripples attached to bothends of it (what we don't want). The ripples will introduce lots of error as we are letting in frequencies wedo not want.Boxcar Filter

In comparison, if we look at the frequency domain of a hamming window, while there are also ripplesattached to the end of our �lter, the ripples are generally much smaller. While not perfect, this is highlypreferable to the boxcar �lter.


13

Hamming Filter


14 CHAPTER 3. FOURIER ANALYSIS OF NOTES


Chapter 4

Harmonic Pitch Class Pro�le (HPCP)1

Harmonic Pitch Class Pro�leThe key to note recognition is analyzing the frequency consistence of the audio clip. Suppose the clip we areanalyzing consists of one or several individual notes, then unsurprisingly, the FFT of the audio should haveseveral "peaks". HPCP provides a way to pin down the note(s) by analyzing these "peaks" in the FFT plot.Big PictureThe HPCP algorithm returns a vector of length 12, representing the 12 notes within an octave. The elementswill in the end be normalized, representing the likelihood that the corresponding note is actually in the audio.The values are obtained as follows:

where n = 1, ... 12, ai is the linear magnitude of the ith peak, and � is the frequency value of the ithpeak. i = 1, ... nPeaks, where nPeaks is the number of spectral peaks that we consider, and w is the weightof the frequency �.

Basically, the note represented by integer n is compared with every peak in FFT. The weighting functionrepresents the similarity between the note and the peak. This correlation is then multiplied by the square ofthe amplitude of the peak. We do this for every peak and add all the correlation up to get the "likelihood"that note n is compatible with the FFT graph. We repeat the same operations for all 12 values of n, andthe HPCP vector is complete.Weighting FunctionThe weighting function mentioned above is determined by the following three steps.

STEP ONE:

note: where size=12. f_ref can be set to 440Hz but doesn't a�ect the general result. Here fn isthe frequency of the note represented by n in a certain octave.

STEP TWO:



15

16 CHAPTER 4. HARMONIC PITCH CLASS PROFILE (HPCP)

note: where m is the integer that minimizes the magnitude of the distance d. The role of m is todrive d to zero as much as possible so that the potential di�erence in vector is eliminated.

STEP THREE:

note: where l is the length of the weighting window. This value is a parameter of the algorithmthat can be adjusted. What this equation says is that when d is small enough, we think there isa correlation between the note and the peak and return a positive value given by cosine square.Otherwise we set the correlation to zero.

NormalizationAfter we have obtained the original HPCP results, we normalize the biggest term to 1. Now we have a vectorof length 12 and each element is between 0 and 1, each representing the "possibility" that the correspondingnote is in that audio clip.ExampleHere is an HPCP vector for a C Major chord:

note: The integers of the x axis represents 12 notes within an octave. 1 represents #A and 12represents A. We can see clearly that C, E, and G are the three notes with the most signi�cant


17

correlation, therefore we can safely conclude that the audio most likely consists of C, E, and G,which is correct.

Here is the input �le:

4.1 Octave Detection

One problem with the HPCP algorithm is that it ignores what octave the original note was in. We havewritten a function �ndOctave to rectify this.

First o�, we exploit the fact that a note's �t is the same regardless of what pitch it is: spikes the thenote's pitch's fundamental frequency and all its multiples. So an A4 starts at 440 Hz, and has spikes at 880,1320, etc. Our HPCP will identify this note as an A. To �nd out what octave it is in, we just look at thelocation of the lowest spike because all other spikes are multiples of this one frequency. So we �nd it at 440Hz.

This process is repeated for every pitch detected by the HPCP, so for the C Major chord above, it'd lookfor the harmonics of C and �nd the lowest at 261 Hz (C4), then E at 330 Hz (E4), and G at 392 Hz (G4).

This method is fast and accurate � however there is one limitation: if there are multiple notes of thesame pitch but in di�erent octaves (e.g. C4 and C5 and C6), their spectra would overlap and the algorithmwould only detect the lowest note, C4.


18 CHAPTER 4. HARMONIC PITCH CLASS PROFILE (HPCP)


Chapter 5

Sources of Error & GeneralImprovements1

5.1 Source of Error

Notes Played On A Grand PianoWhen working with the Fourier transform of musical notes, it's important to remember that each note hasits own harmonic frequency. For example, a C4 would have a harmonic frequency of 440 Hz. Ideally, inthe spectrum, there would be peak at 440 Hz. While this may serve as a good reference for algorithmicpurposes, one also needs to remember that pianos never actually produce perfect notes. The reason is thatthe reverberation generated by the string of a note as it's played a�ects the strings around it. Therefore,when you look at a plot of the Fourier transform of a C4, you'll actually peaks at many di�erent frequencies.This issue is especially exasperated with chords.



19

20 CHAPTER 5. SOURCES OF ERROR & GENERAL IMPROVEMENTS

Figure 5.1: This is what the fourier transform of a C4 note played on a Grand Piano looks like.

HPCP Weighting FunctionHPCP uses a weighting function to determine the relative "strength of presence" of each note within a givenFourier Transform. At the current moment, it uses a curve of cos^2. This works relatively well for just aboutany instrument. However, since we are only focusing on piano for the moment, we can improve the curve tobetter match our needs. By looking at the Fourier transforms of many other piano notes, we should be ableto detect a pattern amongst the distribution of frequencies generated by each note. By using the new curvegenerated by this data analysis, accuracy should be signi�cantly improved.

5.2 General Improvements

Generalizing To Other InstrumentsOur implementation works only with Grand Piano at the moment. The change we would have to make inorder to be able to work with more than one instruments is our approach in detecting note onsets. Forinstruments with clear note attacks, peak detection in the time domain makes perfect sense. However, forinstruments such as �ute, which might have a continuous transition between notes, we will have to tackleonset detection in the frequency domain using image analysis- take the spectrogram of the audio �le anddetect the times when power at frequencies change by a certain threshold.

We implemented this frequency domain analysis and got the same results as with time domain anal-ysis for Grand Piano. You can �nd our MATLAB code for this frequency domain approach here:


21

[Media Object]2

Finding Relationships Between Songs and VariablesAs it stands, our implementation uses many variables that we have used using evidence rather. It would beextremely helpful to the performance of the system if we could �nd exact relationships between the inputsignal and those variables. For example, our gain compression algorithm is non-dynamic in that we use alambda value of 2; we know that each unique song performs better with a certain lambda. Another exampleis the beta value in the threshold for detection of the signal of interest being chord and/or note. This betavalue was manually tweaked: low value if the song has higher likelihood of having chords and vice versa.

2This media object is a downloadable �le. Please view or download it at<notedivider2.m>


22 CHAPTER 5. SOURCES OF ERROR & GENERAL IMPROVEMENTS



23

24 CHAPTER 6. POSTER

Chapter 6

Poster1

6.1

Figure 6.1



Chapter 7

Results & Conclusion1

ResultsWe tested our implementation on a variety of piano songs, created digitally as well as recorded from a GrandPiano. Our metric for accuracy is de�ned to be:

Here x is the correct string of notes/chords and y is the one created by our implementationN is length of xedit_distance gives the number of changes (insertions, substitutions or deletions) needed to go from stringx to y

The average accuracy on testing eight di�erent songs was:

• 92% on songs with single melody line• 86% on songs with single chord line• 78% on songs with homophonies

Noise ResiliencyOne of the features of our implementation is noise-resiliency. With additive Gaussian noise (AWGN), theperformance is maximized for audio with above SNR of -5 dB. The following �gure illustrates this. We testednoise resiliency on a single song, adding AWGN incrementally. We ran the trial ten times.



25

26 CHAPTER 7. RESULTS & CONCLUSION

Figure 7.1: Summary of our testing. The songs can be found in the "Source Code" section. Chorddetection threshold is the beta value that determines which note(s) to pick in the HPCP vector. Thresh-old=mean(HPCP)*Beta. If the value at index i of the HPCP is greater than this threshold, index i ispicked as a note.


27

7.1 Conclusion

Our implementation of a piano note and chord recognition system was successful in varying degrees.Depending upon the texture of the song and independent variables in our algorithm, we were able to get anaverage accuracy of 87.53%.

We applied our knowledge from ELEC 301 to create the onset detection, gain compression, �lteringand spectral analysis algorithms independently. We then augmented our implementation with the use ofHarmonic Pitch Class Pro�le (HPCP) that has the added bene�t of noise resiliency.

By exploring the topic of automatic music transcription, and implementing our own version of ho-mophony detection, we have created a base from where we can only improve the accuracy and performanceof music recognition automation system.


28 CHAPTER 7. RESULTS & CONCLUSION


Chapter 8

Source Code1

MATLAB CodeYou can download our implementation of the algorithm below:[Media Object]2

Test SongsYou can also download the songs we tested our implementation on:[Media Object]3

1This content is available online at <http://cnx.org/content/m48548/1.2/>.2This media object is a downloadable �le. Please view or download it at

<MuScribers MATLAB Code.zip>3This media object is a downloadable �le. Please view or download it at

<Test Songs.zip>


29

30 INDEX

Index of Keywords and Terms

Keywords are listed by the section with that keyword (page numbers are in parentheses). Keywordsdo not necessarily appear in the text of the page. They are merely associated with that section. Ex.apples, � 1.1 (1) Terms are referenced by the page they appear on. Ex. apples, 1

A Automatic music transcription, � 8(29) H Harmonic Pitch Class Pro�le, � 8(29)HPCP, � 8(29)


ATTRIBUTIONS 31

Attributions

Collection: Noise Resilient Piano Note & Chord RecognitionEdited by: Eric Kang, John Yan, Abhipray Sahoo, Chenxi LiuURL: http://cnx.org/content/col11603/1.4/License: http://creativecommons.org/licenses/by/3.0/

Module: "Introduction"By: Eric Kang, Abhipray Sahoo, John Yan, Chenxi LiuURL: http://cnx.org/content/m48363/1.4/Pages: 1-3Copyright: Eric Kang, Abhipray Sahoo, John Yan, Chenxi LiuLicense: http://creativecommons.org/licenses/by/3.0/

Module: "Onset Detection"By: Eric Kang, Abhipray Sahoo, John Yan, Chenxi LiuURL: http://cnx.org/content/m48375/1.3/Pages: 5-10Copyright: Eric Kang, Abhipray Sahoo, John Yan, Chenxi LiuLicense: http://creativecommons.org/licenses/by/3.0/

Module: "Fourier Analysis of Notes"By: Eric Kang, Abhipray Sahoo, John Yan, Chenxi LiuURL: http://cnx.org/content/m48372/1.3/Pages: 11-13Copyright: Eric Kang, Abhipray Sahoo, John Yan, Chenxi LiuLicense: http://creativecommons.org/licenses/by/3.0/

Module: "Harmonic Pitch Class Pro�le (HPCP)"By: Eric Kang, Abhipray Sahoo, John Yan, Chenxi LiuURL: http://cnx.org/content/m48378/1.3/Pages: 15-17Copyright: Eric Kang, Abhipray Sahoo, John Yan, Chenxi LiuLicense: http://creativecommons.org/licenses/by/3.0/

Module: "Sources of Error & General Improvements"By: Eric Kang, Abhipray Sahoo, John Yan, Chenxi LiuURL: http://cnx.org/content/m48386/1.3/Pages: 19-21Copyright: Eric Kang, Abhipray Sahoo, John Yan, Chenxi LiuLicense: http://creativecommons.org/licenses/by/3.0/

Module: "Poster"By: Eric Kang, John Yan, Abhipray Sahoo, Chenxi LiuURL: http://cnx.org/content/m48423/1.3/Pages: 24-25Copyright: Eric Kang, John Yan, Abhipray Sahoo, Chenxi LiuLicense: http://creativecommons.org/licenses/by/3.0/


32 ATTRIBUTIONS

Module: "Results & Conclusion"By: Eric Kang, Abhipray Sahoo, John Yan, Chenxi LiuURL: http://cnx.org/content/m48470/1.3/Pages: 25-27Copyright: Eric Kang, Abhipray Sahoo, John Yan, Chenxi LiuLicense: http://creativecommons.org/licenses/by/3.0/

Module: "Source Code"By: Abhipray Sahoo, Eric Kang, Chenxi Liu, John YanURL: http://cnx.org/content/m48548/1.2/Page: 29Copyright: Abhipray Sahoo, Eric Kang, Chenxi Liu, John YanLicense: http://creativecommons.org/licenses/by/3.0/


About ConnexionsSince 1999, Connexions has been pioneering a global system where anyone can create course materials andmake them fully accessible and easily reusable free of charge. We are a Web-based authoring, teaching andlearning environment open to anyone interested in education, including students, teachers, professors andlifelong learners. We connect ideas and facilitate educational communities.

Connexions's modular, interactive courses are in use worldwide by universities, community colleges, K-12schools, distance learners, and lifelong learners. Connexions materials are in many languages, includingEnglish, Spanish, Chinese, Japanese, Italian, Vietnamese, French, Portuguese, and Thai. Connexions is partof an exciting new information distribution system that allows for Print on Demand Books. Connexionshas partnered with innovative on-demand publisher QOOP to accelerate the delivery of printed coursematerials and textbooks into classrooms worldwide at lower prices than traditional academic publishers.

Documents

Noise Resilient Piano Note & Chord Recognitioncnx.org/.../noise-resilient-piano-note-chord-recognition-4.1.pdf · Noise Resilient Piano Note & Chord Recognition By ... which is the