Onset Detection in Audio Music
J.-S Roger Jang (張智星 )
http://mirlab.org/jang
MIR Lab, CSIE Dept.
National Taiwan University
-2-
What Are Note Onsets?
Energy profile of a percussive instrument is modeled as ADSR stages
Note onset is the time where the slope is the highest, during the attack time.
Soft onsets via violin, etc, are much harder to define and detect.
-3-
Difficulty in Onset Detection
Music types Monophonic Easier
Polyphonic Harder
Instrument types Percussive instruments Easier
String instruments Harder (soft onsets)
-4-
Why Onset Detection is Useful?
It is a basic step in music analysis Music transcription (from wave to midi) Music editing (Song segmentation) Tempo estimation Beat tracking Musical fingerprinting (the onset trace can serve as a robust id for fingerprinting)
-5-
Onset Detection Function
ODF (onset detection function) creates a curve of onset strength, aka Onset strength curve
Novelty curve
Most ODFs are based on time-frequency representation (spectrogram) of Magnitude of STFT (Short-time Fourier transform)
Phase of STFT Mel-band of STFT Constant-Q transform
-6-
ODF: Spectral Flux
Concept sum the positive change in each frequency bin
rectifier wave-half aka ,2
)(
size frame: index, freq: index, time:
)),1(),(()(1
xxxh
Nkn
knXknXhnsfN
k
),1( knX ),( knX
-7-
Flowchart of OSC
Steps of OSC Spectrogram Mel-band spectrogram Spectral flux Smoothed OSC via Gaussian smoothing Trend of OSC via Gaussian smoothing Trend-subtracted OSC
Check out wave2osc.m to see these steps.
-8-
Example of OSC
Try “wave2osc.m”
Time (sec)
Fre
q. b
in in
dex
Spectrogram
0.5 1 1.5 2 2.5 3 3.5 4 4.5
10
20
30
40
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
0.005
0.01
0.015
0.02
0.025
Time (sec)
Am
plitu
de
OSC (original and smoothed)
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
2
4
6
8x 10
-3
Time (sec)
Am
plitu
de
Smoothed OSC and its trend
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
1
2
3
4
5x 10
-3
Time (sec)
Am
plitu
de
Trend-subtracted OSC
-9-
What Can You Do With OSC...
OSC onsets Pick peaks to have onsets
OSC tempo (BPM, beats per minute) Apply ACF (or other PDF) to find the BPM
OSC beat tracking Pick equal-spaced peaks to have beat positions
-10-
Beat Tracking
Demos http://mirlab.org/demo/beatTracking
Try “beatTracking.m” in SAP toolbox
-11-
Example of Beat Tracking
beatTracking.m
-12-
Performance Indices ofBeat Tracking
Many performance indices of BT Check out audio beat tracking task of MIREX
Mostly adopted ones Precision, recall, f-measure, accuracy
Try simSequence.m in SAP toolbox
0 1 2 3 4 5 6
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
TP TP TPFP FP FP
FN FN
Computed
GT
Precision = tp/(tp+fp)=3/(3+3) = 0.5Recall = tp/(tp+fn)=3/(3+2) = 0.6F-measure = tp/(tp+(fn+fp)/2)=3/(3+(2+3)/2) = 0.545Accuracy = tp/(tp+fn+fp)=3/(3+2+3) = 0.375