Onset Detection in Audio Music

J.-S Roger Jang (張智星 )

http://mirlab.org/jang

MIR Lab, CSIE Dept.

National Taiwan University

-2-

What Are Note Onsets?

Energy profile of a percussive instrument is modeled as ADSR stages

Note onset is the time where the slope is the highest, during the attack time.

Soft onsets via violin, etc, are much harder to define and detect.

-3-

Difficulty in Onset Detection

Music types Monophonic Easier

Polyphonic Harder

Instrument types Percussive instruments Easier

String instruments Harder (soft onsets)

-4-

Why Onset Detection is Useful?

It is a basic step in music analysis Music transcription (from wave to midi) Music editing (Song segmentation) Tempo estimation Beat tracking Musical fingerprinting (the onset trace can serve as a robust id for fingerprinting)

-5-

Onset Detection Function

ODF (onset detection function) creates a curve of onset strength, aka Onset strength curve

Novelty curve

Most ODFs are based on time-frequency representation (spectrogram) of Magnitude of STFT (Short-time Fourier transform)

Phase of STFT Mel-band of STFT Constant-Q transform

-6-

ODF: Spectral Flux

Concept sum the positive change in each frequency bin

rectifier wave-half aka ,2

)(

size frame: index, freq: index, time:

)),1(),(()(1

xxxh

Nkn

knXknXhnsfN

k

),1( knX ),( knX

-7-

Flowchart of OSC

Steps of OSC Spectrogram Mel-band spectrogram Spectral flux Smoothed OSC via Gaussian smoothing Trend of OSC via Gaussian smoothing Trend-subtracted OSC

Check out wave2osc.m to see these steps.

-8-

Example of OSC

Try “wave2osc.m”

Time (sec)

Fre

q. b

in in

dex

Spectrogram

0.5 1 1.5 2 2.5 3 3.5 4 4.5

10

20

30

40

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

0.005

0.01

0.015

0.02

0.025

Time (sec)

Am

plitu

de

OSC (original and smoothed)

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

2

4

6

8x 10

-3

Time (sec)

Am

plitu

de

Smoothed OSC and its trend

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50

1

2

3

4

5x 10

-3

Time (sec)

Am

plitu

de

Trend-subtracted OSC

-9-

What Can You Do With OSC...

OSC onsets Pick peaks to have onsets

OSC tempo (BPM, beats per minute) Apply ACF (or other PDF) to find the BPM

OSC beat tracking Pick equal-spaced peaks to have beat positions

-10-

Beat Tracking

Demos http://mirlab.org/demo/beatTracking

Try “beatTracking.m” in SAP toolbox

-11-

Example of Beat Tracking

beatTracking.m

-12-

Performance Indices ofBeat Tracking

Many performance indices of BT Check out audio beat tracking task of MIREX

Mostly adopted ones Precision, recall, f-measure, accuracy

Try simSequence.m in SAP toolbox

0 1 2 3 4 5 6

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

TP TP TPFP FP FP

FN FN

Computed

GT

Precision = tp/(tp+fp)=3/(3+3) = 0.5Recall = tp/(tp+fn)=3/(3+2) = 0.6F-measure = tp/(tp+(fn+fp)/2)=3/(3+(2+3)/2) = 0.545Accuracy = tp/(tp+fn+fp)=3/(3+2+3) = 0.375