A Subsequence Matching with Gaps-Range-Tolerances Framework: A Query-By-Humming Application Alexios Kotsifakos University of Texas at Arlington, University

A Subsequence Matching withGaps-Range-Tolerances Framework:A Query-By-Humming Application

Alexios KotsifakosUniversity of Texas at Arlington, University of Athens (Greece)

Joint work with

Panagiotis Papapetrou (Aalto University, Finland)

Jaakko Hollmen (Aalto University, Finland)

Dimitrios Gunopulos (University of Athens, Greece)

2

Outline

Motivation

Related Work

Problem Setting

SMBGT: The proposed method

Experimental Evaluation

Summary and Conclusions

3

Introduction – MotivationSuppose you hear a song but cannot recall its name… One solution: Hum a short part of it and perform a

search on a music repository to find it or songs similar to it.

Query-By-Humming (QBH): Given a hummed query song search a music db for the K most similar songs.

Subsequence Matching: Given a query sequence and a db of sequences, identify the subsequence in db that best matches the query.

4

Subsequence Matching in a Database

database

query

What subsequence of any database sequence is the best match for Q ?

5

Converting music pieces to time series Each music piece is a sequence of

notes.

Notes consist of pitch (frequency) and duration.

Part of the Happy Birthday music score.

The 2D time series representation of the above music score.Pitch interval : distance between two pitches.

IOIR : duration ratio of two consecutive notes

QBH: A real example

Hummed query:

“Yesterday” (Beatles)

Target: “Yesterday”

QBH: A real example

Hummed query:

“Yesterday” (Beatles)

Matched target subsequence

8

QBH: A noisy application domain Slight or more serious errors due to instant or temporary key or tempo loss error-tolerant method.

Allow skipping a bounded number of elements in both query and target sequences due to serious humming errors.

Constrain the length of the matching subsequence.

Optional constraint to the min. number of matching elements.

9

Our contributions A subsequence matching framework that allows for

a constrained number of gaps on both query and target sequences,

variable tolerance levels in the matching (no probabilistic model),

a matching range that constrains the max. match length, bounding the min. # of matched elements.

SMBGT: Given a query Q and a target sequence X (with |Q| << |X|) finds the subsequence of X that best matches Q, considering the framework.

Extensive comparative evaluation on QBH of several DB-based methods and a probabilistic model-based method.

10

Outline

Motivation

Related Work

Problem Setting




11

Related Work DP-based methods

Whole sequence matching: DTW and variants (cDTW, EDR, ERP) robust to misalignments and warps, LCSS allows for gaps, edit distance, Iliopoulos et al., etc.

Subsequence matching: SPRING finds the subsequences of evolving numerical streams closest to a query, DTWc and DTWs account for local adjustments of tempo, Han et al. (2007) performed uniform segmentation and sliding windows for near exact matching where user selects segments’ length.

n-gram methods for near exact matching All account for 1D + fail to handle noise imposed by

users. Probabilistic-based methods (HMMs)

12

Outline

Motivation

Related Work

Problem Setting




13

Problem Setting: Representing Musical Pieces Expressing pitch

Absolute pitch: frequency of the note. Pitch interval: frequency difference between two adjacent notes.

Encoding duration Inter-Onset-Interval (IOI): difference in time onsets of two

adjacent notes. IOI Ratio (IOIR): ratio of IOIs of two adjacent notes. Log IOI Ratio (LogIOIR): logarithm of IOIR.

To save computational time we deal with note transitions:<pitch interval, IOIR>, <pitch interval, LogIOIR>

Performed quantizations on pitch interval with modulo 12, and on LogIOIR to the closest integer or closest value in [-2, 2].

14

Problem Setting: Definitions

X = {x1, …, xn} representing music piece of length |X|,

pair of real values for pitch and duration, DB={X1, …,

XN }music db,

X[ts : te]={xts , …, xte} subsequence of X (not always continuous), Q = {q1, …, qn }.

Definition 1 (Variable error-tolerant match):

,p rj j jx x x X

and match with variable -tolerance, ,if, for a set of

constraints { , }:

fi j i j

ff fp r

q Q x X q x

( ) ( ) and ( , ) ( , ).f p f r rp p i r r i ji f q i j f q x

15

Problem Setting: Definitions

1 1 2 2

i

Consider [ : ] and [ : ] of equal length. Let , the indices of

those elements in and that are included in the corresponding subsequences.

If , , , 1,...,| | , and πi i

Q X

i Q i X Q

Q ts te X ts te G G

Q X

q x G G i G

+1 i i+1 i

1 1 2 2

2 2

-π , -γ ,

then pair { [ : ], [ : ]} defines a common gapped subsequence of and .

The longest such subsequence with - is called ( , ).

Q ts te X ts te Q X

te ts r SMBGT Q X

Definition 2 (Common bounded-gapped subsequence):

SMBGT: error-tolerant matching denoted as ε-match.

16

Problem Setting:Example and Problem Formulation Example

α = 2, β = 1, r = 6, ε = 1 (absolute tol.) Q = {6, 3, 10, 5, 3, 2, 9}, X = {1, 1, 3, 4, 6, 9, 2, 3, 1} Q[2:6] with GQ = {2, 4, 5, 6} and X[3:8] with GX = {3, 4,

7, 8} Q = {6, 3, 10, 5, 3, 2, 9}X = {1, 1, 3, 4, 6, 9, 2, 3, 1}

Problem: Given a database DB with N sequences of arbitrary lengths, a query sequence Q, and positive integers δ and r, find set

of the top-K subsequences with{ [ : ] | }i s e iS X t t X DB

| ( , [ : ]) | .i s eSMBGT Q X t t

SMBGT(Q, X) = {Q[2:6], X[3:8]}

17

Outline

Motivation

Related Work

Problem Setting




18

SMGT SMGT: α = β = inf. Array a (|Q+1|)*(|X|+1). For all i in {1,…,|Q|}, j in {1,…,|X|}

Matrix s keeps in si,j for each ai,j the start point of its best alignment.

Online computation: two 1D arrays prev, cur, track scores and start points (prev.value, cur.value, prev.start, cur.start) of j-1 and j columns of a.

Best solution (SMGT) in best = (bestvalue, beststart, bestend) accounting for δ.

Check subsequences with length r (Reset()). Update values: Match ai,j = 1 Mismatch left cell can be inherited if length < r-1, top cell

always.

0, ,0

1, 1,

1, , 1

0, 0

1,if

max{ , },otherwise.

j i

fi j i j

i ji j i j

a a

a q xa

a a

19

SMBGT

Mismatch (propagation()): store the largest # of matched elements that can be propagated vertically or horizontally, not violating α and β.

Astart , Bstart : store the latest match positions in X and Q

Additional propagation check in Reset().

( ) (left propagation)

( 1) (top propagation)

prevstart

curstart

j A i

i B i

20

SMBGT: Example - InitializationQ = {0, -4, 1, 2, -2}, X = {0, 0, -4, 3,

0, 2, -3, 1}

α = 2, β = 1, δ = 3, r = |Q| = 5, ε = 0

X 0 0 -4 3 0 2 -3 1

Q 0 0 0 0 0 0 0 0 0

0 0

-4 0

1 0

2 0

-2 0

Alignment array α

21

SMBGT: Example - Computationa X

Q 0 0 -4 3 0 2 -3 1

0 1

-4 1

1 0

2 0

-2 0

Astart X

Q 0 0 -4 3 0 2 -3 1

0 1

-4 1

1 0

2 0

-2 0

s X

Q 0 0 -4 3 0 2 -3 1

0 1

-4 1

1 0

2 0

-2 0

Bstar

t

X

Q 0 0 -4 3 0 2 -3 1

0 1

-4 1

1 0

2 0

-2 0

22


Q 0 0 -4 3 0 2 -3 1

0 1 1

-4 1 1

1 0 0

2 0 0

-2 0 0

Astart X

Q 0 0 -4 3 0 2 -3 1

0 1 2

-4 1 2

1 0 0

2 0 0

-2 0 0

s X

Q 0 0 -4 3 0 2 -3 1

0 1 2

-4 1 2

1 0 0

2 0 0

-2 0 0

Bstar

t

X

Q 0 0 -4 3 0 2 -3 1

0 1 1

-4 1 1

1 0 0

2 0 0

-2 0 0

23


Q 0 0 -4 3 0 2 -3 1

0 1 1 1

-4 1 1 2

1 0 0 2

2 0 0 0

-2 0 0 0

Astart X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2

-4 1 2 3

1 0 0 3

2 0 0 0

-2 0 0 0

s X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2

-4 1 2 2

1 0 0 2

2 0 0 0

-2 0 0 0

Bstar

t

X

Q 0 0 -4 3 0 2 -3 1

0 1 1 1

-4 1 1 2

1 0 0 2

2 0 0 0

-2 0 0 0

24


Q 0 0 -4 3 0 2 -3 1

0 1 1 1 1

-4 1 1 2 2

1 0 0 2 2

2 0 0 0 0

-2 0 0 0 0

Astart X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2 2

-4 1 2 3 3

1 0 0 3 3

2 0 0 0 0

-2 0 0 0 0

s X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2 2

-4 1 2 2 2

1 0 0 2 2

2 0 0 0 0

-2 0 0 0 0

Bstar

t

X

Q 0 0 -4 3 0 2 -3 1

0 1 1 1 1

-4 1 1 2 2

1 0 0 2 2

2 0 0 0 0

-2 0 0 0 0

25


Q 0 0 -4 3 0 2 -3 1

0 1 1 1 1 1

-4 1 1 2 2 2

1 0 0 2 2 2

2 0 0 0 0 0

-2 0 0 0 0 0

Astart X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2 2 5

-4 1 2 3 3 3

1 0 0 3 3 3

2 0 0 0 0 0

-2 0 0 0 0 0

s X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2 2 5

-4 1 2 2 2 2

1 0 0 2 2 2

2 0 0 0 0 0

-2 0 0 0 0 0

Bstar

t

X

Q 0 0 -4 3 0 2 -3 1

0 1 1 1 1 1

-4 1 1 2 2 2

1 0 0 2 2 2

2 0 0 0 0 0

-2 0 0 0 0 0

26


Q 0 0 -4 3 0 2 -3 1

0 1 1 1 1 1 1

-4 1 1 2 2 2 1

1 0 0 2 2 2 0

2 0 0 0 0 0 3

-2 0 0 0 0 0 3

Astart X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2 2 5 5

-4 1 2 3 3 3 5

1 0 0 3 3 3 0

2 0 0 0 0 0 6

-2 0 0 0 0 0 6

s X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2 2 5 5

-4 1 2 2 2 2 5

1 0 0 2 2 2 0

2 0 0 0 0 0 2

-2 0 0 0 0 0 2

Bstar

t

X

Q 0 0 -4 3 0 2 -3 1

0 1 1 1 1 1 1

-4 1 1 2 2 2 1

1 0 0 2 2 2 0

2 0 0 0 0 0 4

-2 0 0 0 0 0 4

27


Q 0 0 -4 3 0 2 -3 1

0 1 1 1 1 1 1

-4 1 1 2 2 2 1

1 0 0 2 2 2 0

2 0 0 0 0 0 3

-2 0 0 0 0 0 3

Astart X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2 2 5 5

-4 1 2 3 3 3 5

1 0 0 3 3 3 0

2 0 0 0 0 0 6

-2 0 0 0 0 0 6

s X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2 2 5 5

-4 1 2 2 2 2 5

1 0 0 2 2 2 0

2 0 0 0 0 0 2

-2 0 0 0 0 0 2

Bstar

t

X

Q 0 0 -4 3 0 2 -3 1

0 1 1 1 1 1 1

-4 1 1 2 2 2 1

1 0 0 2 2 2 0

2 0 0 0 0 0 4

-2 0 0 0 0 0 4

28


Q 0 0 -4 3 0 2 -3 1

0 1 1 1 1 1 1

-4 1 1 2 2 2 1

1 0 0 2 2 2 0

2 0 0 0 0 0 1

-2 0 0 0 0 0 3

Astart X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2 2 5 5

-4 1 2 3 3 3 5

1 0 0 3 3 3 0

2 0 0 0 0 0 6

-2 0 0 0 0 0 6

s X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2 2 5 5

-4 1 2 2 2 2 5

1 0 0 2 2 2 0

2 0 0 0 0 0 6

-2 0 0 0 0 0 2

Bstar

t

X

Q 0 0 -4 3 0 2 -3 1

0 1 1 1 1 1 1

-4 1 1 2 2 2 1

1 0 0 2 2 2 0

2 0 0 0 0 0 4

-2 0 0 0 0 0 4

29


Q 0 0 -4 3 0 2 -3 1

0 1 1 1 1 1 1

-4 1 1 2 2 2 1

1 0 0 2 2 2 0

2 0 0 0 0 0 1

-2 0 0 0 0 0 1

Astart X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2 2 5 5

-4 1 2 3 3 3 5

1 0 0 3 3 3 0

2 0 0 0 0 0 6

-2 0 0 0 0 0 6

s X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2 2 5 5

-4 1 2 2 2 2 5

1 0 0 2 2 2 0

2 0 0 0 0 0 6

-2 0 0 0 0 0 6

Bstar

t

X

Q 0 0 -4 3 0 2 -3 1

0 1 1 1 1 1 1

-4 1 1 2 2 2 1

1 0 0 2 2 2 0

2 0 0 0 0 0 4

-2 0 0 0 0 0 4

30


Q 0 0 -4 3 0 2 -3 1

0 1 1 1 1 1 1 1

-4 1 1 2 2 2 1 1

1 0 0 2 2 2 0 0

2 0 0 0 0 0 1 1

-2 0 0 0 0 0 1 1

Astart X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2 2 5 5 5

-4 1 2 3 3 3 5 5

1 0 0 3 3 3 0 0

2 0 0 0 0 0 6 6

-2 0 0 0 0 0 6 6

s X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2 2 5 5 5

-4 1 2 2 2 2 5 5

1 0 0 2 2 2 0 0

2 0 0 0 0 0 6 6

-2 0 0 0 0 0 6 6

Bstar

t

X

Q 0 0 -4 3 0 2 -3 1

0 1 1 1 1 1 1 1

-4 1 1 2 2 2 1 1

1 0 0 2 2 2 0 0

2 0 0 0 0 0 4 4

-2 0 0 0 0 0 4 4

31


Q 0 0 -4 3 0 2 -3 1

0 1 1 1 1 1 1 1 0

-4 1 1 2 2 2 1 1 0

1 0 0 2 2 2 0 0 2

2 0 0 0 0 0 1 1 2

-2 0 0 0 0 0 1 1 1

Astart X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2 2 5 5 5 0

-4 1 2 3 3 3 5 5 0

1 0 0 3 3 3 0 0 8

2 0 0 0 0 0 6 6 8

-2 0 0 0 0 0 6 6 6

s X

Q 0 0 -4 3 0 2 -3 1

0 1 2 2 2 5 5 5 0

-4 1 2 2 2 2 5 5 0

1 0 0 2 2 2 0 0 5

2 0 0 0 0 0 6 6 5

-2 0 0 0 0 0 6 6 6

Bstar

t

X

Q 0 0 -4 3 0 2 -3 1

0 1 1 1 1 1 1 1 0

-4 1 1 2 2 2 1 1 0

1 0 0 2 2 2 0 0 3

2 0 0 0 0 0 4 4 3

-2 0 0 0 0 0 4 4 4

32

SMBGT: Example - Solution

a X

Q 0 0 -4 3 0 2 -3 1

0 1 1 1 1 1 1 1 0

-4 1 1 2 2 2 1 1 0

1 0 0 2 2 2 0 0 2

2 0 0 0 0 0 1 1 2

-2 0 0 0 0 0 1 1 1

Q = 0-4 1 2 -2

bestvalue = 3, beststart = 2, bestend = 6

X = 0 0 -4 3 0 2 -3 1skipped

33

Outline

Motivation

Related Work

Problem Setting




34

Experimental Setup: Data & Evaluation Music DB: 5643 freely available MIDI files.

6 synthetic query sets, 100 queries/set of lengths 13-137 Q0: exact segments of the db. Q.10 – Q.50: randomly modified 10-50% elements (both

dimensions) of each query in Q0, with at most 3 consecutive elements changed.

100 hummed queries of lengths 14-76.

DP and Model-based competitor methods DTW variants: DTWs, DTWc, SPRING modified to allow r. Edit: Version for music modified for LogIOIR + quantizations Iliopoulos et al.: Elastic version for subsequence matching

supporting constant and variable tolerance, modified to allow r. HMMs: Each db sequence modeled by an HMM.

35

Experimental Setup: Evaluation measures Recall: % of queries for which correct

answer is in top-K. MRR: mean inverse rank of queries in their

top-K results. AR: average rank of all queries in the set,

where rank is the # of sequences with score at least as good as that of the correct match.

36

Experimental Setup: Variable Tolerances Pitch

Absolute and relative tolerances tested, withand for relative tolerance two cases considered

Duration IOIR: LogIOIR:

( ) * , 0.2,0.25,0.5f pp ii q t t

/(1 ) *(1 ), , 0

/(1 ) *(1 ), , 0

p f p p f p pj p i j p i j

p f p p f p pj p i j p i j

x q x q x

x q x q x

( , ) { 2* , 0.5}f r r r rr j i j ii j x q x q

2 2

2 2

{0 log ( / ) 1}, log 0( , )

{| log ( / ) | 1}, log 0

r r rj i jf

r r r rj i j

x q xi j

x q x

37

Experimental Results: Synthetic Queries (1/4) Parameters

r = |Q|, δ = 0.9, 0.7. 0.6, 0.5, 0.35, 0.3 for Q0 - Q.50 , α = β = 3,tested both tolerance schemes, K = 20.

Best accuracy SMBGT: variable abs. tolerance, t = 0.2. SMGT: variable abs. tolerance, t = 0.25. Il. et al.: constant rel. tolerance with εp = 1, εr = 4. HMM: #states = 5, Gaussian observation distribution of the

states.

Representation ↑ noise: high recall for representations subset of lower

noise levels’ DP methods: <mod12, IOIR>, <pitch interval, IOIR> HMM: <mod12, LogIOIR in [-2,2]> smallest alphabet

38

Experimental Results: Synthetic Queries (2/4) Q0

DP-based methods: 100% recall, MRR = 1, AR = 1 for top-5.

DTWs: 96% recall for K = 250.

HMM: 96% recall for K = 5, MRR=0.95, AR = 1.22.

39

Experimental Results: Synthetic Queries (3/4) Q0

DP-based methods: 100% recall, MRR = 1, AR = 1 for top-5.

DTWs: 96% recall for K = 250.

HMM: 96% recall for K = 5, MRR=0.95, AR = 1.22.

40

Experimental Results: Synthetic Queries (4/4)

41

Experimental Results: Hummed Queries Tolerance

Small constant and variable t better than greater values.

Variable t better than constant. Absolute better than relative. SMBGT and SMGT: absolute, t = 0.2 and 0.25.

Any tol. scheme.

42

Outline

Motivation

Related Work

Problem Setting




43

Summary

1. Lack of flexibility for existing subsequence matching methods. Edit most promising, but 30 times lower recall than SM(B)GT for K = 50, α, β improve accuracy.

2. Variable ε-tolerance better than constant tolerance.3. Small range for α, β tested. Other applications

cross validation.4. δ tuned according to hummers’ skills, r not exceed

1.2*|Q|.5. Simple representation favor DP-based methods.6. DP-based methods have similar retrieval time

complexity.

44

Conclusions and Future Work We proposed a novel subsequence matching

framework. We showed that it can be applied to noisy

application domains, such as QBH. We performed an extensive experimental

evaluation on a large collection of music songs using hummed queries.

Directions for future work include Testing the proposed method on other noisy domains. Developing a software application based on SMBGT. Indexing.

Thank you!!!

46

Questions ???

Appendix

48

SMBGT

Theorem: SMBGT is not metric, as it does not follow the triangle inequality. Consider: 1D time series X1 = {3, 2}, X2 = {4}, X3 =

{2, 2, 1} ε-tolerance = 1, r = inf, δ = 0, α = β = 1 SMBGT(X1, X2) = 1, SMBGT(X2, X3) = 0,

SMBGT(X1, X3) = 2, butSMBGT(X1, X2) + SMBGT(X2, X3) < SMBGT(X1, X3).

49

SMBGT: Pseudocode

50

DP Methods – Edit distance Latest version, extended for LogIOIR and

quantizations. PitchRange, DurationRange : max pitch interval, and

LogIOIR range in DB. Reports minj{α|Q|,j}.

51

DP Methods – SPRING

w(qi, xj): Lp of qi and xj

s to keep start points of best alignments. At each iteration, the start point of the element used to produce dbest is propagated.

Reports minj{α|Q|,j}.

52

DP Methods – DTW tempo scaling c: user-defined positive integer w(qip, xjp) = |qip - xjp| Report minj{α|Q|,j}.

53

DP Methods – Iliopoulos et al. Designed for whole sequence matching. Demands all points of Q match within constant ε-

tolerance in a subsequence of X. Allows limited # of gaps in X. Developed a more elastic version suitable for

subsequence matching Single db scan identifies all possible start points of

candidate matches. For each candidate, perform DP computation. Whole query matching requirement eliminated. When

non-matched query element is met, we consider Q[1:i] candidate match.

54

HMM Doubly stochastic process defined by

M distinct states L distinct symbols that can be observed at each state –

alphabet T = {tij} of transition probabilities, where tij = P[st = j|st-1 = i],

1<=i,j<=M E = {ej(k)} of the probabilities of observation symbols at j,

where ej(k) = P[ot = k|st = j] Π = {πj} of prior probabilities, where πj = P[s1 = j], 1<=j<=M.

Reflects the probabilistic relations of symbols within sequences.

Look for the model maximizing the log-likelihood of the query.

Model each db sequence with one HMM.

55

Data Database

Covers various music genres, e.g., blues, rock, classical, jazz, themes from movies and tv series, total of 40891 time series.

Converting music pieces to time series (offline process): All-channels extraction: for all channels, extracted the highest

pitch at every time click. Converted tuples <pitch, click> to <pitch interval, IOIR>.

Synthetic Queries Pitch interval change by ±k in [3, 8] (error by human + audio

tool). IOIR modified by ±k (real) in [2, 4]. At most 3 consecutive elements were replaced by noisy values.

Hummed Queries 4 males (2 with + 2 without musical training), 25 songs each. Conversion to MIDIs with Akoff. Several genres of music, no lyrics, much noise several versions

of each song.

56

Experimental Results: Accuracy

57

Experimental Results: Accuracy

58

Experimental Results: Representation

59

Experimental Results: Complexities All DP-based methods: O(|Q|*|X|). Iliopoulos et al.: O(|X|+|Q|*|X|^2). HMM

Training for a sequence X: O(W*|X|*M^2),W = #iterations for Baum-Welch algorithm.

Log-likelihood of Q : O(|Q|*M^2).

Documents

A Subsequence Matching with Gaps-Range-Tolerances Framework: A Query-By-Humming Application Alexios Kotsifakos University of Texas at Arlington, University