30
Improving the Generation of Ground Truths based on Partially Ordered Lists Julián Urbano, Mónica Marrero, Diego Martín and Juan Lloréns http://julian-urbano.info Twitter: @julian_urbano ISMIR 2010 Utrecht, Netherlands, August 11th

Improving the Generation of Ground Truths based on Partially Ordered Lists

Embed Size (px)

DESCRIPTION

Ground truths based on partially ordered lists have been used for some years now to evaluate the effectiveness of Music Information Retrieval systems, especially in tasks related to symbolic melodic similarity. However, there has been practically no meta-evaluation to measure or improve the correctness of these evaluations. In this paper we revise the methodology used to generate these ground truths and disclose some issues that need to be addressed. In particular, we focus on the arrangement and aggregation of the relevant results, and show that it is not possible to ensure lists completely consistent. We develop a measure of consistency based on Average Dynamic Recall and propose several alternatives to arrange the lists, all of which prove to be more consistent than the original method. The results of the MIREX 2005 evaluation are revisited using these alternative ground truths.

Citation preview

Page 1: Improving the Generation of Ground Truths based on Partially Ordered Lists

Improving the Generation

of Ground Truths based on

Partially Ordered Lists

Julián Urbano, Mónica Marrero, Diego Martín and Juan Lloréns http://julian-urbano.info

Twitter: @julian_urbano

ISMIR 2010 Utrecht, Netherlands, August 11th

Page 2: Improving the Generation of Ground Truths based on Partially Ordered Lists

Outline

• Introduction • Current Methodology • Inconsistencies

▫ Due to Arrangement ▫ Due to Aggregation ▫ Fully Consistent Lists

• Alternative Aggregation Functions ▫ Measure of List Consistency

• Results ▫ MIREX 2005 Results Revisited

• Conclusions and Future Work • Some thoughts on Evaluation in MIR

2

Page 3: Improving the Generation of Ground Truths based on Partially Ordered Lists

Similarity Tasks

• Symbolic Melodic Similarity (SMS)

• Audio Music Similarity (AMS)

▫ Not covered here

• Given a piece of music (i.e. the query) retrieve others musically similar to it

• How do we measure the similarity of a document to a query (i.e. the relevance)?

▫ Traditionally with fixed level-based scales

Similar, not similar

Very similar, somewhat similar, not similar

3

Page 4: Improving the Generation of Ground Truths based on Partially Ordered Lists

Relevance Judgments

• For similarity tasks, they are very problematic

• Relevance is rather continuous [Selfridge-Field, 1998][Typke et al., 2005]

▫ Single melodic changes are not perceived to change the overall melody Move a note up or down in pitch Shorten or enlarge it Add or remove a note

▫ But the similarity is weaker as more changes apply

• Where is the line between relevance levels?

4

Page 5: Improving the Generation of Ground Truths based on Partially Ordered Lists

Partially Ordered Lists

• The relevance of a document is implied by its position in a partially ordered list [Typke et al., 2005]

▫ Does not need any prefixed relevance scale

• Ordered groups of documents equally relevant

▫ Have to keep the order of the groups

▫ Allow permutations within the same group

5

Page 6: Improving the Generation of Ground Truths based on Partially Ordered Lists

Partially Ordered Lists (II)

6

Relevance levels do show up, but they are not pre-fixed beforehand

Page 7: Improving the Generation of Ground Truths based on Partially Ordered Lists

Partially Ordered Lists (III)

• Used in the first edition of MIREX in 2005 [Downie et al., 2005]

• Widely accepted by the MIR community

to report new developments [Urbano et al., 2010][Pinto et al., 2008][Hanna et al., 2007][Gratchen et al., 2006]

• Four-step methodology

1. Filter out non-similar documents in the collection 2. Have the experts rank the candidates 3.Arrange the candidates by their median/mean rank 4.Aggregate candidates whose ranks are not

significantly different (Mann-Whitney U) [Mann et al., 1947]

7

Page 8: Improving the Generation of Ground Truths based on Partially Ordered Lists

Partially Ordered Lists (and IV)

• MIREX was forced to move to traditional level-based relevance since 2006 [Downie et al., 2010]

▫ Partially ordered lists are expensive (step 2) ▫ They have some odd results (step 2) ▫ They are hard to replicate (step 2) ▫ It may leave out relevant results (step 1)

• We have already explored alternatives to step 2

(and by extension 3 and 4) [Urbano et al., SIGIR CSE 2010]

▫ 3-point preference judgments via crowdsourcing

• Here we focus on steps 3 and 4 ▫ The lists have inconsistencies that lead to

incorrect evaluation

8

Page 9: Improving the Generation of Ground Truths based on Partially Ordered Lists

Intra-group Inconsistencies

• Two incipits in the same group were ranked significantly different by the experts

• If a system returns them in reverse order it will be considered correct, despite they were ranked clearly different by the experts

9

2 3 4 5 6 7 8 9

≠ ≠ ≠ ≠ ≠ ≠ ≠ ≠ 1

≠ ≠ ≠ ≠ ≠ ≠ ≠ 2

= = = ≠ ≠ ≠ 3

= = ≠ ≠ ≠ 4

≠ ≠ ≠ ≠ 5

= = ≠ 6

= = 7

= 8

11 of the 21 pairs are incorrectly aggregated

Query 700.010.591-1.4.2

Page 10: Improving the Generation of Ground Truths based on Partially Ordered Lists

Inter-group Inconsistencies

• Two incipits in different groups were not ranked significantly different by the experts

• If a system returns them in reverse order it will not be considered correct, despite no difference could be found between their ranks

10

2 3 4 5 6 7 8

≠ ≠ ≠ ≠ ≠ ≠ ≠ 1

≠ ≠ ≠ ≠ ≠ ≠ 2

≠ = ≠ ≠ ≠ 3

= = = = 4

= = = 5

= = 6

= 7

Query 190.011.224-1.1.1

Page 11: Improving the Generation of Ground Truths based on Partially Ordered Lists

Due to Arrangement

• In step 3 incipits are ordered by median

▫ Mean to break ties

• But in step 4 the Mann-Whitney U test is used

• Central tendency measures (median and mean) might not be appropriate because

▫ They ignore the dispersion in the samples

• Incipits are incorrectly ordered in step 3

▫ Source of inter-group inconsistencies

11

Page 12: Improving the Generation of Ground Truths based on Partially Ordered Lists

Due to Aggregation

• Traverse the list from top to bottom ▫ Begin a new group if the pivot is significantly

different from all incipits in the current group

• This generates very large groups

▫ Incipits at the top are considered similar to the ones at the end just because they are both similar to the ones in the middle

▫ Source of intra-group inconsistencies 178 of the 509 intra-pairs (35%) inconsistent

• The group-initiator has to be very different

12

Page 13: Improving the Generation of Ground Truths based on Partially Ordered Lists

Due to Aggregation (and II)

• The aggregation function may place the pivot in a new group, but the next one is not different from the ones in the group just closed

▫ Source of inter-group inconsistencies

▫ The pivot was just sufficiently different

▫ Or it was incorrectly arranged in step 3

13

2 3 4 5 6 7 8

≠ ≠ ≠ ≠ ≠ ≠ ≠ 1

≠ ≠ ≠ ≠ ≠ ≠ 2

≠ = ≠ ≠ ≠ 3

= = = = 4

= = = 5

= = 6

= 7

Query 190.011.224-1.1.1

Page 14: Improving the Generation of Ground Truths based on Partially Ordered Lists

Fully Consistent Lists

• Two sources of inconsistency ▫ Arrangement (inter-) ▫ Aggregation (inter- and intra-)

• There is a more profound problem ▫ Hypothesis testing is not transitive ▫ Not rejecting H0 does not mean accepting it

• Mann-Whitney U may say something like this ▫ A < B, B < C and A ≥ C (1-tailed test) ▫ A = B, B = C and A ≠ C (2-tailed test)

• We can not ensure fully consistent lists

14

Page 15: Improving the Generation of Ground Truths based on Partially Ordered Lists

Alternative Aggregation

• A function too permissive lead to large groups

▫ Likelihood of intra-group inconsistencies

• A function too restrictive leads to small groups

▫ Likelihood of inter-group inconsistencies

• We consider three rationales to follow

▫ All: a group begins if all incipits are different from the pivot. This should lead to larger groups.

▫ Any: a group begins if any incipit is different from the pivot. This should lead to smaller groups.

▫ Prev: a group begins if the previous incipit is different from the pivot.

15

Page 16: Improving the Generation of Ground Truths based on Partially Ordered Lists

Alternative Aggregation (and II)

• After the arrangement in step 3 we may assume that an incipit ranked higher has a true rank either higher or equal, but not lower

▫ 1-tailed tests are more powerful than the 2-tailed

It is more probable for them to find a difference if there really is one

• Combine the three rationales with the two tests

• All-2, Any-2, Prev-2, All-1, Any-1 and Prev-1

▫ All-2 is the function originally used by Typke et al.

16

Page 17: Improving the Generation of Ground Truths based on Partially Ordered Lists

Measure of List Consistency

• Follow the logics behind ADR [Typke et al., 2006]

• Traverse the list from top to bottom ▫ Calculate the expanded set of allowed incipits

All previous ones and those in the same group

▫ Compute the percentage of correct expansions The pivot is not considered (it is always correct)

▫ Average over all ranks in the list Ignore the last rank (it always expands to all incipits)

• 1 = all expansions are correct

▫ Fully consistent list (not to be expected)

• 0 = that no expansion is correct

17

Page 18: Improving the Generation of Ground Truths based on Partially Ordered Lists

Measure of List Consistency (II)

• Ground truth = ⟨ (A, B), (C), (D, E, F) ⟩, but ▫ A = C (inter-group inconsistency, false negative)

▫ D ≠ F (intra-group inconsistency, false positive)

18

Position Correct

expansion Actual

expansion % of correct expansions

1 B,C B 0.5 2 A A 1 3 A,B A,B 1 4 A,B,C,E A,B,C,E,F 0.8 5 A,B,C,D,F A,B,C,D,F 1

List consistency 0.86

Page 19: Improving the Generation of Ground Truths based on Partially Ordered Lists

Measure of List Consistency (and III)

• Again, it comes in two flavors

• ADR-1 consistency with 1-tailed tests

▫ Accounts for inconsistencies due to arrangement and aggregation

• ADR-2 consistency with 2-tailed tests

▫ Only accounts for inconsistencies due to aggregation

19

Page 20: Improving the Generation of Ground Truths based on Partially Ordered Lists

Results

• Re-generate the 11 lists used in MIREX 2005 with the alternative aggregation functions

• Compare with the original All-2 in terms of

▫ ADR-1 consistency across the 11 queries

▫ Group size across the 11 queries

▫ Are they correlated?

• Re-evaluate the MIREX 2005 SMS task

▫ Would it have been different?

20

Page 21: Improving the Generation of Ground Truths based on Partially Ordered Lists

List Consistency vs Group Size

Aggregation

function

ADR-1

consistency

Incipits

per group Pearson’s r

All-2 0.844 3.752 -0.892***

Any-2 0.913** 2.539* -0.862***

Prev-2 0.857 3.683 -0.937***

All-1 0.881 3.297 -0.954***

Any-1 0.926** 1.981** -0.749***

Prev-1 0.916* 2.858 -0.939***

21

Page 22: Improving the Generation of Ground Truths based on Partially Ordered Lists

List Consistency vs Group Size (and II)

• The original function is outperformed by all the five alternatives proposed

▫ ADR-1 consistency raises from 0.844 to 0.926

Significant at the 0.05 level with just 11 data points

• The relative order is kept within test types

▫ All is worse than Prev, which is worse than Any

• All-x are also more variable across lists

• The smaller its groups, the more consistent the list

▫ This is why Any-x is better than All-x

22

Page 23: Improving the Generation of Ground Truths based on Partially Ordered Lists

Example: Query 600.053.481-1.1.1

All-2 Any-2 Prev-2 All-1 Any-1 Prev-1

1 1 1 1 1 1

2 2 2 2 2 2

3 3 3 3 3 3

3 3 3 3 3 3

3 3 3 3 3 3

3 4 4 3 4 4

3 4 4 4 5 5

3 4 4 4 5 5

3 5 4 4 5 5

ADR-1 consistency 0.782 0.908 0.928 0.95 0.975 0.975

% intra- inconsistencies 0.667 0.333 0.333 0.222 0 0

% inter- inconsistencies 0 0.1 0.037 0 0.033 0.033

23

Page 24: Improving the Generation of Ground Truths based on Partially Ordered Lists

MIREX 2005 Revisited

• The lists could have been more consistent

▫ How would that have affected the evaluation?

• Re-evaluate the 7 systems with the five alternative functions and compare the results

24

System All-2 Any-2 Prev-2 All-1 Any-1 Prev-1

GAM 0.66 0.59 0.66 0.624 0.583 0.605

O 0.65 0.607 0.65 0.643 0.593 0.639

US 0.642 0.604 0.642 0.639 0.594 0.628

TWV 0.571 0.558 0.571 0.566 0.556 0.564

L(P3) 0.558 0.52 0.558 0.54 0.515 0.534

L(DP) 0.543 0.503 0.543 0.511 0.494 0.506

FM 0.518 0.498 0.518 0.507 0.483 0.507

- 0.81 1 0.81 0.714 0.714

Page 25: Improving the Generation of Ground Truths based on Partially Ordered Lists

MIREX 2005 Revisited (and II)

• All systems perform up to 12% worse

▫ The alternatives have smaller groups, which allows fewer false positives due to intra-group inconsistencies

• The ranking of systems would have changed

▫ Kendall’s τ = 0.714 to 0.81

• We overestimated system effectiveness

▫ And not just in MIREX, other papers did too

25

Page 26: Improving the Generation of Ground Truths based on Partially Ordered Lists

Conclusions

• Partially ordered lists make a better ground truth for similarity tasks, but they have problems

• We disclosed new (more fundamental) issues

▫ Intra- and inter-inconsistencies

▫ We can not expect fully consistent lists

The evaluation will always be incorrect to some extent

At least with this methodology

• We proposed several alternatives and a way to measure the consistency of a list

▫ All alternatives yield more consistent ground truths

▫ Proving we have overestimated system performance

26

Page 27: Improving the Generation of Ground Truths based on Partially Ordered Lists

Future Work

• Evaluate other collections • The significance level used was α=0.25

▫ Why? How does it affect the consistency?

• Other effectiveness measures can be proposed

• We believe that partially ordered lists should come back to the official evaluations ▫ First, make them cheaper and solve their problems

• We are working on it! [Urbano et al., SIGIR CSE 2010]

▫ Auto-organizing preference judgments ▫ Crowdsourcing ▫ Pooling ▫ Minimal and incremental test collections

27

Page 28: Improving the Generation of Ground Truths based on Partially Ordered Lists

Evaluation Experiments

• Essential for Information Retrieval

• But somewhat scarce in Music IR

▫ Private collections

Royalties and Copyright do not exactly help…

▫ Non-standard methodologies

▫ Non-standard effectiveness measures

▫ Hard to replicate

▫ Threats to internal and external validity

• MIR community acknowledges the need for these formal evaluation experiments [Downie, 2004]

• MIREX came up in 2005 to help with this, but…

28

Page 29: Improving the Generation of Ground Truths based on Partially Ordered Lists

Meta-Evaluation Analysis

• … now we have to meta-evaluate

▫ How well are we doing?

▫ Are we really improving our systems?

▫ Are we fair with all systems?

▫ Should we try new methodologies?

▫ Are we really measuring what we want to?

▫ How far can we go?

▫ Are we covering all user needs?

▫ Are our assumptions reasonable?

• Can we improve the evaluation itself?

▫ It would make the field improve more rapidly

29