22
Evaluating Hierarchical Clustering of Search Results Departamento de Lenguajes y Sistemas Informáticos UNED, Spain Juan Cigarrán Anselmo Peñas Julio Gonzalo Felisa Verdejo nlp.uned.es SPIRE 2005, Buenos Aires

Evaluating Hiera r chical Clustering of Search Results

  • Upload
    tirza

  • View
    20

  • Download
    0

Embed Size (px)

DESCRIPTION

Evaluating Hiera r chical Clustering of Search Results. SPIRE 2005, Buenos Aires. Departamento de Lenguajes y Sistemas Informáticos UNED, Spain Juan Cigarrán Anselmo Peñas Julio Gonzalo Felisa Verdejo nlp.uned.es. Overview. Scenario Assumptions - PowerPoint PPT Presentation

Citation preview

Page 1: Evaluating Hiera r chical Clustering of Search Results

Evaluating Hierarchical Clustering of Search Results

Departamento de Lenguajes y

Sistemas Informáticos

UNED, Spain

Juan Cigarrán

Anselmo Peñas

Julio Gonzalo

Felisa Verdejo

nlp.uned.es

SPIRE 2005, Buenos Aires

Page 2: Evaluating Hiera r chical Clustering of Search Results

                   

Overview

Scenario Assumptions Features of a Good Hierarchical Clustering Evaluation Measures

– Minimal Browsing Area (MBA)– Distillation Factor (DF)– Hierarchy Quality (HQ)

Conclusion

Page 3: Evaluating Hiera r chical Clustering of Search Results

                   

Scenario Complex information needs

– Compile information from different sources– Inspect the whole list of documents

• More than 100 documents

Help to– Find the relevant topics– Discriminate from unrrelevant documents

Approach– Hierarchical Clustering – Formal Concept Analysis

Page 4: Evaluating Hiera r chical Clustering of Search Results

                   

Page 5: Evaluating Hiera r chical Clustering of Search Results

                   

Page 6: Evaluating Hiera r chical Clustering of Search Results

                   

Problem

How to define and measure the quality of a hierarchical clustering?

How to compare different clustering approaches?

Page 7: Evaluating Hiera r chical Clustering of Search Results

                   

Previous assumptions Each cluster contains only those documents

fully described by its descriptors

d1 d2 d3 d4

Physics X X X X

Nuclear physics

X X

Astrophysics

X

d2, d3

d1Physics

Astrophysicsd4

Nuclear physics

d2, d3

d1, d2, d3, d4 Physics

Astrophysicsd4

Nuclear physics

Page 8: Evaluating Hiera r chical Clustering of Search Results

                   

Previous assumptions ‘Open world’ perspective

d1 d2 d3

Physics X X

Jokes X X

Jokes about physics

X

d1Physics

Jokes about physics

d3

Jokesd2

Jokes about physicsd3

d1Physics

Jokesd2

Jokes about physicsd3

Page 9: Evaluating Hiera r chical Clustering of Search Results

                   

Good Hierarchical Clustering The content of the clusters.

– Clusters should not mix relevant with non relevant information

+ + +

+ + + +

-- -+ - -

+ - + +

-- +

Page 10: Evaluating Hiera r chical Clustering of Search Results

                   

Good Hierarchical Clustering The hierarchical arrangement of the clusters

– Relevant information should be in the same path

+ + + +

- - -++ +

- - -

+ + +

+ + + +

-- -

- - -

Page 11: Evaluating Hiera r chical Clustering of Search Results

                   

Good Hierarchical Clustering The number of clusters

– Number of clusters substantially lower than the number of documents

How clusters are described– Cognitive load of reading a cluster description– Ability to predict the relevance of the information

that it contains (not addressed here)

Page 12: Evaluating Hiera r chical Clustering of Search Results

                   

Evaluation Measures Criterion

– Minimize the browsing effort for finding ALL relevant information

Baseline– The original document list returned by a search

engine

(lattice) load cognitive

list) (ranked load cognitivelatticeQuality

Page 13: Evaluating Hiera r chical Clustering of Search Results

                   

Evaluation Measures Consider

– Content of clusters– Hierarchical arrangement of clusters– Size of the hierarchy– Cognitive load of reading a document (in the

baseline): Kd – Cognitive load of reading a node descriptor (in the

hierarchy): Kn

Requirement– Relevance assessments are available

Page 14: Evaluating Hiera r chical Clustering of Search Results

                   

Minimal Browsing Area (MBA) The minimal set of nodes the user has to traverse to

find ALL the relevant documents minimising the number of irrelevant ones

+

- - +

+

+ -

Page 15: Evaluating Hiera r chical Clustering of Search Results

                   

Distillation Factor (DF) Ability to isolate relevant information compared

with the original document list (Gain Factor, DF>1)

MBAMBAd

d

D

D

Dk

Dk

latticeloadCognitive

listrankedloadCognitiveLDF List RankedList Ranked

)(_

)_(_)(

List Ranked

MBA)(precision

precisionLDF

Considers only the cognitive load of reading documents

Equivalent to:

Page 16: Evaluating Hiera r chical Clustering of Search Results

                   

Distillation Factor (DF) Example

DF(L) = 7/5 = 1.4

Doc 1 +

Doc 2 -

Doc 3 +

Doc 4 +

Doc 5 -

Doc 6 -

Doc 7 +

Document List

Precision = 4/7 Precision MBA = 4/5

+

- - +

+

+ -

Page 17: Evaluating Hiera r chical Clustering of Search Results

                   

Distillation Factor (DF) Counterexample:

+ + - -+ -- +

Precision = 4/8

Precision MBA = 4/4

DF = 8/4 = 2

Bad clustering with good DF

Extend the DF measure considering the cognitive cost of taking browsing decisions HQ

Page 18: Evaluating Hiera r chical Clustering of Search Results

                   

Hierarchy Quality (HQ) Assumption:

– When a node (in the MBA) is explored, all its lower neighbours have to be considered: some will be in turn explored, some will be discarded

– Nview : subset of lower neighbours of each node belonging to the MBA

- +

- +

+

+-

-

-

-

MBA

|Nview|=8

Page 19: Evaluating Hiera r chical Clustering of Search Results

                   

Hierarchy Quality (HQ)

Kn and Kd are directly related with the retrieval scenario in which the experiments take place The researcher must tune K=Kn/Kd before conducting the

experiment

HQ > 1 indicates an improvement of the clustering versus the original list

viewMBA

List Ranked

viewMBA

List Ranked)(N

k

kD

D

NkDk

DkLHQ

d

nnd

d

(lattice) load cognitive

list) (ranked load cognitivelatticeQuality

Page 20: Evaluating Hiera r chical Clustering of Search Results

                   

Hierarchy Quality (HQ)

- +

- +

+

+-

-

-

-nd

d

n

kk

k

kLHQ

66.1

:Value Cutting

85

10)(

+ + - -+ -- +

nd

d

n

kk

k

kLHQ

3

:Value Cutting

124

8)(

Example

Page 21: Evaluating Hiera r chical Clustering of Search Results

                   

Conclusions and Future Work Framework for comparing different clustering

approaches taking into account:– Content of clusters– Hierarchical arrangement of clusters– Cognitive load to read document and node descriptions

Adaptable to the retrieval scenario in which experiments take place

Future work– Conduct user studies to compare their results with the

automatic evaluation• Results will reflect the quality of the descriptors• Will be used to fine-tune the kd and kn parameters

Page 22: Evaluating Hiera r chical Clustering of Search Results

Thank you!