Pragmatic Evaluation of Concept Hierarchies

Graz University of Technology

1

T Trattner C., Singer P., Helic D., Strohmaier M. I-Know 2012

Pragmatic Evaluation of Concept Hierarchies

Christoph Trattner, Philipp Singer

Denis Helic, Markus Strohmaier

Graz University of Technology, Austria


2


What is this talk about

We will introduce a framework to evaluate concept hierarchies that do not rely on a Golden-Standard

Framework determines the pragmatic usefulness of concept hierarchies utilizing Kleinberg’s idea of hierarchical decentralized search

We will show evidence that the framework does not only work in theory but also in practice

Part 1

Part 2


3


What was the motivation of our research?


4


Directories: Categorization by Experts


5


Research question

Can a crowd of users contribute to the creation of such categorizations?

How can we generate such hierarchical structures automatically?


6


Annotation by Users: Tagging

Folksonomy Tuple (U, R, T, Y) User (U) Resource (R) Tag (T) Relation (Y)


7


Folksonomies

Emerge from the process of collaborative tagging

Latent hierarchical structures

Turn flat structure into hierarchy taxonomy induction algorithms Generality-based algorithms (centrality in tag-to-tag networks) Other algorithms possible: k-means, affinity propagation, ... E.g., [Heyman and Garcia-Molina 2006] or [Benz et al. 2010]


8


Problem: How can we evaluate the usefulness of these hierarchies?

Idea: Golden standard based methods Problem: Lack of golden standard [Strohmaier et al. 2012]

little taxonomic overlap => results are not trustworthy

Very small overlap !!!M. Strohmaier, D. Helic, D. Benz, C. Körner and R. Kern, Evaluation of Folksonomy Induction Algorithms, In the ACM Transactions on Intelligent Systems and Technology


9


Question?

Can we somehow find another evaluation method?


10


Stanley Milgram

A social psychologist Yale and Harvard University

Study on the Small World Problem,beyond well defined communities and relations(such as actors, scientists, …)

„An Experimental Study of the Small World Problem”

1933-1984


11


The simplest way of formulating the small-world problem is: Starting with any two people in the world, what is the likelihood that they will know each other?

A somewhat more sophisticated formulation, however, takes account of the fact that while person X and Z may not know each other directly, they may share a mutual acquaintance - that is, a person who knows both of them. One can then think of an acquaintance chain with X knowing Y and Y knowing Z. Moreover, one can imagine circumstances in which X is linked to Z not by a single link, but by a series of links, X-A-B-C-D…Y-Z. That is to say, person X knows person A who in turn knows person B, who knows C… who knows Y, who knows Z.

[Milgram 1967, according to]http://www.ils.unc.edu/dpr/port/socialnetworking/theory_paper.html#2]


12


An Experimental Study of the Small World Problem [Travers and Milgram 1969]

A Social Network Experiment tailored towards Demonstrating Defining And measuring

Inter-connectedness in a large society (USA)

A test of the modern idea of “six degrees of separation”

Which states that: every person on earth is connected to any other person through a chain of acquaintances not longer than 6


13


Set Up

Target person: A Boston stockbroker

Three starting populations 100 “Nebraska stockholders” 96 “Nebraska random” 100 “Boston random”

Nebraska random

Nebraska stockholders

Boston stockbroker

Boston random

Target


14


Results

How many of the starters would be able to establish contact with the target? 64 out of 296 reached the target

How many intermediaries would be required to link starters with the target? Well, that depends: the overall mean 5.2 links Through hometown: 6.1 links Through business: 4.6 links Boston group faster than Nebraska groups Nebraska stockholders not faster than Nebraska random

What form would the distribution of chain lengths take?


15


Decentralized Search

Search in (social) networks people have only local knowledge of the network

People have background knowledge of the network, e.g. geography

Background knowledge defines the notion of distance between nodes

People are greedy: at each step people select a node that has the smallest distance to the target

Kleinberg explained the process of navigating a network and finding others with only local knowledge

Decentralized search with hierarchical background knowledge [Kleinberg 2000]


16


Hierarchical decentralized searcher

InformationNetwork

Hierarchy


17


Idea!

Use Kleinberg‘s model of decentralized search in social networks and apply it to information networks.


18


Framework

Hence, we implemented a framework that takes as input a given hierarchy & network and determines the usefulness of this hierarchy for navigating the network [Helic et al. 2011].

FrameworkUseful? Yes/No

Hierarchy

Network

Hierarchical Decentralized Searcher D. Helic, M. Strohmaier, C. Trattner, M. Muhr, K.

Lerman, Pragmatic Evaluation of Folksonomies, 20th International World Wide Web Conference (WWW2011), Hyderabad, India, March 28 - April 1, ACM, 2011.


19


Question?

To what extent are current tag hierarchy induction algorithms useful for navigation?


20


Evaluating Tag Hierarchy Induction Algorithms

In [Helic et al. 2011 we used this kind of framework to evaluate 5 different hierarchy induction algorithms on 5 different datasets (25 combinations) BibSonomy Delicious CiteUlike Flickr LastFM

Simulations were based on a random sample of 100.000 search pairs

Measuring the success rate and stretch for evaluation


21


Evaluating Tag Hierarchy Induction Algorithms

BibSonomy CiteULikeDelicious

Flickr LastFM

Results:

Centrality-based hierarchy induction algorithms outperform complicated methods such as K-Means or Affinity Propagation


22


Question

What are the differences and similarities of hierarchies based on different types of annotations?

To what extent are hierarchies based on tags more useful for navigation than hierarchies based on keywords?


23


We

Keywords

Tags


24


Results

Results:

Tag-based Hierarchies are more useful for navigation than keyword-based hierarchies


25


Question???

To what extent is it justified to model human navigation in information networks with hierarchical

decentralized search?


26


Idea?

Compare Simulations with real world data!

Exploring the Differences and Similarities between Hierarchical Decentralized Search and Human Navigation in Information Networks


27


Evaluation

We compared simulations with

human click trails of the online Game –

The Wiki Game (http://thewikigame.com/)

Contains 1,500,000

click trails of more

than 500,000 users with

(start; target) information.


28


Hierachy Creation

Two types of hierarchies were evaluated

1.) First type is based on our previous work Categorial Concepts:

Tags from Delicious Category labels from Wikipedia

Similarity GraphLatent Hierarchical Taxonomy

Wikipedia Category Label Dataset: 2,300,000 category labels,4,500,000 articles, 30,000,000 category label assignments

Delicious Tag Dataset: 440,000 tags, 580,000 articles and3,400,000 tag assignments


29


Hierarchy Creation

2.) Second type is based on the work of [Muchnik et al. 2007]

Muchnik, L., Itzhack, R., Solomon S. and Louzoun Y.: Self-emergence of knowledge trees: Extraction of the Wikipedia hierarchies, PHYSICAL REVIEW E 76, 016106 (2007)

Simple idea: Algorithm iterates through all links in the network and decides if that link is of a hierarchical type, in which case it remains in the network otherwise it is removed.

Directed link-network dataset of theEnglish-Wikipedia from February 2012.

All in all, the dataset includesaround 10,000,000 articles and around 250,000,000 links


30


Evaluation Metrics

Success Rate: Percentage of target nodes found Number of Hops: Number of hops needed to reach the target

node Stretch: Fraction of number of the number of steps and global

shortest path Path Similarity: intersection(h_clicks,s_clicks)/s_clicks Degree: median in- and out-degree values of the nodes visited

by the simulator and the human navigator

Transition Similarity


31


What are the results??


32


Results: Hops, Stretch, Success Rate

Humans Searcher with Wikipedia CategoryHierarchy

Success Rate: 31.6%Stretch: 1.7

Success Rate: 100%Stretch: 2.5


33



Humans Searcher with Wikipedia Delicious Hierarchy




34



Humans



Searcher with Wikipedia Network Hierarchy


35


Results: Path Similarity

Humans vs. Humans Humans vs. Simulators

Question: How similar are the paths taken by our searcher compared to the humans


36


Results: Degree

In- Degree Out- Degree


37


Results: Transition Similarity

Humans Searcher


38


Conclusions

We have shown that our approach of hierarchical decentralized search models human navigation in information networks fairly well

Furthermore, we have shown that hierarchies created directly from the link network are better suited for navigation than hierarchies that are created from external knowledge


39


What we plan for the Future?

Enhance the framework to consider not only navigation but also search (= search box)

Evaluation of alternative navigational structures

and many more things


40


Thank you!

Christoph Trattner

[email protected]

@ctrattner

Philipp Singer

[email protected]

@ph_singer

Denis Helic

[email protected]://coronet.iicm.edu/denis/homepage/@dhelic

Markus Strohmaier

[email protected]

@mstrohm

Take home message

Network hierarchies are better suited for navigation than hierarchies created from

external knowledge

mailto:[email protected]

http://www.christophtrattner.info/


http://www.philippsinger.info/


http://coronet.iicm.edu/denis/homepage/







http://www.markusstrohmaier.info/