Evaluating citation functions in CiTO: cognitive issues

STLab University of Bologna#eswc2014Ciancarini

!

Evaluating citation functions in CiTO: cognitive issues

!

!

!!!

28 May 2014 - Heraklion, Crete

Paolo Ciancarini1,2, Angelo Di Iorio1, Andrea Giovanni Nuzzolese1,2, Silvio Peroni1 and Fabio Vitali1

1Department of Computer Science and Engineering, University of Bologna, Italy 2STLab, Institute of Cognitive Science and Technology, National Research Council, Rome, Italy


Outline

2

• Motivations

• CiTO

• Experiment

• Evaluation

• Lessons learnt and conclusions


• Bibliographic citations can be seen as tools for:

• linking, disseminating, exploring, evaluating research

• The task of characterising the functional role of citations in scientific literature is very difficult for agents and humans

• Investigating how an existing reference model for classifying citations, i.e., CiTO, is interpreted and used by human annotators

• We want to study human’s behaviour in order to simulate it within CiTalO, a tool that automatically classifies citations with CiTO

Motivations and goals

3


• An OWL ontology for describing factual as well as rhetorical functions of citations in scholarly articles

• Defines

• a top-level property cites (and its inverse isCitedBy)

• 41 sub-properties of cites that allow users to characterise precisely the semantics of a citation act

• Has been successfully used in large projects, like CiteULike, data.open.ac.uk and the Open Citation Corpus

• Several tools have been developed to annotate citations with CiTO

• e.g., Chrome and WordPress plug-ins

CiTO

4


• The richness of properties in CiTO (CiTO-Ps) is

• a key feature of CiTO: this aspect has contributed to the adoption of the ontology by the Semantic Publishing

Users’ adoption of CiTO

5


• The richness of properties in CiTO (CiTO-Ps) is

• a key feature of CiTO: this aspect has contributed to the adoption of the ontology by the Semantic Publishing

• an hindrance: most tools actually employ a sub-set of the CiTO properties,

• e.g., 6 CiTO-Ps enabled for user annotation by Pensoft Publishers and 9 in the Chrome plug-in

Users’ adoption of CiTO

5

STLab University of Bologna#eswc2014Ciancarini6

user

author

“It extends the research outlined in earlier work [3]”

CiTO annotations and mental models

CiTO


Interpretation of author’s

text

Understanding of CiTO

user

author



CiTO



text


user

mental model

author



CiTO



text


user

mental model

Annotation

cito:extends

author



CiTO


mental model

mental model

6


text


user

mental model

Annotation

cito:extends

author


mental modelmental

model


CiTO


mental model

mental model

6


text


user

mental model

Annotation

cito:extends

author


mental modelmental

model


CiTO!cito:citesForInformation

cito:givesSupportTo …

The mental models of different annotators

hardly ever converge to a single shared opinion


What we did

We performed an experiment to investigate how humans use CiTO to annotate citations with a type

• 20

• 105 citations chosen among the seventh volume of the proceedings of the Balisage Conference Series

subjects


What we did

We performed an experiment to investigate how humans use CiTO to annotate citations with a type

• 20

• 105 citations chosen among the seventh volume of the proceedings of the Balisage Conference Series

(lucky) subjects


• The experiment had one independent variable, i.e., the number of CiTO-Ps available to subjects for the annotation

• Condition T41: 10 subjects used 41 CiTO-Ps

• Condition T10: 10 subjects used a subset of 10 CiTO-Ps

• i.e., citesAsDataSource, citesAsPotentialSolution, citesAsRecommendedReading, citesAsRelated, citesForInformation, credits, critiques, includesQuotationFrom, obtainsBackgroundFrom, usesMethodIn

• T10 CiTO-Ps were chosen among those that had shown a moderate inter-rater agreement (Fleiss’k>0.33) in a preliminary experiment on the same data sample

The experiment

8


!

Evaluation framework

9

Citation context CITO-Ps

Explanations and examples of the usage of the CiTO-Ps

Available on line at http://www.cs.unibo.it/~nuzzoles/cito_1/?user=r

http://www.cs.unibo.it/~nuzzoles/cito_1/?user=r


1. Which properties have been used by subjects during the experiment?

2. Which were the most used properties?

3. What was the global inter-rater agreement of the subjects?

4. Did the number of available choices bias the global inter-rater agreement?

5. Which properties showed an acceptable positive agreement among subjects?

6. Could properties be organized according to their similarity in subjects’ annotations?

7. What was the perceived usability of the CiTO-Ps?

8. Which were the features of CiTO-Ps that subjects perceived as most useful or problematic?

Target questions


!

!

• Condition T41

• used 37 different CiTO-Ps over 41(avg: 21.7 CiTO-Ps per subject)

• 4 properties not selected by any subject

• i.e., parodies, plagiarizes, repliesTo and ridicules

!

• Condition T10

• used all the 10 CiTO-Ps

Results“Which properties have been used by subjects during the experiment?”


Results“Which were the most used properties?”


“What was the global inter-rater agreement of the subjects?” “Did the number of available choices bias the global inter-rater agreement?”

“Which properties showed an acceptable positive agreement among subjects?”

13

• Condition T41

• Global Fleiss’k = 0.13 • 5 CiTO-Ps with moderate local positive agreement (k > 0.5)

• i.e., citesAsPotentialSolution (0.66), citesAsRecommendedReading (0.6), agreesWith (0.54), citesAsDataSource (0.52), usesMethodIn (0.54)

• Condition T10

• Global Fleiss’k = 0.15 • 4 CiTO-Ps with moderate local positive agreement

• i.e., citesAsPotentialSolution (0.71), citesAsDataSource (0.63), citesAsRecommendedReading (0.52), includesQuotationFrom (0.69)

Data evaluation




13

• Condition T41



• Condition T10



Data evaluation

The global agreement is

very low




13

• Condition T41



• Condition T10



Data evaluation


very low

The # of CiTO-Ps does not affect the

agreement




13

• Condition T41



• Condition T10



Data evaluation


very low

The # of CiTO-Ps does not affect the

agreementThe set of CiTO-PS with moderate local positive

agreement is little affected by the # of

CiTO-Ps


• We applied the Chinese Whispers clustering algorithm

• Input: 2 graphs built by combining all the pairs of different CiTO-Ps as annotated by subjects for each citation

• Gr - it takes into account repetitions in annotations for a each CiTO property

• e.g., “extends”, “extends”, and “updates” on a citation generate (extends,updates) and (extends,updates)

• Gn - it does not take into account repetitions

• e.g., “extends”, “extends”, and “updates” generate (extends,updates)

14

Clustering CiTO-Ps“Could properties be organized

according to their similarity in subjects’ annotations?”




Gr Gn

disputes

critiques

deridesrefutes confirms

credits

obtainsSupportFrom




Gr Gn

disputes

critiques

deridesrefutes confirms

credits

obtainsSupportFrom

There exist some sort of relations (e.g., taxonomical,

equivalence) among the CiTO-Ps of each cluster


• We computed the System Usability Scale (SUS)

Measuring the usability of CiTO-Ps

0.00!

10.00!

20.00!

30.00!

40.00!

50.00!

60.00!

70.00!

80.00!

90.00!

100.00!

SUS mean! Usability mean! Learnability mean!

T41!

T10!

“What was the perceived usability of the CiTO-Ps? ”


• We computed the System Usability Scale (SUS)

Measuring the usability of CiTO-Ps

0.00!

10.00!

20.00!

30.00!

40.00!

50.00!

60.00!

70.00!

80.00!

90.00!

100.00!

SUS mean! Usability mean! Learnability mean!

T41!

T10!

“What was the perceived usability of the CiTO-Ps? ”

Only the usability score approaches the statistical significance


Grounded theory analysis

• The subjects filled a final text questionnaire aimed at capturing positive and negative aspects of CiTO-Ps

• We used the text answers for performing a grounded theory analysis, used in Social Science to extract relevant concepts from unstructured text

“Which were the features of CiTO-Ps that subjects perceived as most useful or problematic?”


Grounded theory analysis“Which were the features of CiTO-Ps that subjects perceived as most useful or

problematic?”


Conclusions• Lessons learnt and suggestions to improve CiTO

• Reduce the number of less-used properties

• Identify the most-used neutral properties

• Investigate motivations for low inter-rater agreement

• Define explicit relations between CiTO properties

• Add support for customised properties

• Extend examples, labels and explanations

• Future work

• Improve CiTalO, a tool for identifying automatically the nature of citations

• e.g., by investigating cognitive architectures in order to simulate humans’ behaviour


Thank you!