Katrin Tomanek Udo Hahnlrec-conf.org/proceedings/lrec2010/slides/652.pdf · 2010. 6. 28. · Katrin Tomanek and Udo Hahn Annotation Time Stamps 13 / 15. Conclusions & Outlook Summary

Annotation Time Stamps — Temporal Metadatafrom the Linguistic Annotation Process

Katrin Tomanek Udo Hahn

Jena Language & Information Engineering (JULIE) LabFriedrich-Schiller-Universität Jena, Germany

http://www.julielab.de

Katrin Tomanek and Udo Hahn Annotation Time Stamps 1 / 15


Introduction

Economizing the Creation of Training Material

Standard Procedure


Introduction

Economizing the Creation of Training Material

Standard Procedure

Active Learning


Introduction

Evaluation of Active Learning

“Does Active Learning really reduce annotation time ?”requires cost-sensitive evaluation of Active Learningbut: how to simulate AL with true annotation cost?→ corpus with annotation time stamps


Timed Annotations

The MUC7T Annotation Project

re-annotation of well-known corpusMUC7 corpus (news-wire)ENAMEX types (PER, LOC, ORG)reproducable annotation guidelines(hopefully) reasonably large for AL simulations

store annotation time information for each annotation unit


Timed Annotations

Annotation Units

Sentencesmost natural linguistic unitmight be too coarse for some applications

Complex Noun Phrases (CNPs)top-level NPs derived from sentence constituency structureby definition MUC7 entities occur within CNPssmallest syntactic unit completely covering entity mentions

98.95% of MUC7’s ENAMEX entities contained in CNPsremaining 1.05% mostly due to parsing errors


Timed Annotations

Complex Noun Phrases


Timed Annotations

Annotation Principles

one annotation example shown at a timeMUC7 documentsingle annotation unit (sentence or CNP) highlighted andannotatable

annotation examples randomly shuffledin order to guarantee independence of single annotations (avoidlearning/synergy effects due to consecutive reading of a text)

annotation in blocks of 500/100 annotation examplesto be annotated without breaks and under quiet noise conditionsto avoid exhaustion effects

annotation GUI controlled by keyboard shortcutsavoids “mechanical” annotation overheadassumption: measured time reflects only cognitive process


Timed Annotations



Timed Annotations


one annotation example shown at a timeMUC7 documentsingle annotation unit (sentence or CNP) highlighted andannotatable

annotation examples randomly shuffledin order to guarantee independence of single annotations (avoidlearning/synergy effects due to consecutive reading of a text)

annotation in blocks of 500/100 annotation examplesto be annotated without breaks and under quiet noise conditionsto avoid exhaustion effects

annotation GUI controlled by keyboard shortcutsavoids “mechanical” annotation overheadassumption: measured time reflects only cognitive process


Timed Annotations

Annotators

2 students with good general English skillsoriginal MUC7 guidelinesextensive training on MUC7 test setfinal annotations: MUC7 set on airplain crashes completelyannotated by both annotators


Analysis of MUC7T

Annotation Performance

agreement with MUC7 annotationsκA = 0.95 and κB = 0.96FA = 0.92 and FB = 0.94

both annotators perform similarly on different blocksannotation performance largely stationary

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●●

● ●

●

●

0 5 10 15 20 25 30

0.80

0.85

0.90

0.95

1.00

sentence−level annotation

blocks

kapp

a

●

● ●

●● ●

●

●

● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

annotator Aannotator B


Analysis of MUC7T

Annotation Time Measurements

times similar for both annotatorslearning effect for annotator B up to block 9quite stationary annotation times

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

● ●

●

●

0 5 10 15 20 25 30

4.0

4.5

5.0

5.5

6.0

sentence−level annotation

blocks

seco

nds

●

● ● ●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

● ●

●

●

●●

●

●

●

●

●

●

●


●●

●●

●

●

●●

● ●

●

● ●

● ●

●●

●

● ●

● ●●

●

●

● ● ●

●

●

0 5 10 15 20 25 30

1.0

1.2

1.4

1.6

1.8

2.0

CNP−level annotation

blocks

seco

nds

●

●●

●

●

●

●

● ●

●●

●● ● ●

●

● ●

●

● ●

●

● ●

●

●

●

●

●

●

●

●



Analysis of MUC7T

Variability of Annotation Times

sentence-level annotation

●●●●

●

●●●

●●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●●

●

●

●

●●

●

●

●

●

●●●

●●

●

●

●

●

●

●

●●

●

●●●●

●

●●●

●

●●

●●

●

●●

●

●

●

●●

●●

●

●

●

●

●

●

●●

●

●

●

●

●●●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●●

●

●

●

●●

●●●●

●

●●

●

●

●

●●●●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●●

●●●

●

●

●

●●

●

●●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

A B

05

1015

20

annotator

seco

nds

CNP-level annotation

●●

●

●

●

●

●●

●

●●

●

●●

●

●

●

●

●

●

●●

●●

●

●

●

●

●●

●

●

●●

●

●●

●

●

●

●●

●

●●●

●

●

●●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●●●

●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●●●

●●

●

●

●

●●●

●

●

●

●

●

●

●

●●●

●●

●●●●

●

●

●

●

●●

●

●●

●

●

●

●

●●●●

●●

●●

●

●●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●

●

●●●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●●

●

●●

●

●

●

●

●●●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●●●●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●●

●

●

●

●

●

●●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●

●

●●●●●●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●●●●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●●●

●

●

●●

●

●●

●

●●

●●

●

●

●●

●●●

●

●

●●●

●

●●

●

●

●●

●

●●

●

●

●

●

●

●

●●

●

●●

●

●●●

●

●

●

●●

●

●

●

●

●

●●

●

●●●

●●

●

●

●

●

●

●

●●●●

●

●●

●

●

●●

●

●●●

●

●●●●

●

●●●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●●

●

●●

●

●●

●

●●●

●

●

●

●

●

●

●

●

●●●●●

●

●

●

●●

●

●●

●●

●●

●

●

●●

●

●●●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●●●

●

●●

●

●

●●●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●●

●

●●

●●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●●●●

●

●

●

●●

●●

●

●●

●

●●

●

●●●●

●

●

●

●

●●

●

●

●

●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●●

●

●

●●

●●

●

●

●

●

●●

●

●●

●

●●●

●

●

●

●●●●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●●

●

●●

●●

●●

●

●

●

●

●

●

●

●●●

●

●

●

●●

●●

●●

●

●

●

●

●

●

●●●●

●●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●●

●●●

●

●●

●

●●

●

●●

●●

●

●●

●

●

●●●

●●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●●

●

●

●

●●●

●

●

●

●

●●

●

●

●●

●

●●

●●●

●

●

●

●

●●●

●

●

●●

●●●

●

●

●

●

●

●

●

●●●

●

●

●●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●●

●

●●

●

●

●●

●

●

●●

●

●

●

●

●●●

●

●

●

●●●

●

●

●

●

●●

●●

●

●

●

●

●

●●

●

●

●

●

●

●●●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●●

●

●

●

●●●

●

●●●

●

●●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●

●

●

●●

●●

●●

●

●●●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●●●

●

●

●

●

●●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●●

●●

●

●

●●●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●●●

●

●

●

●

●

●

●

●●

●

●

●●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●●

●

●

●●●

●

●●

●

●

●

●

●

●●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●●

●●●

●

●

●

●

●

●

●●●

●

●

●

●●●

●●

●

●

●

●●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●●●

●●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●

●

●

●

●●●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●●

●

●●

●

●●

●

●●

●

●●

●●

●

●

●

●

●●

●

●

●

●

●●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●●●●●

●

●

●

●

●

●

●●●

●

●●

●

●

●●

●●

●●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●●

●

●

●

●

●

●

●●

●●

●

●

●●

●●

●

●

●

●

●●

●

●●

●

●

●

●●●

●

●

●

●

●

●

●●

●

●

●●

●

●●

●

●

●

●●●

●●●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●●

●

●●●

●

●

●

●

●●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●●

●

●

●

●●

●●

●

●

●

●●

●●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●●●

●

●

●●

●●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●●●

●

●

●

●

●

●

●

●

●

●●

●

●

●●●

●●

●

●●●

●

●

●

●

●

●●

●

●

●

●

●●●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●●●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●●●

●●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●●●

●

●

●

●

●●●●

●

●

●

●

●●

●●

●●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●

●

●●●

●

●

●●●●

●

●●

●

●●

●●

●

●

●●

●

●●●

●

●●

●

●●●

●

●

●

●●

●

●●

●

●

●

●

●●

●●

●●

●

●

●●

●●

●●

●

●

●

●●

●

●

●

●

●

●●●

●

●

●

●

●

●

●●●

●

●●●

●

●

●

●●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●●

●

●

●

●●

●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●●

●●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

A B

05

1015

20

annotator

seco

nds

annotation times subject to high varianceconfirms findings of Settles et al. 2008 and Ringger et al. 2008


Cost-sensitive Evaluation of Active Learning

Application of MUC7TCost-sensitive Evaluation of Active Learning

Experimental settingsNamed Entity Recognition with Conditional Random Fields

Pθ(~y |~x) =1

Zθ(~x)·

n∏i=1

exp

k∑j=1

λj fj(

yi−1, yi , ~x , i)

straight-forward approach to Active LearningUncertainty Samplingutility function

u(θ, ~x) = 1−max~y ′∈Y

Pθ(~y ′|~x)


Cost-sensitive Evaluation of Active Learning

Evaluation

0 10000 20000 30000 40000 50000

0.70

0.75

0.80

0.85

0.90

effort as annotated tokens

tokens

F−

scor

e

Standard AL

Random Selection

0 2000 4000 6000 8000 10000

0.70

0.75

0.80

0.85

0.90

effort as annotation time

seconds

F−

scor

e

Standard AL

Random Selection

Savings of Active Learning over Random Selectionnumber of tokens: 67.8 %annotation time: 44.0 %


Conclusions & Outlook

Summary

MUC7T is a corpus with information on annotation timeannotation time stamps are new breed of linguistic metadatacoarse- and fine-grained time measurement

allows for more realistic evaluation of selective sampling strategies(e.g., Active Learning)currently also used to learn predictive annotation cost models(ongoing research, see Tomanek et al. 2010)


Acknowledgements




Documents

Katrin Tomanek Udo Hahnlrec-conf.org/proceedings/lrec2010/slides/652.pdf · 2010. 6. 28. · Katrin Tomanek and Udo Hahn Annotation Time Stamps 13 / 15. Conclusions & Outlook Summary