Incorporating Contextual Cues in Trainable Models for Coreference Resolution

Incorporating Contextual Cues in Incorporating Contextual Cues in Trainable Models for Coreference Trainable Models for Coreference

Resolution Resolution

Incorporating Contextual Cues in Incorporating Contextual Cues in Trainable Models for Coreference Trainable Models for Coreference

Resolution Resolution

14 April 2003

Ryu IidaComputational Linguistic Laboratory

Graduate School of Information ScienceNara Institute of Science and

Technology

2

BackgroundBackground

Two approaches to coreference resolutionRule-based approach [Mitkov 97, Baldwin 95, Nakaiwa 96, Okumura 95, Murata 97]

Many attempted to encode linguistic cues into rules This was significantly influenced by Centering Theory[Grosz 95, Walker et al. 94, Kameyama, 86]

Best-achieved performance in MUC: Precision roughly 70% (Message Understanding Conference) Recall roughly 60%

Corpus-based machine learning approach[Aone and Bennett 95, Soon et al. 01, Ng and Cardie 02, Seki 02]

Cost effectiveThey have achieved a performance comparable to best performing rule-based systems

Problem: Further manual refinement is needed in this study but it will be prohibitively costly

Problem: These previous work tend to lack an appropriate reference to the theoretical linguistic work on coherence and coreference

3

BackgroundBackground

Challenging issue

Achieving a good union between theoretical linguistic findings and corpus-based empirical methods

4

Outline of this TalkOutline of this Talk

Background

Problems with previous statistical approaches

Two methodsCentering featuresTournament-based search model

Experiments

Conclusions

5

Statistical approaches [Soon et al. ‘01, Ng and Cardie Statistical approaches [Soon et al. ‘01, Ng and Cardie ‘02]‘02]

Reach a level of performance comparable to state-of-the-art rule-based systems

Recast the task of anaphora resolution as a sequence of classification problems

6

Statistical approaches [Soon et al. ‘01, Ng and Cardie Statistical approaches [Soon et al. ‘01, Ng and Cardie ‘02] ‘02]

the task is to classify these pairs of noun phrases into positive or negative

positive instance: Pair of an anaphor and the antecedentnegative instance: Pairs of an anaphor and the NPs located between the anaphor and the antecedent

A federal judge in Pittsburgh issued a temporary restraining order preventing Trans World Airlines from buying additional shares of USAir Group Inc.The order, requested in a suit filed by USAir, dealt another blow to TWA's bid to buy the company for $52 a share.


[MUC-6]

USAirUSAirsuitsuit

USAirUSAirorderorder

USAirUSAirUSAir Group IncUSAir Group Inc positive

negative

negative

outputclass

antecedent

anaphor×

×

〇

7


Feature set [Ng and Cardie 02]

features

training (C4.5)

Model (decision tree)

Person:1 ハ :1先行詞候補照応詞

Person:1

Pronoun:1

ハ :1

Pronoun:0

SENT_DIST:0 negativePerson:1 ハ :1先行詞候補照応詞

Person:1

Pronoun:1

ハ :1

Pronoun:0

SENT_DIST:0 negativeOrganization:1Prp_noun:1

candidate anaphorOrganization:1

Pronoun:0Pronoun:0

SENT_DIST:0 positive

POS 　DEMONSTRATIVE STRING_MATCHNUMBERGENDERSEM_CLASSDISTANCESYNTACTIC ROLE

STR_MATCH:0

USAirUSAirsuitsuit



negative

negative

8

1.5


Precision 　 78.0%, Recall 　 64.2%Slightly better than best-performing rule-based model at MUC-7

-3.5

-2.5

-0.3

-1.0

-0.4

-1.1

-2.0

1.5

NP6

antecedent

anaphoranaphor

NP8NP8

NP7NP7

NP6NP6

NP5NP5

NP4NP4

NP3NP3

NP2NP2

NP1NP1extract NPscandidates

Test Phase [Ng and Cardie, 02]

We refer to Ng and Cardie’s model as the baseline of our empirical

evaluation

We refer to Ng and Cardie’s model as the baseline of our empirical

evaluation

Select the best-scored candidateas the output

Input each pair of given anaphorand one of these candidates tothe decision tree

9

Sarah went downstairs and received another curious shock, for when Glendora flapped into the dining room in her home made moccasins, Sarah asked her when she had brought coffee to her room, and Glendora said she hadn't.


A drawback of the previous statistical modelsA drawback of the previous statistical models

Positive and negative instances may have the identical feature vector

Sarah she

positiveGlendora she

negative

POS : NounProp_Noun : YesPronoun : NoNE : PERSONSEM_CLASS : PersonSENT_DIST : 0

features[Kameyama 98]

antecedent

anaphor

The previous models do not capture local context

appropriately

The previous models do not capture local context

appropriately


Two methodsTwo methods

11


Use more sophisticated linguistic cues: centering features

Augmentation of a set of new features inspired by Centering Theory that implement local contextual factors

Improve the search algorithm: tournament modelA new model which makes pair-wise comparisons between candidates

12

Centering FeaturesCentering Features

Sarah went downstairs and received another curious shock,

……

she hadn't.

CHAIN(Cb = Cp = Sarah)

CHAIN(Cb = Cp = Glendora)

transition

the problem is that the current feature set does not tell the

difference between these two candidates

Sarah she

positiveGlendora she

negative


features

Glendoraantecede

nt

Introduce extra devices such as the forward-looking center listEncode state transitions on them into a set of additional features


13


Use more sophisticated linguistic cues: centering features

We augment the feature set with a set of new features inspired by Centering theory that implement local contextual factors

Improve the search algorithm: tournament modelWe propose a new model which makes pair-wise comparisons between antecedent candidates

14

Tournament modelTournament model

What we want to do is to answer a question which is more likely to be coreferent, Sarah or Glendora

Conduct a tournament consisting of a series of matches in which candidates compete with each other

Match victory is determined by a pairwise comparison between candidates as a binary classification problemMost likely candidate is selected through a single-elimination tournament of matches



dining room

〇

×downstairs

〇

×Sarah

〇×

15

Tournament modelTournament modelTraining Phase

Training instances

NP7NP7NP5NP5

NP5NP5NP4NP4

NP5NP5NP1NP1 ANPANP

ANPANP

ANPANP

features

rightright

class

rightright

leftleft

NP8NP8NP5NP5 ANPANP leftleft

In the tournament, the correct antecedent NP5 must prevail over any of the other four candidates

Extract four training instances

Induce a pairwise classifier from a set of 　 extracted training instances

The classifier classifies a given pair of 　　 candidates into left or right

NP7NP7

coreferent

coreferent

ANPANPNP6NP6NP5NP5NP4NP4NP3NP3NP2NP2NP1NP1

coreferent

anaphor

NP8NP8

beginning of document

antecedent

the right hand side of a given pair wins (is more

likely to be the antecedent)

16

NP7NP7

coreferent

coreferent

Test PhaseTournament modelTournament model


coreferent


anaphor

NP8NP8

1. the first match is arranged between the nearest candidates (NP7 and NP8)

2. each of the following matches arranged in turn between the winner (NP8) of the previous match and a new challenger (NP5)

17

NP7NP7

coreferent

coreferent


NP5

antecedent


coreferent


anaphor

NP8NP8

3. the winner is next matched against the next challenger (NP4)4. this process is repeated until the last one participate5. the model selects the candidate that prevails through the final round as the answer

ExperimentsExperiments

19

ExperimentsExperiments

Empirical evaluation on Japanese zero-anaphora resolution

Japanese does not normally use personal pronoun as anaphorInstead, Japanese uses zero-pronouns

Comparison among four models1. Baseline model2. Baseline model with Centering Features3. Tournament model4. Tournament model with Centering Features

20

Centering Features in JapaneseCentering Features in Japanese

Japanese anaphora resolution model [Nariyama 02]Expansion of Kameyama’s work on the application of Centering Theory to Japanese zero-anaphora resolution

Expanding the original forward-looking center listinto Salience Reference List (SRL) to take into account broader contextual information

More use of linguistic information

In the experiments, we introduced two features to reflect the SRL-related contextual factors

21

DataGDA-tagged Japanesenewspaper article corpus

Texts ： 2,176 　　 60

Sentences ： 24,475 -Tags of anaphoric relation 　　　： 14,743 8,946Tags of ellipsis (Zero-anaphor) ： 5,966 0

As a preliminarily test, only resolving subject zero-anaphors, 2,155 instances in total

Conduct five fold cross-validation on that data set with support vector machines

GDA MUC-6

MethodMethod

22

1. Features for simulating Ng and Cardie’s feature set

2. Centering Features

3. Features for capturing the relations between two candidates

Feature set Feature set (see our paper for details)

Order in SRLHeuristic rule of preference

Preference of SRL in two candidatesPreference of Animacy in two candidatesDistance between two candidates

POS PronounParticleNamed-EntitySemantic classAnimacy

Selectional RestrictionsDistance between an anaphor and the candidateNumber of anaphoric relations

introduce only in tournament model

but not in the baselinemodel

23

ResultsResults

Tournament model + Centering Features

Tournament model

Baseline model +Centering Features

Baseline model

24

Results (1/3) Results (1/3) the effect of incorporating centering features

Baseline model


67.0%

64.0%

centering features were reasonably effective

25

Results (2/3)Results (2/3)

Tournament model

70.8%

67.0%

64.0%

Introducing the tournament model significantly improved the performance regardless the size of training data

Baseline model


26

most complex model did not outperform the tournament model without centering features

Results (3/3)Results (3/3)


Tournament model

70.8%

69.7%

67.0%

64.0%

The improvement ratio of this model against the data size is the best of all

Baseline model


27

Results Results after cleaning data (March ‘03)

74.3%

72.5%


Tournament model

the tournament model with centering featuresis more effective than the one without centering features

28

ConclusionsConclusions

Our concern is achieving a good union between theoretical linguistic findings and corpus-based empirical methods

We presented a trainable coreference resolution model that is designed to incorporate contextual cues by means of centering features and a tournament-based search algorithm. These two improvements worked effectively in our experiments on Japanese zero-anaphora resolution.

29

Future WorkFuture Work

In Japanese zero-anaphora resolution,1. Identification of relations between the topic and subtopics2. Analysis of complex and quoted sentences3. Refinement of the treatment of selectional restrictions

30

31

Tournament modelTournament modelTraining Phase

ANPANP

NP7NP7

NP6NP6

NP5NP5

NP4NP4

NP3NP3

NP2NP2

NP1NP1

coreferent

coreferent

antecedent

coreferent


anaphor

NP8NP8

Training instances

NP7NP7NP5NP5

NP5NP5NP4NP4

NP5NP5NP1NP1 ANPANP

ANPANP

ANPANP

features

rightright

class

rightright

leftleft

NP8NP8NP5NP5 ANPANP leftleft

In the tournament, the correct antecedent NP5 must prevail over any of the other four candidates

extract four training instances

Induce from a set of extracted training instances a pairwise classifier

32


NP5

antecedent

A tournament consists of a series of matchesin which candidates

compete with each other

A tournament consists of a series of matchesin which candidates

compete with each other

ANPANP

NP7NP7

NP6NP6

NP5NP5

NP4NP4

NP3NP3

NP2NP2

NP1NP1

coreferent

coreferent

coreferent


anaphor

NP8NP8<

<

<>

33


What we want to do is to answer a question which is more likely to be coreferent, Sarah or Glendora

Implement a pairwise comparison between candidates as a binary classification problem



CHAIN(Cb = Cp = Sarah):

CHAIN(Cb = Cp = Glendora):

Sarah Glendora she<

transition

<

<

<

Sarahdining room

downstairs

34


Training Phase

She

moccasins Glendora she<

coffee Glendora she<

Sarah Glendora she<

room Glendora she<

output class

downstairs

Glendora

moccasins

Sarah

she

coffee

room

Glendora

she

her

her

coreferred

downstairs Glendora she<

extract NPs

coreferent



Training instances

35

ConclusionsConclusions

To incorporate linguistic cues into trainable approaches:

Add features which takes into consideration linguistic cues such as Centering Theory: Centering Features

Propose the novel search model which the candidates are compared in terms of the likelihood of antecedents:Tournament model

In Japanese zero-anaphora resolution task,Tournament model significantly outperforms earliermachine learning approaches [Ng and Cardie 02]Incorporating linguistic cues

in machine learning models is effectiveIncorporating linguistic cues

in machine learning models is effective

36

GDA-tagged Japanesenewspaper article corpus

Texts ： 2,176 60Sentences ： 24,475 -Tags of anaphoric relation 　　　： 14,743 8,946Tags of ellipsis (Zero-anaphor) ： 5,966 0

DataData

<n id=“tagid1”> クリントン米大統領 </n> の内政の最大課題のひとつである <n id=“tagid2”> 包括犯罪対策法案 </n> が十一日の下院本会議で、審議・表決に移ることを承認する動議が、反対二二五対賛成二一〇で否決された。これで <n eq=“tagid2”> 同法案 </n> は事実上、大幅修正または廃案に追い込まれた。 <n eq=“tagid1”>同大統領 </n> は緊急会見で怒りをあらわにして、法案の復活を要求。 <n eq=“tagid1”> 同大統領 </n> は中間選挙を前に得点を<v agt=“tagid1”> あげる <v> ことを目指したが、逆に大きな痛手を受けた。

<n id=“tagid1”> クリントン米大統領 </n> の内政の最大課題のひとつである <n id=“tagid2”> 包括犯罪対策法案 </n> が十一日の下院本会議で、審議・表決に移ることを承認する動議が、反対二二五対賛成二一〇で否決された。これで <n eq=“tagid2”> 同法案 </n> は事実上、大幅修正または廃案に追い込まれた。 <n eq=“tagid1”>同大統領 </n> は緊急会見で怒りをあらわにして、法案の復活を要求。 <n eq=“tagid1”> 同大統領 </n> は中間選挙を前に得点を<v agt=“tagid1”> あげる <v> ことを目指したが、逆に大きな痛手を受けた。

GDA MUC-6

Extract 2,155 example

coreferent

Ellipsis (AGENT)

coreferent

37

Statistical approaches [Soon et al. 01, Ng and Cardie Statistical approaches [Soon et al. 01, Ng and Cardie 02]02]

Reach a level of performance comparable to state-of-the-art rule-based systemsRecast the task of anaphora resolution as a sequence of classification problems

Pair of an anaphor and the antecedent:positive instancePairs of an anaphor and the NPs located between the anaphor and the antecedent: negative instance

USAirUSAirsuitsuit



negative

negative



[MUC-6]

outputclass

the task is to classify these pairs of noun phrases into positive or negative.

38

*Centering Features*Centering Features

Centering Theory [Grosz 95, Walker et al. 94, Kameyama, 86]

Part of an overall theory of discourse structure and meaningTwo levels of discourse coherence: global and localCentering models the local-level component of

attentional state

e.g. Intrasentential centering [Kameyama 97]



39

*Centering Features in English [Kameyama *Centering Features in English [Kameyama 97]97]


for when Glendora flapped into the dining room in her home made moccasins,

Sarah asked her

when she had brought coffee to her room,

and Glendora said

she hadn't.

CHAIN(Cb = Cp = Sarah):

ESTABLISH(Cb = Cp = Glendora):

CHAIN(Cb = Glendora, Cp = Sarah):


CHAIN(Cb = NULL, Cp = Glendora):


[Kameyama 97]

40

*Centering Features in English [Kameyama *Centering Features in English [Kameyama 97]97]

The essence is that takes into account the preference between candidates

Cb and Cp distinguish the two candidates


……

she hadn't.

CHAIN(Cb = Cp = Sarah)

CHAIN(Cb = Cp = Glendora)

transition

Implement local contextual factor: centering features

Implement local contextual factor: centering features

41

She

downstairs

shock

Glendora

room

moccasins

Sarah

she

coffee

room

Glendora

she

her

her

her

*Tournament model*Tournament model

Test Phase

Glendora

antecedent

room

<

her

<

coffee<

moccasins

<

room

<

shock<

downstairs< A tournament consists of a series of matchesin which candidates compete with each

other

42

Rule-based ApproachesRule-based Approaches

Encoding linguistic cues into rules manually Thematic roles of the candidatesOrder of the candidatesSemantic relation between anaphors and antecedentsetc..

This approaches are influenced by Centering Theory[Grosz 95, Walker et al. 94, Kameyama, 86]

The Coreference Resolution Task of Message Understanding Conference (MUC-6 / MUC-7)

Precision: roughly 70%Recall: roughly 60%

Further manual refinement of rule-based modelswill be prohibitively costly

Further manual refinement of rule-based modelswill be prohibitively costly

43

Statistical Approaches with Tagged-CorpusStatistical Approaches with Tagged-Corpus

The statistical approaches have achieved a performance comparable to the best-performing rule-based systems

Lack an appropriate reference to theoretical linguisticwork on coherence and coreference

Making a good marriage between theoretical linguistic findings and corpus-based empirical methods

Making a good marriage between theoretical linguistic findings and corpus-based empirical methods

44

*Test Phase [Soon et al. 01]*Test Phase [Soon et al. 01]

Precision 　 67.3%, Recall 　 58.6% on MUC data set

USAirUSAir

a suita suit

orderorder

USAir Group IncUSAir Group Inc

anaphor

〇

×

×

USAir Group Inc

antecedent

extracting NP

shareshare

Trans World Airlines


orderorder

PittsburghPittsburgh

judgejudge



candidates

45

Improving Soon’s modelImproving Soon’s model

[Ng and Cardie 02]

Expanding the feature set12 features 　⇒　 53 features

Introducing a new search algorithm

POS 　DEMONSTRATIVE STRING_MATCHNUMBERGENDERSEM_CLASSDISTANCESYNTACTIC ROLE

46

Test Phase [Soon et al. 01]Test Phase [Soon et al. 01]

Precision 　 67.3%, Recall 　 58.6% on MUC data set

USAirUSAir

a suita suit

orderorder

USAir Group IncUSAir Group Inc

anaphor

〇

×

×

USAir Group Inc

antecedent

extracting NP

shareshare



orderorder

PittsburghPittsburgh

judgejudge



candidates

47


Task of Coreference ResolutionsTask of Coreference Resolutions

Two processResolution of anaphors 　Resolution of antecedents

applicationsMachine Translation, IR, etc

antecedent

anaphor [MUC-6](Same color NPs are coreferred)

48

Future WorkFuture Work

Evaluate some examplesTournament model doesn’t deal with Direct quote

Proposed methods cannot deal with different discourse structures

……獄に下るモハンメドは妻にこう言い残した。「おれが刑務所にいる間、外で働いてはいけない」。貞節を守れ、という意味だ。さすがに刑務所で新しい子供に恵まれる可能性はないと思ったのだろうか。

Topic モハンメド

Focus おれ

I-Obj 刑務所

D-Obj NULL

Others 外

SRL

49

Centering Features of JapaneseCentering Features of Japanese

Adding the likelihood of antecedents into featuresIn Japanese, wa-marked NPs tend to be topicsTopics tend to be omitted

Salience Reference List (SRL) [Nariyama 02]Store NPs in SRL from the beginning of textOverwrite the old entity if new entity fills same point

Topic/φ (wa) > Focus (ga) > I-Obj (ni) > D-Obj (wo) > Others

…NP1-waNP2-wo… 。…NP3-ga… 、 NP4-ha… 。…NP5-ni……(φ-ga)V 。

Topic NULL

Focus NULL

I-Obj NULL

D-Obj NULL

Others NULL

Topic NP1

Focus NULL

I-Obj NULL

D-Obj NULL

Others NULL

Topic NP1

Focus NULL

I-Obj NULL

D-Obj NP2

Others NULL

Topic NP1

Focus NP3

I-Obj NULL

D-Obj NP2

Others NULL

Topic NP4

Focus NP3

I-Obj NULL

D-Obj NP2

Others NULL

Topic NP4

Focus NP3

I-Obj NP5

D-Obj NP2

Others NULL

preferred

50

Evaluation of modelsEvaluation of models

Introduce a confidence measureConfidence coefficient is the value when two candidatesare the nearest at the tournament

corefered

President A

armistice

President B

this

he

action

he

President B

< this

>

action

<

armistice<

President A

3.8

3.2

2.4

0.9

51

Evaluation of Tournament modelEvaluation of Tournament model investigate the Tournament model (investigate the Tournament model (the best

performance )

52

ドゥダエフ大統領 > NULL >NULL > NULL > NULL

エリツィン･ロシア大統領 > NULL >行動 > NULL > NULL

Centering FeaturesCentering Features

example

SRL

ドゥダエフ大統領は、正月休戦を提案したが、エリツィン・ロシア大統領はこれを黙殺し、(φ ガ ) 行動を開始した。

President A proposed the armistice, but President B ignored this. And he started action.

53

*Features (1/3) Ng’s model, Tournament *Features (1/3) Ng’s model, Tournament modelmodel

Features are decided by one candidate

candidate1 candidate2 anaphor

POS Pronoun Particle Named-Entity the number of anaphoric relations First NP in a sentence Order of SRL

54

*Features (2/3) Ng’s model, Tournament *Features (2/3) Ng’s model, Tournament modelmodel

Features are decided by a pair of an anaphor and the candidate


Selectional restrictions the pair of candidate and anaphor satisfies selectional restriction in Nihongo Goi Taikei log-likelihood ratio calculated from cooccurrence data

Distance in terms of sentence between an anaphor and the candidate

55

*Features (3/3) only Tournament model*Features (3/3) only Tournament model

Features are decided by the relation between two candidates


Distance in terms of sentence between two candidatesAnimacy

Whether or not one candidate belongs to the class of PERSON or ORGANIZATION

Which candidate is preferred in SRL

Topic NULL

Focus NP1

I-Obj NULL

D-Obj NP2

Others NULL

56

Anaphoric relationsAnaphoric relations

Variety of antecedents

Noun Phrase (NP), Sentence, text, etc

Many previous works deal with anaphora resolutionsThe number of antecedent-anaphor examples is the most of all

endophora

exophora

anaphoracataphora

Antecedent exists in a context

Antecedents precede anaphors

Antecedent doesn’t exist in context

Anaphors precede antecedents

57

Results Results (examples: 2155 　⇒ 2681)

Tournament modelThe Model using Centering Features gets worse thanthe model without Centering Features

Modify some tagging errors by handexamples: 2155 　⇒ 2681


Tournament model

Ng’s model +Centering Features

Ng’s model

Documents

Incorporating Contextual Cues in Trainable Models for Coreference Resolution