Aiding WSD by exploiting hypo/hypernymy relationsin a restricted framework

MEANING projectExperiment 6.H(d)

Luis Villarejo and Lluís Màrquez

Preface

This document stands for the 1st draft of the description of Experiment 6.H(d): “Aiding WSD by exploiting hypo/hypernymy relations in a restricted framework”. To be included in the Working Paper describing experiment 6.H: Bootstrapping.

Outline

• Introduction

• Our approach vs Mihalcea’s

• Technical details

• Experiments

• Results

• Conclusions

• Future work

Introduction

“The Role of Non-Ambiguous Words in Natural Language Disambiguation”

Rada Mihalcea

University of North Texas

• Task: Automatic resolution of ambiguity in NL.

• Problem: The lack of large amounts of annotated data.

• Proposal: Inducing knowledge from non-ambiguous words via equivalence classes to automatically build an annotated corpus.

Introduction

In this experiment we explore whether training example sets can be enlarged with automatically extracted examples associated to each sense. Some work has been recently done in the direction of extracting examples, in a non/supervised manner, associated to word senses like the one presented by Mihalcea. R. in The Role of Non-Ambiguous Words in Natural Language Disambiguation where POS tagging, Named entity tagging and WSD were approached. We will only tackle here WSD. We not only did our best to reproduce the conditions, in which experiments were developed, described in the paper by Mihalcea. R. but also explored new possibilities not taken into account in the paper. However, our scenario is better since our unique source for obtaining the extra training examples is SemCor, and therefore, we do not need to perform any kind of disambiguation. The semantic relations, used to acquire the target words for the extra examples, were taken from the MCR.

“Equivalence classes consist of words semantically related.”

Introduction

Focuses on:

• Part of Speech Tagging

• Named Entity Tagging

• Word Sense Disambiguation

WordNet 1.6

• Synonyms ?

• Hyperonyms ?

• Hyponyms ?

• Holonyms ?

• Meronyms ?

Target Word Meanings Monosemous equivalents

plant

living_organism flora

manufacturing_plant industrial_plant

What happens with the other two meanings of plant?? (actor and trick)

• Manually selected word set:

• Our list: child, day, find, keep, live, material, play and serve.

• Rada’s: bass, crane, motion, palm, plant and tank.

• Only two, specially selected, and clearly differentiated senses per word:

• {play 00727813}: play games, play sports “Tom plays tennis”

• {play 01177316}: play a role or part “Gielgud played Hamlet”

• Source for examples on the equivalent words:

• Relation of equivalence between words:

Our approach vs Mihalcea’s

5 verbs + 3 nouns 6 ¿nouns?

SemCor Raw corpus (monosemous words, no need for annotation)

Synonymy, Hyperonymy, Hyponymy and mixes

(levels 1, 2 and both)

¿Synonymy?

Our approach Mihalcea

• Equivalent corpus sizes.

• Features used:

•Technique used to learn:

• Use of the examples coming from the equivalent words:

Our approach vs Mihalcea’s

Two left words, Two right words

Two left POS, Two right POS

One left word, One right word

One left POS, One right POS

Bag of words

(WSD task in Meaning project)

Two left words, Two right words

Nouns before and after

Verbs before and after

Sense specific keywords??

Our approach Rada’s

SVM Timbl Memory Based Learner

Added to the original word set examples and training a

classifier on itTraining a classifier on it

Technical details

• SVM_light (Joachims):

• Each word is a binary classification problem which has to decide between two possible labels (senses).

• Positive examples of one sense are negative for the other.

• 10-fold cross validation

• Testing with a random folder from the originals.

• Training with the rest of the originals plus the equivalents.

• C parameter tuning by margin (5 pieces and 2 rounds) for each classification problem.

• Linear kernel

Experiments

• Baselines:

• Examples from Original word set codified with all features.

• Examples from Original word set codified only with BOW feature.

• Experiments over each Baseline:

• Examples from Equivalents codified with all features.

• Examples from Equivalents codified only with BOW feature.

• Examples from Equivalents added in equal proportions.

• Relations explored over each experiment:

• Hyponymy levels 1, 2 and both

• Hyperonymy levels 1, 2 and both

• Synonymy

• Mixes: SHypo1, SHypo2, SHypo12, SHype1, SHype2, SHype12

• Total: 8 words * 2 baselines * 3 experim * 13 relations = 624 results

Results – Originals BOW, Added BOW (I)

MFS Accuracy

Baseline 1 (Originals only BOW feature)

72.74 77.78

In detail #origs MFS Baseline #addedBest accuracy &

relation

Child 206 70.86 74.27 62 76.21 BagSHype1

Day 195 83.61 85.64 6 86.15 BagSinon

Find 27 59.67 70.37 8 74.07 BagSHype2

Keep 24 56.67 75.00 611 66.67 BagSHype1

Live 88 64.64 75.00 4 76.14 BagHypo2

Material 107 68.41 72.90 196 78.50 BagHype12

Play 54 75.05 79.63 4 81.48 BagSinon

Serve 46 74.00 80.43 34 91.30 BagHipo1

Results – Originals BOW, Added BOW (II)

MFS Accuracy

Baseline 1 (Originals only BOW feature)

72.74 77.78

Best Global Results #original exs #exs added Accuracy

BagSHype1 747 1545 77.64

BagSinon 747 119 77.38

BagSHypo1 747 608 77.14

BagHypo1 747 489 77.05

Results – Originals All Feats, Added Both (I)

MFS Accuracy

Baseline 2 (Originals All features)

72.74 78.71

In detail #origs MFS Baseline #addedBest accuracy &

relation

Child 206 70.86 75.73 79 79.61 BagSHypo1

Day 195 83.61 85.64 145 88.21 Hypo1

Find 27 59.67 74.07 2 77.78 Hypo1

Keep 24 56.67 75.00 611 79.17 BagSHype1

Live 88 64.64 76.14 476 78.41 BagHype1

Material 107 68.41 72.90 201 79.44 SHype12

Play 54 75.05 77.78 12 83.33 SHypo1

Serve 46 74.00 86.96 34 91.30 Hypo1

Results – Originals All Feats, Added Both (II)

MFS Accuracy

Baseline 2 (Originals All features)

72.74 78.71

Best Global Results #original exs #exs added Accuracy

BagSHype1 747 1545 82.40

BagSHype12 747 8730 82.22

BagSinon 747 119 82.21

SHype1 747 1545 81.68

Results – O-all, A-both, keeping proportions I

MFS Accuracy

Baseline 2 (Originals All features)

72.74 78.71

In detail #origs MFS Baseline #addedBest accuracy &

relation

Child 206 70.86 75.73 51 80.10 BagSHypo1

Day 195 83.61 85.64 4 86.67 Sinon

Find 27 59.67 74.07 6 70.37 Hype1

Keep 24 56.67 75.00 44 87.50 SHype1

Live 88 64.64 76.14 0 76.14 -----------

Material 107 68.41 72.90 54 75.70 BagHype2

Play 54 75.05 77.78 6 81.48 BagSHypo1

Serve 46 74.00 86.96 14 91.30 Hypo1

Results – O-all, A-both, keeping proportions II

MFS Accuracy

Baseline 2 (Originals All features)

72.74 78.71

Best Global Results #original exs #exs added Accuracy

BagSHypo1 747 226 79.84

SHype12 747 179 79.65

SHype1 747 112 79.38

Hype2 747 121 79.38

Note: Examples not randomly added.

Results

• Other results:

• Accuracy improves (slightly) when choosing manually which equivalents to take into account.

• Experiments with a set of 41 words (nouns and verbs) with all senses per word (varying from 2 to 20) proved to have worse results (accuracies on all mixes of relations and features are below the baseline).

ConclusionsThe work presented by R. Mihalcea has some dark points. The criteria used to select the words, involved in the experiment, the criteria to select word senses, the restriction on the number of senses per word, the semantic relations used to get the monosemous equivalents or the features used to learn are not satisfactorily described. Although this, Mihalcea’s results showed that equivalents carry useful information to do WSD (better than MFS 76.60% against 70.61%). But is this information useful to improve the state-of-the-art in WSD?. Experiments carried out here showed that adding examples coming from the equivalents seems to improve the results in a restricted framework. This means using a small word set, only two senses per word and clearly differentiated senses. When we moved to an open framework, this means using a bigger word set is used (41 words), no special selection of words, no special selection of senses and no restriction on the number of senses per word (varying from 2 to 20), results proved to be worse.

MFSClassifier trained on the automatically generated

corpora

Classifier trained on the manually generated

corpora

Accuracy 70.61 76.60 83.35

Future Work

• Do the differences between the features set used by Mihalcea and the one we used critically affected the results on the 41 words experiment?

• Exploiting the feature extractor over SemCor to enrich the feature set used. Ideas are welcome.

• Study the correlation between the number of examples added and the accuracy obtained.

• Restrict the addition of examples coming from the equivalent words (second class examples).

• Randomly select which examples to add when keeping proportions or restricting the addition.