Incremental, Predictive Parsing with Psycholinguistically Motivated … · 2011-04-13 · Incrementality Prediction Case Study: Syntactic Prediction Incrementality Text and speech

IntroductionPrediction and Grammar

Predictive ParsingEvaluation

Incremental, Predictive Parsing withPsycholinguistically Motivated Tree-Adjoining

Grammar

Frank Keller

Institute for Language, Cognition and ComputationSchool of Informatics, University of Edinburghhttp://homepages.inf.ed.ac.uk/keller/

Joint work with Vera Demberg and Alexander Koller

Frank Keller Incremental, Predictive Parsing with PLTAG 1

http://homepages.inf.ed.ac.uk/keller/



1 IntroductionIncrementalityPredictionCase Study: Syntactic Prediction

2 Prediction and GrammarConceptual IssuesFormalismComparison with TAGModeling Prediction

3 Predictive ParsingTreebank Conversion and Lexicon InductionParsing Algorithm and Probability ModelLinking Theory

4 EvaluationParsing PerformanceCognitive Plausibility




IncrementalityPredictionCase Study: Syntactic Prediction









Incrementality

Text and speech are perceived serially.

Human language processing is adapted to this: sentencecomprehension proceeds incrementally:

the interpretation of a sentence is built word by word;

each new word is integrated as fully as possible into arepresentation of the sentence thus far;

processing effort depends on the properties of the word and itsrelationship to the preceding context.





Discourse Prediction

Not only is processing word-by-word, it is also predictive:comprehenders anticipate upcoming linguistic material.

van Berkum et al. (2005) show that contextual information is usedto predict specific lexical items; processing difficulty ensues if inputis incompatible with the prediction (ERP study).





Structural Prediction

Staub & Clifton (2006) show that the sentence processor can alsomake structural predictions:

(1) Peter read either a book or an essay in the school magazine.

(2) Peter read a book or an essay in the school magazine.

The presence of either leads to shorter reading times on or and onthe NP that follows it (eye-tracking study).

The word either makes it possible to anticipate an upcoming NPconjunction (rather than VP conjunction).





Semantic Prediction

Visual world paradigm:

image and speech presented synchronously;

eye-movements reflect listeners’ interpretation of input;

they can also indicate predictions about upcoming input.

Altmann & Kamide (1999) use this paradigm to provided evidencefor semantic prediction. They presented sentences such as:

(3) a. The boy will eat . . .b. The boy will move . . .

together with a scene that contained one edible but severalmovable objects.





Semantic Prediction

When participants heard eat, they looked more at the cake.Evidence for prediction induced by semantic restrictions of the verb.





Granularity of Prediction

What is the granularity of prediction? We saw predictions can betriggered by:

discourse context;

specific collocations (either . . . or);

semantic restrictions of a lexical item.

But can we get predictions from lexically specific syntacticinformation?

We will look at an experiment in detail that shows prediction basedon verb subcategorization (joint work with Manabu Arai).





Design

Compare obligatory transitive verbs (e.g., offend) and intransitiveverbs (e.g., frown):

(4) a. All of the sudden, the inmate offended the judge.b. All of the sudden, the inmate frowned at the judge.c. All of the sudden, the inmate frowned and the judge threw

the gloves.

Listeners’ looks at the verb indicate which verb frame they assume.





Materials

(5) All of the sudden, the inmate offended the judge.

Listeners predict upcoming patient information on hearing the verband look at the judge.





Materials

(5) All of the sudden, the inmate frowned at the judge.

No prediction on hearing the verb. But listeners predict upcomingpatient information on hearing at and look at the judge.





Materials

(5) All of the sudden, the inmate frowned and the judge threwthe gloves.

No prediction on hearing the verb and and.





Results

Gazes after the verb onset:

-0.8

-0.7

-0.6

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.22

0

10

0

18

0

26

0

34

0

42

0

50

0

58

0

66

0

74

0

82

0

90

0

98

0

10

60

11

40

12

20

13

00

13

80

14

60

Time from the verb onset (ms)

log

it o

f p

ati

en

t lo

ok

Transitive Intransitive + preposition Intransitive + conjunction

ROI: PATIENT (judge)

Participants looked more at a patient picture on hearing transitiveverbs (offended) than intransitive verbs (frowned).





Results

Gazes after hearing at or and :

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.520

100

180

260

340

420

500

580

660

740

820

900

980

1060

1140

1220

1300

1380

1460

Time from the preposition/conjunction onset (ms)

log

ito

fp

ati

en

tlo

ok

s

Intransitive + preposition Intransitive + conjunction

ROI: PATIENT (judge)

Participants predicted and looked more at a patient picture onhearing at than on hearing and.





Summary

Evidence for the use of verb-specific subcategorization informationin prediction:

participants predicted a direct object following a transitiveverb more than following an intransitive verb;

they made a similar predictions at the preposition following anintransitive verb;

but not if a conjunction followed the intransitive verb.




Conceptual IssuesFormalismComparison with TAGModeling Prediction









Conceptual Issues

Challenge: develop a model of prediction in sentence processingthat accounts for these experimental results. Assumptions:

structures are built incrementally (word by word);

partial structures do not contain unconnected nodes;

upcoming syntactic material is predicted.

Evidence for connectedness: Sturt & Lombardo (2005). Existingincremental parsers don’t build fully connected structures.

Our approach: devise a grammar formalism that supportsincrementality and connectedness; prediction then follows.





Implementing Prediction

Experimental results inform our model regarding the granularity ofprediction. The model predicts:

lexical items when they are syntactically required (e.g., either. . . or, pick . . . up);

syntactic structure when required by subcat frames;

syntactic structure when required by connectedness.





Formalism

We propose Psycholinguistically Motivated TAG (PLTAG), avariant of tree-adjoining grammar:

in standard TAG, the lexicon consists of initial trees andauxiliary trees (both are lexicalized);

we add unlexicalized predictive trees to achieve connectivity;

the standard TAG operations are substitution and adjunction;

we add verification to verify predictive trees;

we use TAG’s extended domain of locality for lexicalprediction.

PLTAG supports parsing with incremental, fully connectedstructures (Demberg & Keller 2008).





Formalism

Lexicon:

Standard TAG lexicon

Predictive lexicon(PLTAG)

Operations:

Substitution

Adjunction

Verification (PLTAG)





Formalism

Lexicon:



Operations:

Substitution

Adjunction


Example

Initial Tree: S

��

HHH

NP↓ VP

�� HHV

meets

NP↓

Auxiliary Tree: VP

�� HHAP

yesterday

VP*





Formalism

Lexicon:



Operations:

Substitution

Adjunction


Example

DT

the

substitutes into NP�� HH

DT↓ NN

man

resulting in NP�� HH

DT

the

NN

man





Formalism

Lexicon:



Operations:

Substitution

Adjunction


Example

VP

�� HHVP* AP

yesterday

adjoins into S

��

HHH

NP��HH

DT

the

NN

man

VP

V

slept

resulting in S

��

HHH

NP��HH

DT

the

NN

man

VP

�� HHVP

V

slept

AP

yesterday





Formalism

Lexicon:



Operations:

Substitution

Adjunction


Example

Initial Prediction Tree: Ss

�� HHNPs VPs

s

Auxiliary Prediction Tree: VPs

�� HHAPs

s VP*s

Index s marks predicted node.





Formalism

Lexicon:



Operations:

Substitution

Adjunction


Example

Ss

�� HHNPs

the man

VPss

is verified by S

��

HHH

NP↓ VP

�� HHV

meets

NP↓

resulting in S

��

HHH

NP

the man

VP

�� HHV

meets

NP↓

All nodes indexed with s have to be verified.





Comparison with and TAG

TAG derivations are not always incremental.

a

DT

initial(subst)

Peter

NP VP

AP

often

VP

S

V

reads

NP

DT

book

NN

a

VP

S

V NP

reads

NP

Peter

NP

V

reads

VP

NP

S

Peter

NP VP

AP

often

VP

S

V

reads

NP

Peter

NP VP

AP

often

VP

S

V

reads

NP

DT

book

NN

substitution

VPNP

S

V NP

reads

VP

VP*AP

often

adjunction

NP

Peter

NP

NNDT

book

substitution substitution

TAG





Comparison with and TAG

PLTAG derivation are always incremental and fully connected.

Peter

NP

ssVPNP

S

s

s

initial(subst)

s

ss

NP

NNDTs

Peter

NP VP

AP

often

VP

S

s s

s

s

Peter

NP VP

AP

often

VP

S

V

reads

NP

Peter

NP VP

AP

often

VP

S

sNPV

reads DT NNsss

Peter

NP VP

AP

often

VP

S

sNPV

reads DT NNsss

a

Peter

NP VP

AP

often

VP

S

V

reads

NP

DT

book

NN

a

Peter

NP

Peter

NP

PLTAG

substitution

VP

VP*AP

often

adjunction

VPNP

S

V NP

reads

verification substitution

a

DT

substitution

NP

NNDT

book

verification

s s

s

VP

S





Modeling Prediction

PLTAG assumes three types of prediction:

predictive nodes (required for connectivity);

open substitution nodes (subcategorization);

lexical prediction (e.g., either . . . or).

Connectedness and prediction interact closely:

in order to achieve incrementality with full connectedness,upcoming nodes have to be predicted;

in a fully connected structure, predictions can be read offstraightforwardly (all open prediction and substitution nodes).





Modeling Prediction

Predictive Nodes:

Peter

NP

initial(subst)

s

ss

NP

NNDTs

Peter

NP VP

AP

often

VP

S

sNPV

reads DT NNsss

Peter

NP VP

AP

often

VP

S

sNPV

reads DT NNsss

a

Peter

NP VP

AP

often

VP

S

V

reads

NP

DT

book

NN

a

Peter

NP

Peter

NP

Peter

NP

Peter

NP

PLTAGVP

VP*AP

often

adjunction

VPNP

S

V NP

reads

substitution

a

DT

substitution

NP

NNDT

book

verification

NP VPsss

Ss

s

Ss

sVPs

NP

AP

often

VP

V

reads

s

AP

often

VP

Ss

VPs

s

verificationsubstitution

S

VP





Modeling Prediction

Open Substitution Nodes:

Peter

NP

ssVPNP

S

s

s

initial(subst)

s

ss

NP

NNDTs

Peter

NP VP

AP

often

VP

S

s s

s

s

Peter

NP VP

AP

often

VP

S

sNPV

reads DT NNsss

a

Peter

NP VP

AP

often

VP

S

V

reads

NP

DT

book

NN

a

Peter

NP

Peter

NP

Peter

NP

Peter

NP VP

AP

often

VP

S

V

reads

PLTAG

substitution

VP

VP*AP

often

adjunction

VPNP

S

V NP

reads

verification substitution

a

DT

substitution

NP

NNDT

book

verification

s s

s

VP

S

ssNPV

reads NNss

VP

AP

often

VP

S

NP

DT





Modeling Prediction

Lexical prediction based on TAG’s extended domain of locality:

NP CC NP

NP

DT

either or

Peter

NP VP

read a book

NPV

S

Peter

NP

DT NP CC NP

either a book or

V

read

VP

S

NP

CC SS*or

CC NPNP*or

NP

(b) derivation at "or" in either−case(a) lexicon entry for "either"

(c) ambiguity at "or"adjunction

(d) lexicon entries for "or"S

S1

S1S1




Treebank Conversion and Lexicon InductionParsing Algorithm and Probability ModelLinking Theory









An Incremental Parser for PLTAG

In order to construct an incremental parser for PLTAG, we need to:

1 convert the Penn Treebank into PLTAG format;

2 induce a lexicon from it;

3 develop an incremental parsing algorithm;

4 devise a probability model;

5 formulate a linking theory.





Step 1: Treebank Conversion

Convert Penn Treebank into TAG format (Xia et al. 2000) using:

head percolation table for determining how to cut up a treeinto elementary trees (Magerman 1995);

Propbank for distinguishing arguments and modifiers (Palmeret al. 2003);

noun phrase annotation to derive NP-internal structure(Vadas & Curran 2007).

The resulting trees are less flat, contain head information, andargument/modifier distinction.





Step 2: Lexicon Induction

A standard TAG lexicon can be derived from the TAG Treebank bycutting up the trees into initial trees and adjunction trees.

For the predictive lexicon, we need the notion of connection path.

Connection Path

The connection path of w1 is the minimal amount of structureneeded to connect words w1 . . .wi under one node (Sturt et al.2003).

Essentially, we determine which parts of the tree we need topredict to achieve connectivity.






predictive lexiconentries generatedfrom tree

The

DET

Italian

ADJ

N

people

N

NP

vote

V

VP

The

DET

Italian

ADJ

N

people

N

NP

vote

V

VP

Berlusconi

NPoften

ADVP

VP

Berlusconi

NPoften

ADVP

VP

SS

none







The

DET

Italian

ADJ

N

people

N

NP

vote

V

VP

Berlusconi

NPoften

ADVP

VP

S

The

DET

Italian

ADJ

N

people

N

NP

vote

V

VP

Berlusconi

NPoften

ADVP

VP

S







vote

V

VP

vote

V

VP

Berlusconi

NPoften

ADVP

VP

S

people

N

NP

DET

NP

Ns

s

ss







The

DET

Italian

ADJ

N

people

N

NP

vote

V

VP

Berlusconi

NPoften

ADVP

VP

S

The

DET

Italian

ADJ

N

people

N

NP

vote

V

VP

Berlusconi

NPoften

ADVP

VP

S

DET

NP

Ns

s

ss







Berlusconi

NP

no new predictive entry

SS

Berlusconi

NP

vote

V

VP

vote

V

VP

often

ADVP

VP

often

ADVP

VP

DET

NP

Ns

s

ss







The

DET

Italian

ADJ

N

people

N

NP

vote

V

VP

The

DET

Italian

ADJ

N

people

N

NP

vote

V

VP

Berlusconi

NPoften

ADVP

VP

Berlusconi

NPoften

ADVP

VP

SS

DET

NP

Ns

s

ss







vote

V

VP

vote

V

VP

Berlusconi

NP

Berlusconi

NP

SS

DET

NP

N

S

VPNP s

s

ss

s

s

ss







The

DET

Italian

ADJ

N

people

N

NP

vote

V

VP

Berlusconi

NPoften

ADVP

VP

S

The

DET

Italian

ADJ

N

people

N

NP

vote

V

VP

Berlusconi

NPoften

ADVP

VP

S

DET

NP

Ns

s

ss

S

VPNP s

s

ss





Step 3: Parsing Algorithm

Properties:

incrementally builds fully connected partial structures;

only allows valid partial PLTAG structures;

constructs all possible structures in parallel.

At word wi , retrieve elementary tree ε for wi and connect it to theprefix tree β for w1 . . .wi−1:

parsing operations: substitution, adjunction, verification;

dependent on status of β and ε: standard or predictive tree.





Step 4: Probability Model

The following properties need to hold (Chiang 2000):

Substitution:∑ε

P(ε|ηβ) = 1

Adjunction:∑ε

P(ε|ηβ) + P(NONE |ηβ) = 1

Verification:∑ε

P(ε|πβ) = 1

where P(ε|ηβ) = P(τε|ηβ)P(λε|τε, λβ)and P(ε|πβ) = P(τε|πβ)P(λε|τε)

elementary tree ε prefix tree β prediction tree πtree structure τ integration point node η a tree’s head leaf λ





Step 5: Linking Theory

The linking theory translates parser states into processing difficulty:

elementary tree εwi is integrated with prefix tree βw1...wi−1 ;

processing difficulty proportional to change in distributionP(β) from wi−1 to wi ;

each predicted tree π has a time-stamp t;

at verification, decay d is calculated based on t (recentlyaccessed structures are easier to integrate).

Surprisal

Dwi =

︷︸︸︷− log

∑βw1...wi

P(βw1...wi ) + log∑

βw1...wi−1

P(βw1...wi−1)

− log∑π

P(π)(1−d tπ )

}Verification Cost




Parsing PerformanceCognitive Plausibility









Parsing Performance

Computational evaluation of PLTAG parser:

train and test on standard Penn Treebank data (converted toPLTAG), with sentences of length 40 or less;

assume gold-standard POS tags;

use a supertagger to choose prediction trees (one wordlookahead);

coverage on the test set is not perfect: beam search; missinglexical entries.





Results

Model Precision Recall F-score CoverageBaseline 44.39 52.38 48.06 85.10PLTAG Parser 78.01 78.83 78.42 92.73Prediction Tree Oracle 79.88 80.51 80.19 89.54

Baseline: pick most frequent tree (highest combined frequency ofall subtrees).

Oracle: assume correct prediction tree (instead of supertagging).





Comparison with other TAG Parsers

imple- incre-Model mented mental connected predictive F-scoreMazzei et al. (2007) – + + + n/aThis work + + + + 78.4Kato et al. (2004) + + + – 79.7Sarkar (2001) + – – – 79.8Chiang (2000) + – – – 86.7Shen & Joshi (2005) + + – – 87.4∗

∗evaluated on dependencies





Comparison with other TAG Parsers

Performance not directly comparable with parsers that use theoriginal Treebank structure (simpler NP structure, etc.);

there are structural differences even with other TAG parsers(LTAG, spinal TAG);

Mazzei et al. (2007) parser conceptually most similar, but notimplemented and evaluated;

Kato et al. (2004) make strong simplifying assumptions (nomodifier/argument distinction);

Sarkar (2001) and Chiang (2000) parsers are not incremental;

Shen & Joshi (2005) don’t build connected structures.





Cognitive Plausibility

Psycholinguistic evaluation of PLTAG parser:

train on Penn Treebank;

take experimental materials from psycholinguistic experiments;

parse them using the PLTAG parser, compute processingdifficulty values for each sentence;

compare to published reading time results.

Baseline: standard surprisal model (PLTAG without prediction andverification component).





Either . . . or Constructions

PLTAG model predicts difficulty in either . . . or constructions:

(6) Peter read either a book or an essay in the school magazine.

(7) Peter read a book or an essay in the school magazine.





Relative Clause Asymmetry

Classic result in psycholinguistics: subject relative clauses are easierto process than object relative clauses.

(8) SRC: The reporter that attacked the senator admitted the error.

(9) ORC: The reporter that the senator attacked admitted the error.

340

360

380

400

420

440

460

Rea

ding

tim

e in

mse

c

Reading Times for SRC vs. ORC (Gibson, 1998)

SRCORC

admitted the error.who attacked thewho the senator

senatorattacked

The reporter





The relative clause asymmetry

PLTAG model predicts difficulty at verb region:

Correct predictions, but verification component necessary, resultsnot predicted by surprisal-only baseline.





Conclusions

Human sentence processing is incremental and predictive;

evidence for lexical syntactic prediction (subcat frames);

we presented a version of TAG that models these properties;

the model comes with a parser, a probability model, and alinking theory;

performance comparable to parsers with similar properties inthe TAG literature;

cognitive evaluation using experimental data: either . . . orprediction and relative clauses.





References

Altmann, G. T. M., & Kamide, Y. (1999). Incremental interpretation at verbs: Restricting the domain ofsubsequent reference. Cognition, 73 , 247–264.

Chiang, D. (2000). Statistical parsing with an automatically-extracted tree adjoining grammar. In Proceedings ofthe 38th Annual Meeting of the Association for Computational Linguistics, (pp. 463–470), Hong Kong.

Demberg, V., & Keller, F. (2008). A psycholinguistically motivated version of TAG. In Proceedings of the 9thInternational Workshop on Tree Adjoining Grammars and Related Formalisms, (pp. 25–32), Tubingen.

Kato, Y., Matsubara, S., & Inagaki, Y. (2004). Stochastically evaluating the validity of partial parse trees inincremental parsing. In F. Keller, S. Clark, M. Crocker, & M. Steedman (eds.), Proceedings of the ACLWorkshop on Incremental Parsing: Bringing Engineering and Cognition Together , (pp. 9–15), Barcelona.

Magerman, D. (1995). Statistical decision-tree models for parsing. In Proceedings of the 33rd Annual Meeting ofthe Association for Computational Linguistics, (pp. 276–283), Cambridge, MA.

Mazzei, A., Lombardo, V., & Sturt, P. (2007). Dynamic TAG and lexical dependencies. Research on Language andComputation, 5 , 309–332.

Palmer, M., Gildea, D., & Kingsbury, P. (2003). The proposition bank: An annotated corpus of semantic roles.Computational Linguistics, 31 , 71–106.

Sarkar, A. (2001). Applying co-training methods to statistical parsing. In Proceedings of the 2nd Conference of theNorth American Chapter of the Association for Computational Linguistics, (pp. 175–182), Pittsburgh, PA.Association for Computational Linguistics.

Shen, L., & Joshi, A. K. (2005). Incremental LTAG parsing. In Proceedings of the Human Language TechnologyConference of the North American Chapter of the Association for Computational Linguistics, (pp. 811–818),Vancouver.

Staub, A., & Clifton, C. (2006). Syntactic prediction in language comprehension: Evidence from either . . . or.Journal of Experimental Psychology: Learning, Memory, and Cognition, 32 , 425–436.

Sturt, P., Costa, F., Lombardo, V., & Frasconi, P. (2003). Learning first-pass structural attachment preferenceswith dynamic grammars and recursive neural networks. Cognition, 88 , 133–169.

Sturt, P., & Lombardo, V. (2005). Processing coordinated structures: Incrementality and connectedness. CognitiveScience, 29 , 291–305.

Vadas, D., & Curran, J. (2007). Adding noun phrase structure to the Penn Treebank. In Proceedings of the 45thAnnual Meeting of the Association for Computational Linguistics, (pp. 240–247), Prague.

van Berkum, J. J. A., Brown, C. M., Zwitserlood, P., Kooijman, V., & Hagoort, P. (2005). Anticipating upcomingwords in discourse: Evidence from erps and reading times. Journal of Experimental Psychology: Learning,Memory, and Cognition, 31 , 443–467.

Xia, F., Palmer, M., & Joshi, A. (2000). A uniform method of grammar extraction and its applications. InProceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and VeryLarge Corpora, (pp. 53–62), Hong Kong.


Documents

Incremental, Predictive Parsing with Psycholinguistically Motivated … · 2011-04-13 · Incrementality Prediction Case Study: Syntactic Prediction Incrementality Text and speech