Lect10 Earley

8/12/2019 Lect10 Earley

1/96

600.465 - Intro to NLP - J. Eisner 1

Earleys Algorithm (1970)

Nice combo of our parsing ideas so far:

no restrictions on the form of the grammar:

A B C spoon D x

incremental parsing (left to right, like humans)

left context constrains parsing of subsequent words

so waste less time building impossible things

makes it faster than O(n3) for many grammars


2/96



3/96


Overview of Earleys Algorithm

Finds constituents and partial constituents in input

A B C . D E is partial: only the first half of theA

A

B C D E

A B C . D E

D+ =A

B C D E

A B C D . E

i j i k

j k


4/96


Overview of Earleys Algorithm

Proceeds incrementally, left-to-right

Before it reads word 5, it has already built allhypotheses that are consistent with first 4 words

Reads word 5 & attaches it to immediately preceding

hypotheses. Might yield new constituents that are thenattached to hypotheses immediately preceding them

E.g., attaching D toA B C . D E givesA B C D . E

Attaching E to that givesA B C D E .

Now we have a completeA that we can attach tohypotheses immediately preceding theA, etc.


5/96


Our Usual Example Grammar

ROOT S

S NP VP NP PapaNP Det N N caviar

NP NP PP N spoon

VP VP PP V ateVP V NP P with

PP P NP Det the

Det a


6/96


First Try: Recursive Descent

0 ROOT . S0

0 S . NP VP0

0NP

. Papa0

0 NP Papa . 1

0 S NP . VP1

1 VP . VP PP1

1 VP . VP PP1

1 VP . VP PP1

1 VP . VP PP1

oops, stack overflowed

OK, lets pretend that didnt happen.

Lets suppose we didnt see VP VP PP, and used VP V NP instead.

ROOT S VP VP PP NP Papa V ate

S NP VP VP V NP N caviar P with

NP Det N PP P NP N spoon Det the

NP NP PP Det a

0Papa 1ate 2the 3caviar 4with 5a 6spoon 7

goal stack


7/96



1VP . V NP1 after dot = nonterminal, so recursively look for it (predict)

1V . ate 1 after dot = terminal, so look for it in the input (scan)

1 V ate . 2 after dot = nothing, so parents subgoal is completed (attach) 1 VP V . NP2 predict (next subgoal)

2NP . ... 2 do some more parsing and eventually ...

2NP ... . 7 we complete the parents NP subgoal, so attach

1 VP V NP . 7 attach again

0 S NP VP . 7 attach again

ROOT S VP V NP NP Papa V ate

S NP VP VP VP PP N caviar P with


NP NP PP Det a


0 ROOT . S0

0 S . NP VP0

0 NP . Papa 0

0 NP Papa . 1

0 S NP . VP1 after dot = nonterminal, so recursively look for it (predict)


8/96



0 ROOT . S0

0 S . NP VP0

0 NP . Papa 0

0 NP Papa . 1

0 S NP . VP1

1VP . V NP1

1V . ate 1

1 V ate . 2

1 VP V . NP2

2NP . ... 2

2NP ... . 7

1 VP V NP . 7




NP NP PP Det a

But how about the other parse?

must backtrack to try predictinga different VP rule here instead


implement by function calls:

S() calls NP() and VP(), which recurse


9/96



1VP . V NP1

1V . ate 1

1 V ate . 2

1 VP V . NP2

2NP . ... 2 do some more parsing and eventually ...

2NP ... . 4 ... the correct NP is from 2 to 4 this timebut mi ht we find the one from 2 to 7 instead?




NP NP PP Det a

wed better backtrack here too!(why?)


0 ROOT . S0

0 S . NP VP0

0 NP . Papa 0

0 NP Papa . 1

0 S NP . VP1

1VP . VP PP1


10/96



1VP . VP PP1

1VP . VP PP1

1VP . VP PP1

oops, stack overflowed

no fix after allmust transform grammar to eliminate left-recursive rules




NP NP PP Det a


0 ROOT . S0

0 S . NP VP0

0 NP . Papa 0

0 NP Papa . 1

0 S NP . VP1

1VP . VP PP1

1VP . VP PP1


11/96


Use a Parse Table (= Chart)

Earleys algorithm resembles recursive descent, but solvesthe left-recursion problem. No recursive function calls.

Use a parse table as we did in CKY, so we can look upanything weve discovered so far.

Dynamic programming.

Entries in column 5 look like (3, S NP . VP)(but well omit the etc. to save space)

Built while processing word 5

Means that the input substring from 3 to 5matches the initial NP portion of a S NP VP rule

Dot shows how much weve matched as of column 5

Perfectly fine to have entries like (3, S is it . true that S)


12/96


Use a Parse Table (Chart) Entries in column 5 look like (3, S NP . VP)

What does it mean if we have this entry? Unknown right context: Doesntmean well necessarily be

able to find a VP starting at column 5 to complete the S.

Known left context: Does mean that some dotted rule back

in column 3 is looking for an S that starts at 3. So if we actually do find a VP starting at column 5, allowing us to

complete the S, then well be able to attach the S to something.

And when that something is complete, it too will have a customerto itsleft just as in recursive descent!

In short, a top-down (i.e., goal-directed) parser: it chooses to startbuilding a constituent not because of the input but because thatswhat the left context needs. In the spoon, wont build spoon as averb because theres no way to use a verb there.

So any hypothesis in column 5 couldget used in the correct parse,if words 1-5 are continued in just the right way by words 6-n.


13/96


Operation of the Algorithm

Process all hypotheses one at a time in order.

(Current hypothesis is shown in blue, with substring.) This may add to the end of the

to-do list, or try to add again.new hypotheses

old hypotheses

Process a hypothesis according to what followsthe dot just as in recursive descent: If a word, scan input and see if it matches

If a nonterminal, predict ways to match it(well predict blindly, but could reduce # of predictions by

looking ahead k symbols in the input and only makingpredictions that are compatible with this limited right context)

If nothing, then we have a complete constituent, so

attach it to all its customers (shown in purple).


14/96


B C D E

i j

A

(i, A B C . D E)

column j

current hypothesis (incomplete)which action?

One entry (hypothesis)

All entries ending at j stored in column j, as in CKY


15/96


(j, D . bludger)new entry to process later

Predict

B C D E

i j

A

(i, A B C . D E)

column j

current hypothesis (incomplete)

D

j j bludger


16/96


Scan

(i, A B C . D E)

column j

D

j j bludger

(j, D bludger .)

column k

new entry to process later

k=j+1 in this example

Current hypothesis (incomplete)(j, D . bludger)

+ j kbludger = D

j kbludger


17/96


Attach

D

j k

(i, A B C . D E)

column j column k

(j, D bludger .)

current hypothesis (complete)


18/96


Attach

D

j k

B C D E

+

i j

A

(i, A B C . D E)

=

B C D E

i k

A

(i, A B C D . E)

column j column k

(j, D bludger .)

current hypothesis (complete)

customer (incomplete)

new entry to process later


19/96


Our Usual Example GrammarROOT S

S NP VP NP PapaNP Det N N caviar

NP NP PP N spoon

VP VP PP V ateVP V NP P with

PP P NP Det the

Det a



20/96

0

0 ROOT . S

initialize

Remember this stands for (0, ROOT . S)


21/96

0

0 ROOT . S

0 S . NP VP

predict the kind of S we are looking for

Remember this stands for (0, S . NP VP)


22/96

0

0 ROOT . S

0 S . NP VP

0 NP . Det N0 NP . NP PP

0 NP . Papa

predict the kind of NP we are looking for

(actually well look for 3 kinds: any of the 3 will do)


23/96

0

0 ROOT . S

0 S . NP VP


0 NP . Papa

0 Det . the

0 Det . a

predict the kind of Det we are looking for (2 kinds)


24/96

0

0 ROOT . S

0 S . NP VP


0 NP . Papa

0 Det . the

0 Det . a

predict the kind of NP were looking forbut we were already looking for these sodont add duplicate goals! Note that this happened

when we were processing a left-recursive rule.


25/96


26/96

0 Papa 1

0 ROOT . S 0 NP Papa .

0 S . NP VP


0 NP . Papa

0 Det . the

0 Det . a

scan: failure


27/96

0 Papa 1


0 S . NP VP


0 NP . Papa

0 Det . the

0 Det . a scan: failure


28/96

0 Papa 1


0 S . NP VP 0 S NP . VP

0 NP . Det N 0 NP NP . PP0 NP . NP PP

0 NP . Papa

0 Det . the

0 Det . a

attach the newly created NP(which starts at 0) to its customers

(incomplete constituents that end at 0and have NP after the dot)


29/96

0 Papa 1



0 NP . Det N 0 NP NP . PP0 NP . NP PP 1 VP . V NP

0 NP . Papa 1 VP . VP PP

0 Det . the

0 Det . a

predict


30/96

0 Papa 1





0 Det . the 1 PP . P NP

0 Det . a

predict


31/96

0 Papa 1






0 Det . a 1 V . ate

predict


32/96

0 Papa 1






0 Det . a 1 V . ate

predict


33/96

0 Papa 1






0 Det . a 1 V . ate

1 P . with

predict


34/96

0 Papa 1 ate 2

0 ROOT . S 0 NP Papa . 1 V ate .





0 Det . a 1 V . ate

1 P . with

scan: success!


35/96

0 Papa 1 ate 2






0 Det . a 1 V . ate

1 P . with scan: failure


36/96

0 Papa 1 ate 2


0 S . NP VP 0 S NP . VP 1 VP V . NP




0 Det . a 1 V . ate

1 P . with

attach


37/96

0 Papa 1 ate 2



0 NP . Det N 0 NP NP . PP 2 NP . Det N0 NP . NP PP 1 VP . V NP 2 NP . NP PP

0 NP . Papa 1 VP . VP PP 2 NP . Papa


0 Det . a 1 V . ate

1 P . with

predict


38/96

0 Papa 1 ate 2





0 Det . the 1 PP . P NP 2 Det . the

0 Det . a 1 V . ate 2 Det . a

1 P . with

predict (these next few stepsshould look familiar)


39/96

0 Papa 1 ate 2







1 P . with

predict


40/96

0 Papa 1 ate 2







1 P . with

scan (this time we fail sincePapa is not the next word)


41/96

0 Papa 1 ate 2 the 3

0 ROOT . S 0 NP Papa . 1 V ate . 2 Det the .






1 P . with

scan: success!


42/96








1 P . with


43/96



0 S . NP VP 0 S NP . VP 1 VP V . NP 2 NP Det . N





1 P . with


44/96




0 NP . Det N 0 NP NP . PP 2 NP . Det N 3 N . caviar0 NP . NP PP 1 VP . V NP 2 NP . NP PP 3 N . spoon




1 P . with


45/96

0 Papa 1 ate 2 the 3 caviar 4

0 ROOT . S 0 NP Papa . 1 V ate . 2 Det the . 3 N caviar .






1 P . with


46/96








1 P . with


47/96



0 S . NP VP 0 S NP . VP 1 VP V . NP 2 NP Det . N 2 NP Det N .





1 P . with

attach


48/96




0 NP . Det N 0 NP NP . PP 2 NP . Det N 3 N . caviar 1 VP V NP .0 NP . NP PP 1 VP . V NP 2 NP . NP PP 3 N . spoon 2 NP NP . PP




1 P . with

attach

(again!)


49/96





0 NP . Papa 1 VP . VP PP 2 NP . Papa 0 S NP VP .

0 Det . the 1 PP . P NP 2 Det . the 1 VP VP . PP


1 P . with

attach(again!)


50/96







0 Det . a 1 V . ate 2 Det . a 4 PP . P NP

1 P . with


51/96








1 P . with 0 ROOT S .

attach(again!)


52/96










53/96









4 P . with


54/96









4 P . with


55/96

0 Papa 1 ate 2 the 3 caviar 4 with 5

0 ROOT . S 0 NP Papa . 1 V ate . 2 Det the . 3 N caviar . 4 P with .







4 P . with


56/96



0 S . NP VP 0 S NP . VP 1 VP V . NP 2 NP Det . N 2 NP Det N . 4 PP P . NP






4 P . with


57/96




0 NP . Det N 0 NP NP . PP 2 NP . Det N 3 N . caviar 1 VP V NP . 5 NP . Det N0 NP . NP PP 1 VP . V NP 2 NP . NP PP 3 N . spoon 2 NP NP . PP 5 NP . NP PP

0 NP . Papa 1 VP . VP PP 2 NP . Papa 0 S NP VP . 5 NP . Papa




4 P . with


58/96




0 NP . Det N 0 NP NP . PP 2 NP . Det N 3 N . caviar 1 VP V NP . 5 NP . Det N

0 NP . NP PP 1 VP . V NP 2 NP . NP PP 3 N . spoon 2 NP NP . PP 5 NP . NP PP


0 Det . the 1 PP . P NP 2 Det . the 1 VP VP . PP 5 Det . the

0 Det . a 1 V . ate 2 Det . a 4 PP . P NP 5 Det . a


4 P . with


59/96










4 P . with


60/96










4 P . with


61/96










4 P . with


62/96

ate 2 the 3 caviar 4 with 5 a 6

. 1 V ate . 2 Det the . 3 N caviar . 4 P with . 5 Det a .

1 VP V . NP 2 NP Det . N 2 NP Det N . 4 PP P . NP

P 2 NP . Det N 3 N . caviar 1 VP V NP . 5 NP . Det N

2 NP . NP PP 3 N . spoon 2 NP NP . PP 5 NP . NP PP

P 2 NP . Papa 0 S NP VP . 5 NP . Papa

2 Det . the 1 VP VP . PP 5 Det . the

2 Det . a 4 PP . P NP 5 Det . a

0 ROOT S .

4 P . with


63/96



1 VP V . NP 2 NP Det . N 2 NP Det N . 4 PP P . NP 5 NP Det . N

P 2 NP . Det N 3 N . caviar 1 VP V NP . 5 NP . Det N

2 NP . NP PP 3 N . spoon 2 NP NP . PP 5 NP . NP PP




0 ROOT S .

4 P . with


64/96




P 2 NP . Det N 3 N . caviar 1 VP V NP . 5 NP . Det N 6 N . caviar

2 NP . NP PP 3 N . spoon 2 NP NP . PP 5 NP . NP PP 6 N . spoon




0 ROOT S .

4 P . with


65/96









0 ROOT S .

4 P . with


66/96


67/96

ate 2 the 3 caviar 4 with 5 a 6 spoon 7

. 1 V ate . 2 Det the . 3 N caviar . 4 P with . 5 Det a . 6 N spoon .

1 VP V . NP 2 NP Det . N 2 NP Det N . 4 PP P . NP 5 NP Det . N 5 NP Det N .






0 ROOT S .

4 P . with


68/96

ate 2 the 3 caviar 4 with 5 a 6 spoon 7. 1 V ate . 2 Det the . 3 N caviar . 4 P with . 5 Det a . 6 N spoon .

1 VP V . NP 2 NP Det . N 2 NP Det N . 4 PP P . NP 5 NP Det . N 5 NP Det N .

P 2 NP . Det N 3 N . caviar 1 VP V NP . 5 NP . Det N 6 N . caviar 4 PP P NP .

2 NP . NP PP 3 N . spoon 2 NP NP . PP 5 NP . NP PP 6 N . spoon 5 NP NP . PP




0 ROOT S .

4 P . with


69/96

0 Papa 1 ate 2 the 3 caviar 4 with a spoon 70 ROOT . S 0 NP Papa . 1 V ate . 2 Det the . 3 N caviar . 6 N spoon .

0 S . NP VP 0 S NP . VP 1 VP V . NP 2 NP Det . N 2 NP Det N . 5 NP Det N .

0 NP . Det N 0 NP NP . PP 2 NP . Det N 3 N . caviar 1 VP V NP . 4 PP P NP .

0 NP . NP PP 1 VP . V NP 2 NP . NP PP 3 N . spoon 2 NP NP . PP 5 NP NP . PP

0 NP . Papa 1 VP . VP PP 2 NP . Papa 0 S NP VP . 2 NP NP PP .

0 Det . the 1 PP . P NP 2 Det . the 1 VP VP . PP 1 VP VP PP .



4 P . with


70/96







0 Det . a 1 V . ate 2 Det . a 4 PP . P NP 7 PP . P NP


4 P . with


71/96








1 P . with 0 ROOT S . 1 VP V NP .

4 P . with 2 NP NP . PP


72/96










0 S NP VP .

1 VP VP . PP


73/96










0 S NP VP .

1 VP VP . PP

7 P . with


74/96










0 S NP VP .

1 VP VP . PP

7 P . with


75/96










0 S NP VP .

1 VP VP . PP

7 P . with


76/96










0 S NP VP .

1 VP VP . PP

7 P . with

0 ROOT S .


77/96










0 S NP VP .

1 VP VP . PP

7 P . with

0 ROOT S .


78/96










0 S NP VP .

1 VP VP . PP

7 P . with

0 ROOT S .


79/96










0 S NP VP .

1 VP VP . PP

7 P . with

0 ROOT S .


80/96

Left Recursion Kills Pure

Top-Down Parsing


81/96


p g

VP


Top-Down Parsing


82/96


p g

VP

VP PP


Top-Down Parsing


83/96


VP PP

VP

VP PP


Top-Down Parsing


84/96


VP PP

VP

VP PP

VP PP

makes new hypothesesad infinitum before weveseen the PPs at all

hypotheses try to predictin advance how manyPPs will arrive in input

but Earleys Alg is Okay!


85/96


VP

PPVP

1 VP . VP PP

(in column 1)



86/96


VP

V NP

1 VP V NP .

ate the caviar

VP

PPVP

1 VP . VP PP

(in column 1)

(in column 4)



87/96


VP

V NP

VP

PPVP

V NP

attach

ate the caviar

1 VP VP . PP

VP

PPVP

1 VP . VP PP

(in column 1)

(in column 4)



88/96


VP

V NP

VP

PPVP

V NPate the caviar

with a spoon

1 VP VP PP .

VP

PPVP

1 VP . VP PP

(in column 1)

(in column 7)



89/96


VP

V NP

VP

PPVP

V NPate the caviar

with a spoon

1 VP VP PP .

VP

PPVP

1 VP . VP PP

can be reused(in column 1)

(in column 7)



90/96


VP

V NP

VP

PPVP

V NP

VP

PP

1 VP VP . PP

ate the caviar

with a spoon

VP

PPVP

1 VP . VP PP

can be reused(in column 1)

(in column 7)

attach


91/96



92/96


VP

V NP

VP

PPVP

V NP

VP

PP

ate the caviar

with a spoon

in his bed

1 VP VP PP .

VP

PPVP

1 VP . VP PP

can be reused again(in column 1)

(in column 10)



93/96


VP

V NP

VP

PPVP

V NP

VP

PPVP

1 VP . VP PP

can be reused again

VP

PP

VP

PP

1 VP VP . PP

ate the caviar

with a spoon

in his bed

(in column 1)

(in column 10)

attach

0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7


94/96







0 Det . a 1 V . ate 2 Det . a 4 PP . P NP 7 PP . P NP1 P . with 0 ROOT S . 1 VP V NP .


0 S NP VP .

1 VP VP . PP

7 P . with

0 ROOT S .

completed a VP in col 4col 1 lets us use it in a VP PP structure

0 Papa 1 ate 2 the 3 caviar 4 with a spoon 7


95/96






0 Det . the 1 PP . P NP 2 Det . the 1 VP VP . PP 1 VP VP PP.

0 Det . a 1 V . ate 2 Det . a 4 PP . P NP 7 PP . P NP1 P . with 0 ROOT S . 1 VP V NP .


0 S NP VP .

1 VP VP . PP

7 P . with

0 ROOT S .

completed that VP = VP PP in col 7col 1 would let us use itin a VP PP structurecan reuse col 1 as often as we need

Whats the Complexity?


96/96


Documents

Lect10 Earley