Part 3: Main StatisticalMethods

2017-11-27 IJCNLP 2017 Tutorial 1

Part 3.1: Graph-based Methods

2017-11-27 IJCNLP 2017 Tutorial 2

Graph-based Methods• Training: Score⇔ Correctness• Decoding model score

• enumerate all candidate structures; scoring each and selecting the highestscored structure

2017-11-27 IJCNLP 2017 Tutorial 3

Sequence Labeling Models

2017-11-27 IJCNLP 2017 Tutorial 4

𝑃 𝑦 #:% 𝑥 #:% ∝1

𝑍*[,:-]0

exp∑ 𝜆6𝑓6 𝑦8, 𝑦8:#�6

+∑ 𝜇>𝑔> 𝑦8, 𝑥8�>

%

8@#

yt yt+1yt-1

xt xt+1xt-1

CRF

CRF Decoding

where GEN(𝑥 #:% ) is all possible tag sequences

2017-11-27 IJCNLP 2017 Tutorial 5

argmax*[,:-]∈KLM(N ,:- )

O𝐰 Q 𝐟(𝑥 #:% , 𝑦S, 𝑦S:#)%

S@#

Viterbi Algorithm• Define a dynamic programming table

• 𝜋(𝑖, 𝑦) = maximum score of a tag sequence ending in tag y at position i

• Recursive definition: 𝜋 𝑖, 𝑦 = max8

𝜋 𝑖 − 1, 𝑡 + 𝐰 Q 𝐟 𝑥 #:% , 𝑦, 𝑡

2017-11-27 IJCNLP 2017 Tutorial 6

START

V

N

P

D

Time

V

N

P

D

flies

V

N

P

D

like

V

N

P

D

an

V

N

P

D

arrow

1

0.7

0.2

0.3

0.1

1.7

1.2

1.3

1.10.8

0.2

0.2

0.5

1.9

1.3

1.2

2.5

15.9

10.3

12.2

11.2

ConstituencyParsing

2017-11-27 IJCNLP 2017 Tutorial 7

Constituency Parsing with CRF

• Probabilityofatree T conditionedonasentencew

• More features

2017-11-27 IJCNLP 2017 Tutorial 8

Hall D, Durrett G, Klein D (2014) Less grammar, more features. ACL.

fish peoplefishtanks

RuleProb θi

S® NPVP θ0NP® NPNP θ1…

N® fish θ42N® people θ43V® fish θ44

PCFG

• E.g.Cocke–Younger–Kasami algorithm (CYKor CKY)• AkindofDynamicProgramming

2017-11-27 IJCNLP 2017 Tutorial 9

Chart-based Method

2017-11-27 IJCNLP 2017 Tutorial 10

CKY Parsing Algorithm

Graph-basedDependencyParsing

• Findthehighestscoringtreefromacompletedependencygraph

2017-11-27 IJCNLP 2017 Tutorial 11

He1 does2 it3 here4$0 He1 does2 it3 here4$0

*

( )argmax ( , )Y X

Y score X YÎF

=

First-orderasanExample

• Thefirst-ordergraph-basedmethodassumesthatarcs inatreeareindependentfromeachother(arc-factorization)

2017-11-27 IJCNLP 2017 Tutorial 12

He1 does2 it3 here4$0

( , )( , ) ( , , )

h m Yscore X Y score X h m

Î

= å

Featuresforan Arc

2017-11-27 IJCNLP 2017 Tutorial 13

Decodingforfirst-ordermodel

•MaximumSpanningTree (MST) Algorithm•Eisner(2000)describedadynamicprogrammingbaseddecodingalgorithmforbilexical grammar

•McDonald et al.(2005)appliedthisalgorithmtothesearchproblemofthesecond-ordermodel

•Koo et al. (2010) investigate third-order model

2017-11-27 IJCNLP 2017 Tutorial 14

Part 3.2: Transition-basedMethods

2017-11-27 IJCNLP 2017 Tutorial 15

Transition Systems

2017-11-27 IJCNLP 2017 Tutorial 16

Atransitionsystem

• Automata•State

• Startstate—— anemptystructure• Endstate—— theoutputstructure• Intermediatestates—— partiallyconstructedstructures

•Actions• Changeonestatetoanother

2017-11-27 IJCNLP 2017 Tutorial 17

Atransitionsystem

• Automata

2017-11-27 IJCNLP 2017 Tutorial 18

start

Atransitionsystem

• Automata

2017-11-27 IJCNLP 2017 Tutorial 19

start

a0

S1

Atransitionsystem

• Automata

2017-11-27 IJCNLP 2017 Tutorial 20

start …

a0

S1

a1

Atransitionsystem

• Automata

2017-11-27 IJCNLP 2017 Tutorial 21

start …

a0

S1 Si

a1 ai-1

Atransitionsystem

• Automata

2017-11-27 IJCNLP 2017 Tutorial 22

start …

a0

S1 Si …

a1 ai-1 ai

Atransitionsystem

• Automata

2017-11-27 IJCNLP 2017 Tutorial 23

start …

a0

S1 Si … Sn

a1 ai-1 ai an-1

Atransitionsystem

• Automata

2017-11-27 IJCNLP 2017 Tutorial 24

start …

a0

S1 Si … Sn end

a1 ai-1 ai an-1 an

Atransitionsystem

•State•Correspondstopartialresultsduringdecoding

• startstate,endstate,Si

•Actions•Theoperationsthatcanbeappliedforstatetransition

•Constructoutputincrementally2017-11-27 IJCNLP 2017 Tutorial 25

start …

a0

S1 Si … Sn end

a1 ai-1 ai an-1 an

Examples

2017-11-27 IJCNLP 2017 Tutorial 26

Atransition-basedPOS-taggingexample

• POStaggingIlikereadingbooks→ I/PRONlike/VERBreading/VERBbooks/NOUN

• Transitionsystem• State

• Partiallylabeledword-POSpairs• Unprocessedwords

• Actions• TAG(t)𝑤1/𝑡1⋯𝑤𝑖/𝑡𝑖 → 𝑤1/𝑡1⋯𝑤𝑖/𝑡𝑖𝑤S]#/𝑡

2017-11-27 IJCNLP 2017 Tutorial 27

Atransition-basedPOS-taggingexample

• StartState

2017-11-27 IJCNLP 2017 Tutorial 28

I like reading books

Atransition-basedPOS-taggingexample

• TAG(PRON)

2017-11-27 IJCNLP 2017 Tutorial 29

I/PRON like reading books

Atransition-basedPOS-taggingexample

• TAG(VERB)

2017-11-27 IJCNLP 2017 Tutorial 30

I/PRON like/VERB reading books

Atransition-basedPOS-taggingexample

• TAG(VERB)

2017-11-27 IJCNLP 2017 Tutorial 31

booksI/PRON like/VERB reading/VERB

Atransition-basedPOS-taggingexample

• TAG(NOUN)

2017-11-27 IJCNLP 2017 Tutorial 32

I/PRON like/VERB reading/VERB books/NOUN

Atransition-basedPOS-taggingexample

• EndState

2017-11-27 IJCNLP 2017 Tutorial 33

I/PRON like/VERB reading/VERB books/NOUN

Wordsegmentation

• State• Partiallysegmentedresults• Unprocessedcharacters

• Twocandidateactions• Separate####→ #####• Append####→ #####

2017-11-27 IJCNLP 2017 Tutorial 34Zhang, Y., & Clark, S. (2007). Chinese Segmentation Using a Word-Based Perceptron Algorithm. ACL.

Wordsegmentation

• InitialState

2017-11-27 IJCNLP 2017 Tutorial 35

我喜欢读书

I like reading books

Zhang, Y., & Clark, S. (2007). Chinese Segmentation Using a Word-Based Perceptron Algorithm. ACL.

Wordsegmentation

• Separate

2017-11-27 IJCNLP 2017 Tutorial 36

喜欢读书我

Zhang, Y., & Clark, S. (2007). Chinese Segmentation Using a Word-Based Perceptron Algorithm. ACL.

Wordsegmentation

• Separate

2017-11-27 IJCNLP 2017 Tutorial 37

欢读书我 喜

Zhang, Y., & Clark, S. (2007). Chinese Segmentation Using a Word-Based Perceptron Algorithm. ACL.

Wordsegmentation

• Append

2017-11-27 IJCNLP 2017 Tutorial 38

读书我 喜欢

Zhang, Y., & Clark, S. (2007). Chinese Segmentation Using a Word-Based Perceptron Algorithm. ACL.

Wordsegmentation

• Separate

2017-11-27 IJCNLP 2017 Tutorial 39

书我 喜欢 读

Zhang, Y., & Clark, S. (2007). Chinese Segmentation Using a Word-Based Perceptron Algorithm. ACL.

Wordsegmentation

• Separate

2017-11-27 IJCNLP 2017 Tutorial 40

我 喜欢 读 书

Zhang, Y., & Clark, S. (2007). Chinese Segmentation Using a Word-Based Perceptron Algorithm. ACL.

Wordsegmentation

• EndState

2017-11-27 IJCNLP 2017 Tutorial 41

我 喜欢 读 书

Zhang, Y., & Clark, S. (2007). Chinese Segmentation Using a Word-Based Perceptron Algorithm. ACL.

Transition-basedDependency Parsing

• State• A stack to hold partial structures• A queue of next incoming words

• Actions• SHIFT, REDUCE, ARC-LEFT, ARC-RIGHT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

• State

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

STSTP...

STLC STRCThe stack

The input

N0 N1 N2 N3 ...

N0LC

Transition-basedDependency Parsing

• Actions • SHIFT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

STSTP...

STLC STRCThe stack

The input

N0 N1 N2 N3 ...

N0LC

Transition-basedDependency Parsing

• Actions • SHIFT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

ST STP...

STLC STRCThe stack

The input

N1 N2 N3 ...N0

N0LC

Transition-basedDependency Parsing

• Actions • REDUCE

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

STSTP...

STLC STRCThe stack

The input

N0 N1 N2 N3 ...

N0LC

Transition-basedDependency Parsing

• Actions • REDUCE

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

ST

STP...

STLC STRC

The stack

The input

N0 N1 N2 N3 ...

N0LC

Transition-basedDependency Parsing

• Actions • ARC-LEFT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

STSTP...

STLC STRCThe stack

The input

N0 N1 N2 N3 ...

N0LC

Transition-basedDependency Parsing

• Actions • ARC-LEFT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

STP...The stack The

input

N0 N1 N2 N3 ...

N0LCST

STLC STRC

Transition-basedDependency Parsing

• Actions • ARC-RIGHT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

STSTP...

STLC STRCThe stack

The input

N0 N1 N2 N3 ...

N0LC

Transition-basedDependency Parsing

• Actions • ARC-RIGHT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

ST STP...

STLC STRCThe stack

The input

N1 N2 N3 ...

N0LC

N0

Transition-basedDependency Parsing

• An Example • S-SHIFT• R-REDUCE• AL-ARC-LEFT• AR-ARC-RIGHT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

He does it here

Transition-basedDependency Parsing

• An Example • S-SHIFT• R-REDUCE• AL-ARC-LEFT• AR-ARC-RIGHT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

He does it here does it hereHe S

Transition-basedDependency Parsing

• An Example • S-SHIFT• R-REDUCE• AL-ARC-LEFT• AR-ARC-RIGHT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

He does it here does it hereHe S does it hereAL

He

Transition-basedDependency Parsing

• An Example • S-SHIFT• R-REDUCE• AL-ARC-LEFT• AR-ARC-RIGHT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

He does it here does it hereHe S does it hereAL

He

it hereS

He

does

Transition-basedDependency Parsing

• An Example • S-SHIFT• R-REDUCE• AL-ARC-LEFT• AR-ARC-RIGHT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

He does it here does it hereHe S does it hereAL

He

it hereS

He

does

He

does it

AR

here

Transition-basedDependency Parsing

• An Example • S-SHIFT• R-REDUCE• AL-ARC-LEFT• AR-ARC-RIGHT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

He does it here does it hereHe S does it hereAL

He

it hereS

He

does

He

does it

AR

hereR

He

does here

it

Transition-basedDependency Parsing

• An Example • S-SHIFT• R-REDUCE• AL-ARC-LEFT• AR-ARC-RIGHT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

He does it here does it hereHe S does it hereAL

He

it hereS

He

does

He

does it

AR

hereR

He

does here

it

AR

He

does here

it

Transition-basedDependency Parsing

• An Example • S-SHIFT• R-REDUCE• AL-ARC-LEFT• AR-ARC-RIGHT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

He does it here does it hereHe S does it hereAL

He

it hereS

He

does

He

does it

AR

hereR

He

does here

it

AR

He

does here

it

R

He

does

it here

Transition-basedDependency Parsing

Transition-basedConstituentParsing

• Actions• SHIFT

stack queue

NR布朗 VV访问 NR上海

布朗(Brown) 访问(visits) 上海(Shanghai)

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

• Actions• SHIFT

stack queue

NR布朗 VV访问 NR上海

布朗(Brown) 访问(visits) 上海(Shanghai)

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

Transition-basedConstituentParsing

• Actions• REDUCE-UNARY-X

stack queue

NR布朗 VV访问 NR上海

布朗(Brown) 访问(visits) 上海(Shanghai)

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

Transition-basedConstituentParsing

• Actions• REDUCE-UNARY-X

stack queue

VV访问 NR上海

NR布朗

布朗(Brown) 访问(visits) 上海(Shanghai)

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

Transition-basedConstituentParsing

• Actions• REDUCE-UNARY-X

stack queue

X VV访问 NR上海

NR布朗

布朗(Brown) 访问(visits) 上海(Shanghai)

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

Transition-basedConstituentParsing

• Actions• REDUCE-BINARY-{L/R}-X

stack queue

NP VV访问

NR布朗

NP

NR上海

布朗(Brown) 访问(visits) 上海(Shanghai)

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

Transition-basedConstituentParsing

• Actions• REDUCE-BINARY-{L/R}-X

stack queue

NP

VV访问NR布朗 NP

NR上海

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

Transition-basedConstituentParsing

• Actions• REDUCE-BINARY-{L/R}-X

stack queue

NP

NR布朗 NP

NR上海

VP

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

VV访问

Transition-basedConstituentParsing

• Actions• TERMINATE

stack queue

S

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

Transition-basedConstituentParsing

• Actions• TERMINATE

stack queue

S

ans

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

Transition-basedConstituentParsing

• Example• SHIFT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

stack queue

DT ADJ NN

The little boy

VV

likes

NNSADJ

tomatoesred

.

.

Transition-basedConstituentParsing

• Example• SHIFT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

stack queue

DT ADJ NN

The little boy

VV

likes

NNSADJ

tomatoesred

.

.

Transition-basedConstituentParsing

• Example• SHIFT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

stack queue

DT ADJ NN

The little boy

VV

likes

NNSADJ

tomatoesred

.

.

Transition-basedConstituentParsing

• Example• REDUCE-R-NP

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

stack queue

DT ADJ NN

The little boy

VV

likes

NNSADJ

tomatoesred

.

.

Transition-basedConstituentParsing

• Example• REDUCE-R-NP

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

stack queue

DT

ADJ NN

The

little boy

VV

likes

NNSADJ

tomatoesred

.

.

NP-r

Transition-basedConstituentParsing

• Example• SHIFT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

stack queue

DT

ADJ NNThe

little boy

VV

likes

NNSADJ

tomatoesred

.

.

NP-r

NP-r

Transition-basedConstituentParsing

• Example• SHIFT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

stack queue

DT

ADJ NNThe

little boy

VV

likes

NNSADJ

tomatoesred

.

.

NP-r

NP-r

Transition-basedConstituentParsing

• Example• SHIFT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

stack queue

DT

ADJ NNThe

little boy

VV

likes

NNSADJ

tomatoesred

.

.

NP-r

NP-r

Transition-basedConstituentParsing

• Example• REDUCE-R-NP

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

stack queue

DT

ADJ NNThe

little boy

VV

likes

NNSADJ

tomatoesred

.

.

NP-r

NP-r

Transition-basedConstituentParsing

• Example• REDUCE-L-NP

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

stack queue

DT

ADJ NNThe

little boy

VV

likes

NNSADJ

tomatoesred

.

.

NP-r

NP-r NP-r

Transition-basedConstituentParsing

• Example• SHIFT

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

stack queue

DT

ADJ NNThe

little boy

VV

likesNNSADJ

tomatoesred

.

.

NP-r

NP-r

NP-r

VP-l

Transition-basedConstituentParsing

• Example• REDUCE-L-S

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

stack queue

DT

ADJ NNThe

little boy

VV

likesNNSADJ

tomatoesred

.

.

NP-r

NP-r

NP-r

VP-l

Transition-basedConstituentParsing

• Example• REDUCE-R-S

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

stack queue

DT

ADJ NNThe

little boy

VV

likesNNSADJ

tomatoesred

.

.

NP-r

NP-r

NP-r

VP-l

S-l

Transition-basedConstituentParsing

• Example• TERMINATE

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

stack queue

DT

ADJ NNThe

little boy

VV

likesNNSADJ

tomatoesred

.

.

NP-r

NP-r

NP-r

VP-l

S-l

S-r

Transition-basedConstituentParsing

Features• Non-local Features

• Extracted from top nodes on the stack S0, S1, S2, S3, the left and right or single child of S0 and S1, and the first words on the queue N0, N1, N2, N3.

stack queue

… S1 S0

S0l S0rS1u

N0 …

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

• Rich features• Fast speed• Search error

Transition-basedMethods

Beam-search Decoding

2017-11-27 IJCNLP 2017 Tutorial 86

Search Space• Find the best sequence of actions • Exponential

S0

S1

S’1

S’’1

a’0

S2

S’2

S’’2

•••

•••

⁞ ⁞

S’n

S’’n

•••

•••

Sn

•••

⁞⁞⁞

⁞⁞⁞

⁞⁞⁞

⁞

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

Beam-search decoding

start

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

Beam-search decoding

start

S11

S12

S13

S1k

……

agenda

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

start

S11

S12

S13

S1k

……

…

…

…

…

…

…

……

agenda

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

Beam-search decoding

start

S11

S12

S13

S1k

……

…

…

…

…

…

…

……

Si1

Si2

Si3

Sik

…

…

……

agenda

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

Beam-search decoding

start

S11

S12

S13

S1k

……

…

…

…

…

…

…

……

Si1

Si2

Si3

Sik

…

…

……

…

…

…

…

…

…

……

End1

End2

End3

EndK

…

…

……

agenda

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

Beam-search decoding

Beam-search decoding

• Dependency Parsing Example• Decoding

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

He does it here

Beam-search decoding

• Dependency Parsing Example• Decoding

He does it here does it hereHe S

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

Beam-search decoding

• Dependency Parsing Example• Decoding

He does it here does it hereHe S does it hereAL

He

it hereHe does

it hereHe does

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

Beam-search decoding

• Dependency Parsing Example• Decoding

He does it here does it hereHe S does it hereAL

He

it hereHe does

it hereHe does

it hereS

He

does

S hereHe does it

hereHe does it

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

Beam-search decoding

• Dependency Parsing Example• Decoding

He does it here

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

does it hereHe S does it hereAL

He

it hereHe does

it hereHe does

it hereS

He

does

S hereHe does it

hereHe does it

He

does it here

He does it here

hereHe does

it

Beam-search decoding

• Dependency Parsing Example• Decoding

He does it here

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

does it hereHe S does it hereAL

He

it hereHe does

it hereHe does

it hereS

He

does

S hereHe does it

hereHe does it

He

does it here

He does it here

hereHe does

it

He

He

does it here

He

does here

it

He

does it here

He

Beam-search decoding

• Dependency Parsing Example• Decoding

He does it here

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

does it hereHe S does it hereAL

He

it hereHe does

it hereHe does

it hereS

He

does

S hereHe does it

hereHe does it

He

does it here

He does it here

hereHe does

it

He

He

does it here

He

does here

it

He

does it here

He He

does here

it

He

does it hereHe

does here

it

Beam-search decoding

• Dependency Parsing Example• Decoding

He does it here

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

does it hereHe S does it hereAL

He

it hereHe does

it hereHe does

it hereS

He

does

S hereHe does it

hereHe does it

He

does it here

He does it here

hereHe does

it

He

He

does it here

He

does here

it

He

does it here

He He

does here

it

He

does it hereHe

does here

it

He

does

it

He here

does it

here

Beam-searchdecoding

Zhang, Y., & Clark, S. (2011). Syntactic Processing Using the Generalized Perceptron and Beam Search. CL.

Global Training

2017-11-27 IJCNLP 2017 Tutorial 102

Structured Learning

• Model whole sequences of actions• Correspond to structures

Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. In Computational Linguistics, 37(1), March.

Learning guided search

• Search not optional (vs graph-based structured prediction)• Learn to fix search errors

start

S11

S12

S13

S1k…

…

S21

S22

S23

…

…

…

……

Si1

Si2

Si3

Sik

Sjk+1

…

……

agenda

Positiveexample

start S11 S22 Sjk+1……

Negativeexample

start S11 S21 Sj1……