July 11, 2010ACL 2010 Tutorial, Uppsala, Sweden1 Tree-based and Forest-based Translation Yang LiuLiang Huang Institute of Computing Technology Chinese

July 11, 2010ACL 2010 Tutorial, Uppsala,

Sweden 1

Tree-based and Forest-based Translation

Yang Liu Liang HuangInstitute of Computing

TechnologyChinese Academy of Sciences

Information Sciences InstituteUniversity of Southern

California


Sweden 2

Outline

Part 1: Tree-based Translation Overview and Motivation Tree-to-String Model and Decoding Tree-to-String Rule Extraction Language Model-Integrated Decoding: Cube Pruning

Part 2: Forest-based Translation Packed Forest Forest-based Decoding Forest-based Rule Extraction

Part 3: Extensions Tree-to-Tree Translation Tree Sequence-based Translation Joint Parsing and Translation

Part 4: Conclusion


Sweden 3

Natural Languages are Different

I love you

By Google Translate

我爱你

Je t'aime

Ich liebe dich

Eu te amo

Te quiero

Я люблю тебя

わたしは、あなたを愛しています

당신을 사랑합니다

دارم دوست را شما من

Jag älskar dig

Ti amo Miluji tě

Ik hou van je

Tôi yêu bạn

אני אוהב אותך

ผมรั�กคุ�ณ


Sweden 4

Translation is Hard!

connocting poopie

HELP ONESELF TERMINATING MACHINE


Sweden 5

Machine Translation

布什与沙龙举行了会谈bushi yu shalong juxing le huitan

Bush held a talk with Sharon


Sweden 6

Word-based MT

bushi yu shalong juxing le huitan


(Brown et al., 1993)


Sweden 7

Phrase-based MT



(Koehn et al., 2003; Och and Ney, 2004)


Sweden 8

Hierarchical Phrase-based MT



(Chiang, 2005; Chiang, 2007)

X1

X1

X2

X2

X3

X3X3->(X1 yu shalong X2, X1 X2 with

Sharon)


Sweden 9

Syntax-based MT



NNP VBD DT NN IN NNP

NP NP

PP

VP

NP

S(Yamda and Knight, 2001; Galley et al., 2006; Shen et

al., 2008)


Sweden 10

Motivation

Human Translation Understand the source sentence Generate the target sentence

Compiling Parse input program into a syntax tree Generate code in machine language


Sweden 11

Syntax-Directed Translation for Compiling

Input: y:=3*x+z Parsing:

:=

id +

* id

const id

(y)

(3) (x)

(z)

(Irons, 1961; Lewis and Stearns, 1968; Aho and Ullman., 1972)


Sweden 12

Motivation

Human Translation Understand the source sentence Generate the target sentence

Compiling Parse input program into a syntax tree Generate code in machine language

Machine Translation Parse the source sentence into a tree Recursively transfer the tree into the target

language


Sweden 13

Syntax-Directed Translation for MT Input: bushi yu shalong juxing le huitan Parsing:


NPB P NPB VS AS NPB

PP VPB

VP

IP

(Huang et al., 2006)


Sweden 14

Outline




Part 4: Conclusion


Sweden 15

Tree-to-String Translation Recursive rewrite by pattern-matching


NPB P NPB VS AS NPB

PP VPB

VP

IP

IP

X1:NPB X2:VP

X1 X2

(Liu et al., 2006; Huang et al., 2006)


Sweden 16



NPB P NPB VS AS NPB

PP VPB

VP


bushi

NPB

Bush


Sweden 17


Bush yu shalong juxing le huitan

P NPB VS AS NPB

PP VPB

VP


VP

X1:PP X2:VPB

X2 X1


Sweden 18


Bush yu shalong

P NPB

PP

juxing le huitan

VS AS NPB

VPB


VPB

VS AS X1:NPB

lejuxing

held a X1


Sweden 19


Bush yu shalong

P NPB

PP

held a huitan

NPB


huitan

NPB

talk


Sweden 20


Bush yu shalong

P NPB

PP

held a talk


yu

P X1:NPB

PP

with X1


Sweden 21


Bush with shalong

NPB

held a talk


shalong

NPB

Sharon


Sweden 22


Bush with Sharonheld a talk


Syntax-directed translation (e.g., Irons, 1961)

Tree transducer (e.g., Knight and Graehl, 2005)

Synchronous grammar (e.g., Eisner, 2003)

…

Tree-to-string translation


Sweden 23

Expressive Power

PP

P NPB

yu shalong

with Sharon

phrasal translation non-constituent phrase non-contiguous phrase

VPB

VS AS X1:NPB

lejuxing

held a X1

PP

PLCP

dangX1:IP LC

hou

when X1

word omission

QP

X1:CD CLP

ben

X1

multilevel re-ordering

IP

X1:NP

VP

X2:IP X3:VPB

X1 X3 X2

lexicalized re-ordering

NP

DNP X2:NP

X1:NP DEG

X2 of X1

de

(Knight and Graehl, 2005)


Sweden 24

Outline




Part 4: Conclusion


Sweden 25

NPBASVSNPBP

Tree-to-String Rule Extraction Compute target spans


bushi yu shalong

juxing

le huitan

“Bush”

“with”

“Sharon”

“held”

“held”

“talk”

“with Sharon”

“held a talk”

“held … Sharon”

“Bush … Sharon”

NPB

VPB

VP

IP

PP

(Galley et al., 2004)


Sweden 26

NPBASVSNPBP

Tree-to-String Rule Extraction Find admissible nodes


bushi yu shalong

juxing

le huitan

“Bush”

“with”

“Sharon”

“held”

“held”

“talk”

“with Sharon”

“held a talk”



NPB

VPB

VP

IP

PP



Sweden 27

NPBASVSNPBP

Tree-to-String Rule Extraction Extract minimal rules


bushi yu shalong

juxing

le huitan

“Bush”

“with”

“Sharon”

“held”

“held”

“talk”

“with Sharon”

“held a talk”



NPB

VPB

VP

IP

PP


bushi

NPB

Bush


Sweden 28

NPBASVSNPBP



bushi yu shalong

juxing

le huitan

“Bush”

“with”

“Sharon”

“held”

“held”

“talk”

“with Sharon”

“held a talk”



NPB

VPB

VP

IP

PP


VP

X1:PP X2:VPB

X2 X1


Sweden 29

NPBASVSNPBP



bushi yu shalong

juxing

le huitan

“Bush”

“with”

“Sharon”

“held”

“held”

“talk”

“with Sharon”

“held a talk”



NPB

VPB

VP

IP

PP


VPB

VS AS X1:NPB

lejuxing

held a X1


Sweden 30

Tree-to-String Rule Extraction Get composed rules

VPB

VS AS X1:NPB

lejuxing

held a X1

huitan

NPB

talk

VPB

VS AS NPB

lejuxing

held a

huitan

talk

+ =


tree substitution


Sweden 31

Outline

Part 1: Tree-based Translation Overview and Motivation Tree-to-String Model and Decoding Tree-to-String Rule Extraction Language Model-Integrated Decoding: Cube

Pruning Part 2: Forest-based Translation

Packed Forest Forest-based Decoding Forest-based Rule Extraction


Part 4: Conclusion


Sweden 32

Bottom-up Decoding


NPB P NPB VS AS NPB

PP VPB

VP

IP

bushi

NPB

Bush

Bush



Sweden 33

Bottom-up Decoding


NPB P NPB VS AS NPB

PP VPB

VP

IP

Bush

shalong

NPB

Sharon

Sharon



Sweden 34

Bottom-up Decoding


NPB P NPB VS AS NPB

PP VPB

VP

IP

BushSharon

huitan

NPB

talk

talk



Sweden 35

Bottom-up Decoding


NPB P NPB VS AS NPB

PP VPB

VP

IP

BushSharon talk

yu

P X1:NPB

PP

with X1

with Sharon



Sweden 36

Bottom-up Decoding


NPB P NPB VS AS NPB

PP VPB

VP

IP

BushSharon talk

with Sharon

VPB

VS AS X1:NPB

lejuxing

held a X1

held a talk



Sweden 37

Bottom-up Decoding


NPB P NPB VS AS NPB

PP VPB

VP

IP

BushSharon talk

with Sharonheld a talk

VP

X1:PP X2:VPB

X2 X1

held a talk with Sharon



Sweden 38

Bottom-up Decoding


NPB P NPB VS AS NPB

PP VPB

VP

IP

BushSharon talk

with Sharonheld a talk


IP

X1:NPB X2:VP

X1 X2




Sweden 39

Beam Search


NPB P NPB VS AS NPB

PP VPB

VP

IP



Sweden 40

Exhaustive Search

VP

X1:PP X2:VPB

X2 X1

held a talkheld talkshold a talkhold talks

VPB3,6PP1,3

VP1,6

held a talk with Sharonheld a talk and Sharonheld talks with Sharonheld talks and Sharon…

with Sharonand SharonSharon withSharon and


Sweden 41

Update Bigram LM Probability

with Sharon held a talk

p1=p(“with’’) * p(“Sharon”|”with”)

p2=p(“held’’) * p(“a”|”held”)*p(“talk”|”a”)



p1*p2*p(“held”|”Sharon”)/p(“held”)

p1*p2*p(“with”|”talk”)/p(“with”)

Only boundary words are used to update LM probability!


Sweden 42

Exhaustive Search with a Bigram Language Model

VP

X1:PP X2:VPB

X2 X1

held * talkheld * talkshold * talkhold * talks

VPB3,6PP1,3

VP1,6

held * Sharonhold * Sharonheld * withhold * with…

with * Sharonand * SharonSharon * withSharon * and


Sweden 43

Monotonicity

VPB3,6PP1,3

VP1,6

1.0 3.0 4.0

1.0 2.0 4.0 5.0

1.1 2.1 4.1 5.1

2.0 6.0

held * talk

held * talkshold * talkhold * talks

with

* Sha

ron

and

* Sha

ron

Sharo

n *

with

Sharo

n *

and

6.5

7.5

7.6

8.5

3.5 4.5 6.5 7.5 10.0

3.0 5.0VPB3,6

PP1,3

(Huang and Chiang, 2005, 2007; Chiang, 2007)

monotonic


Sweden 44

Non-Monotonicity

VPB3,6PP1,3

VP1,6

1.0 3.0 4.0

1.0 2.0 + 0.54.0 + 2.05.0 + 4.0

1.1 2.1 + 0.34.1 + 1.55.1 + 3.5

2.0 6.0 + 4.0

held * talk


with

* Sha

ron

and

* Sha

ron

Sharo

n *

with

Sharo

n *

and

6.5

7.5 + 4.0

7.6 + 3.0

8.5 + 4.0

3.5 4.5 + 0.36.5 + 1.57.5 + 3.510 + 3.5

3.0 + 0.55.0 + 2.0VPB3,6

PP1,3


log(p(with|talk))-log(p(with))

LM introduces non-monotonicity


Sweden 45

Cube Pruning

1.0 3.0 4.0

1.0 2.5 6.0 9.0

1.1 2.4 5.6 8.6

2.0 10.0

held * talk


with

* Sha

ron

and

* Sha

ron

Sharo

n *

with

Sharo

n *

and

6.5

11.5

10.6

12.5

3.5 4.8 8.0 11.0 13.5

3.5 7.0VPB3,6

PP1,3

queue

4-best



Sweden 46

Cube Pruning

1.0 3.0 4.0

1.0 2.5 6.0 9.0

1.1 2.4 5.6 8.6

2.0 10.0

held * talk


with

* Sha

ron

and

* Sha

ron

Sharo

n *

with

Sharo

n *

and

6.5

11.5

10.6

12.5

3.5 4.8 8.0 11.0 13.5

3.5 7.0VPB3,6

PP1,3

queue

4-best

2.5



Sweden 47

Cube Pruning

1.0 3.0 4.0

1.0 2.5 6.0 9.0

1.1 2.4 5.6 8.6

2.0 10.0

held * talk


with

* Sha

ron

and

* Sha

ron

Sharo

n *

with

Sharo

n *

and

6.5

11.5

10.6

12.5

3.5 4.8 8.0 11.0 13.5

3.5 7.0VPB3,6

PP1,3

queue

4-best 2.5

2.4 6.0



Sweden 48

Cube Pruning

1.0 3.0 4.0

1.0 2.5 6.0 9.0

1.1 2.4 5.6 8.6

2.0 10.0

held * talk


with

* Sha

ron

and

* Sha

ron

Sharo

n *

with

Sharo

n *

and

6.5

11.5

10.6

12.5

3.5 4.8 8.0 11.0 13.5

3.5 7.0VPB3,6

PP1,3

queue

4-best 2.4

3.5 5.6

2.5

6.0



Sweden 49

Cube Pruning

1.0 3.0 4.0

1.0 2.5 6.0 9.0

1.1 2.4 5.6 8.6

2.0 10.0

held * talk


with

* Sha

ron

and

* Sha

ron

Sharo

n *

with

Sharo

n *

and

6.5

11.5

10.6

12.5

3.5 4.8 8.0 11.0 13.5

3.5 7.0VPB3,6

PP1,3

queue

4-best 2.4

4.8

2.5

5.6

3.5


6.0 7.0


Sweden 50

Cube Pruning

1.0 3.0 4.0

1.0 2.5 6.0 9.0

1.1 2.4 5.6 8.6

2.0 10.0

held * talk


with

* Sha

ron

and

* Sha

ron

Sharo

n *

with

Sharo

n *

and

6.5

11.5

10.6

12.5

3.5 4.8 8.0 11.0 13.5

3.5 7.0VPB3,6

PP1,3

queue

4-best 2.4 4.82.5

5.6

3.5


6.07.0


Sweden 51

Cube Pruning within Rule GroupVPB

VS AS X1:NPB

lejuxing

held a X1

VPB

VS AS X1:NPB

lejuxing

held X1

VPB

VS AS X1:NPB

lejuxing

hold a X1

1.0

1.4

2.0

1.0 2.0 2.5

2.1

3.2

3.1

5.0

4.0

6.0

3.7

5.0

4.7

held a X1

held X1

hold a X1

talk

talk

sm

eetin

g

Group rules that have the same LHS

NPB5,6



Sweden 52

Cube Pruning within Node

VP1,6

PP1,3 VPB3,6 NPB2,3 NPB5,6 NPB2,3 VPB3,6

process all rules simultaneously!significant savings of computation



Sweden 53

Outline




Part 4: Conclusion


Sweden 54

Syntactic Ambiguity


NPB P NPB VS AS NPB

PP VPB

VP

IP


NPB CC NPB VS AS NPB

NP VPB

IP

with and

``Bush held a talk with Sharon’’

``Bush and Sharon held a talk’’

It is important to choose a correct tree for producing a good translation!


Sweden 55

Parsing Mistake Propagation

parsing mistakes potentially introduce translation mistakes!

(Quirk and Corston-Oliver, 2006)

parse translatesourcetarget

string tree

string


Sweden 56

1-best Trees => n-best Trees?


NPB P NPB VS AS NPB

PP VPB

VP

IP



NP VPB

IP

Very few variations among the n-best trees!


Sweden 57

Packed Forest


NPB0,1CC1,2 NPB2,3 VS3,4 AS4,5 NPB5,6

PP1,3 VPB3,6

VP1,6

IP0,6

P1,2

NP0,3

(Billot and Lang, 1989; Klein and Manning, 2001; Huang and Chiang, 2005)


Sweden 58

Outline




Part 4: Conclusion


Sweden 59

Pattern Matching on Forest



PP1,3 VPB3,6

VP1,6

IP0,6

P1,2

NP0,3

IP

NP X3:VPB

X1:NPB CC X2:NPB

yu

X1 X3 with X2

(Mi et al., 2008)


Sweden 60

Translation Forest

(Mi et al., 2008)bushi yu shalong juxing le huitan


PP1,3 VPB3,6

VP1,6

IP0,6

P1,2

NP0,3

bushi

NPB

Bush

NPB0,1


Sweden 61

Translation Forest



PP1,3 VPB3,6

VP1,6

IP0,6

P1,2

NP0,3

bushi

NPB

Bush

NPB0,1

shalong

NPB

Sharon

NPB2,3


Sweden 62

Translation Forest



PP1,3 VPB3,6

VP1,6

IP0,6

P1,2

NP0,3

bushi

NPB

Bush

NPB0,1

shalong

NPB

Sharon

NPB2,3

huitan

NPB

talk

NPB5,6


Sweden 63

Translation Forest



PP1,3 VPB3,6

VP1,6

IP0,6

P1,2

NP0,3

bushi

NPB

Bush

NPB0,1

shalong

NPB

Sharon

NPB2,3

huitan

NPB

talk

NPB5,6

VPB

VS ASX1:NPB

lejuxing

held a X1

VPB3,6


Sweden 64

Translation Forest



PP1,3 VPB3,6

VP1,6

IP0,6

P1,2

NP0,3

bushi

NPB

Bush

NPB0,1

shalong

NPB

Sharon

NPB2,3

huitan

NPB

talk

NPB5,6

VPB

VS ASX1:NPB

lejuxing

held a X1

VPB3,6

IP

NP X3:VPB

X1:NPB CC X2:NPB

yu

X1 X3 with X2

IP0,6

``Bush’’

``Sharon’’

``talk’’

``held a talk’’

``Bush held a talk with Sharon’’


Sweden 65

N-best Trees Vs. Forest

(Mi et al., 2008)


Sweden 66

Forest as Virtual ∞-best list How often is the ith-best tree picked by the

decoder?

(Mi et al., 2008)


Sweden 67

Outline




Part 4: Conclusion


Sweden 68

Forest-based Rule Extraction



PP1,3 VPB3,6

VP1,6

IP0,6

P1,2

NP0,3

Compute target spans


“Bush”

“with”

“with”

“Sharon”

“held”

“held”

“talk”

“Bush □ with Sharon”

“with Sharon” “held … talk”



(Mi and Huang, 2008)


Sweden 69




PP1,3 VPB3,6

VP1,6

IP0,6

P1,2

NP0,3

Compute admissible nodes


“Bush”

“with”

“with”

“Sharon”

“held”

“held”

“talk”







Sweden 70




PP1,3 VPB3,6

VP1,6

IP0,6

P1,2

NP0,3

Extract minimal rules


“Bush”

“with”

“with”

“Sharon”

“held”

“held”

“talk”




“Bush … Sharon” bushi

NPB

Bush



Sweden 71




PP1,3 VPB3,6

VP1,6

IP0,6

P1,2

NP0,3



“Bush”

“with”

“with”

“Sharon”

“held”

“held”

“talk”





VP

X1:PP X2:VPB

X2 X1



Sweden 72




PP1,3 VPB3,6

VP1,6

IP0,6

P1,2

NP0,3



“Bush”

“with”

“with”

“Sharon”

“held”

“held”

“talk”





VPB

VS AS X1:NPB

lejuxing

held a X1



Sweden 73




PP1,3 VPB3,6

VP1,6

IP0,6

P1,2

NP0,3



“Bush”

“with”

“with”

“Sharon”

“held”

“held”

“talk”





IP

X1:NPB X2:VP

X1 X2



Sweden 74




PP1,3 VPB3,6

VP1,6

IP0,6

P1,2

NP0,3



“Bush”

“with”

“with”

“Sharon”

“held”

“held”

“talk”





IP

NP X4:VPB

X1:NPBX2:CCX3:NPB

X1 X3 X4 X2



Sweden 75

Rule Probabilities and Rule Count

': '

': '

': '

|'

|'

|'

r lhs r lhs r

r rhs r rhs r

r root lhs r root lhs r

c rP r lhs r

c r

c rP r rhs r

c r

c rP r root lhs r

c r

How often does a rule occur in training examples?


Sweden 76

Fractional Count



PP1,3 VPB3,6

VP1,6

IP0,6

P1,2

NP0,3


“Bush”

“with”

“with”

“Sharon”

“held”

“held”

“talk”





VP

X1:PP X2:VPB

X2 X1


Q: What ‘s the count of this rule on this training example?


Sweden 77

Fractional Count



PP1,3 VPB3,6

VP1,6

IP0,6

P1,2

NP0,3


“Bush”

“with”

“with”

“Sharon”

“held”

“held”

“talk”






1,6 1,3 3,6VP PP VPB

TOP

e p e

lhs rc r

1,6VP

1,3PP 3,6VPB

p e


Sweden 78

Results

1-best tree forest

1-best tree 0.2560 0.2674

forest 0.2679 0.2816

decoding

rule

extra

ction



Sweden 79

Outline




Part 4: Conclusion


Sweden 80

Tree-to-Tree Translation Recursive rewrite by pattern-matching


NPB P NPB VS AS NPB

PP VPB

VP

IP

NP

VP

S

IP

X1:NPB X2:VP

S

X1:NP X2:VP

(Eisner 2003, Zhang, 2007)


Sweden 81



NPB P NPB VS AS NPB

PP VPB

VP

NP

VP

S

NP

Bush

NNP

bushi

NPB

Bush

NNP

NP



Sweden 82


yu shalong juxing le huitan

P NPB VS AS NPB

PP VPB

VP

NP

VP

S

NP

Bush

NNP

juxing le

VS ASX2:NPB

X1:PP VPB

VP

held a

VBD DT X2:NN

NP

X1:PP

VP

held a

VBD DT NN

NP

PP



Sweden 83


yu shalong huitan

P NPB NPB

PP

NP

VP

S

NP

Bush

NNP

held a

VBD DT NN

NP

PP

yu shalong

P NPB

PP

with Sharon

IN NNP

NP

PP

with Sharon

IN NNP

NP



Sweden 84


huitan

NPB

NP

VP

S

NP

Bush

NNP

held a

VBD DT NN

NP

PP

with Sharon

IN NNP

NP

huitan

NPB

talk

NN

talk



Sweden 85


NP

VP

S

NP

Bush

NNP

held a

VBD DT NN

NP

PP

with Sharon

IN NNP

NP

talk



Sweden 86

Tree-to-Tree Rule Extraction Find admissible node pairs


NPB P NPB VS AS NPB

PP VPB

VP

IP



NP NP NP

PP

VP

S (Zhang, 2007, Liu et al., 2009a)


Sweden 87

Tree-to-Tree Rule Extraction Extract minimal rules


NPB P NPB VS AS NPB

PP VPB

VP

IP



NP NP NP

PP

VP

S

bushi

NPB

Bush

NNP

NP

(Zhang, 2007, Liu et al., 2009a)


Sweden 88



NPB P NPB VS AS NPB

PP VPB

VP

IP



NP NP NP

PP

VP

S

juxing le

VS ASX2:NPB

X1:PP VPB

VP

held a

VBD DT X2:NN

NP

X1:PP

VP



Sweden 89



NPB P NPB VS AS NPB

PP VPB

VP

IP



NP NP NP

PP

VP

S

IP

X1:NPB X2:VP

S

X1:NP X2:VP



Sweden 90

Tree-to-Tree Rule Extraction Get composed rules

juxing le

VS ASX2:NPB

X1:PP VPB

VP

held a

VBD DT X2:NN

NP

X1:PP

VP

huitan

NPB

talk

NN

juxing le

VS AS NPB

X1:PP VPB

VP

held a

VBD DT NN

NP

X1:PP

VP

huitan

talk

+ =



Sweden 91

Challenges

Tree-to-tree translation is over-constrained Poorest rule coverage Suffers from parsing mistake propagation

on both sides Recent advances

Use tree sequence (Zhang et al., 2008) Use packed forest (Liu et al., 2009a) Fuzzy extraction and decoding (Chiang,

2010)


Sweden 92

Outline




Part 4: Conclusion


Sweden 93

Non-Constituent Phrase Pairs

bushi yu shalongjuxing le huitan


NP VPB

IP


(Marcu et al., 2006)


Sweden 94





NP NP NP

PP

VP

S(Marcu et al., 2006)


Sweden 95



NPB P NPB VS AS NPB

PP VPB

VP

IP



NP NP NP

PP

VP

S



Sweden 96

Rule Coverage


NPB P NPB VS AS NPB

PP VPB

VP

IP



NP NP NP

PP

VP

S

(bushi, Bush)

(yu, with)

(shalong, Sharon)(huitan, talk)

(yu shalong, with Sharon)(juxing le, held)

(juxing … huitan, held … talk)(yu … huitan, held … Sharon)(bushi … huitan, Bush … Sharon)

s2s t2s s2t t2t√√√

√

√

√

√

√

√

√√√

√

√

√

√

√

√√√

√

√

√

√

√

√√√

√

√

√

√

×

×

×

×

phrase pair

100%89% 89%78%


Sweden 97

Rule Coverage

model human automatic

string-to-string 100% 100%

tree-to-string 78% 75%

string-to-tree 76% 72%

tree-to-tree 68% 60%Results from (Chiang, 2010)


Sweden 98

Solutions Extend to larger rules



NP VPB

IP


VPB

VS AS X1:NPB

lejuxing

held a X1



Sweden 99

Solutions Add pseudo nodes



NP VPB

IP


VPB

X1:*VPB_*NPB X2:NPB

X1 a X2

*VPB_*NPB

VS AS

lejuxing

held



Sweden 100

Solutions Use tree sequences



NP VPB

IP


VPB

X1:VS X2:NPB

X1 a X2

VS AS

lejuxing

held

X1:AS

(Liu et al., 2007; Zhang et al., 2008)


Sweden 101

Tree-Sequence + Forest

system input ruleBLEU

Moses string string-to-string 25.7

tree-to-string

treetree-to-string 26.1

tree-sequence-to-string 27.0

foresttree-to-string 27.7

tree-sequence-to-string 28.8

Results from (Zhang et al., 2009)


Sweden 102

Other Solutions

Re-structure syntax-trees (Wang et al., 2007)

Offer more trees (Mi and Huang, 2008) Re-align syntax trees and strings (May

and Knight, 2007) Well-formed dependency structures

(Shen et al., 2008) Gibbs sampling (Cohn and Blunsom,

2009) Joint decoding (Liu et al., 2009b)


Sweden 103

Outline




Part 4: Conclusion


Sweden 104

Separate Parsing and Translation

parse translatesourcetarget

string tree/forest

string

Separate grammar for parsing and translation

decoding is fast!


Sweden 105

Joint Parsing and Translation

parse + translatesourcetarget

string tree/forest

string

• Its search space is larger than tree/forest• It is a translator as well as a parser• Parsing interacts with translation

(Liu and Liu, 2010)


Sweden 106

Tree-to-String Translation as Parsing


bushi

NPB

Bush

NPB

Bush(Liu and Liu, 2010)


Sweden 107



yu

P

withNPB P

withBush(Liu and Liu, 2010)


Sweden 108



shalong

NPB

Sharon

NPB P

withBush

NPB

Sharon(Liu and Liu, 2010)


Sweden 109



huitan

NPB

talk

NPB P

withBush

NPB

Sharon

NPB

talk(Liu and Liu, 2010)


Sweden 110



NPB P

Bush

NPB

with Sharon

NPB

talk

X1:P

PP

X1 X2

X2:NPBPP

(Liu and Liu, 2010)


Sweden 111



NPB P

Bush

NPB

with Sharon

NPB

talk

X1:P

PP

X1 X2

X2:NPBPP

(Liu and Liu, 2010)


Sweden 112



NPB P

Bush

NPB

with Sharon

NPB

talk

PP

VPB

VS AS X1:NPB

lejuxing

held a X1

VS AS

VPB

(Liu and Liu, 2010)


Sweden 113



NPB P

Bush

NPB

with Sharon

NPB

PP

VPB

VS AS X1:NPB

lejuxing

held a X1

VS AS

VPB

held a talk(Liu and Liu, 2010)


Sweden 114



NPB P

Bush

NPB NPB

PP

VS AS

VPB


X1:PP

VP

X2 X1

X2:VPBVP

(Liu and Liu, 2010)


Sweden 115



NPB P

Bush

NPB NPB

PP

VS AS

VPB

X1:PP

VP

X2 X1

X2:VPBVP

held a talk with Sharon(Liu and Liu, 2010)


Sweden 116



NPB P NPB NPB

PP

VS AS

VPB

X1:NPB

IP

X1 X2

X2:VPVP


IP

(Liu and Liu, 2010)


Sweden 117



NPB P NPB NPB

PP

VS AS

VPB

X1:NPB

IP

X1 X2

X2:VPVP


IP

Bush(Liu and Liu, 2010)


Sweden 118

Translation Evaluationalgorithm input

parsing model rules BLEU time

matching

tree none 1.2M 29.8 0.56

forest PCFG 1.9M 31.6 9.49

parsing string

none

7.7M

32.0 51.41

PCFG 32.4 55.52

Lex 32.6 89.35

PCFG+Lex 32.7 91.72

(Liu and Liu, 2010)


Sweden 119

Parsing Evaluation

parsing model F1 time

none 62.7 23.9

PCFG 65.4 24.7

Lex 79.8 48.8

PCFG + Lex 80.6 50.4

(Liu and Liu, 2010)


Sweden 120

Results on Tree-to-Tree

task extraction rules features BLEU

Chinese

string-to-string 440M 1K 23.7

tree-to-tree 50M 5K 23.9

Arabic

string-to-string 790M 1K 48.9

tree-to-tree 38M 5K 47.5Results from (Chiang, 2010)


Sweden 121

Outline




Part 4: Conclusion


Sweden 122

Conclusion Statistical machine translation

Word-based Phrase-based Syntax-based

String-to-String String-to-Tree Tree-to-String Tree-to-Tree

flat

hierarchical


Sweden 123

Conclusion

Tree-based translation Pros: simplicity, faster decoding, expressive

grammar, no need for binarization Cons: commits to 1-best tree

Forest-based translation Compromise between tree-based and

string-based, combining the advantages of both Fast decoding, but does not commit to 1-best

trees Significant improvement of translation

performance over tree-based


Sweden 124

Thank you

谢谢大家

VV PN

VP


Sweden 125

Bibliography Alfred V. Aho and Jeffrey D. Ullman. 1972. The Theory of Parsing,

Translation, and Compiling, volume I: parsing. Prentice Hall, Englewood Cliffs, New Jersey.

Peter F. Brown, Stephan A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2): 263-311.

Sylvie Billot and Bernard Lang. 1989. The structure of shared forests in ambiguous parsing. In Proceedings of ACL 1989.

David Chiang, 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of ACL 2005.

David Chiang, 2007. Hierarchical phrase-based translation. Computational Linguistics, 33(2): 201-228.

David Chiang, 2010. Learning to translate with source and target syntax. In Proceedings of ACL 2010.

Trevor Cohn and Phil Blunsom. 2009. A bayesian model for syntax-directed tree to string grammar induction. In Proceedings of EMNLP 2009.

Michel Galley, Mark Hopkins, Kevin Knight, and Daniel Marcu. 2004. What’s in a translation rule? In Proceedings of HLT-NAACL 2004.

Michel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve DeNeefe, Wei Wang, and Ignacio Thayer. 2006. Scalable inference and training of context-rich syntactic translation models. In Proceedings of COLING-ACL 2006.


Sweden 126

Bibliography Liang Huang and David Chiang. 2005. Better k-best parsing. In

Proceedings of IWPT 2005. Liang Huang and David Chiang. 2007. Forest rescoring: Faster

decoding with integrated language models. In Proceedings of ACL 2007.

Liang Huang, Kevin Knight, and Aravind Joshi. 2006. Statistical syntax-directed translation with extended domain of locality. In Proceedings of AMTA 2006.

Liang Huang. 2008. Forest reranking: Discriminative parsing with non-local features. In Proceedings. of ACL-HLT 2008.

E. T. Irons. 1961. A syntax-directed compiler for ALGOL 60. Comm. ACM, 4(1): 51-55.

Kevin Knight and Jonathan Graehl. 2005. An overview of probabilistic tree transducers for natural language processing. In Proceedings of CICLing 2005.

Dan Klein and Christopher D. Manning. 2001. Parsing and hypergraphs. In Proceedings of IWPT 2001.

Philipp Koehn, Franz Och, and Daniel Marcu. 2003. Statistical phrase-based translation In Proceedings of HLT-NAACL 2003.

P. M. Lewis and R. E. Stearns. 1968. Syntax-directed transduction. Journal of the ACM, 15(3): 465-488.

Yang Liu, Qun Liu, and Shouxun Lin. 2006. Tree-to-string alignment template for statistical machine translation. In Proceedings of COLING-ACL 2006.


Sweden 127

Bibliography Yang Liu, Yun Huang, Qun Liu, and Shouxun Lin. 2007. Forest-to-

string statistical translation rules. In Proceedings of ACL 2007. Yang Liu, Yajuan Lu, and Qun Liu. 2009a. Improving tree-to-tree

translation with packed forests. In Proceedings of ACL-IJCNLP 2009. Yang Liu, Haitao Mi, Yang Feng, and Qun Liu. 2009b. Joint decoding

with multiple translation models. In Proceeding of ACL-IJCNLP 2009. Yang Liu and Qun Liu. 2010. Joint parsing and translation. Submitted

to COLING 2010. Daniel Marcu, Wei Wang, Abdessamad Echihabi, and Kevin Knight.

2006. SPMT: Statistical machine translation with syntactified target language phrases. In Proceedings of EMNLP 2006.

Jonathan May and Kevin Knight. 2007. Syntax re-alignment models for machine translation. In Proceedings of EMNLP 2007.

Haitao Mi and Liang Huang. 2008. Forest-based translation rule extraction. In Proceedings of EMNLP 2008.

Haitao Mi, Liang Huang, and Qun Liu. 2008. Forest-based translation. In Proceedings of ACL-HLT 2008.

Franz Och and Hermann Ney. 2004. The alignment template approach to statistical machine translation. Computational Linguistics, 30(4): 417-449.


Sweden 128

Bibliography Libin Shen, Jinxi Xu, and Ralph Weischedel. 2008. A new string-to-

dependency machine translation algorithm with a target dependency language model. In Proceedings of ACL-HLT 2008.

Ashish Venugopal, Andreas Zollmann, Noah Smith, and Stephan Vogel. 2008. Wider pipelines: n-best alignments and parses in mt training. In Proceedings of AMTA 2008.

Wei Wang, Kevin Knight, and Daniel Marcu. 2007. Binarizing syntax trees to improve syntax-based machine translation accuracy. In Proceedings of EMNLP 2007.

Kenji Yamada and Kevin Knight. 2001. A syntax-based statistical machine translation model. In Proceedings of ACL 2001.

Hui Zhang, Min Zhang, Haizhou Li, Aiti Aw, and Chew Lin Tan. 2009. Forest-based tree sequence to string translation model. In Proceedings of ACL-IJCNLP 2009.

Min Zhang, Hongfei Jiang, Aiti Aw, Jun Sun, Sheng Li, and Chew Lin Tan. 2007. A tree-to-tree alignment-based model for statistical machine translation. In Proceedings of MT Summit 2007.

Min Zhang, Hongfei Jiang, Aiti Aw, Haizhou Li, Chew Lin Tan, and Sheng Li. 2008. A tree sequence alignment-based tree-to-tree translation model. In Proceedings of ACL-HLT 2008.

Documents

July 11, 2010ACL 2010 Tutorial, Uppsala, Sweden1 Tree-based and Forest-based Translation Yang LiuLiang Huang Institute of Computing Technology Chinese