Cluster Computing for Statistical Machine

TranslationQin Gao, Kevin Gimpel, Alok Palikar, Andreas Zollmann

Stephan Vogel, Noah Smith

Outline

Introduction Statistical Machine Translation How can parallel processing help MT?

INCA: An Integrated Cluster Computing Architecture for Machine Translation Roadmap Progress Ongoing work

Statistical Machine Translation

澳洲 是 少数与 北朝鲜 建交 的 国家 之一Source Sentence

Hello John …The United State government is going to…Australia is one of afew countries that has relationship with North Korea…Australia is North Korean has relationship a few…Sarah Palin's neighbors saluted their hometown girl with Alaskan Amber The idea, especially from the Democrats that I know

…..…..

Possible Translations

Which one to choose?

Components of Statistical Machine Translation

Source Language Target Language

Interlingua

Analysis

GenerationTransfer

Components of Statistical Machine Translation

Source Language Target Language

Interlingua

Analysis

Generation

?

Semantic (?)

Syntactic Tree-to-Tree

Syntactic Tree-to-StringString-to-String

Analysis Transfer Generation

Components of Statistical Machine Translation

Source Language Target Language

Interlingua

Analysis

Generation

?

Semantic (?)

Syntactic Tree-to-Tree

Syntactic Tree-to-StringString-to-StringText Normalizing

Syntactic Parsing

Semantic Parsing

Tasks for Analysis:• Text normalization• Tokenization• Syntactic Parsing• Dependency Parsing• Semantic Parsing….

Components of Statistical Machine Translation

Source Language Target Language

Interlingua (?)

Analysis

Generation

?

Tasks for Transfer:• Phrase extraction• Rule extraction• Feature value estimation

….

Phrase extraction

Rule extraction

Components of Statistical Machine Translation

Target Language

Tasks for Generation:• Target language modeling• Minimal Error Rate

Training• Decoding

….

Source Language

Interlingua (?)

Analysis

Generation

?

Analysis Transfer Generation

Tasks for Analysis:• Text normalization• Tokenization• Syntactic Parsing• Dependency Parsing• Semantic Parsing

….

Tasks for Transfer:• Phrase extraction• Generation rule extraction• Feature value estimation….

Tasks for Generation:• Target language modeling• Minimal Error Rate

Training• Decoding….

Data-Driven

Bilingual Data Phrase Table/Rule Database

Language Model Monolingual Data

g

INCA: An Integrated Cluster Computing Architecture for Machine Translation

With the help of computer clusters, we want to make the statistical machine translation tasks:

Faster Easier

• Parallelize time-consuming

components• Automate the

procedure

• Make the experiment management easier• Make the substitution of

components more convenient

Moses is an excellent example how such tool can benefit the research community!

Target: Integrated Cluster ComputingArchitecture for MT

Simplify experiment process with integrated framework

Experiment Management : Reproduce the results

Progress• POS tagging• Parsing• Semantic Role

labeling

• Parallel GIZA++ (To be ported to Hadoop)

• Distributed Phrase extraction (Chaski)

• Rule extraction for SAMT

• Distributed Tuning/Decoding framework (Trambo)

Data Pre-processing

Word Alignment

Phrase/Rule Extraction

Tuning and

Decoding

Data Preprocessing

Examples: Tokenization Word segmentation (e.g. for Chinese) POS tagging Parsing Semantic role labeling

Naturally parallelizable, perfect for MapReduce

However, there are still interesting problems

Error Tolerance? Hadoop has the functionality to re-run failed

jobs. It works fine for hardware errors or

“unpredictable” software errors But it does not work for predictable errors

E.g: A certain sentence may crash the parser, restart the parser does not help Fail-safe mechanism: Falling back to simple models Currently implemented through individual scripts,

aiming at providing the frame work

Parallelized Word AlignmentWord

Alignment

Collect Counts

Has More Sentences?

Y

NormalizeCounts

Has More Iterations?

Y

Algorithm of wordalignment:

EM(Iterative Optimization)

N

N

Parallel GIZA++

Model Parameters

Proc 1 Proc 2 Proc n

Count 1 Count 2 Count n

Master Process

Initialization

Loading previous model

Distributed Alignment

Normalizing and Updating Model Parameters

Performance of PGIZA++

Running Time

BLEU Score(Tuning)

BLEU Score(Testing)

CPUs

GIZA++ 169h 32.34 29.43 2

PGIZA++ 39h 32.20 30.14 11

Per-Iteration Total

Model 1 47.0 min 235 min (3.9 h)

HMM 31.8 min 159 min (2.6 h)

Model 3/4 25.2 min 151 min (2.5 h)

Comparison of Speed and BLEU score using PGIZA++ and GIZA++

Normalization time in PGIZA++

Corpus Size: 10 million sentence pairs, Chinese-English

I/O bottleneck

Currently the tool run on NFS I/O of counts becomes the major bottleneck The plan is to port it to Hadoop The locality of HDFS should help

Ongoing Work:Distributed Asynchronous Online EM

Master node runs MapReduce jobs to perform E-step on mini-batches of data

When each mini-batch completes, master node updates parameters, starts a new mini-batch immediately

E-steps performed using slightly stale parameters → “asynchronous”

Phrase Extraction Tool Chaski

Chaski Postmen of Inca Empire They run really fast!

Phrase Model Training

Problems in phrase extraction: Disk space

700GB when extracting phrase up to length 10, on 10 million sentences

Sorting phrases External sorting must be used, which adds up to the

disk usage

Phrase Translation Feature and Lexicon Translation Feature

Source Target

Phrase Translation Feature and Lexicon Translation Feature

Target Source

Phrase PairsTarget Source

Phrase Model Training (Moses)

Bidirectional Word Alignment

Phrase PairsSource Target

Extract Phrases

Reorder InfoTarget Source

Reorder Info

Sort on Source

Sort on Target

Sort on Source

Reordering Table

Merge Phrase Table

Chaski

Mapper Sorting Reducer

Extract phrases

Sort on target

Score phrases

Dummy Mapper

Sort on source

Score phrases

Dummy Mapper

Sort on sourceLearn Reordering

• Hadoop and HDFS handle large intermediate file and sorting• Kept away from the merge operation which MapReduce is not good at

Performance of Chaski

Phrase Length 7 Phrase Length 10

Moses 43 128M-R 50 Nodes 0.7 1.9M-R 100 Nodes 0.57 1.2

0.1

1

10

100

1000

Tim

e U

sed

(Log

ged

Hou

rs)

Rule Extraction for Syntactic Augmented Machine Translation

System Extract probabilistic synchronous context-free

grammar (PSCFG) rules in in a Map step from all the target-parsed training sentence pairs

From syntax-annotated phrase pairs…S -> Il ne marche pas / It does not work

NP -> Il / ItVB -> marche / work

…create rule:S -> NP ne VB pas / NP does not VB

Merge and compute weights for them in the Reduce step

Distributed Tuning/Decoding (Trambo)

Translate one sentence at a time Split up decoding into sub-processes Collect the output for MERT

Test Set

Decoder Decoder Decoder…Split

MERT Merge

Trambo

Filter the phrase table and language models on a per-sentence basis -- beforehand Each decoder instance loads faster Memory usage is kept in check

Tuning time (with Moses decoder): 12.5 hrs on desktop machine →70 mins using 50 nodes

Filtering as (temporary?) Solution

For language model In Trambo, filtering a 11GB language model for

1600 sentences, ends up with 1TB temporary storage, with the limitation of 60 candidate target phrases for each source phrase

Distributed Language Model?

Conclusion

Working towards integrated cluster computing architecture for MT by providing cluster-enabled components Data preprocessing Word alignment Phrase/Rule extraction Tuning/Decoding

On-going Work

Data preprocessing

Word Alignment

Phrase/Rule Extraction

Tuning/Decoding

System Integration

Research Side Engineering Side

Error tolerant framework through fall-back strategies

Distributed Asynchronous

Online EM

Porting PGIZA++ to Hadoop

Merge Chaski and SAMT rule extractor

into single framework

Efficient access of language models

User study: Which kinds of automation can

benefit researchers most

Master script to integrate components

Experiment management

Improved phrase/rule extraction

Thanks!Questions & Suggestion?