Machine Translationufal.mff.cuni.cz/mtm13/files/01-mtm-intro-chris-dyer.pdf · • 1993 -...

Machine Translation

Chris DyerLanguage Technologies InstituteMachine Learning Department

MT Marathon 2013September 9, 2013

ltiMonday, September 9, 13

Overview

• A Brief History of MT

• Modeling: Language & Translation

• Using MT

• Evaluating MT

• Coffee

Monday, September 9, 13

MT Timeline• 1940s - WW2 code breaking

• 1947 - Weaver letter outlining translation as a problem in cryptography

• 1954 - Georgetown Experiments showed “promise” of Russian-English MT

• 1966 - ALPAC report shifts funding to basic research in computational linguistics

• 1968 - MT company SYSTRAN founded (still in existence)

• 1970s - advances in formal languages and automata theory; development of statistical speech recognition techniques at IBM and Princeton

• 1993 - Weaver’s model of translation prototyped by IBM; statistical revolution

• 1999 - Open source reimplementation of IBM models

• 2000s - Major modeling advances, rediscovery of syntax, large scale funding

• 2006 - Open source Moses decoder development begins

• 2006 - Google Translate launches

• 2010 - SDL acquires Language Weaver

• 2013 - Prague MT Marathon!

One naturally wonders if the problem of translation could conceivably be

treated as a problem in cryptography. When I look at an article in Russian, I

say: ‘This is really written in English, but it has been coded in some strange symbols. I will now

proceed to decode.’

Warren Weaver to Norbert Wiener, March, 1947

Claude Shannon. “A Mathematical Theory of Communication” 1948.

Message

EncoderM

Message

EncoderM

Message

Senttransmission

EncoderM

Message

Senttransmission

“Noisy” channel

p(x|y)

EncoderM

Message

Senttransmission

Receivedtransmission

“Noisy” channel

p(x|y)

EncoderM

Message

DecoderM 0

Recoveredmessage

Senttransmission

“Noisy” channel

p(x|y)

EncoderM

Message

“Noisy” channel Decoder

Y X M 0

Senttransmission

Recoveredmessage

p(y) p(x)p(x|y)

EncoderM

Message

Y X M 0

Senttransmission

Recoveredmessage

p(y) p(x)p(x|y)

EncoderM

Message

Y X M 0

Senttransmission

Recoveredmessage

p(y) p(x|y)

Shannon’s theory tells us:1) how much data you can send2) the limits of compression3) why your download is so slow4) how to translate

Y X M 0

Senttransmission

Recoveredmessage

p(y) p(x|y)

Y X M 0

Senttransmission

Recoveredmessage

p(y) p(x|y)

Y X M 0

Senttransmission

Recoveredmessage

p(y) p(x|y)

Y X M 0

Senttransmission

Recoveredmessage

p(y) p(x|y)

0= argmax

yp(y|x)

= argmax

p(x|y)p(y)p(x)

= argmax

yp(x|y)p(y)

Y X M 0

Senttransmission

Recoveredmessage

p(y) p(x|y)

0= argmax

yp(y|x)

= argmax

p(x|y)p(y)p(x)

= argmax

yp(x|y)p(y)

Y X M 0

Senttransmission

Recoveredmessage

p(y) p(x|y)

0= argmax

yp(y|x)

= argmax

p(x|y)p(y)p(x)

= argmax

yp(x|y)p(y)

I can help.

Y X M 0

Senttransmission

Recoveredmessage

0= argmax

yp(y|x)

= argmax

p(x|y)p(y)p(x)

= argmax

yp(x|y)p(y)

Y X M 0

Senttransmission

Recoveredmessage

0= argmax

yp(y|x)

= argmax

p(x|y)p(y)p(x)

= argmax

yp(x|y)p(y)

Denominator doesn’t depend on .y

Y X M 0

Senttransmission

Recoveredmessage

0= argmax

yp(y|x)

= argmax

p(x|y)p(y)p(x)

= argmax

yp(x|y)p(y)

Y X M 0

Senttransmission

Recoveredmessage

0= argmax

yp(x|y)p(y)

Senttransmission

Recoveredmessage

Y X M 0Y 0

English Csesky English’

0= argmax

yp(x|y)p(y)

e0 = argmax

ep(f|e)p(e)

Senttransmission

Recoveredmessage

Y X M 0Y 0

0= argmax

yp(x|y)p(y)

e0 = argmax

ep(f|e)p(e)

translation model

Senttransmission

Recoveredmessage

Y X M 0Y 0

0= argmax

yp(x|y)p(y)

e0 = argmax

ep(f|e)p(e)

translation model language model

Division of labor• Translation model

• translation back into the source

• learned from (source, target) translations

• adequacy of translation

• Language model

• probability of the output sentence

• learned from any target language corpus

• fluency of translation

Intuition for Division of Labor

• Better use of data

• We have parallel data

• We also have (lots of) parallel data

• ... use both

• Use weaker translation models

• Language modeling is hard ...

• Translation modeling is language modeling ++

• Language models are used in many applications

The Big Question in MT

• How do we design language and translation models?

• Considerations

• Is the model correct?

• Is prediction (inference) tractable?

• Is there data to learn the parameters?

Language Models• What is the probability of a sentence (in a particular

language)?

• There are an infinite number of grammatical sentences in any language

• This is the era of Big Data...but infinity is bigger

• For machine translation

• Naive model (model language)

• Naive parameterization (no relationship between words like walk walked walking walks)

• Clever estimation of parameters (Lecture later this week)

My legal name is Alexander Perchov. But all of my many friends dub me Alex, because that is a more flaccid-to-utter version of my legal name. Mother dubs me Alexi-stop-spleening-me!, because I am always spleening her. If you want to know why I am always spleening her, it is because I am always elsewhere with friends, and disseminating so much currency, and performing so many things that can spleen a mother. Father used to dub me Shapka, for the fur hat I would don even in the summer month. He ceased dubbing me that because I ordered him to cease dubbing me that. It sounded boyish to me, and I have always thought of myself as very potent and generative. I have many many girls, believe me, and they all have a different name for me. One dubs me Baby, not because I am a baby, but because she attends to me.

Probability Models of Text

• Sequence of symbols (bytes, letters, characters, morphemes, words, ...)

• Let denote the set of symbols

• Lots of possible sequences ( is infinitely large!)

• Probability distributions over ?

⌃⇤

Probability Models of Text

• How are probability models built?

• Make some independence assumptions

• Make assumptions about the (conditional) distributions of the events

• Estimate parameters from a sample of data

P (x, y) = P (x)⇥ P (y) () x, y are independent

I want

I want a

I want a flight

I want a flight to

I want a flight to Prague

I want a flight to Prague STOP

I have a bad headache

Chukchi (Siberian language):

English:

Aggultinative and polysynthetic languages have rich word-formation processes.

English:

Aggultinative and polysynthetic languages have rich word-formation processes.

Təmeyŋəlevtpəγtərkən

Translation Models

• Is a string of words e a meaning-preserving translation of a string of words f?

• Conditional model p(f | e)

• Challenges

• There are an infinite number of sentences

• How do we learn parameters of the models?

In der Innenstadt explodierte eine Autobombe

• Make independence assumptions

• Permute the source words into the target language order

• Pick translations for individual words / phrases

• Probability of particular translations

• Look at source context

• Ensure that the output is fluent & idiomatic

In der Innenstadt explodierte eine Autobombe A car bomb exploded downtown

• Computational challenges

• Searching all word permutations is NP-hard

• Massive numbers of translation alternatives

A car bomb exploded downtownIn der Innenstadt explodierte eine Autobombe

• String-to-string translation

• Became popular in the 1990’s with statistical MT

• State-of-the-art for many (most?) language pairs

• Especially: Closely related language pairs

• Especially: Typologically similar language pairs

• ~ Google Translate / Bing Translator

• Limitations

• Independence assumptions are wrong (too strong and too weak)

• No structural information available to improve modeling

• Models can be learned directly from parallel data

Egyptian

A car bomb exploded downtownIn der Innenstadt explodierte eine Autobombe

• Tree-to-string translation

• Syntactic analysis of source (parse)

• Transfer from tree to string

• Source trees have some benefits

• Proxy for semantic relationships

• Syntax is a natural source of reordering constraints

A car bomb exploded downtown

• String-to-tree translation

• Transfer from source tree to target string

• Formally a generalization of monolingual parsing

• Intuition: it is more important to know the language you are translating into well than the source language

• String-to-tree translation

• Transfer from source tree to target string

• Formally a generalization of monolingual parsing

• Intuition: it is more important to know the language you are translating into well than the source language

The best Chinese-English systems are string-to-tree

• Tree to tree translation

• Use syntax to predict syntax

• Benefits

• As parsers improve, MT will improve (we hope)

• Rich information for modeling in source and target

• Downside: where does the syntax come from?

detonate :arg0 bomb :arg1 car :loc downtown :time past

Semantics“logical form”

Syntax

report_event[ factivity=true explode(e, bomb, car) loc(e, downtown)]

explodieren :arg0 Bombe :arg1 Auto :loc Innenstadt :tempus imperf

Interlingua“meaning”

More Abstract Models

More Abstract Models• Modeling challenges

• What are the right abstract representations?

• How do we support more abstraction without sacrificing accuracy on frequent elements?

• Large search spaces

• Error propagation in pipelines

• Large search spaces

• Error propagation in pipelines

• Learning challenges

• Nonconvexity

• Where does the data come from?

Hidden

Other ChallengesEuropean parliament language (training):

I declare resumed the session of the European Parliament adjourned on Friday 17 December 1999, and I would like once again to wish you a happy new year in the hope that you enjoyed a pleasant festive period.

Human language (testing):

spelling error

abbreviationsMonday, September 9, 13

nonstandard contractionsMonday, September 9, 13

Questions?

Machine Translationufal.mff.cuni.cz/mtm13/files/01-mtm-intro-chris-dyer.pdf · • 1993 -...

Documents

License Management - Security Weaver...Security Weaver’s License Management module delivers role-based license management that allows your organization to optimize the value of SAP

TAUS%Moses%Roundtable% - Univerzita Karlovaufal.mff.cuni.cz/mtm13/files/mtm13-taus-round-table-slides.pdf · evaluate’the’postediMng’eﬀorton’the’basis’of’how’much

Business Process Improvement - Weaver Process Improv… · Business Process Improvement Weaver’s Business Process Improvement (BPI) service begins with working closely with clients

California Crafts Competition 2016 Awards - Cal Expo · Katie Martin Waves 4 ... Kelly Hechinger Baby Blanket 1 Sacramento Spinner’s & Weaver’s Award Pamela McSmith Summer/Winter

CSE 592 INTERNET CENSORSHIP (FALL 2015) LECTURE 03 PHILLIPA GILL STONY BROOK UNIVERSITY, COMPUTER SCIENCE ACKS: SLIDES BASED ON MATERIAL FROM NICK WEAVER’S

The Power of Intention - crossways.com.auPower+of+Intention+by+Wayne+Dyer.pdf · Dr Darryl Cross 1 BOOK REVIEW "The Power of Intention: Learning to Co-Create Your World Your Way"

Translation Quality Assessment: Evaluation and Estimationufal.mff.cuni.cz/mtm13/files/02-qe-lucia-specia.pdf · Translation QualityQTLaunchPad ProjectQuality EstimationState of the

Scaling Your Content Audit A recap of Misty Weaver’s Confab Presentation

This is a repository copy of Introductioneprints.whiterose.ac.uk/108222/3/Chapter 1 Hadley and Dyer.pdf · vigorous and wealthy aristocracy, occupying the ranks of the hierarchy below

Black Thrive in Lambeth - Beyond The Roombeyondtheroom.net/wp-content/uploads/2019/04/03-Dyer.pdf · Co-chaired by Jacqui Dyer, set up in response to the death of local residentSean

The Weaver’s Bagmmawg.org/newsletters/201502Newsletter.pdf · introduction to Swedish Weaving, but it gave so much more. The adventure began with participants arriving with warps

The Weaver’s Loom: A Conceptual Framework for Facilitating ...jbm.johogo.com/pdf/volume/2001/JBM-2001-05-full.pdfTransformational Human Resource Management When HRM practices and

Announcements Political Science Student Association PUB NIGHT! Political Science Student Association PUB NIGHT! TONIGHT, 8:30 PM TONIGHT, 8:30 PM Weaver’s

Silencing of Mary Dyer for classroom reading and discussion Dyer.pdf · Men took their guns to church with ... 1631 and eventually served 12 annual terms in the position. In 1637,

CSE 592 INTERNET CENSORSHIP (FALL 2015) LECTURE 04 PHILLIPA GILL, STONY BROOK UNIVERSITY ACKS: SLIDES BASED ON MATERIAL FROM NICK WEAVER’S PRESENTATION

Weaver’s Cove Reuse Task Force Reportfroed.org/pdf/Weaver'sCoveReuseTaskForceReport.pdfWeaver’s Cove Reuse Task Force Report. ... The Committee assigned scores relative to each

Louisville, Kentucky Incorporated 1989 February 2008members.aye.net/~kbsblues/Newsletters/2008/KBS_BN_2008-02.pdfPen Bogert played Sylvester Weaver’s “Guitar Rag” at the ceremo-In

Natalie Kertes Weaver’s - smp.org › dynamicmedia › files › fa3cb94ba...introductory theology textbook, reading Christian Thought and Practice gave me a “eureka” (“I found

Selected Images of Bambusa textilisdavisbamboo.com/graphics/Bambusa-textilis.pdf · 2018-04-08 · Selected Images of Bambusa textilis (common name: Weaver’s Bamboo) Figure 1. Bambusa

el poder de la intencion - wayne dyer.pdf