Embed Size (px)
Citation preview
1/22
Introduction The actors The scenarios 5 counterobjections Conclusions
The translation gameMachine translation evaluation without prejudice
Federico [email protected]
University of Insubria, Varese, ItalyCC© Some rights reserved.
ECAP09, UAB, Barcelona, July 2009
2/22
Introduction The actors The scenarios 5 counterobjections Conclusions
Why machine translation evaluation is interesting?
Machine Translation (MT): a coherent chain of grammaticalsentences written in a given natural language (NL) rendered by amachine from a source language (Ls) into a reliable text written ina target language (Lt).
Remarks:
only asynchronous written texts (otherwise, interpretation).
without any human aid (otherwise, Computer-AidedTranslation).
MT can be seen as a corpus-based test of the appropriateness ofmodels we have of the language faculty, if the evaluation isperformed in the appropriate setting.
2/22
Introduction The actors The scenarios 5 counterobjections Conclusions
Why machine translation evaluation is interesting?
Machine Translation (MT): a coherent chain of grammaticalsentences written in a given natural language (NL) rendered by amachine from a source language (Ls) into a reliable text written ina target language (Lt).
Remarks:
only asynchronous written texts (otherwise, interpretation).
without any human aid (otherwise, Computer-AidedTranslation).
MT can be seen as a corpus-based test of the appropriateness ofmodels we have of the language faculty, if the evaluation isperformed in the appropriate setting.
2/22
Introduction The actors The scenarios 5 counterobjections Conclusions
Why machine translation evaluation is interesting?
Machine Translation (MT): a coherent chain of grammaticalsentences written in a given natural language (NL) rendered by amachine from a source language (Ls) into a reliable text written ina target language (Lt).
Remarks:
only asynchronous written texts (otherwise, interpretation).
without any human aid (otherwise, Computer-AidedTranslation).
MT can be seen as a corpus-based test of the appropriateness ofmodels we have of the language faculty, if the evaluation isperformed in the appropriate setting.
2/22
Introduction The actors The scenarios 5 counterobjections Conclusions
Why machine translation evaluation is interesting?
Machine Translation (MT): a coherent chain of grammaticalsentences written in a given natural language (NL) rendered by amachine from a source language (Ls) into a reliable text written ina target language (Lt).
Remarks:
only asynchronous written texts (otherwise, interpretation).
without any human aid (otherwise, Computer-AidedTranslation).
MT can be seen as a corpus-based test of the appropriateness ofmodels we have of the language faculty, if the evaluation isperformed in the appropriate setting.
3/22
Introduction The actors The scenarios 5 counterobjections Conclusions
The problem of MT evaluation
In the development process of an MT engine, automatic evaluatorsare used: they implement algorithms that measure the distance ofthe target text (Tt) from a gold standard reference corpus, trainedthrough machine learning techniques.
Nonetheless, it is well known among specialists that automatedevaluation is not enough, especially in the production phase of thelife cycle of the MT software, where informants are used.
4/22
Introduction The actors The scenarios 5 counterobjections Conclusions
Why using a Gedankenexperiment?
I argue that the setting where the MT evaluation is performed isnot free from psychological and epistemological prejudice byinformants, a-priori invalidating their judgement.
Therefore, I compare two MT evaluation settings as aGedankenexperiment, a la Turing (Imitation Game, 1950) or Searle(Chinese Room, 1980).
The first setting is called the default scenario.
The second setting is called the neutral scenario.
4/22
Introduction The actors The scenarios 5 counterobjections Conclusions
Why using a Gedankenexperiment?
I argue that the setting where the MT evaluation is performed isnot free from psychological and epistemological prejudice byinformants, a-priori invalidating their judgement.
Therefore, I compare two MT evaluation settings as aGedankenexperiment, a la Turing (Imitation Game, 1950) or Searle(Chinese Room, 1980).
The first setting is called the default scenario.
The second setting is called the neutral scenario.
4/22
Introduction The actors The scenarios 5 counterobjections Conclusions
Why using a Gedankenexperiment?
I argue that the setting where the MT evaluation is performed isnot free from psychological and epistemological prejudice byinformants, a-priori invalidating their judgement.
Therefore, I compare two MT evaluation settings as aGedankenexperiment, a la Turing (Imitation Game, 1950) or Searle(Chinese Room, 1980).
The first setting is called the default scenario.
The second setting is called the neutral scenario.
5/22
Introduction The actors The scenarios 5 counterobjections Conclusions
A. & B.: the sender and the receiver
Alice is a native speaker of Spanish (Ls) and she wantsto translate a newspaper article in Tamil (Lt).
Bob is a native speaker of Tamil (Lt) and he wants toread Alice’s newspaper article in his own language.
6/22
Introduction The actors The scenarios 5 counterobjections Conclusions
C. & D.: MT designing & evaluation
Charles is the designer of the Spanish-Tamil MTsystem. He is a software engineer specialized in MT, not atranslator (nor a linguist).
Dave is a bilingual Spanish-Tamil and a professionaltranslator, i.e., he is skilled to evaluate translationese – the set oflinguistic indicators of a text being a translation.
7/22
Introduction The actors The scenarios 5 counterobjections Conclusions
The default scenario: the sender and the receiver
Alice is expected to write the original text in Spanish (Ts), whileBob will read the MT in Tamil (Tt)
8/22
Introduction The actors The scenarios 5 counterobjections Conclusions
The default scenario: the evaluation process
To evaluate the reliability of the translation, Charles wants Dave toread the original text in Spanish (Ts) and the MT in Tamil (Tt).
Charles asks to Dave the following question:
Is this translation reliable?
9/22
Introduction The actors The scenarios 5 counterobjections Conclusions
How to avoid the psychological fallacy
If Dave does not know that the translation is a machinetranslation, he is free from the psychological bias towards MT:
“MT is stupid, only professional translators can be realtranslator”, etc.
10/22
Introduction The actors The scenarios 5 counterobjections Conclusions
The default sceario cannot avoid the epistemic fallacy
In fact, Dave is evaluating:
the text in Spanish (Ts) by a human agent;
the text in Tamil (Ts) by an artificial agent.
In other words, the epistemic fallacy is still valid: in the defaultscenario MT can only mimick human translation, therefore itcannot be truly evaluated per se.
11/22
Introduction The actors The scenarios 5 counterobjections Conclusions
The neutral scenario: the Entry Language
Let’s suppose that Charles asks Alice not to write directly inSpanish (Ls), but in a special controlled language, i.e., aQuasi-Natural Language (QNL, Lyons 2006):
it is double articulated (i.e., phonemes vs. morphemes)
its semantics is not domain-specific;
it is highly regular in morphology (low homophony degree);
POS-tagging is easy (low allomorphy degree).
Let us call this QNL the “Entry Language” (Le).
11/22
Introduction The actors The scenarios 5 counterobjections Conclusions
The neutral scenario: the Entry Language
Let’s suppose that Charles asks Alice not to write directly inSpanish (Ls), but in a special controlled language, i.e., aQuasi-Natural Language (QNL, Lyons 2006):
it is double articulated (i.e., phonemes vs. morphemes)
its semantics is not domain-specific;
it is highly regular in morphology (low homophony degree);
POS-tagging is easy (low allomorphy degree).
Let us call this QNL the “Entry Language” (Le).
11/22
Introduction The actors The scenarios 5 counterobjections Conclusions
The neutral scenario: the Entry Language
Let’s suppose that Charles asks Alice not to write directly inSpanish (Ls), but in a special controlled language, i.e., aQuasi-Natural Language (QNL, Lyons 2006):
it is double articulated (i.e., phonemes vs. morphemes)
its semantics is not domain-specific;
it is highly regular in morphology (low homophony degree);
POS-tagging is easy (low allomorphy degree).
Let us call this QNL the “Entry Language” (Le).
12/22
Introduction The actors The scenarios 5 counterobjections Conclusions
What is a QNL?
For example, QNL has a subclass Quasi-English, one of whosememebers is like English in all respects except that it isinflectionally regular, all plurals of nouns being formed with the-s suffix (childs, sheeps, gooses, etc.), all past-tense forms ofverbs with -ed (goed, runned, beed, etc.), and so on. This is alanguage part of which children construct of themselves (andthen in part decostruct – if I may so express it at) at a certainstage in the normal (natural3) process of acquiring English. Itis also the language into which English would presumably havedeveloped under particular environmental conditions whichmaximized the effect of what is traditionally referred to asanalogy.
Lyons, Natural language and universal grammar, 2006:69–70.
13/22
Introduction The actors The scenarios 5 counterobjections Conclusions
The MT interface for writing in the neutral scenario
When Alice writes the text in the Entry Language (Te), the MTsystems generates automatically the translation in Spanish (Ts)and in Tamil (Tt) and she can adjust her writing to obtain a betterresult controlling the Ts output. In practice, two cognitiveprocesses are performed:
A writing process in the Entry Language input;
An evaluation process, through the reading of the Spanishoutput.
14/22
Introduction The actors The scenarios 5 counterobjections Conclusions
The receiver isn’t aware of the new scenario
While Alice is expected to write differently, Bob will still read theMT in Tamil (Tt) as in the default scenario.
The text in the Entry Language is only part of the writing interfacefor Alice: only Charles is aware of it, unlike Bob and Dave.
15/22
Introduction The actors The scenarios 5 counterobjections Conclusions
The neutral sceario does avoid the epistemic fallacy
For Dave, apparently nothing changed, but in reality heis evaluating:
the text in Spanish (Ts) by an artificial agent;
the text in Tamil (Ts) by an artificial agent.
I argue that in this way, the epistemic fallacy is avoided.
16/22
Introduction The actors The scenarios 5 counterobjections Conclusions
1. The chinese room argument
“Whatever linguistic model you put into the machine, wecannot consider it really cognition, as the meaning of thelinguistic model is only in the brain of Charles, thesystem designer”.
Counterobjection:
based on the Chinese Room argument;
the system is made explicit by the computer program;
once formulated, a theorem should also be proved;
analogously, the MT engine is like a proof (not the sameknowledge!);
software is a “cognitive mediator” (Magnani, 2007).
16/22
Introduction The actors The scenarios 5 counterobjections Conclusions
1. The chinese room argument
“Whatever linguistic model you put into the machine, wecannot consider it really cognition, as the meaning of thelinguistic model is only in the brain of Charles, thesystem designer”.
Counterobjection:
based on the Chinese Room argument;
the system is made explicit by the computer program;
once formulated, a theorem should also be proved;
analogously, the MT engine is like a proof (not the sameknowledge!);
software is a “cognitive mediator” (Magnani, 2007).
17/22
Introduction The actors The scenarios 5 counterobjections Conclusions
2. The engineer’s reaction
“Machine translation is not a testbed of any linguistictheory or anything else. What we need is somethingpractical, i.e., commercially valuable as useful in somedomains where we want fast translation of large amountof data.”
Counterobjection:
seldom said openly;
having no linguistic theory is a linguistic choice;
computational brute force is not enough for achieving goodresults;
machine learning techniques should be poised in annotatedcorpora.
17/22
Introduction The actors The scenarios 5 counterobjections Conclusions
2. The engineer’s reaction
“Machine translation is not a testbed of any linguistictheory or anything else. What we need is somethingpractical, i.e., commercially valuable as useful in somedomains where we want fast translation of large amountof data.”
Counterobjection:
seldom said openly;
having no linguistic theory is a linguistic choice;
computational brute force is not enough for achieving goodresults;
machine learning techniques should be poised in annotatedcorpora.
18/22
Introduction The actors The scenarios 5 counterobjections Conclusions
3. The desperantist’s argument
“A QNL is an artificial language, like Esperanto,Interlingua, Lojban, or Klingon. Artificial language areliving dead languages, unfit for your purposes!”
Counterobjection:
at least Esperanto proved to work and evoluate as any NL;
non-naturalness is not unnaturalness (such as for C, Java orProlog);
true: is there anyone out there willing to test concretely thisGedankenexperiment?
Probably not.
18/22
Introduction The actors The scenarios 5 counterobjections Conclusions
3. The desperantist’s argument
“A QNL is an artificial language, like Esperanto,Interlingua, Lojban, or Klingon. Artificial language areliving dead languages, unfit for your purposes!”
Counterobjection:
at least Esperanto proved to work and evoluate as any NL;
non-naturalness is not unnaturalness (such as for C, Java orProlog);
true: is there anyone out there willing to test concretely thisGedankenexperiment?
Probably not.
18/22
Introduction The actors The scenarios 5 counterobjections Conclusions
3. The desperantist’s argument
“A QNL is an artificial language, like Esperanto,Interlingua, Lojban, or Klingon. Artificial language areliving dead languages, unfit for your purposes!”
Counterobjection:
at least Esperanto proved to work and evoluate as any NL;
non-naturalness is not unnaturalness (such as for C, Java orProlog);
true: is there anyone out there willing to test concretely thisGedankenexperiment?
Probably not.
19/22
Introduction The actors The scenarios 5 counterobjections Conclusions
4. The typologist’s argument
“Whatever QNL you choose, it will be typologicallydetermined, according to the native tongue of Alice –e.g., you will use a Quasi-Natural Spanish. Your scenariomay work with English, French or Spanish, but not withnon-European languages, such as Chinese, Arabic, orTamil.”
Counterobjection:
to be verified: this is a very serious variable in implementingthe MT engine; but please, give us a chance!
19/22
Introduction The actors The scenarios 5 counterobjections Conclusions
4. The typologist’s argument
“Whatever QNL you choose, it will be typologicallydetermined, according to the native tongue of Alice –e.g., you will use a Quasi-Natural Spanish. Your scenariomay work with English, French or Spanish, but not withnon-European languages, such as Chinese, Arabic, orTamil.”
Counterobjection:
to be verified: this is a very serious variable in implementingthe MT engine; but please, give us a chance!
20/22
Introduction The actors The scenarios 5 counterobjections Conclusions
5. The human interface argument
“Your assumption is too strong. You force Alice not touse her mother tongue, i.e., Spanish, and you ask her tolearn Charles’ system too. Furthermore, as your approachimplies a strong supervision, I think that it will be easier,faster and cheapier to translate source and targetlanguage by professionals instead of using your system.”
Counterobjection:
a pragmatic argument, moved from economics;
the comparison between Te and Ts should compensate Alice’sadditional effort;
only a monolingual parser is needed (for Le);
translation memories can be stored so that the MT systembecomes more and more precise according to its use.
20/22
Introduction The actors The scenarios 5 counterobjections Conclusions
5. The human interface argument
“Your assumption is too strong. You force Alice not touse her mother tongue, i.e., Spanish, and you ask her tolearn Charles’ system too. Furthermore, as your approachimplies a strong supervision, I think that it will be easier,faster and cheapier to translate source and targetlanguage by professionals instead of using your system.”
Counterobjection:
a pragmatic argument, moved from economics;
the comparison between Te and Ts should compensate Alice’sadditional effort;
only a monolingual parser is needed (for Le);
translation memories can be stored so that the MT systembecomes more and more precise according to its use.
21/22
Introduction The actors The scenarios 5 counterobjections Conclusions
Paraphrasing Turing...
Only a machine can really appreciate amachine translation.
Perhaps.
21/22
Introduction The actors The scenarios 5 counterobjections Conclusions
Paraphrasing Turing...
Only a machine can really appreciate amachine translation.
Perhaps.
22/22
Introduction The actors The scenarios 5 counterobjections Conclusions
Thanks. Any questions?
Download these slides here:
http://www.slideshare.net/goberiko/
CC© BY:© $\© C© Federico Gobbo 2009. Pubblicato in Italia.Attribuzione – Non commerciale – Condividi allo stesso modo 2.5
