Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1591–1604July 5 - 10, 2020. c©2020 Association for Computational Linguistics

1591

Character-Level Translation with Self-attention

Yingqiang Gao†‡, Nikola I. Nikolov‡, Yuhuang Hu‡, Richard H.R. Hahnloser‡†Department of Informatics, Technical University of Munich

‡Institute of Neuroinformatics, University of Zurich and ETH Zurichyingqiang.gao@in.tum.de {niniko, yuhuang.hu, rich}@ini.ethz.ch

Abstract

We explore the suitability of self-attentionmodels for character-level neural machinetranslation. We test the standard transformermodel, as well as a novel variant in whichthe encoder block combines information fromnearby characters using convolutions. Weperform extensive experiments on WMT andUN datasets, testing both bilingual and mul-tilingual translation to English using up tothree input languages (French, Spanish, andChinese). Our transformer variant consis-tently outperforms the standard transformer atthe character-level and converges faster whilelearning more robust character-level align-ments.1

1 Introduction

Most existing Neural Machine Translation (NMT)models operate on the word or subword-level,which tends to make these models memory ineffi-cient because of large vocabulary sizes. Character-level models (Lee et al., 2017; Cherry et al., 2018)instead work directly on raw characters, result-ing in a more compact language representation,while mitigating out-of-vocabulary (OOV) prob-lems (Luong and Manning, 2016). Character-level models are also very suitable for multilingualtranslation since multiple languages can be mod-eled using the same character vocabulary. Multi-lingual training can lead to improvements in theoverall performance without an increase in modelcomplexity (Lee et al., 2017), while also circum-venting the need to train separate models for eachlanguage pair.

Models based on self-attention have achievedexcellent performance on a number of tasks, in-cluding machine translation (Vaswani et al., 2017)and representation learning (Devlin et al., 2019;

1Code available at https://github.com/CharizardAcademy/convtransformer

Multi-Head Attention 

Add & Norm

Feed Forward

Add & Norm

Conv 1Dwidth 3

Conv 1Dwidth 5

Conv 1Dwidth 7

concat.

Conv 1Dwidth 3

Parallel Conv 1D layer

Parallel Conv 1D Layer

Multi-Head Attention 

Add & Norm

Feed Forward

Add & Norm

a) b)

Figure 1: A comparison of the encoder blocks in thestandard transformer (a) and our novel modification,the convtransformer (b), which uses 1D convolutionsto facilitate character interactions.

Yang et al., 2019). Despite the success of thesemodels, their suitability for character-level trans-lation remains largely unexplored, with most ef-forts having focused on recurrent models (e.g., Leeet al. (2017); Cherry et al. (2018)).

In this work, we perform an in-depth investiga-tion of the suitability of self-attention models forcharacter-level translation. We consider two mod-els: the standard transformer from Vaswani et al.(2017) and a novel variant that we call the con-vtransformer (Figure 1, Section 3). The convtrans-former uses convolutions to facilitate interactionsamong nearby character representations.

We evaluate these models on both bilingualand multilingual translation to English, using upto three input languages: French (FR), Spanish(ES), and Chinese (ZH). We compare the perfor-mance when translating from close (e.g., FR andES) and on distant (e.g., FR and ZH) input lan-guages (Section 5.1) and we analyze the learnedcharacter alignments (Section 5.2). We find thatself-attention models work surprisingly well forcharacter-level translation, achieving competitive

1592

performance to equivalent subword-level mod-els while requiring up to 60% fewer parameters(under the same model configuration). At thecharacter-level, the convtransformer outperformsthe standard transformer, converges faster, andproduces more robust alignments.

2 Background

2.1 Character-level NMT

Fully character-level translation was first tackledin Lee et al. (2017), who proposed a recurrentencoder-decoder model. Their encoder combinesconvolutional layers with max-pooling and high-way layers to construct intermediate representa-tions of segments of nearby characters. Their de-coder network autoregressively generates the out-put translation one character at a time, utilizing at-tention on the encoded representations.

Lee et al. (2017)’s approach showed promis-ing results on multilingual translation in partic-ular. Without any architectural modifications orchanges to the character vocabularies, training onmultiple source languages yielded performanceimprovements while also acting as a regularizer.Multilingual training of character-level models ispossible not only for languages that have almostidentical character vocabularies, such as Frenchand Spanish, but even for distant languages thatcan be mapped to a common character-level vo-cabulary, for example, through latinizing Russian(Lee et al., 2017) or Chinese (Nikolov et al., 2018).

More recently, (Cherry et al., 2018) per-formed an in-depth comparison between differ-ent character- and subword-level models. Theyshowed that, given sufficient computational timeand model capacity, character-level models canoutperform subword-level models, due to theirgreater flexibility in processing and segmentingthe input and output sequences.

2.2 The Transformer

The transformer (Vaswani et al., 2017) is anattention-driven encoder-decoder model that hasachieved state-of-the-art performance on a numberof sequence modeling tasks in NLP. Instead of us-ing recurrence, the transformer uses only feedfor-ward layers based on self-attention. The standardtransformer architecture consists of six stackedencoder layers that process the input using self-attention and six decoder layers that autoregres-sively generate the output sequence.

The original transformer (Vaswani et al., 2017)computes a scaled dot-product attention by takingas input query Q, key K, and value V matrices:

Attention(Q,K, V ) = softmax(QKT

√dk

)V,

where√dk is a scaling factor. For the encoder,

Q, K and V are equivalent, thus, given an inputsequence with length N , Attention performs N2

comparisons, relating each word position with therest of the words in the input sequence. In practice,Q, K, and V are projected into different represen-tation subspaces (called heads), to perform Multi-Head Attention, with each head learning differ-ent word relations, some of which might be inter-pretable (Vaswani et al., 2017; Voita et al., 2019).

Intuitively, attention as an operation might notbe as meaningful for encoding individual charac-ters as it is for words, because individual charac-ter representations might provide limited semanticinformation for learning meaningful relations onthe sentence level. However, recent work on lan-guage modeling (Al-Rfou et al., 2019) has surpris-ingly shown that attention can be very effective formodeling characters, raising the question of howwell the transformer would work on character-level bilingual and multilingual translation, andwhat architectures would be suitable for this task.These are the questions this paper sets out to in-vestigate.

3 Convolutional Transformer

To facilitate character-level interactions in thetransformer, we propose a modification of thestandard architecture, which we call the convtrans-former. In this architecture, we use the same de-coder as the standard transformer, but we adapteach encoder block to include an additional sub-block. The sub-block (Figure 1, b), inspired fromLee et al. (2017), is applied to the input repre-sentations M , before applying self-attention. Thesub-block consists of three 1D convolutional lay-ers, Cw, with different context window sizes w. Inorder to maintain the temporal resolution of con-volutions, the padding is set to bw−12 c.

We apply three separate convolutional layers,C3, C5 and C7, in parallel, using context windowsizes of 3, 5 and 7, respectively. The differentcontext window sizes aim to resemble character-level interactions of different levels of granular-ity, such as on the subword- or word-level. To

1593

compute the final output of the convolutional sub-block, the outputs of the three layers are concate-nated and passed through an additional 1D con-volutional layer with context window size 3, C

′3,

which fuses the representations:

Conv(M) =

M + C′3(Concat(C3(M), C5(M), C7(M))).

For all convolutional layers, we set the numberof filters to be equal to the embedding dimensionsize dmodel, which results in an output of equaldimension as the input M . Therefore, in con-trast to Lee et al. (2017), who use max-poolingto compress the input character sequence into seg-ments of characters, here we leave the resolutionunchanged, for both transformer and convtrans-former models. Finally, for additional flexibility,we add a residual connection (He et al., 2016)from the input to the output of the convolutionalblock.

4 Experimental Set-up

Datasets. We conduct experiments on twodatasets. First, we use the WMT15 DE→ENdataset, on which we test different model con-figurations and compare our results to previouswork on character-level translation. We followthe preprocessing in Lee et al. (2017) and usethe newstest-2014 dataset for testing. Second, weconduct our main experiments using the UnitedNations Parallel Corporus (UN) (Ziemski et al.,2016), for two reasons: (i) UN contains a largenumber of parallel sentences from six languages,allowing us to conduct multilingual experiments;(ii) all sentences in the corpus are from the samedomain. We construct our training corpora by ran-domly sampling one million sentence pairs fromthe FR, ES, and ZH parts of the UN dataset, tar-geting translation to English. To construct multi-lingual datasets, we combine the respective bilin-gual datasets (e.g., FR→EN, and ES→EN) andshuffle them. To ensure all languages share thesame character vocabulary, we latinize the Chi-nese dataset using the Wubi encoding method, fol-lowing (Nikolov et al., 2018). For testing, we usethe original UN test sets provided for each pair.

Tasks. Our experiments are designed as follows:(i) bilingual scenario, in which we train a modelwith a single input language; (ii) multilingual sce-nario, in which we input two or three languages

Model BLEU #par

char

acte

r-le

vel Lee et al. (2017) 25.77 69M

transformer-6-layer 28.8 49Mconvtransformer-6-layer 29.23 68Mtransformer-12-layer 29.81 93Mconvtransformer-12-layer 30.16 131M

bpe transformer-6-layer 30.06 121M

transformer-12-layer 31.60 165M

Table 1: Comparison of architecture variants on theWMT15 DE→EN dataset. #par is the number ofmodel parameters.

at the same time without providing any languageidentifiers to the models and without increasingthe number of parameters. We test combining in-put languages that can be considered as more sim-ilar in terms of syntax and vocabulary (e.g. FR andES) as well as more distant (e.g., ES and ZH).

5 Results

5.1 Automatic evaluationModel comparison. In Table 1, we compare theBLEU performance (Papineni et al., 2002) of di-verse character-level architectures trained on theWMT dataset. For reference, we include the recur-rent character-level model from Lee et al. (2017),as well as transformers trained on the subwordlevel using a vocabulary of 50k byte-pair encoding(BPE) tokens (Sennrich et al., 2016). All modelswere trained on four Nvidia GTX 1080X GPUs for20 epochs.

We find character-level training to be 3 to 5times slower than subword-level training due tomuch longer sequence lengths. However, thestandard transformer trained at the character levelalready achieves very good performance, out-performing the recurrent model from Lee et al.(2017). On this dataset, our convtransformer vari-ant performs on par with the character-level trans-former. Character-level transformers also per-form competitively with equivalent BPE mod-els while requiring up to 60% fewer parameters.Furthermore, our 12-layer convtransformer modelmatches the performance of the 6-layer BPE trans-former, which has a comparable number of param-eters.

Multilingual experiments. In Table 2, we re-port our BLEU results on the UN datasetusing the 6-layer transformer/convtransformercharacter-level models (Appendix A contains ex-ample model outputs). All of our models weretrained for 30 epochs. Multilingual models are

1594

Model #P transformer convtransformerInput lang. t-FR t-ES t-ZH t-FR t-ES t-ZH

bilin

gual FR 1M 32.48 - - 33.69 - -

ES 1M - 39.90 - - 41.41 -ZH 1M - - 38.70 - - 41.01

mul

tilin

gual FR+ES 2M 33.51 40.83 - 34.69 41.84 -

FR+ZH 2M 32.89 - 37.92 33.98 - 40.56ES+ZH 2M - 40.43 38.23 - 41.49 40.41

FR+ES+ZH 3M 33.69 40.71 38.01 34.38 41.73 39.87

Table 2: BLEU scores on the UN dataset, for differentinput training languages (first column), and evaluatedon three different test sets (t-FR, t-ES and t-ZH). Thetarget language is always English. #P is the numberof training pairs. The best overall results for each lan-guage are in bold.

evaluated on translation from all possible inputlanguages to English.

Although multilingual translation can be real-ized using subword-level models through extract-ing a joint segmentation for all input languages(e.g., as in Firat et al. (2016); Johnson et al.(2017)), here we do not include any subword-levelmultilingual baselines, for two reasons. First, ex-tracting a good multilingual segmentation is muchmore challenging for our choice of input lan-guages, which includes distant languages such asChinese and Spanish. Second, as discussed pre-viously, subword-level models have a much largernumber of parameters, making a balanced compar-ison with character-level models difficult.

The convtransformer consistently outperformsthe character-level transformer on this dataset,with a gap of up to 2.3 BLEU on bilingual trans-lation (ZH→EN) and up to 2.6 BLEU on multi-lingual translation (FR+ZH→EN). Training mul-tilingual models on similar input languages (FR +ES→EN) leads to improved performance for bothlanguages, which is consistent with (Lee et al.,2017). Training on distant languages is surpris-ingly still effective in some cases. For exam-ple, the models trained on FR+ZH→EN outper-form the models trained just on FR→EN; how-ever they perform worse than the bilingual modelstrained on ZH→EN. Thus, distant-language train-ing seems to be helpful mainly when the input lan-guage is closer to the target translation language(which is English here).

The convtransformer is about 30% slower totrain than the transformer (see Figure 2). Nev-ertheless, the convtransformer reaches compara-ble performance in less than half of the numberof epochs, leading to an overall training speedup

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30Epoch

36

37

38

39

40

41

42

BLEU

scor

e

conv. ES→ENconv. FR+ES→ENconv. ES+ZH→ENconv. FR+ES+WB→ENtrans. ES→ENtrans. FR+ES→ENtrans. ES+ZH→ENtrans. FR+ES+ZH→EN

Figure 2: BLEU scores on the UN dataset as a func-tion of epoch number, for bilingual and multilingualcharacter-level translation from ES to EN. conv. isthe convtransformer, while trans. is the originaltransformer.

compared to the transformer.

5.2 Analysis of Learned Alignments

To gain a better understanding of the multilingualmodels, we analyze their learned character align-ments as inferred from the model attention prob-abilities. For each input language (e.g., FR), wecompare the alignments learned by each of ourmultilingual models (e.g., FR + ES → EN model)to the alignments learned by the correspondingbilingual model (e.g., FR → EN). Our intuition isthat the bilingual models have the greatest flexibil-ity to learn high-quality alignments because theyare not distracted by other input languages. Mul-tilingual models, by contrast, might learn lowerquality alignments because either (i) the architec-ture is not robust enough for multilingual training;or (ii) the languages are too dissimilar to allowfor effective joint training, prompting the modelto learn alternative alignment strategies to accom-modate for all languages.

We quantify the alignments using canonical cor-relation analysis (CCA) (Morcos et al., 2018).First, we sample 500 random sentences from eachof our UN testing datasets (FR, ES, or ZH) andthen produce alignment matrices by extracting theencoder-decoder attention from the last layer ofeach model. We use CCA to project each align-ment matrix to a common vector space and inferthe correlation. We analyze our transformer andconvtransformer models separately. Our resultsare in Figure 3, while Appendix B contains exam-ple alignment visualizations.

For similar source and target languages (e.g.,the FR+ES→EN model), we observe a strong pos-

1595

FR FR ES ES1.000.750.500.250.000.250.500.751.00

Cano

nica

l cor

rela

tion

(CCA

) FR + ES EN

FR FR ZH ZH

FR + ZH EN

ES ES ZH ZH

ES + ZH EN

FR FR ES ES ZH ZH

Training pairs:

Input language during testing (FR, ES or ZH).

FR + ES + ZH EN

transformer convtransformer

Figure 3: Canonical correlation between multilingual and bilingual translation models trained on the UN dataset.

itive correlation to the bilingual models, indicat-ing that alignments can be simultaneously learned.When introducing a distant source language (ZH)in the training, we observe a drop in correlation,for FR and ES, and an even larger drop for ZH.This result is in line with our BLEU results fromSection 5.1, suggesting that multilingual trainingon distant input languages is more challengingthan multilingual training on similar input lan-guages. The convtransformer is more robust to theintroduction of a distant language than the trans-former (p < 0.005 for FR and ES inputs, accord-ing to a one-way ANOVA test). Our results alsosuggest that more sophisticated attention architec-tures might need to be developed when trainingmultilingual models on several distant input lan-guages.

6 Conclusion

We performed a detailed investigation of the utilityof self-attention models for character-level trans-lation. We test the standard transformer architec-ture, as well as introduce a novel variant whichaugments the transformer encoder with convolu-tions, to facilitate information propagation acrossnearby characters. Our experiments show thatself-attention performs very well on character-level translation, with character-level architec-tures performing competitively when compared toequivalent subword-level architectures while re-quiring fewer parameters. Training on multipleinput languages is also effective and leads to im-provements across all languages when the sourceand target languages are similar. When the lan-guages are different, we observe a drop in perfor-mance, in particular for the distant language.

In future work, we will extend our analysisto include additional source and target languages

from different language families, such as moreAsian languages. We will also work towards im-proving the training efficiency of character-levelmodels, which is one of their main bottlenecks,as well as towards improving their effectivenessin multilingual training.

Acknowledgements

We acknowledge support from the Swiss NationalScience Foundation (grant 31003A 156976) andthe National Centre of Competence in Research(NCCR) Robotics. We also thank the anonymousreviewers for their useful comments.

ReferencesRami Al-Rfou, Dokook Choe, Noah Constant, Mandy

Guo, and Llion Jones. 2019. Character-level Lan-guage Modeling with Deeper Self-attention. In Pro-ceedings of the AAAI Conference on Artificial Intel-ligence, volume 33, pages 3159–3166.

Colin Cherry, George Foster, Ankur Bapna, OrhanFirat, and Wolfgang Macherey. 2018. RevisitingCharacter-Based Neural Machine Translation withCapacity and Compression. In Proceedings of the2018 Conference on Empirical Methods in Natu-ral Language Processing, pages 4295–4305, Brus-sels, Belgium. Association for Computational Lin-guistics.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofDeep Bidirectional Transformers for Language Un-derstanding. In Proceedings of the 2019 Conferenceof the North American Chapter of the Associationfor Computational Linguistics: Human LanguageTechnologies, Volume 1 (Long and Short Papers),pages 4171–4186, Minneapolis, Minnesota. Associ-ation for Computational Linguistics.

Orhan Firat, Kyunghyun Cho, and Yoshua Bengio.2016. Multi-Way, Multilingual Neural Machine

1596

Translation with a Shared Attention Mechanism. InProceedings of the 2016 Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics: Human Language Technologies,pages 866–875, San Diego, California. Associationfor Computational Linguistics.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and JianSun. 2016. Deep Residual Learning for ImageRecognition. In Proceedings of the IEEE conferenceon computer vision and pattern recognition, pages770–778.

Melvin Johnson, Mike Schuster, Quoc V Le, MaximKrikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat,Fernanda Viegas, Martin Wattenberg, Greg Corrado,et al. 2017. Google’s Multilingual Neural MachineTranslation System: Enabling Zero-shot Transla-tion. Transactions of the Association for Compu-tational Linguistics, 5:339–351.

Jason Lee, Kyunghyun Cho, and Thomas Hofmann.2017. Fully Character-level Neural Machine Trans-lation without Explicit Segmentation. Transactionsof the Association for Computational Linguistics,5:365–378.

Minh-Thang Luong and Christopher D. Manning.2016. Achieving Open Vocabulary Neural MachineTranslation with Hybrid Word-Character Models.In Proceedings of the 54th Annual Meeting of the As-sociation for Computational Linguistics (Volume 1:Long Papers), pages 1054–1063, Berlin, Germany.Association for Computational Linguistics.

Ari Morcos, Maithra Raghu, and Samy Bengio. 2018.Insights on Representational Similarity in NeuralNetworks with Canonical Correlation. In Advancesin Neural Information Processing Systems, pages5727–5736.

Nikola Nikolov, Yuhuang Hu, Mi Xue Tan, andRichard H.R. Hahnloser. 2018. Character-levelChinese-English Translation through ASCII Encod-ing. In Proceedings of the Third Conference onMachine Translation, Volume 1: Research Papers,pages 10–16, Belgium, Brussels. Association forComputational Linguistics.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for AutomaticEvaluation of Machine Translation. In Proceedingsof the 40th annual meeting on association for com-putational linguistics, pages 311–318. Associationfor Computational Linguistics.

Rico Sennrich, Barry Haddow, and Alexandra Birch.2016. Neural Machine Translation of Rare Wordswith Subword Units. In Proceedings of the 54th An-nual Meeting of the Association for ComputationalLinguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computa-tional Linguistics.

Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N Gomez, ŁukaszKaiser, and Illia Polosukhin. 2017. Attention is AllYou Need. In Advances in neural information pro-cessing systems, pages 5998–6008.

Elena Voita, David Talbot, Fedor Moiseev, Rico Sen-nrich, and Ivan Titov. 2019. Analyzing multi-headself-attention: Specialized heads do the heavy lift-ing, the rest can be pruned. In Proceedings of the57th Annual Meeting of the Association for Com-putational Linguistics, pages 5797–5808, Florence,Italy. Association for Computational Linguistics.

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car-bonell, Russ R Salakhutdinov, and Quoc V Le. 2019.XLNet: Generalized Autoregressive Pretraining forLanguage Understanding. In Advances in Neural In-formation Processing Systems 32, pages 5754–5764.Curran Associates, Inc.

Michał Ziemski, Marcin Junczys-Dowmunt, and BrunoPouliquen. 2016. The United Nations Parallel Cor-pus v1. 0. In Proceedings of the Tenth InternationalConference on Language Resources and Evaluation(LREC 2016), pages 3530–3534.

A Example model outputs

Tables 3, 4, and 5 contain example translationsproduced by our different bilingual and multilin-gual models trained on the UN datasets.

B Visualization of Attention

In Figures 4,5, 6 and 7 we plot example alignmentsproduced by our different bilingual and multilin-gual models trained on the UN datasets, alwaystesting on translation from FR to EN. The align-ments are produced by extracting the encoder-decoder attention of the last decoder layer of ourtransformer/convtransformer models.

We observe the following patterns: (i) for bilin-gual translation (Figure 4), the convtransformerhas a sharper weight distribution on the match-ing characters and words than the transformer;(ii) for multilingual translation of close languages(FR+ES→EN, Figure 5), both transformer andconvtransformer are able to preserve the wordalignments, but the alignments produced by theconvtransformer appear to be slightly less noisy;(iii) for multilingual translation of distant lan-guages (FR+ZH→EN, Figure 6), the characteralignments of the transformer become visuallymuch noisier and concentrate on a few individ-ual characters, with many word alignments dis-solving. The convtransformer character align-ments remain more spread out, and word align-

1597

ment appears to be better preserved. This is an-other indication that the convtransformer is morerobust for multilingual translation of distant lan-guages. (iv) for multilingual translation with threeinputs, where two of the three languages are close(FR+ES+ZH→EN, Figure 7), we observe a simi-lar pattern, with word alignments being better pre-served by the convtransformer.

1598

source Pour que ce cadre institutionnel soit efficace, il devra remedier aux lacunes enmatiere de reglementation et de mise en œuvre qui caracterisent a ce jour la gouver-nance dans le domaine du developpement durable.

reference For this institutional framework to be effective, it will need to fill the regulatoryand implementation deficit that has thus far characterized governance in the area ofsustainable development.

FR→EN

transformer To ensure that this institutional framework is effective, it will need to address reg-ulatory and implementation gaps that characterize governance in sustainable devel-opment.

convtransformer In order to ensure that this institutional framework is effective, it will have to ad-dress regulatory and implementation gaps that characterize governance in the areaof sustainable development.

FR+ES→EN

transformer To ensure that this institutional framework is effective, it will need to address gapsin regulatory and implementation that characterize governance in the area of sus-tainable development.

convtransformer In order to ensure that this institutional framework is effective, it will be neces-sary to address regulatory and implementation gaps that characterize governance insustainable development so far.

FR+WB→EN

transformer To ensure that this institutional framework is effective, gaps in regulatory and imple-mentation that have characterized governance in sustainable development to date.

convtransformer For this institutional framework to be effective, it will need to address gaps in reg-ulatory and implementation that characterize governance in the area of sustainabledevelopment.

FR+ES+WB→EN

transformer To ensure that this institutional framework is effective, it will need to address regu-latory and implementation gaps that are characterized by governance in the area ofsustainable development.

convtransformer If this institutional framework is to be effective, it will need to address gaps inregulatory and implementation that are characterized by governance in the area ofsustainable development.

Table 3: Example character-level translation outputs on the UN dataset, FR→EN.

1599

source Estamos convencidos de que el futuro de la humanidad en condiciones de seguridad,la coexistencia pacıfica, la tolerancia y la reconciliacion entre las naciones se veranreforzados por el reconocimiento de los hechos del pasado.

reference We strongly believe that the secure future of humanity, peaceful coexistence, toler-ance and reconciliation between nations will be reinforced by the acknowledgementof the past.

ES→EN

transformer We are convinced that the future of humanity in conditions of security, peacefulcoexistence, tolerance and reconciliation among nations will be strengthened byrecognition of the facts of the past.

convtransformer We are convinced that the future of humanity under conditions of safe, peacefulcoexistence, tolerance and reconciliation among nations will be reinforced by therecognition of the facts of the past.

FR+ES→EN

transformer We are convinced that the future of mankind under security, peaceful coexistence,tolerance and reconciliation among nations will be strengthened by the recognitionof the facts of the past.

convtransformer We are convinced that the future of humanity in safety, peaceful coexistence, toler-ance and reconciliation among nations will be reinforced by the recognition of thefacts of the past.

ES+WB→EN

transformer We are convinced that the future of humanity in safety, peaceful coexistence, tol-erance and reconciliation among nations will be strengthened by the recognition ofthe facts of the past.

convtransformer We are convinced that the future of humanity in safety, peaceful coexistence, tol-erance and reconciliation among nations will be strengthened by the recognition ofthe facts of the past.

FR+ES+WB→EN

transformer We are convinced that the future of mankind in safety, peaceful coexistence, toler-ance and reconciliation among nations will be strengthened by the recognition ofthe facts of the past.

convtransformer We are convinced that the future of mankind in security, peaceful coexistence, tol-erance and reconciliation among nations will be strengthened by the recognition ofthe facts of the past.

Table 4: Example character-level translation outputs on the UN dataset, ES→EN.

1600

source ZH 利用专家管理农场对于最大限度提高生产率和灌溉水使用效率也是重要的。source ZH tjh|et fny|pe tp|gj pei|fnrt cf|gf jb|dd bv|ya rj|ym tg|u|yx t iak|ivc|ii wgkq0|et uqt|yx

bn j tgj|s r .reference EN The use of expert farm management is also important to maximize land productivity

and efficiency in the use of irrigation water.ZH→EN

transformer The use of expert management farms is also important for maximizing productivityand irrigation use.

convtransformer The use of experts to manage farms is also important for maximizing efficiency inproductivity and irrigation water use.

FR+ZH→EN

transformer The use of expert management farms is also important for maximizing productivityand efficiency in irrigation water use.

convtransformer The use of expert management farms is also important for maximizing productivityand irrigation water efficiency.

ES+ZH→EN

transformer The use of expert farm management is also important for maximizing productivityand irrigation water use efficiency.

convtransformer The use of expert management farms to maximize efficiency in productivity andirrigation water use is also important.

FR+ES+ZH→EN

transformer The use of expert management farms is also important for maximizing productivityand irrigation water use.

convtransformer It is also important that expert management farms be used to maximize efficiencyin productivity and irrigation use.

Table 5: Example character-level translation outputs on the UN dataset, ZH→EN.

1601

(a) transformer trained on FR→EN, testing with FR as input.

(b) convtransformer trained on FR→EN, testing with FR as input.

Figure 4: Example alignments produced by character-level models trained on FR→EN.

1602

(a) transformer trained on FR+ES→EN, testing with FR as input.

(b) convtransformer trained on FR+ES→EN, testing with FR as input

Figure 5: Example alignments produced by character-level models trained on FR+ES→EN.

1603

(a) transformer FR+ZH→EN, test on FR

(b) convtransformer FR+ZH→EN, test on FR

Figure 6: Example alignments produced by character-level models trained on FR+ZH→EN.

1604

(a) transformer FR+ES+ZH→EN, test on FR

(b) convtransformer FR+ES+ZH→EN, test on FR

Figure 7: Example alignments produced by character-level models trained on FR+ES+ZH→EN.