The DOI matches its Crossref record; 72 of the 72 references it cites were found in a registry or an index.
A Survey of Multilingual Neural Machine Translation
Open accessAbstract
We present a survey on multilingual neural machine translation (MNMT), which has gained a lot of traction in recent years. MNMT has been useful in improving translation quality as a result of translation knowledge transfer (transfer learning). MNMT is more promising and interesting than its statistical machine translation counterpart, because end-to-end modeling and distributed representations open new avenues for research on machine translation. Many approaches have been proposed to exploit multilingual parallel corpora for improving translation quality. However, the lack of a comprehensive survey makes it difficult to determine which approaches are promising and, hence, deserve further exploration. In this article, we present an in-depth survey of existing literature on MNMT. We first categorize various approaches based on their central use-case and then further categorize them based on resource scenarios, underlying modeling principles, core-issues, and challenges. Wherever possible, we address the strengths and weaknesses of several techniques by comparing them with each other. We also discuss the future directions for MNMT. This article is aimed towards both beginners and experts in NMT. We hope this article will serve as a starting point as well as a source of new ideas for researchers and engineers interested in MNMT.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches
Cho, K., van Merrienboer, B., Bahdanau, D., & Bengio, Y. (2014) · Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation · doi:10.3115/v1/w14-4012 · Cited by 7,099
Moses: open source toolkit for statistical machine translation
Koehn, P., Zens, R., Dyer, C., Bojar, O., Constantin, A., Herbst, E., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., et al. (2007) · Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions - ACL '07 · doi:10.3115/1557769.1557821 · Cited by 4,735
Neural Architecture Search with Reinforcement Learning
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 3,974
Kudo, T., & Richardson, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/d18-2012 · Cited by 2,869
Convolutional Sequence to Sequence Learning
Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017) · arXiv · doi:10.48550/arxiv.1705.03122 · Cited by 2,850 · cited 4×
“Recurrent neural network (RNN) [8], convolutional neural network (CNN) [52] and self-attention/feed-forward network (SA/FFN) [148] architectures are most commonly used approaches based on this paradigm.”
BackgroundMethodInfluentialA Generalized Solution of the Orthogonal Procrustes Problem
Schönemann, P. H. (1966) · Psychometrika · doi:10.1007/bf02289451 · Cited by 2,040
Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
Johnson, M., Schuster, M., Le, Q. V., Krikun, M., Wu, Y., Chen, Z., Thorat, N., Viégas, F., Wattenberg, M., Corrado, G., et al. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00065 · Cited by 1,747
Cross-lingual Language Model Pretraining
Lample, G., & Conneau, A. (2019) · arXiv · doi:10.48550/arxiv.1901.07291 · Cited by 1,586
How Multilingual is Multilingual BERT?
Pires, T., Schlinger, E., & Garrette, D. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1493 · Cited by 1,351
Domain-adversarial training of neural networks
GaninYaroslav, UstinovaEvgeniya, AjakanHana, GermainPascal, LarochelleHugo, LavioletteFrançois, MarchandMario, & LempitskyVictor (2016) · Journal of Machine Learning Research · doi:10.5555/2946645.2946704 · Cited by 1,201
Statistical Phrase-Based Translation
Koehn, P., Och, F. J., & Marcu, D. (2003) · doi:10.21236/ada461156 · Cited by 965
Neural Machine Translation of Rare Words with Subword Units
Sennrich, R., Haddow, B., & Birch, A. (2015) · arXiv · doi:10.48550/arxiv.1508.07909 · Cited by 950
XNLI: Evaluating Cross-lingual Sentence Representations
Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S., Schwenk, H., & Stoyanov, V. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1269 · Cited by 915
Japanese and Korean voice search
Schuster, M., & Nakajima, K. (2012) · 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2012.6289079 · Cited by 862
Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond
Artetxe, M., & Schwenk, H. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00288 · Cited by 791
Transfer Learning for Low-Resource Neural Machine Translation
Zoph, B., Yuret, D., May, J., & Knight, K. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1163 · Cited by 788
Clause restructuring for statistical machine translation
Collins, M., Koehn, P., & Kučerová, I. (2005) · Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics - ACL '05 · doi:10.3115/1219840.1219906 · Cited by 585
Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanism
Firat, O., Cho, K., & Bengio, Y. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1101 · Cited by 562
Massively Multilingual Neural Machine Translation
Aharoni, R., Johnson, M., & Firat, O. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1388 · Cited by 402
Fully Character-Level Neural Machine Translation without Explicit Segmentation
Lee, J., Cho, K., & Hofmann, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00067 · Cited by 394
Learning principled bilingual mappings of word embeddings while preserving monolingual invariance
Artetxe, M., Labaka, G., & Agirre, E. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1250 · Cited by 369
Meta-Learning for Low-Resource Neural Machine Translation
Gu, J., Wang, Y., Chen, Y., Li, V. O. K., & Cho, K. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1398 · Cited by 325
Multi-Source Neural Translation
Zoph, B., & Knight, K. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1004 · Cited by 300
Simple, Scalable Adaptation for Neural Machine Translation
Bapna, A., & Firat, O. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1165 · Cited by 297
Improving a statistical MT system with automatically learned rewrite patterns
Xia, F., & McCord, M. (2004) · Proceedings of the 20th international conference on Computational Linguistics - COLING '04 · doi:10.3115/1220355.1220428 · Cited by 266
Zero-Resource Translation with Multi-Lingual Neural Machine Translation
Firat, O., Sankaran, B., Al-Onaizan, Y., Yarman Vural, F. T., & Cho, K. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1026 · Cited by 261
On the Limitations of Unsupervised Bilingual Dictionary Induction
Søgaard, A., Ruder, S., & Vulić, I. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1072 · Cited by 254
Unsupervised Pretraining for Sequence to Sequence Learning
Ramachandran, P., Liu, P., & Le, Q. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1039 · Cited by 251
An Empirical Comparison of Domain Adaptation Methods for Neural Machine Translation
Chu, C., Dabre, R., & Kurohashi, S. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p17-2061 · Cited by 215
Self-citationGuzmán, F., Chen, P.-J., Ott, M., Pino, J., Lample, G., Koehn, P., Chaudhary, V., & Ranzato, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1632 · Cited by 203
Pivot language approach for phrase-based statistical machine translation
Wu, H., & Wang, H. (2007) · Machine Translation · doi:10.1007/s10590-008-9041-6 · Cited by 199
A massively parallel corpus: the Bible in 100 languages
Christodouloupoulos, C., & Steedman, M. (2015) · Language Resources and Evaluation · doi:10.1007/s10579-014-9287-y · Cited by 190
Rapid Adaptation of Neural Machine Translation to New Languages
Neubig, G., & Hu, J. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1103 · Cited by 185
Contextual Parameter Generation for Universal Neural Machine Translation
Platanios, E. A., Sachan, M., Neubig, G., & Mitchell, T. M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1039 · Cited by 147
Multi-Source Cross-Lingual Model Transfer: Learning What to Share
Chen, X., Awadallah, A. H., Hassan, H., Wang, W., & Cardie, C. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1299 · Cited by 139
Multilingual Neural Machine Translation with Language Clustering
Tan, X., Chen, J.-L., He, D., Xia, Y., Qin, T., & Liu, T.-Y. (2019) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d19-1089 · Cited by 138
A neural interlingua for multilingual machine translation
Lu, Y., Keung, P., Ladhak, F., Bhardwaj, V., Zhang, S., & Sun, J. (2018) · Conference on Machine Translation · doi:10.18653/v1/w18-6309 · Cited by 136
Instance Weighting for Neural Machine Translation Domain Adaptation
Wang, R., Utiyama, M., Liu, L., Chen, K., & Sumita, E. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1155 · Cited by 130
Learning Language Representations for Typology Prediction
Malaviya, C., Neubig, G., & Littell, P. (2017) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d17-1268 · Cited by 121
Parameter Sharing Methods for Multilingual Self-Attentional Translation Models
Sachan, D. S., & Neubig, G. (2018) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w18-6327 · Cited by 119
Investigating Multilingual NMT Representations at Scale
Kudugunta, S., Bapna, A., Caswell, I. R., & Fırat, O. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1167 · Cited by 104
Sentence Embedding for Neural Machine Translation Domain Adaptation
Wang, R., Finch, A., Utiyama, M., & Sumita, E. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-2089 · Cited by 90
Effective Cross-lingual Transfer of Neural Machine Translation Models without Shared Vocabularies
Kim, Y., Gao, Y., & Ney, H. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1120 · Cited by 90
Vilar, D., Peter, J.-T., & Ney, H. (2007) · Proceedings of the Second Workshop on Statistical Machine Translation - StatMT '07 · doi:10.3115/1626355.1626360 · Cited by 87
Scheduled Multi-Task Learning: From Syntax to Translation
Kiperwasser, E., & Ballesteros, M. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00017 · Cited by 80
Three Strategies to Improve One-to-Many Multilingual Translation
Wang, Y., Zhang, J., Zhai, F., Xu, J., & Zong, C. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1326 · Cited by 77
Pivot-based Transfer Learning for Neural Machine Translation between Non-English Languages
Kim, Y., Petrov, P., Petrushkov, P., Khadivi, S., & Ney, H. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1080 · Cited by 77
Neural System Combination for Machine Translation
Zhou, L., Hu, W., Zhang, J., & Zong, C. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-2060 · Cited by 74
España-Bonet, C., Varga, A. C., Barrón‐Cedeño, A., & van Genabith, J. (2017) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2017.2764273 · Cited by 73
Nakov, P., & Ng, H. T. (2009) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/1699648.1699682 · Cited by 72
Continuous multilinguality with language vectors
Östling, R., & Tiedemann, J. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-2102 · Cited by 72
Learning Multilingual Word Embeddings in Latent Metric Space: A Geometric Approach
Jawanpuria, P., Balgovind, A., Kunchukuttan, A., & Mishra, B. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00257 · Cited by 72
Self-citationAn overview of the European Union’s highly multilingual parallel corpora
Steinberger, R., Ebrahim, M., Poulis, A., Benitez, M. T. C., Schlüter, P., Przybyszewski, M., & Gilbro, S. (2014) · Language Resources and Evaluation · doi:10.1007/s10579-014-9277-0 · Cited by 66
Improving Zero-shot Translation with Language-Independent Constraints
Pham, N.-Q., Niehues, J., Ha, T.-L., & Waibel, A. H. (2019) · Conference on Machine Translation · doi:10.18653/v1/w19-5202 · Cited by 65
Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation
Siddhant, A., Johnson, M., Tsai, H., Ari, N., Riesa, J., Bapna, A., Fırat, O., & Raman, K. (2020) · AAAI Publications (The Association for the Advancement of Artificial Intelligence (AAAI)) · doi:10.1609/aaai.v34i05.6414 · Cited by 64
Consistency by Agreement in Zero-Shot Neural Machine Translation
Al-Shedivat, M., & Parikh, A. P. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1121 · Cited by 60
Dabre, R., Fujita, A., & Chu, C. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1146 · Cited by 56
Self-citationCross-Lingual Pre-Training Based Transfer for Zero-Shot Neural Machine Translation
Ji, B., Zhang, Z., Duan, X., Zhang, M., Chen, B., & Luo, W. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5341 · Cited by 56
Adaptive Knowledge Sharing in Multi-Task Learning: Improving Low-Resource Neural Machine Translation
Zaremoodi, P., Buntine, W., & Haffari, G. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-2104 · Cited by 51
Word lattices for multi-source translation
Schroeder, J., Cohn, T., & Koehn, P. (2009) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.3115/1609067.1609147 · Cited by 41
Multi-Source Neural Machine Translation with Missing Data
Nishimura, Y., Sudoh, K., Neubig, G., et al. (2018) · doi:10.18653/v1/w18-2711 · Cited by 41
Multilingual Unsupervised NMT using Shared Encoder and Language-Specific Decoders
Sen, S., Gupta, K. K., Ekbal, A., & Bhattacharyya, P. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1297 · Cited by 40
A Compact and Language-Sensitive Multilingual Translation Method
Wang, Y., Zhou, L., Zhang, J., Zhai, F., Xu, J., & Zong, C. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1117 · Cited by 40
Murthy, R., Kunchukuttan, A., & Bhattacharyya, P. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1387 · Cited by 37
Self-citationOrthographic Syllable as basic unit for SMT between Related Languages
Kunchukuttan, A., & Bhattacharyya, P. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1196 · Cited by 30
Self-citationLearning variable length units for SMT between related languages via Byte Pair Encoding
Kunchukuttan, A., & Bhattacharyya, P. (2017) · Workshop on Subword and Character Level Models in NLP (SCLeM) · doi:10.18653/v1/w17-4102 · Cited by 29
Self-citationMulti-Round Transfer Learning for Low-Resource NMT Using Multiple High-Resource Languages
Maimaiti, M., Liu, Y., Luan, H., & Sun, M. (2019) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3314945 · Cited by 28
Zero-Resource Neural Machine Translation with Monolingual Pivot Data
Currey, A., & Heafield, K. (2019) · Workshop on Neural Generation and Translation (WNGT) · doi:10.18653/v1/d19-5610 · Cited by 20
Evaluating the Supervised and Zero-shot Performance of Multi-lingual Translation Models
Hokamp, C., Glover, J., & Ghalandari, D. G. (2019) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w19-5319 · Cited by 13
Cited by
Structured information extraction from scientific text with large language models
Dagdelen, J., Dunn, A., Lee, S., Walker, N., Rosen, A. S., Ceder, G., Persson, K. A., & Jain, A. (2024) · Nature Communications · doi:10.1038/s41467-024-45563-x · Cited by 771
Neural Machine Translation for Low-resource Languages: A Survey
Ranathunga, S., Lee, E.-S. A., Prifti Skenduli, M., Shekhar, R., Alam, M., & Kaur, R. (2023) · ACM Computing Surveys · doi:10.1145/3567592 · Cited by 313
The Political Biases of ChatGPT
Rozado, D. (2023) · Social Sciences · doi:10.3390/socsci12030148 · Cited by 281
Overview and challenges of machine translation for contextually appropriate translations
Naveen, P., & Trojovský, P. (2024) · iScience · doi:10.1016/j.isci.2024.110878 · Cited by 103
The Threat of Offensive AI to Organizations
Mirsky, Y., Demontis, A., Kotak, J., Shankar, R., Gelei, D., Yang, L., Zhang, X., Pintor, M., Lee, W., Elovici, Y., et al. (2023) · Computers & Security · doi:10.1016/j.cose.2022.103006 · Cited by 97
Metadata
- Authors
- Dabre, R., Chu, C., & Kunchukuttan, A.
- Venue
- ACM Computing Surveys
- Published
- 2020-09-28
- Volume
- 53
- Issue
- 5
- Pages
- 1-38
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W2945415576
- Cited by
- 294
- References
- 72
- FWCI
- 14.49
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.