Learning Deep Transformer Models for Machine Translation
Open accessThe DOI matches its Crossref record; 35 of the 35 references it cites were found in a registry or an index.
Abstract
Transformer is the state-of-the-art model in recent machine translation evaluations. Two strands of research are promising to improve models of this kind: the first uses wide networks (a.k.a. Transformer-Big) and has been the de facto standard for the development of the Transformer system, and the other uses deeper language representation but faces the difficulty arising from learning deep networks. Here, we continue the line of research on the latter. We claim that a truly deep Transformer model can surpass the Transformer-Big counterpart by 1) proper use of layer normalization and 2) a novel way of passing the combination of previous layers to the next. On WMT'16 English- German, NIST OpenMT'12 Chinese-English and larger WMT'18 Chinese-English tasks, our deep system (30/25-layer encoder) outperforms the shallow Transformer-Big/Base baseline (6-layer encoder) by 0.4-2.4 BLEU points. As another bonus, the deep model is 1.6X smaller in size and 3X faster in training than Transformer-Big.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Identity mappings in deep residual networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46493-0_38 · Cited by 10,741
Identity Mappings in Deep Residual Networks
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · arXiv · doi:10.48550/arxiv.1603.05027 · Cited by 10,665
Effective Approaches to Attention-based Neural Machine Translation
Luong, T., Pham, H., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1166 · Cited by 8,382
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319
On the difficulty of training recurrent neural networks
Pascanu, R., Mikolov, T., & Bengio, Y. (2012) · arXiv · doi:10.48550/arxiv.1211.5063 · Cited by 4,310
Numerical Methods for Ordinary Differential Equations
Butcher, J. C. (2016) · Wiley eBooks · doi:10.1002/9781119121534 · Cited by 2,986
Self-Attention with Relative Position Representations
Shaw, P., Uszkoreit, J., & Vaswani, A. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers) · doi:10.18653/v1/n18-2074 · Cited by 2,505
Computer Methods for Ordinary Differential Equations and Differential-Algebraic Equations
Ascher, U. M., & Petzold, L. R. (1998) · doi:10.1137/1.9781611971392 · Cited by 2,403 · cited 4×
“• Inspired by the linear multi-step method in numerical analysis (Ascher and Petzold, 1998), we propose an approach based on dynamic linear combination of layers (D LCL ) to memorizing the features extracted from all preceding layers.”
MethodInfluentialOpenNMT: Open-Source Toolkit for Neural Machine Translation
Klein, G., Kim, Y., Deng, Y., Senellart, J., & Rush, A. (2017) · Proceedings of ACL 2017, System Demonstrations · doi:10.18653/v1/p17-4012 · Cited by 1,749
Numerical Methods for Ordinary Differential Equations
Butcher, J. C. (2008) · Wiley eBooks · doi:10.1002/9780470753767 · Cited by 1,614
Scaling Neural Machine Translation
Ott, M., Edunov, S., Grangier, D., & Auli, M. (2018) · Proceedings of the Third Conference on Machine Translation: Research Papers · doi:10.18653/v1/w18-6301 · Cited by 544
Lu, Y., Zhong, A., Li, Q., & Dong, B. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.10121 · Cited by 484
Massive Exploration of Neural Machine Translation Architectures
Britz, D., Goldie, A., Luong, M.-T., & Le, Q. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1151 · Cited by 482
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation
Chen, M. X., Firat, O., Bapna, A., Johnson, M., Macherey, W., Foster, G., Jones, L., Schuster, M., Shazeer, N., Parmar, N., et al. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1008 · Cited by 418
A Deep Architecture for Matching Short Texts
Lu, Z., & Li, H. (2013) · Cited by 225
Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation
Zhou, J., Cao, Y., Wang, X., Li, P., & Xu, W. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00105 · Cited by 216
Training Deeper Neural Machine Translation Models with Transparent Attention
Bapna, A., Chen, M., Fırat, O., Cao, Y. J., & Wu, Y. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1338 · Cited by 165
Layer-Wise Coordination between Encoder and Decoder for Neural Machine Translation
He, T., Tan, X., Xia, Y., He, D., Qin, T., Chen, Z., & Liu, T. (2018) · neural information processing systems · Cited by 102
Exploiting Deep Representations for Neural Machine Translation
Dou, Z.-Y., Tu, Z., Wang, X., Shi, S., & Zhang, T. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1457 · Cited by 91
How Much Attention Do You Need? A Granular Analysis of Neural Machine Translation Architectures
Domhan, T. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1167 · Cited by 74
NiuTrans: An Open Source Toolkit for Phrase-based and Syntax-based Machine Translation
Xiao, T., Zhu, J., Zhang, H., & Li, Q. (2012) · Cited by 71
Self-citationCharacter-Level Language Modeling with Deeper Self-Attention
Al-Rfou, R., Choe, D., Constant, N., Guo, M., & Jones, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33013159 · Cited by 56
Deep Neural Machine Translation with Linear Associative Unit
Wang, M., Lu, Z., Zhou, J., & Liu, Q. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1013 · Cited by 55
Multi-layer Representation Fusion for Neural Machine TranslationPreprint
Wang, Q., Li, F., Xiao, T., Li, Y., Li, Y., & Zhu, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.06714 · Cited by 48
Self-citationThe NiuTrans Machine Translation System for WMT18
Wang, Q., Li, B., Liu, J., Jiang, B., Zhang, Z., Li, Y., Lin, Y., Xiao, T., & Zhu, J. (2018) · doi:10.18653/v1/w18-6430 · Cited by 12
Self-citationExact optimal values of step-size coefficients for boundedness of linear multistep methods
Lóczi, L. (2018) · ELTE Digital Institutional Repository (EDIT) (Eötvös Loránd University) · doi:10.1007/s11075-017-0354-5 · Cited by 3
Cited by
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
Conformer: Convolution-augmented Transformer for Speech Recognition
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. (2020) · Interspeech 2020 · doi:10.21437/interspeech.2020-3015 · Cited by 3,150
Conformer: Convolution-augmented Transformer for Speech Recognition
Gulati, A., Qin, J., Chiu, C.-C., Parmar, N., Zhang, Y., Yu, J., Han, W., Wang, S., Zhang, Z., Wu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.08100 · Cited by 3,150
Pre-Trained Image Processing Transformer
Chen, H., Wang, Y., Guo, T., Xu, C., Deng, Y., Liu, Z., Ma, S., Xu, C., Xu, C., & Gao, W. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01212 · Cited by 2,308
“The seminal work Transformers [61] have been widely used in many natural language processing (NLP) tasks, such as translation [64] and questionanswering [58].”
Method
Metadata
- Authors
- Wang, Q., Li, B., Xiao, T., Zhu, J., Li, C., Wong, D. F., & Chao, L. S.
- Affiliations
- Northeastern University (CN)
- University of Macau (MO)
- Kingsoft (China) (CN)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 1810-1822
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2963542740
- Cited by
- 673
- References
- 35
- Citations by year
- 2026: 27 · 2025: 83 · 2024: 102 · 2023: 157 · 2022: 98 · 2021: 119 · 2020: 79 · 2019: 8
- FWCI
- 29.08
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (61672555, 61432013, 61876035, 045/2017/AFJ, 61732005)
- Universidade de Macau (MYRG2017-00087-FST)
- Fundo para o Desenvolvimento das Ciências e da Tecnologia (MYRG2017-00087-FST, 045/2017/AFJ)
- Fundamental Research Funds for the Central Universities (N181602013)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- machine translation, layer normalization, English-German translation, Chinese-English translation, model size reduction, training efficiency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.