What do Neural Machine Translation Models Learn about Morphology?
Open accessAbstract
Neural machine translation (MT) models obtain state-of-the-art performance while maintaining a simple, end-to-end architecture.However, little is known about what these models learn about source and target languages during the training process.In this work, we analyze the representations learned by neural MT models at various levels of granularity and empirically evaluate the quality of the representations for learning morphology through extrinsic part-of-speech and morphological tagging tasks.We conduct a thorough investigation along several parameters: word-based vs. character-based representations, depth of the encoding layer, the identity of the target language, and encoder vs. decoder representations.Our data-driven, quantitative evaluation sheds light on important aspects in the neural MT system and its ability to capture word structure.1
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For POS tagging, accuracies using the first biLM layer are higher than the top layer.
“However, unlike WSD, accuracies using the first biLM layer are higher than the top layer, consistent with results from deep biL-STMs in multi-task training (Søgaard and Goldberg, 2016;Hashimoto et al., 2017) and MT (Belinkov et al., 2017).”
From Deep Contextualized Word Representations (2018) · cites this paper as Belinkov et al., 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Distributed representations, simple recurrent networks, and grammatical structure
Elman, J. L. (1991) · Machine Learning · doi:10.1007/bf00114844 · Cited by 1,354 · see citing works
Flagged first, then by citations
Research works printed but not linked: 39
- Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2016. Fine-grained Anal- ysis of Sentence Embeddings Using Auxiliary Pre- diction Tasks. arXiv preprint arXiv:1608.04207 . · arXiv
- Ibrahim Badr, Rabih Zbib, and James Glass. 2008. Segmentation for English-to-Arabic Statisti- cal Machine Translation. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Lan- guage Technologies: Short Papers. Colum- bus, Ohio, HLT-Short '08, pages 153-156. http://dl.acm.org/citation.cfm?id=1557690.1557732.
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv preprint arXiv:1409.0473 . · arXiv
- Yonatan Belinkov and James Glass. 2016. Large-Scale Machine Translation between Arabic and Hebrew: Available Corpora and Initial Results. In Proceed- ings of the Workshop on Semitic Machine Trans- lation. Association for Computational Linguistics, Austin, Texas, pages 7-12.
- Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016. Neural versus Phrase- Based Machine Translation Quality: a Case Study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics, Austin, Texas, pages 257-267. https://aclweb.org/anthology/D16- 1025.
- Mauro Cettolo. 2016. An Arabic-Hebrew parallel cor- pus of TED talks. In Proceedings of the Work- shop on Semitic Machine Translation. Association for Computational Linguistics, Austin, Texas, pages 1-6.
- Mauro Cettolo, Christian Girardi, and Marcello Fed- erico. 2012. WIT 3 : Web Inventory of Transcribed and Translated Talks. In Proceedings of the 16 th Conference of the European Association for Ma- chine Translation (EAMT). Trento, Italy, pages 261- 268.
- Marta R. Costa-jussà and José A. R. Fonollosa. 2016. Character-based Neural Machine Translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol- ume 2: Short Papers). Association for Computa- tional Linguistics, Berlin, Germany, pages 357-361. http://anthology.aclweb.org/P16-2058.
- Nadir Durrani, Philipp Koehn, Helmut Schmid, and Alexander Fraser. 2014. Investigating the Usefulness of Generalized Word Representations in SMT. In Proceedings of COLING 2014, the 25th International Conference on Compu- tational Linguistics: Technical Papers. Dublin City University and Association for Computa- tional Linguistics, Dublin, Ireland, pages 421-432. http://www.aclweb.org/anthology/C14-1041.
- Nadir Durrani, Hassan Sajjad, Alexander Fraser, and Helmut Schmid. 2010. Hindi-to-Urdu Machine Translation through Transliteration. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Compu- tational Linguistics, Uppsala, Sweden, pages 465- 474. http://www.aclweb.org/anthology/P10-1048.
- Lieke Gelderloos and Grzegorz Chrupała. 2016. From phonemes to images: levels of representation in a recurrent neural model of visually-grounded lan- guage learning. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. The COLING 2016 Organizing Committee, Osaka, Japan, pages 1309- 1319. http://aclweb.org/anthology/C16-1124.
- Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9(8):1735-1780.
- Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2016. Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Transla- tion. arXiv preprint arXiv:1611.04558 . · arXiv
- Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the Limits of Language Modeling. arXiv preprint arXiv:1602.02410 . · arXiv
- Ákos Kádár, Grzegorz Chrupała, and Afra Alishahi. 2016. Representation of linguistic form and func- tion in recurrent neural networks. arXiv preprint arXiv:1602.08952 . · arXiv
- Andrej Karpathy, Justin Johnson, and Fei-Fei Li. 2015. Visualizing and Understanding Recurrent Networks. arXiv preprint arXiv:1506.02078 . · arXiv
- Yoon Kim, Yacine Jernite, David Sontag, and Alexan- der M Rush. 2015. Character-aware Neural Lan- guage Models. arXiv preprint arXiv:1508.06615 . · arXiv
- Philipp Koehn and Hieu Hoang. 2007. Fac- tored Translation Models. In Proceedings of the 2007 Joint Conference on Empirical Meth- ods in Natural Language Processing and Com- putational Natural Language Learning (EMNLP- CoNLL). Association for Computational Linguis- tics, Prague, Czech Republic, pages 868-876. http://www.aclweb.org/anthology/D07-1091.
- Philipp Koehn and Kevin Knight. 2003. Empirical Methods for Compound Splitting. In 10th Con- ference of the European Chapter of the Associa- tion for Computational Linguistics. pages 187-194. http://www.aclweb.org/anthology/E03-1076.
- Arne Köhn. 2015. What's in an Embedding? An- alyzing Word Embeddings through Multilingual Evaluation. In Proceedings of the 2015 Con- ference on Empirical Methods in Natural Lan- guage Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 2067-2073. http://aclweb.org/anthology/D15-1246.
- Minh-Thang Luong and Christopher D. Manning. 2015. Stanford Neural Machine Translation Sys- tems for Spoken Language Domains. In Proceed- ings of the International Workshop on Spoken Lan- guage Translation. Da Nang, Vietnam.
- Minh-Thang Luong and D. Christopher Manning. 2016. Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Mod- els. In Proceedings of the 54th Annual Meet- ing of the Association for Computational Lin- guistics (Volume 1: Long Papers). Association for Computational Linguistics, pages 1054-1063. https://doi.org/10.18653/v1/P16-1100. · DOI
- Minh-Thang Luong, Preslav Nakov, and Min-Yen Kan. 2010. A Hybrid Morpheme-Word Representation for Machine Translation of Morphologically Rich Languages. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Pro- cessing. Association for Computational Linguistics, pages 148-157. http://aclweb.org/anthology/D10- 1015.
- Thomas Mueller, Helmut Schmid, and Hinrich Schütze. 2013. Efficient Higher-Order CRFs for Morphological Tagging. In Proceedings of the 2013 Conference on Empirical Methods in Natural Lan- guage Processing. Association for Computational Linguistics, Seattle, Washington, USA, pages 322- 332. http://www.aclweb.org/anthology/D13-1032.
- Preslav Nakov and Jörg Tiedemann. 2012. Com- bining Word-Level and Character-Level Models for Machine Translation Between Closely-Related Lan- guages. In Proceedings of the 50th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers). Jeju, Korea, ACL '12, pages 301-305. http://aclweb.org/anthology/P12- 2059.
- Sonja Nieflen and Hermann Ney. 2000. Improv- ing SMT quality with morpho-syntactic analysis. In COLING 2000 Volume 2: The 18th Interna- tional Conference on Computational Linguistics. http://www.aclweb.org/anthology/C00-2162.
- Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14). Reykjavik, Iceland, pages 1094-1101.
- Peng Qian, Xipeng Qiu, and Xuanjing Huang. 2016a. Analyzing Linguistic Knowledge in Se- quential Model of Sentence. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing. Association for Compu- tational Linguistics, Austin, Texas, pages 826-835. https://aclweb.org/anthology/D16-1079.
- Peng Qian, Xipeng Qiu, and Xuanjing Huang. 2016b. Investigating Language Universal and Spe- cific Properties in Word Embeddings. In Pro- ceedings of the 54th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 1478-1488. http://www.aclweb.org/anthology/P16-1140.
- Adwait Ratnaparkhi. 1998. Maximum Entropy Models for Natural Language Ambiguity Resolution. Ph.D. thesis, University of Pennsylvania, Philadelphia, PA.
- Hassan Sajjad, Fahim Dalvi, Nadir Durrani, Ahmed Abdelali, Yonatan Belinkov, and Stephan Vogel. 2017. Challenging Language-Dependent Segmenta- tion for Arabic: An Application to Machine Trans- lation and Part-of-Speech Tagging. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Association for Compu- tational Linguistics, Vancouver, Canada.
- Helmut Schmid. 1994. Part-of-Speech Tagging with Neural Networks. In Proceedings of the 15th Inter- national Conference on Computational Linguistics (Coling 1994). Coling 1994 Organizing Committee, Kyoto, Japan, pages 172-176.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceed- ings of the 54th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 1715-1725. http://www.aclweb.org/anthology/P16-1162.
- Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does String-Based Neural MT Learn Source Syn- tax? In Proceedings of the 2016 Con- ference on Empirical Methods in Natural Lan- guage Processing. Association for Computational Linguistics, Austin, Texas, pages 1526-1534. https://aclweb.org/anthology/D16-1159.
- Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to Sequence Learning with Neural Net- works. In Advances in neural information process- ing systems. pages 3104-3112.
- Antonio Toral and Víctor M. Sánchez-Cartagena. 2017. A Multifaceted Evaluation of Neural ver- sus Phrase-Based Machine Translation for 9 Lan- guage Directions. In Proceedings of the 15th Conference of the European Chapter of the As- sociation for Computational Linguistics: Volume 1, Long Papers. Association for Computational Linguistics, Valencia, Spain, pages 1063-1073. http://aclweb.org/anthology/E17-1100.
- Ekaterina Vylomova, Trevor Cohn, Xuanli He, and Gholamreza Haffari. 2016. Word Representa- tion Models for Morphologically Rich Languages in Neural Machine Translation. arXiv preprint arXiv:1606.04217 . · arXiv
- Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv preprint arXiv:1609.08144 . · arXiv
- Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. 2016. Deep Recurrent Mod- els with Fast-Forward Connections for Neural Machine Translation. Transactions of the Asso- ciation for Computational Linguistics 4:371-383. https://transacl.org/ojs/index.php/tacl/article/view/863.
Cited by
Sorted by citations
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591 · see citing works
How Can We Know What Language Models Know?
Jiang, Z., Xu, F. F., Araki, J., & Neubig, G. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00324 · Cited by 1,002 · see citing works
Dissecting Contextual Word Embeddings: Architecture and Representation
Peters, M., Neumann, M., Zettlemoyer, L., & Yih, W.-T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1179 · Cited by 434 · see citing works
Designing and Interpreting Probes with Control Tasks
Hewitt, J., & Liang, P. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1275 · Cited by 393 · see citing works
Direct Fit to Nature: An Evolutionary Perspective on Biological and Artificial Neural Networks
Hasson, U., Nastase, S. A., & Goldstein, A. (2020) · Neuron · doi:10.1016/j.neuron.2019.12.002 · Cited by 387 · see citing works
Factual Probing Is [MASK]: Learning vs. Learning to Recall
Zhong, Z., Friedman, D., & Chen, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.398 · Cited by 274 · see citing works
Locating and Editing Factual Associations in GPT
Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022) · arXiv · doi:10.48550/arxiv.2202.05262 · Cited by 230 · see citing works
Self-citationMeasuring and improving consistency in pretrained language models
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00410 · Cited by 220 · see citing works
A Systematic Assessment of Syntactic Generalization in Neural Language Models
Hu, J., Gauthier, J., Qian, P., Wilcox, E., & Levy, R. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.158 · Cited by 134 · see citing works
Hosseini, E. A., Schrimpf, M., Zhang, Y., Bowman, S., Zaslavsky, N., & Fedorenko, E. (2024) · Neurobiology of Language · doi:10.1162/nol_a_00137 · Cited by 63 · see citing works
Unlearn What You Want to Forget: Efficient Unlearning for LLMs
Chen, J., & Yang, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.738 · Cited by 52 · see citing works
Roles and Utilization of Attention Heads in Transformer-based Neural Language Models
Jo, J.-Y., & Myaeng, S.-H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.311 · Cited by 33 · see citing works
AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders
Wu, Z., Arora, A., Geiger, A., Wang, Z., Huang, J., Jurafsky, D., Manning, C. D., & Potts, C. (2025) · arXiv · doi:10.48550/arxiv.2501.17148
“Past work in LM interpretability trained probes (Conneau et al., 2018; Hewitt and Manning, 2019; Belinkov et al., 2017) to measure to what extent LM representations encode properties of interest, e.g. linguistic features.”
- Authors
- Belinkov, Y., Durrani, N., Dalvi, F., Sajjad, H., & Glass, J.
- Affiliations
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Hamad bin Khalifa University (QA)
- Venue
- Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2017
- Pages
- 861-872
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2605717780
- Cited by
- 307
- References
- 2
- Citations by year
- 2026: 8 · 2025: 5 · 2024: 13 · 2023: 30 · 2022: 39 · 2021: 45 · 2020: 62 · 2019: 49 · 2018: 50 · 2017: 6
- FWCI
- 20.95
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Hamad Bin Khalifa University
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- neural machine translation, morphology (linguistics), learned representations, part-of-speech tagging, morphological tagging, word structure, multilingual translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
