Analyzing Uncertainty in Neural Machine Translation
Open accessAbstract
Machine translation is a popular test bed for research in neural sequence-to-sequence models but despite much recent research, there is still a lack of understanding of these models. Practitioners report performance degradation with large beams, the under-estimation of rare words and a lack of diversity in the final translations. Our study relates some of these issues to the inherent uncertainty of the task, due to the existence of multiple valid translations for a single source sentence, and to the extrinsic uncertainty caused by noisy training data. We propose tools and metrics to assess how uncertainty in the data is captured by the model distribution and how it affects search strategies that generate translations. Our results show that search works remarkably well but that models tend to spread too much probability mass over the hypothesis space. Next, we propose tools to assess model calibration and show how to easily fix some shortcomings of current models. As part of this study, we release multiple human reference translations for two popular benchmarks.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
High certainty does not imply truthfulness.
“However, high certainty does not mean truthful (Ott et al., 2018;Xiao & Wang, 2021;Kadavath et al., 2022)”
From CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023) · cites this paper as Ott et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 30 · Cited by 140
Research works printed but not linked: 27
- Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 , 2014. · arXiv
- Bentivogli, L., Bisazza, A., Cettolo, M., and Federico, M. Neural versus phrase-based machine translation quality: a case study. In EMNLP , 2016.
- Blunsom, P., Cohn, T., and Osborne, M. A discriminative latent variable model for statistical machine translation. In ACL-HLT , 2008.
- Cho, K. Noisy parallel approximate decoding for conditional recurrent language model. arXiv:1605.03835 , 2016. · arXiv
- Dreyer, M. and Marcu, D. Hyter: Meaning-equivalent semantics for translation evaluation. In ACL , 2012.
- Edunov, S., Ott, M., Auli, M., Grangier, D., et al. Classical structured prediction losses for sequence to sequence learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , volume 1, pp. 355-364, 2018.
- Galley, M., Brockett, C., Sordoni, A., Ji, Y ., Auli, M., Quirk, C., Mitchell, M., Gao, J., and Dolan, B. deltableu: A discriminative metric for generation tasks with intrinsically diverse targets. arXiv preprint arXiv:1506.06863 , 2015. · arXiv
- Gimpel, K., Batra, D., Dyer, C., and Shakhnarovich, G. A systematic exploration of diversity in machine translation. In Conference on Empirical Methods in Natural Language Processing , 2013.
- Isabelle, P., Cherry, C., and Foster, G. A challenge set approach to evaluating machine translation. arXiv:1704.07431 , 2017. · arXiv
- Koehn, P. and Knowles, R. Six challenges for neural machine translation. arXiv:1706.03872 , 2017. · arXiv
- Kuleshov, V. and Liang, P. Calibrated structured prediction. In Neural Information Processing Systems , 2015.
- Li, J. and Jurafsky, D. Mutual information and diverse decoding improve neural machine translation. arXiv:1601.00372 , 2016. · arXiv
- Lin, C.-Y. and Och, F. J. Orange: a method for evaluating automatic evaluation metrics for machine translation. In Proc. of COLING , 2004.
- Luong, M.-T., Pham, H., and Manning, C. D. Effective approaches to attention-based neural machine translation. In Proc. of EMNLP , 2015. URL http://nlp. stanford.edu/projects/nmt .
- Nguyen, K. and O'Connor, B. Posterior calibration and exploratory analysis for natural language processing models. In Proc. of EMNLP , 2015.
- Niehues, J., Cho, E., Ha, T.-L., and Waibel, A. Analyzing neural mt search and model performance. arXiv:1708.00563 , 2017. · arXiv
- Ranzato, M., Chopra, S., Auli, M., and Zaremba, W. Sequence level training with recurrent neural networks. In ICLR , 2016.
- Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. In Association for Computational Linguistics , 2016.
- Smith, J. R., Saint-Amand, H., Plamada, M., Koehn, P., Callison-Burch, C., and Lopez, A. Dirt cheap web-scale parallel text from the common crawl. In ACL , 2013.
- Sokolov, A., Wisniewski, G., and Yvon, F. Lattice bleu oracles in machine translation. ACM Transactions on Speech and Language Processing , 2008.
- Sutskever, I., Martens, J., Dahl, G. E., and Hinton, G. E. On the importance of initialization and momentum in deep learning. In ICML , 2013.
- Toral, A. and Sanchez-Cartagena, V. M. A multifaceted evaluation of neural versus phrase-based machine translation for 9 language directions. In ACL , 2017.
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need. arXiv , 2017.
- Venuti, L. The Translator's Invisibility . Routledge, 2008.
- Vijayakumar, A. K., Cogswell, M., Selvaraju, R. R., Sun, Q., Lee, S., Crandall, D., and Batra, D. Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv:1610.02424 , 2016. · arXiv
- Zhang, B., Xiong, D., Su, J., Duan, H., and Zhang, M. Variational neural machine translation. In EMNLP , 2016.
- Zoph, B., Ghazvininejad, M., and Knight, K. How much information does a human translator add to the original? In EMNLP , 2015.
- Authors
- Ott, M., Auli, M., Grangier, D., & Ranzato, M.
- Affiliations
- Meta (Israel) (IL)
- Google (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- Pages
- 3956-3965
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1803.00047
- OpenAlex
- W2790319220
- Cited by
- 140
- References
- 30
- Citations by year
- 2025: 3 · 2024: 7 · 2023: 16 · 2022: 29 · 2021: 25 · 2020: 30 · 2019: 17 · 2018: 13
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- International Conference on Machine Learning
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- neural machine translation, uncertainty estimation, model calibration, beam search, noisy training data, rare words, translation diversity, sequence-to-sequence models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
