Six Challenges for Neural Machine Translation
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
We explore six challenges for neural machine translation: domain mismatch, amount of training data, rare words, long sentences, word alignment, and beam search.We show both deficiencies and improvements over the quality of phrasebased statistical machine translation.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Still, similar issues have been reported when using large beam sizes (Koehn & Knowles, 2017) and more recently with exact inference (Stahlberg & Byrne, 2019), a counter-intuitive observation since more comprehensive search helps maximize probability.”
From The Curious Case of Neural Text Degeneration (2019) · cites this paper as Koehn & Knowles, 2017
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This is because there is another contributor of hallucinations - training and modeling choices of neural models [121, 215, 222, 263].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [121]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 2×
“…has emerged as the most promising machine translation approach in recent years, showing superior performance on public benchmarks (Bojar et al., 2016) and rapid adoption in deployments by, e.g., Google (Wu et al., 2016), Systran (Crego et al., 2016), and WIPO (Junczys-Dowmunt et al., 2016).”
BackgroundMoses: open source toolkit for statistical machine translation
Koehn, P., Zens, R., Dyer, C., Bojar, O., Constantin, A., Herbst, E., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., et al. (2007) · Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions - ACL '07 · doi:10.3115/1557769.1557821 · Cited by 4,735 · cited 4×
“n that were deleted. Nematus9 (Sennrich et al., 2017), with byte-pair encodings (Sennrich et al., 2016b) to allow for open-vocabulary NMT. The phrase-based model that we used was trained using Moses (Koehn et al., 2007), and the training data and parameters match those described in Johns Hopkins University’s submissio…”
MethodInfluentialSelf-citationParallel Data, Tools and Interfaces in OPUS
Tiedemann, J. (2012) · Language Resources and Evaluation · Cited by 1,421
Galley, M., Hopkins, M., Knight, K., & Marcu, D. (2004) · doi:10.21236/ada460212 · Cited by 460 · cited 1×
“…we note that there are other statistical machine translation approaches such as hierarchical phrase-based models (Chiang, 2007) and syntax-based models (Galley et al., 2004, 2006) that have been shown to give superior performance for language pairs such as Chinese– English and German–English.”
BackgroundNematus: a Toolkit for Neural Machine Translation
Sennrich, R., Firat, O., Cho, K., Birch, A., Haddow, B., Hitschler, J., Junczys-Dowmunt, M., Läubli, S., Miceli Barone, A. V., Mokry, J., et al. (2017) · Proceedings of the Software Demonstrations of the 15th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-3017 · Cited by 291
Incorporating Discrete Translation Lexicons into Neural Machine Translation
Arthur, P., Neubig, G., & Nakamura, S. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1162 · Cited by 203
Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation DirectionsPreprint
Junczys-Dowmunt, M., Dwojak, T., & Hoang, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.01108 · Cited by 170
Fast Domain Adaptation for Neural Machine Translation
Freitag, M., & Al-Onaizan, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.06897 · Cited by 151 · cited 1×
“A currently popular approach is to train a general domain system, followed by training on in-domain data for a few epochs (Luong and Manning, 2015; Freitag and Al-Onaizan, 2016).”
BackgroundNeural Machine Translation with Supervised Attention
Liu, L., Utiyama, M., Finch, A., & Sumita, E. (2016) · arXiv · doi:10.48550/arxiv.1609.04186 · Cited by 146 · cited 2×
“Note that the attention model may produce better word alignments by guided alignment training (Chen et al., 2016; Liu et al., 2016) where supervised word alignments (such as the ones produced by fast-align) are provided to model training.”
Background
Research works printed but not linked: 19
- [3]3. Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016. Neural versus phrase- based machine translation quality: a case study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Associ- ation for Computational Linguistics, Austin, Texas, pages 257-267. https://aclweb.org/anthology/D16- 1025.
- [4]4. Ondřej Bojar, Rajen Chatterjee, Christian Feder- mann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Ma- chine Translation. Association for Computational Linguistics, Berlin, Germany, pages 131-198. http://www.aclweb.org/anthology/W/W16/W16- 2301.
- [6]6. David Chiang. 2007. Hierarchical phrase-based translation. Computational Linguistics 33(2). http://www.aclweb.org/anthology-new/J/J07/J07- 2003.pdf.
- [7]7. Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bah- danau, and Yoshua Bengio. 2014. On the proper- ties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. Association for Computa- tional Linguistics, Doha, Qatar, pages 103-111. http://www.aclweb.org/anthology/W14-4012.
- [10]10. Shuoyang Ding, Kevin Duh, Huda Khayrallah, Philipp Koehn, and Matt Post. 2016b. The JHU machine translation systems for WMT 2016. In Proceed- ings of the First Conference on Machine Translation (WMT).
- [11]11. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameteriza- tion of ibm model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computa- tional Linguistics, Atlanta, Georgia, pages 644-648. http://www.aclweb.org/anthology/N13-1073.
- [13]13. Michel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve DeNeefe, Wei Wang, and Igna- cio Thayer. 2006. Scalable inference and train- ing of context-rich syntactic translation models. In Proceedings of the 21st International Confer- ence on Computational Linguistics and 44th An- nual Meeting of the Association for Computa- tional Linguistics. Association for Computational Linguistics, Sydney, Australia, pages 961-968. http://www.aclweb.org/anthology/P/P06/P06-1121.
- [15]15. Ann Irvine and Chris Callison-Burch. 2013. Com- bining bilingual and comparable corpora for low resource machine translation. In Pro- ceedings of the Eighth Workshop on Statistical Machine Translation. Association for Computa- tional Linguistics, Sofia, Bulgaria, pages 262-270. http://www.aclweb.org/anthology/W13-2233.
- [17]17. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natu- ral Language Processing. Association for Computa- tional Linguistics, Seattle, Washington, USA, pages 1700-1709. http://www.aclweb.org/anthology/D13- 1176.
- [18]18. Philipp Koehn and Barry Haddow. 2012. Interpolated backoff for factored translation models. In Proceed- ings of the Tenth Conference of the Association for Machine Translation in the Americas (AMTA).
- [21]21. Minh-Thang Luong and Christopher D Manning. 2015. Stanford neural machine translation systems for spo- ken language domains. In Proceedings of the In- ternational Workshop on Spoken Language Transla- tion.
- [22]22. Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. 2015. Addressing the rare word problem in neural machine transla- tion. In Proceedings of the 53rd Annual Meet- ing of the Association for Computational Lin- guistics and the 7th International Joint Con- ference on Natural Language Processing (Vol- ume 1: Long Papers). Association for Computa- tional Linguistics, Beijing, China, pages 11-19. http://www.aclweb.org/anthology/P15-1002.
- [23]23. Jean Pouget-Abadie, Dzmitry Bahdanau, Bart van Merrienboer, KyungHyun Cho, and Yoshua Ben- gio. 2014. Overcoming the curse of sentence length for neural machine translation using au- tomatic segmentation. CoRR abs/1409.1257. http://arxiv.org/abs/1409.1257. · arXiv
- [25]25. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine trans- lation systems for WMT 16. In Proceedings of the First Conference on Machine Transla- tion (WMT). Association for Computational Linguistics, Berlin, Germany, pages 371-376. http://www.aclweb.org/anthology/W/W16/W16- 2323.
- [26]26. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceed- ings of the 54th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 1715-1725. http://www.aclweb.org/anthology/P16-1162.
- [28]28. Antonio Toral and Víctor M. Sánchez-Cartagena. 2017. A multifaceted evaluation of neural versus phrase- based machine translation for 9 language directions. In Proceedings of the 15th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. Association for Computational Linguistics, Valencia, Spain, pages 1063-1073. http://www.aclweb.org/anthology/E17- 1100.
- [29]29. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiao- hua Liu, and Hang Li. 2016. Modeling cov- erage for neural machine translation. In Pro- ceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers). Association for Computa- tional Linguistics, Berlin, Germany, pages 76-85. http://www.aclweb.org/anthology/P16-1008.
- [30]30. Marco Turchi, Tijl De Bie, and Nello Cristianini. 2008. Learning performance of a machine translation system: a statistical and computational analysis. In Proceedings of the Third Workshop on Statistical Machine Translation. Association for Computa- tional Linguistics, Columbus, Ohio, pages 35-43. http://www.aclweb.org/anthology/W/W08/W08- 0305.
- [31]31. Philip Williams, Rico Sennrich, Maria Nadejde, Matthias Huck, Barry Haddow, and Ondřej Bojar. 2016. Edinburgh's statistical machine translation systems for wmt16. In Proceed- ings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 399-410. http://www.aclweb.org/anthology/W/W16/W16- 2327.
Cited by
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
“This is because there is another contributor of hallucinations – training and modeling choices of neural models [81, 145, 150, 181].”
BackgroundThe Curious Case of Neural Text Degeneration
Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09751 · Cited by 1,304
“Still, similar issues have been reported when using large beam sizes (Koehn & Knowles, 2017) and more recently with exact inference (Stahlberg & Byrne, 2019), a counter-intuitive observation since more comprehensive search helps maximize probability.”
BackgroundNeural Machine Translation: A Review
Stahlberg, F. (2020) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.12007 · Cited by 397
Unsupervised Statistical Machine Translation
Artetxe, M., Labaka, G., & Agirre, E. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1399 · Cited by 396
Progress in Machine Translation
Wang, H., Wu, H., He, Z., Huang, L., & Church, K. W. (2022) · Engineering · doi:10.1016/j.eng.2021.03.023 · Cited by 226
Metadata
- Authors
- Koehn, P., & Knowles, R.
- Affiliations
- Johns Hopkins University (US)
- ORCID
- Knowles, Rebecca
- Venue
- Proceedings of the First Workshop on Neural Machine Translation
- Published
- 2017
- Pages
- 28-39
- OpenAlex
- W2625092622
- Semantic Scholar
- 106d5e0cf44ea08500adc91c4d5bb3e6c8a4d627
- Cited by
- 184
- References
- 12
- Citations by year
- 2026: 11 · 2025: 12 · 2024: 10 · 2023: 18 · 2022: 14 · 2021: 24 · 2020: 27 · 2019: 20 · 2018: 44 · 2017: 4
- FWCI
- 11.59
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- arXiv (Cornell University) (submittedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- National Science Foundation (DGE-1,232,825, 1232825)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- neural machine translation, domain mismatch, training data size, rare words, long sentences, word alignment, beam search, phrase-based statistical machine translation, translation quality
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Six Challenges for Neural Machine Translation. https://app.cochat.ai/papers/10.18653/v1/w17-3204
- The paper
- Koehn, P., & Knowles, R. (2017). Six Challenges for Neural Machine Translation. Proceedings of the First Workshop on Neural Machine Translation. https://doi.org/10.18653/v1/w17-3204
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.