The DOI matches its Crossref record; 33 of the 36 references it cites were found in a registry or an index.
On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation
Open accessAbstract
The standard training algorithm in neural machine translation (NMT) suffers from exposure bias, and alternative algorithms have been proposed to mitigate this.However, the practical impact of exposure bias is under debate.In this paper, we link exposure bias to another well-known problem in NMT, namely the tendency to generate hallucinations under domain shift.In experiments on three datasets with multiple test domains, we show that exposure bias is partially to blame for hallucinations, and that training with Minimum Risk Training, which avoids exposure bias, can mitigate this.Our analysis explains why exposure bias is more problematic under domain shift, and also links exposure bias to the beam search problem, i.e. performance deterioration with increasing beam size.Our results provide a new justification for methods that reduce exposure bias: even if they do not increase performance on in-domain test sets, they can increase model robustness to domain shift.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Further, in longform tasks consisting of generating multiple sentences or paragraphs, the hallucination problem can be exacerbated due to the issue of exposure bias (Wang & Sennrich, 2020).”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2023) · cites this paper as Wang & Sennrich, 2020
In the citing paper
“Durmus et al. (2020) and Wang et al. (2020) suggested using QG and QA to identify factual inconsistencies in abstractive summaries, which is more Precisionoriented.”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Wang et al. (2020)
In the citing paper
“Their approach is based on the intuition that if a summary is consistent with its source, questions asked on the summary and the source should result in similar answers.”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Wang et al. (2020)
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213
Convolutional Sequence to Sequence Learning
Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017) · arXiv · doi:10.48550/arxiv.1705.03122 · Cited by 2,850
Minimum error rate training in statistical machine translation
Och, F. J. (2003) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1075096.1075117 · Cited by 2,728
A Call for Clarity in Reporting BLEU Scores
Post, M. (2018) · Proceedings of the Third Conference on Machine Translation: Research Papers · doi:10.18653/v1/w18-6319 · Cited by 2,112
Assessing Agreement on Classification Tasks: The Kappa Statistic
Carletta, J. (1996) · arXiv · doi:10.48550/arxiv.cmp-lg/9602004 · Cited by 2,088
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015) · arXiv · doi:10.48550/arxiv.1506.03099 · Cited by 1,621
Moses : open source toolkit for statistical machine translation
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., et al. (2007) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 1,547
Sequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2015) · arXiv · doi:10.48550/arxiv.1511.06732 · Cited by 1,426
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates
Kudo, T. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1007 · Cited by 1,021
OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles
Lison, P., & Tiedemann, J. (2016) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3fi26b3nobqg · Cited by 657
Pay Less Attention with Lightweight and Dynamic ConvolutionsPreprint
Wu, F., Fan, A., Baevski, A., Dauphin, Y., & Auli, M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.10430 · Cited by 585
Sequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2016) · International Conference on Learning Representations · Cited by 572
Sequence-to-Sequence Learning as Beam-Search Optimization
Wiseman, S., & Rush, A. M. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1137 · Cited by 549
Minimum Risk Training for Neural Machine Translation
Shen, S., Cheng, Y., He, Z., He, W. P., Wu, H. R., Sun, M., & Liu, Y. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1159 · Cited by 377
(2004) · doi:10.3115/1220355.1220427 · Cited by 294
Nematus: a Toolkit for Neural Machine Translation
Sennrich, R., Fırat, O., Cho, K., Birch, A., Haddow, B., Hitschler, J., Junczys-Dowmunt, M., Läubli, S., Barone, A. V. M., Mokry, J., et al. (2017) · doi:10.18653/v1/e17-3017 · Cited by 291
Self-citationBridging the Gap between Training and Inference for Neural Machine Translation
Zhang, W., Feng, Y., Meng, F., You, D., & Liu, Q. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1426 · Cited by 212
Classical Structured Prediction Losses for Sequence to Sequence Learning
Edunov, S., Ott, M., Auli, M., Grangier, D., & Ranzato, M. (2017) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n18-1033 · Cited by 190
Neural Machine Translation with Reconstruction
Tu, Z., Liu, Y., Shang, L., Liu, X., & Li, H. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10950 · Cited by 185
Report on the 11th IWSLT Evaluation Campaign, IWSLT 2014
Cettolo, M., Niehues, J., Stüker, S., Bentivogli, L., & Federico, M. (2014) · Cited by 158
Mutual Information and Diverse Decoding Improve Neural Machine TranslationPreprint
Li, J., & Jurafsky, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.00372 · Cited by 141
Analyzing Uncertainty in Neural Machine Translation
Ott, M., Auli, M., Grangier, D., & Ranzato, M. (2018) · arXiv · doi:10.48550/arxiv.1803.00047 · Cited by 140
Correcting Length Bias in Neural Machine Translation
Murray, K., & Chiang, D. (2018) · Proceedings of the Third Conference on Machine Translation: Research Papers · doi:10.18653/v1/w18-6322 · Cited by 133 · cited 2×
“The beam search problem has previously been linked to length bias (Yang et al., 2018; Murray and Chiang, 2018) and the copy mode (Ott et al.”
BackgroundYang, Y., Huang, L., & Ma, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1342 · Cited by 121
On NMT Search Errors and Model Errors: Cat Got Your Tongue?
Stahlberg, F., & Byrne, B. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1331 · Cited by 107
Domain Robustness in Neural Machine Translation
Müller, M., Rios, A., & Sennrich, R. (2019) · arXiv · doi:10.48550/arxiv.1911.03109 · Cited by 68
Self-citationBeyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter
Wu, L., Tan, X., He, D., Tian, F., Qin, T., Lai, J., & Liu, T. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1396 · Cited by 54
Bi-Directional Differentiable Input Reconstruction for Low-Resource Neural Machine Translation
Niu, X., Xu, W., & Carpuat, M. J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1043 · Cited by 21
The Trilingual ALLEGRA Corpus: Presentation and Possible Use for Lexicon Induction
Scherrer, Y., & Cartoni, B. (2012) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2q4x6wm48kzn · Cited by 7
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
“Such inconsistency can result in hallucinations (Wang and Sennrich, 2020), especially when an erroneous token generated by the model cascades errors throughout the subsequent sequence, akin to a snowball effect (Zhang et al., 2023c).”
ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B. O., Ricke, J., et al. (2023) · European Radiology · doi:10.1007/s00330-023-10213-1 · Cited by 549
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. (2025) · Computational Linguistics · doi:10.1162/coli.a.16 · Cited by 534
Chain-of-Verification Reduces Hallucination in Large Language Models
Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., & Weston, J. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.212 · Cited by 282
Metadata
- Authors
- Wang, C., & Sennrich, R.
- Affiliations
- University of Edinburgh (GB)
- University of Zurich (CH)
- Corresponding author
- Sennrich, Rico
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 3544-3552
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3035214886
- Cited by
- 118
- References
- 36
- Citations by year
- 2026: 7 · 2025: 14 · 2024: 11 · 2023: 31 · 2022: 22 · 2021: 28 · 2020: 5
- FWCI
- 8.21
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by)
- Funders
- National Science Foundation
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (176727)
- Samsung
- Engineering and Physical Sciences Research Council (EP/S001271/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- exposure bias, hallucinations, domain shift, neural machine translation, beam search, spot size, model robustness, training algorithms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
