On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translation
Open accessAbstract
The standard training algorithm in neural machine translation (NMT) suffers from exposure bias, and alternative algorithms have been proposed to mitigate this.However, the practical impact of exposure bias is under debate.In this paper, we link exposure bias to another well-known problem in NMT, namely the tendency to generate hallucinations under domain shift.In experiments on three datasets with multiple test domains, we show that exposure bias is partially to blame for hallucinations, and that training with Minimum Risk Training, which avoids exposure bias, can mitigate this.Our analysis explains why exposure bias is more problematic under domain shift, and also links exposure bias to the beam search problem, i.e. performance deterioration with increasing beam size.Our results provide a new justification for methods that reduce exposure bias: even if they do not increase performance on in-domain test sets, they can increase model robustness to domain shift.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In longform tasks, the hallucination problem can be exacerbated due to exposure bias.
“Further, in longform tasks consisting of generating multiple sentences or paragraphs, the hallucination problem can be exacerbated due to the issue of exposure bias (Wang & Sennrich, 2020).”
From Chain-of-Verification Reduces Hallucination in Large Language Models (2023) · cites this paper as Wang & Sennrich, 2020
Durmus et al. (2020) and Wang et al. (2020) suggested using QG and QA to identify factual inconsistencies in abstractive summaries, which is more Precision-oriented.
“Durmus et al. (2020) and Wang et al. (2020) suggested using QG and QA to identify factual inconsistencies in abstractive summaries, which is more Precisionoriented.”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Wang et al. (2020)
The QG-QA approach is based on the intuition that if a summary is consistent with its source, questions asked on the summary and the source should result in similar answers.
“Their approach is based on the intuition that if a summary is consistent with its source, questions asked on the summary and the source should result in similar answers.”
From Q2: : Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering (2021) · cites this paper as Wang et al. (2020)
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Rethinking the Inception Architecture for Computer Vision
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., & Wojna, Z. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.308 · Cited by 32,138 · see citing works
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · see citing works
Sequence to Sequence Learning with Neural Networks
Sutskever, I., Vinyals, O., & Le, Q. V. (2014) · arXiv · doi:10.48550/arxiv.1409.3215 · Cited by 17,766 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Convolutional Sequence to Sequence Learning
Gehring, J., Auli, M., Grangier, D., Yarats, D., & Dauphin, Y. N. (2017) · arXiv · doi:10.48550/arxiv.1705.03122 · Cited by 2,850 · see citing works
Minimum error rate training in statistical machine translation
Och, F. J. (2003) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1075096.1075117 · Cited by 2,728 · see citing works
A Call for Clarity in Reporting BLEU Scores
Post, M. (2018) · Proceedings of the Third Conference on Machine Translation: Research Papers · doi:10.18653/v1/w18-6319 · Cited by 2,112 · see citing works
Assessing Agreement on Classification Tasks: The Kappa Statistic
Carletta, J. (1996) · arXiv · doi:10.48550/arxiv.cmp-lg/9602004 · Cited by 2,088 · see citing works
Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
Bengio, S., Vinyals, O., Jaitly, N., & Shazeer, N. (2015) · arXiv · doi:10.48550/arxiv.1506.03099 · Cited by 1,621 · see citing works
Moses : open source toolkit for statistical machine translation
Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., et al. (2007) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 1,547 · see citing works
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates
Kudo, T. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1007 · Cited by 1,021 · see citing works
Pay Less Attention with Lightweight and Dynamic ConvolutionsPreprint
Wu, F., Fan, A., Baevski, A., Dauphin, Y., & Auli, M. (2019) · International Conference on Learning Representations · doi:10.48550/arxiv.1901.10430 · Cited by 669 · see citing works
OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles
Lison, P., & Tiedemann, J. (2016) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3fi26b3nobqg · Cited by 657 · see citing works
Sequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2016) · International Conference on Learning Representations · Cited by 572 · see citing works
Sequence-to-Sequence Learning as Beam-Search Optimization
Wiseman, S., & Rush, A. M. (2016) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d16-1137 · Cited by 485 · see citing works
Minimum Risk Training for Neural Machine Translation
Shen, S., Cheng, Y., He, Z., He, W. P., Wu, H. R., Sun, M., & Liu, Y. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1159 · Cited by 377 · see citing works
(2004) · doi:10.3115/1220355.1220427 · Cited by 294 · see citing works
Nematus: a Toolkit for Neural Machine Translation
Sennrich, R., Fırat, O., Cho, K., Birch, A., Haddow, B., Hitschler, J., Junczys-Dowmunt, M., Läubli, S., Barone, A. V. M., Mokry, J., et al. (2017) · doi:10.18653/v1/e17-3017 · Cited by 291 · see citing works
Self-citationBridging the Gap between Training and Inference for Neural Machine Translation
Zhang, W., Feng, Y., Meng, F., You, D., & Liu, Q. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1426 · Cited by 212 · see citing works
Classical Structured Prediction Losses for Sequence to Sequence Learning
Edunov, S., Ott, M., Auli, M., Grangier, D., & Ranzato, M. (2017) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n18-1033 · Cited by 190 · see citing works
Neural Machine Translation with Reconstruction
Tu, Z., Liu, Y., Shang, L., Liu, X., & Li, H. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.10950 · Cited by 188 · see citing works
Report on the 11th IWSLT Evaluation Campaign, IWSLT 2014
Cettolo, M., Niehues, J., Stüker, S., Bentivogli, L., & Federico, M. (2014) · Cited by 158 · see citing works
Analyzing Uncertainty in Neural Machine Translation
Ott, M., Auli, M., Grangier, D., & Ranzato, M. (2018) · arXiv · doi:10.48550/arxiv.1803.00047 · Cited by 140 · see citing works
Correcting Length Bias in Neural Machine Translation
Murray, K., & Chiang, D. (2018) · Proceedings of the Third Conference on Machine Translation: Research Papers · doi:10.18653/v1/w18-6322 · Cited by 133 · cited 2× · see citing works
“The beam search problem has previously been linked to length bias (Yang et al., 2018; Murray and Chiang, 2018) and the copy mode (Ott et al.”
BackgroundMutual Information and Diverse Decoding Improve Neural Machine TranslationPreprint
Li, J., & Jurafsky, D. (2016) · arXiv.org · doi:10.48550/arxiv.1601.00372 · Cited by 131 · see citing works
On NMT Search Errors and Model Errors: Cat Got Your Tongue?
Stahlberg, F., & Byrne, B. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1331 · Cited by 107 · see citing works
Yang, Y., Huang, L., & Ma, M.-B. (2018) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1342 · Cited by 103 · see citing works
Domain Robustness in Neural Machine Translation
Müller, M., Rios, A., & Sennrich, R. (2019) · arXiv · doi:10.48550/arxiv.1911.03109 · Cited by 68 · see citing works
Self-citationBeyond Error Propagation in Neural Machine Translation: Characteristics of Language Also Matter
Wu, L., Tan, X., He, D., Tian, F., Qin, T., Lai, J., & Liu, T. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1396 · Cited by 54 · see citing works
Bi-Directional Differentiable Input Reconstruction for Low-Resource Neural Machine Translation
Niu, X., Xu, W., & Carpuat, M. J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1043 · Cited by 21 · see citing works
The Trilingual ALLEGRA Corpus: Presentation and Possible Use for Lexicon Induction
Scherrer, Y., & Cartoni, B. (2012) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2q4x6wm48kzn · Cited by 7 · see citing works
Sequence Level Training with Recurrent Neural Networks
Ranzato, M., Chopra, S., Auli, M., & Zaremba, W. (2015) · arXiv · doi:10.48550/arxiv.1511.06732
Flagged first, then by citations
3 of the 36 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · see citing works
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
“Such inconsistency can result in hallucinations (Wang and Sennrich, 2020), especially when an erroneous token generated by the model cascades errors throughout the subsequent sequence, akin to a snowball effect (Zhang et al., 2023c).”
ChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B. O., Ricke, J., et al. (2023) · European Radiology · doi:10.1007/s00330-023-10213-1 · Cited by 549 · see citing works
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models
Zhang, Y., Li, Y., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y., Chen, Y., et al. (2025) · Computational Linguistics · doi:10.1162/coli.a.16 · Cited by 534 · see citing works
Chain-of-Verification Reduces Hallucination in Large Language Models
Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., & Weston, J. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.212 · Cited by 282 · see citing works
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516 · see citing works
“Such inconsistency can result in hallucinations (Wang and Sennrich, 2020), especially when an erroneous token generated by the model cascades errors throughout the subsequent sequence, akin to a snowball effect (Zhang et al., 2023c).”
Survey of Low-Resource Machine Translation
Haddow, B., Bawden, R., Barone, A. V. M., Helcl, J., & Birch, A. (2022) · Computational Linguistics · doi:10.1162/coli_a_00446 · Cited by 185 · see citing works
The Curious Case of Hallucinations in Neural Machine Translation
Raunak, V., Menezes, A., & Junczys-Dowmunt, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.92 · Cited by 142 · see citing works
Towards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141 · see citing works
Hallucinations in Large Multilingual Translation Models
Guerreiro, N. M., Alves, D. M., Waldendorf, J., Haddow, B., Birch, A., Colombo, P., & Martins, A. F. T. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00615 · Cited by 124 · see citing works
Detecting Hallucinated Content in Conditional Neural Sequence Generation
Zhou, C., Neubig, G., Gu, J., Diab, M., Guzmán, F., Zettlemoyer, L., & Ghazvininejad, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.120 · Cited by 122 · see citing works
On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?
Dziri, N., Milton, S., Yu, M., Zaiane, O., & Reddy, S. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.387 · Cited by 121 · see citing works
The limitations and ethical considerations of ChatGPT
Hua, S., Jin, S., & Jiang, S. (2024) · Data Intelligence · doi:10.1162/dint_a_00243 · Cited by 120 · see citing works
Honovich, O., Choshen, L., Aharoni, R., Neeman, E., Szpektor, I., & Abend, O. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.619 · Cited by 113 · see citing works
Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey
Saunders, D. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13566 · Cited by 96 · see citing works
Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarization
Cao, M., Dong, Y., & Cheung, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.236 · Cited by 87 · see citing works
Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding
Dziri, N., Madotto, A., Zaïane, O., & Bose, A. J. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.168 · Cited by 81 · see citing works
Symbolic Music Generation with Transformer-GANs
Muhamed, A., Li, L., Shi, X., Yaddanapudi, S., Chi, W., Jackson, D., Suresh, R., Lipton, Z. C., & Smola, A. J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i1.16117 · Cited by 72 · see citing works
Contrastive Learning Reduces Hallucination in Conversations
Sun, W., Shi, Z., Gao, S., Ren, P., De Rijke, M., & Ren, Z. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i11.26596 · Cited by 69 · see citing works
FaithDial : A Faithful Benchmark for Information-Seeking Dialogue
Dziri, N., Kamalloo, E., Milton, S., Zai͏̈ane, O. R., Yu, M., Ponti, E. M., & Reddy, S. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00529 · Cited by 68 · see citing works
Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation
Voita, E., Sennrich, R., & Titov, I. (2021) · doi:10.18653/v1/2021.acl-long.91 · Cited by 66 · see citing works
Self-citationGuerreiro, N. M., Voita, E., & Martins, A. (2023) · Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.75 · Cited by 63 · see citing works
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation
Liu, T., Zhang, Y., Brockett, C., Mao, Y., Sui, Z., Chen, W., & Dolan, B. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.464 · Cited by 63 · see citing works
Automatic Evaluation of Attribution by Large Language Models
Yue, X., Wang, B., Chen, Z., Zhang, K., Su, Y., & Sun, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.307 · Cited by 62 · see citing works
Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection
Xu, W., Agrawal, S., Briakou, E., Martindale, M. J., & Carpuat, M. J. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00563 · Cited by 59 · see citing works
Faithful AI in Medicine: A Systematic Review with Large Language Models and BeyondPreprint
Xie, Q., Schenck, E. J., Yang, H. S., Chen, Y., Peng, Y., & Wang, F. (2023) · medRxiv · doi:10.1101/2023.04.18.23288752 · Cited by 57 · see citing works
HaluCheck: Explainable and verifiable automation for detecting hallucinations in LLM responses
Heo, S., Son, S., & Park, H. (2025) · Expert Systems with Applications · doi:10.1016/j.eswa.2025.126712 · Cited by 47 · see citing works
Quality-Aware Decoding for Neural Machine Translation
Fernandes, P., Farinhas, A., Rei, R., de Souza, J. G. C., Ogayo, P., Neubig, G., & Martins, A. F. T. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.100 · Cited by 46 · see citing works
Truth-O-Meter: Collaborating with LLM in fighting its hallucinations
Galitsky, B. A. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-29246-0.00004-3 · Cited by 45 · see citing works
Text Generation by Learning from DemonstrationsPreprint
Pang, R. Y., & He, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.07839 · Cited by 45 · see citing works
Arora, K., Asri, L. E., Bahuleyan, H., & Cheung, J. C. K. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.58 · Cited by 44 · see citing works
Do Long-Range Language Models Actually Use Long-Range Context?
Sun, S., Krishna, K., Mattarella-Micke, A., & Iyyer, M. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.62 · Cited by 43 · see citing works
Dale, D. C., Voita, E., Barrault, L., & Costa‐jussà, M. R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.3 · Cited by 43 · see citing works
Chain-of-Verification Reduces Hallucination in Large Language Models
Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., & Weston, J. (2023) · arXiv · doi:10.48550/arxiv.2309.11495 · Cited by 42 · see citing works
“Further, in longform tasks consisting of generating multiple sentences or paragraphs, the hallucination problem can be exacerbated due to the issue of exposure bias (Wang and Sennrich, 2020).”
BackgroundLarge language models in clinical trials: applications, technical advances, and future directions
Lin, A., Wang, Z., Jiang, A., Chen, L., Qi, C., Zhu, L., Mou, W., Gan, W., Zeng, D., Xiao, M., et al. (2025) · BMC Medicine · doi:10.1186/s12916-025-04348-9 · Cited by 37 · see citing works
Machine Translation in the Era of Large Language Models:A Survey of Historical and Emerging Problems
Ataman, D., Birch, A., Habash, N. Y., Federico, M., Koehn, P., & Cho, K. (2025) · Information · doi:10.3390/info16090723 · Cited by 36 · see citing works
Mutual Information Alleviates Hallucinations in Abstractive Summarization
van der Poel, L., Cotterell, R., & Meister, C. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.399 · Cited by 35 · see citing works
Understanding and Improving Sequence-to-Sequence Pretraining for Neural Machine Translation
Wang, W., Jiao, W., Hao, Y., Wang, X., Shi, S., Tu, Z., & Lyu, M. R.-T. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.185 · Cited by 34 · see citing works
Prevent the Language Model from being Overconfident in Neural Machine Translation
Miao, M., Meng, F., Liu, Y., Zhou, X., & Zhou, J. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.268 · Cited by 30 · see citing works
He, T., Zhang, J., Zhou, Z., & Glass, J. R. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.415 · Cited by 30 · see citing works
Mitigating exposure bias in large language model distillation: an imitation learning approach
Pozzi, A., Incremona, A., Tessera, D., & Toti, D. (2025) · Neural Computing and Applications · doi:10.1007/s00521-025-11162-0 · Cited by 28 · see citing works
Language Models in Dialogue: Conversational Maxims for Human-AI Interactions
Miehling, E., Nagireddy, M., Sattigeri, P., Daly, E., Piorkowski, D., & Richards, J. T. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.843 · Cited by 26 · see citing works
Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation
Kiegeland, S., & Kreutzer, J. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.133 · Cited by 25 · see citing works
Úbeda, P. L., Martín‐Noguerol, T., Díaz-Angulo, C., & Luna, A. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105443 · Cited by 24 · see citing works
Quantifying Exposure Bias for Neural Language GenerationPreprint
He, T., Zhang, J., Zhou, Z., & Glass, J. R. (2019) · arXiv (Cornell University) · Cited by 23 · see citing works
Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR
Opitz, J., & Frank, A. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.129 · Cited by 22 · see citing works
Sánchez-Cartagena, V. M., Esplà-Gomis, M., Pérez-Ortiz, J. A., & Sánchez-Martínez, F. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.669 · Cited by 22 · see citing works
AI Large Models for Power System: A Survey and Outlook
Yao, Q. J., Fang, F., Chen, Y., Liu, J., Mo, H., & Ao, Y. (2025) · IET Smart Energy Systems · doi:10.1049/ses2.70000 · Cited by 20 · see citing works
Alleviating the Long-Tail Problem in Conversational Recommender Systems
Zhao, Z., Zhou, K., Wang, X., Zhao, W. X., Pan, F., Cao, Z., & Wen, J.-R. (2023) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3604915.3608812 · Cited by 18 · see citing works
Understanding and Bridging the Modality Gap for Speech Translation
Fang, Q., & Feng, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.884 · Cited by 18 · see citing works
Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translation
Eikema, B., & Aziz, W. F. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.754 · Cited by 17 · see citing works
Improving Automated Program Repair with Domain Adaptation
Zirak, A., & Hemmati, H. (2023) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3631972 · Cited by 17 · see citing works
Language Models As or For Knowledge Bases
Razniewski, S., Yates, A., Kassner, N., & Weikum, G. (2021) · arXiv · doi:10.48550/arxiv.2110.04888 · Cited by 16 · see citing works
BLEURT Has Universal Translations: An Analysis of Automatic Metrics by Minimum Risk Training
Yan, Y., Wang, T., Zhao, C., Huang, S., Chen, J., & Wang, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.297 · Cited by 16 · see citing works
IGA: An Intent-Guided Authoring Assistant
Sun, S., Zhao, W., Manjunatha, V., Jain, R., Morariu, V. I., Dernoncourt, F., Srinivasan, B. V., & Iyyer, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.483 · Cited by 16 · see citing works
Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation
Gu, S., & Feng, Y. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.381 · Cited by 15 · see citing works
Holtzman, A., West, P., & Zettlemoyer, L. (2025) · Journal of Social Computing · doi:10.23919/jsc.2025.0009 · Cited by 14 · see citing works
Addressing domain shift in neural machine translation via reinforcement learning
Kumar, A., Pratap, A., Singh, A. K., & Saha, S. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.117039 · Cited by 14 · see citing works
Towards understanding and mitigating the hallucinations in NLP and Speech
Mittal, A., Murthy, R., Kumar, V., & Bhat, R. A. (2024) · Joint International Conference on Data Science & Management of Data (CODS-COMAD) · doi:10.1145/3632410.3633297 · Cited by 14 · see citing works
Understanding and Improving Encoder Layer Fusion in Sequence-to-Sequence LearningPreprint
Liu, X., Wang, L., Wong, D. F., Ding, L., Chao, L. S., & Tu, Z. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.14768 · Cited by 14 · see citing works
Modelling Latent Translations for Cross-Lingual TransferPreprint
Ponti, E. M., Kreutzer, J., Vulić, I., & Reddy, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.11353 · Cited by 14 · see citing works
WeChat Neural Machine Translation Systems for WMT21Preprint
Zeng, X., Liu, Y., Li, E., Ran, Q., Meng, F., Li, P., Xu, J., & Zhou, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.02401 · Cited by 13 · see citing works
WeChat Neural Machine Translation Systems for WMT20
Meng, F., Yan, J., Liu, Y., Gao, Y., Zeng, X., Zeng, Q., Li, P., Chen, M., Zhou, J., Liu, S., et al. (2020) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2020.wmt-1.24 · Cited by 12 · see citing works
Data augmented large language models for medical record generation
Zhang, X., Zhao, G., Ren, Y., Wang, W., Cai, W., Zhao, Y., Zhang, X., & Liu, J. (2024) · Applied Intelligence · doi:10.1007/s10489-024-05934-9 · Cited by 11 · see citing works
AgreeSum: Agreement-Oriented Multi-Document Summarization
Pang, R. Y., Lelkes, A. D., Tran, V., & Yu, C. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.299 · Cited by 11 · see citing works
Bridging the Data Gap between Training and Inference for Unsupervised Neural Machine Translation
He, Z., Wang, X., Wang, R., Shi, S., & Tu, Z. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.456 · Cited by 11 · see citing works
(2021) · doi:10.18653/v1/2021.findings-acl.427 · Cited by 10 · see citing works
MLE-Guided Parameter Search for Task Loss Minimization in Neural Sequence Modeling
Welleck, S., & Cho, K. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17652 · Cited by 10 · see citing works
Scientific document processing: challenges for modern learning methods
Kashyap, A. R., Yang, Y., & Kan, M. (2023) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00352-7 · Cited by 9 · see citing works
Liu, Z., Kou, J., Zhang, W., Gu, C., Fang, X., Huang, Z., Hao, Y., Li, H., Lu, X., Yin, A., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189137 · Cited by 9 · see citing works
The AI Annotator: Large Language Models’ Potential in Scoring Sustainability Reports
Wu, Y., Hu, P., & Wang, D. D. (2025) · Systems · doi:10.3390/systems13100899 · Cited by 9 · see citing works
Optimal Transport for Unsupervised Hallucination Detection in Neural Machine Translation
Guerreiro, N. M., Colombo, P., Piantanida, P., & Martins, A. F. T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.770 · Cited by 9 · see citing works
Better Language Models of Code through Self-Improvement
To, H. Q., Bui, N., Guo, J., & Nguyen, T. N. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.823 · Cited by 9 · see citing works
Relating Neural Text Degeneration to Exposure Bias
Chiang, T.-R., & Chen, Y.-N. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.16 · Cited by 9 · see citing works
Efficient Fine-Tuning Large Language Models for Knowledge-Aware Response Planning
Nguyen, M., Kishan, K. C., Nguyen, T., Chadha, A., & Vu, T. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-43415-0_35 · Cited by 8 · see citing works
Calderon, N., Mukherjee, S., Reichart, R., & Kantor, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.818 · Cited by 8 · see citing works
Non-Fluent Synthetic Target-Language Data Improve Neural Machine Translation
Sánchez-Cartagena, V. M., Esplà-Gomis, M., Pérez-Ortiz, J. A., & Sánchez-Martínez, F. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3333949 · Cited by 8 · see citing works
Machine-in-the-Loop Rewriting for Creative Image Captioning
Padmakumar, V., & He, H. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.42 · Cited by 7 · see citing works
Thinking Hallucination for Video Captioning
Ullah, N., & Mohanta, P. P. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-26316-3_37 · Cited by 7 · see citing works
Saunders, D., & Byrne, B. (2020) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2020.wmt-1.94 · Cited by 7 · see citing works
Disentangled Action Recognition with Knowledge Bases
Luo, Z., Ghosh, S., Guillory, D., Kato, K., Darrell, T., & Xu, H. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.41 · Cited by 7 · see citing works
Hallucination in Large Language Models: From Mechanistic Understanding to Novel Control Frameworks
Zhang, W., Zhang, C., Gu, C., Kou, J., Yuan, H., Fang, X., Duan, X., & Fang, Y. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189226 · Cited by 6 · see citing works
ESRL: Efficient Sampling-Based Reinforcement Learning for Sequence Generation
Wang, C., Zhou, H., Hu, Y., Huo, Y., Li, B., Liu, T., Xiao, T., & Zhu, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29878 · Cited by 6 · see citing works
Self-citationEnhancing the Transformer Decoder with Transition-based Syntax
Choshen, L., & Abend, O. (2022) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2022.conll-1.27 · Cited by 5 · see citing works
Inverse Reinforcement Learning for Text Summarization
Fu, Y., Xiong, D., & Dong, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.436 · Cited by 4 · see citing works
SURF: Semantic-level Unsupervised Reward Function for Machine Translation
Anuchitanukul, A., & Ive, J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.334 · Cited by 4 · see citing works
Few-shot LLM Synthetic Data with Distribution Matching
Ren, J., Du, Z., Wen, Z., Jia, Q., Dai, S., Wu, C., & Dong, Z. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3715245 · Cited by 4 · see citing works
Adaptor: Objective-Centric Adaptation Framework for Language Models
Štefánik, M., Novotný, V. S., Groverová, N., & Sojka, P. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.26 · Cited by 4 · see citing works
Fang, X., Yuan, H. R., Li, H., Kou, J., Gu, C., Zhang, W., Duan, X., & Fang, Y. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189217 · Cited by 3 · see citing works
Provilkov, I., & Malinin, A. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.677 · Cited by 3 · see citing works
Lan, Z., Li, W., Su, J., Xiao, X., Liu, J., Wu, W., & Lyu, Y. (2023) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.14267 · Cited by 3 · see citing works
Transition based Graph Decoder for Neural Machine Translation.Preprint
Choshen, L., & Abend, O. (2021) · arXiv (Cornell University) · Cited by 3 · see citing works
Identifying large language model hallucinations in health communication
Galitsky, B. A. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-30046-2.00012-0 · Cited by 2 · see citing works
Kou, J., Lu, X., Yin, A., Zhang, W., Fang, X., Li, Y., Pan, T., Zhang, L., Cheng, G., Yuan, H., et al. (2024) · International Conference on Universal Village (UV) · doi:10.1109/uv63228.2024.11189154 · Cited by 2 · see citing works
Open-World Factually Consistent Question Generation
Maheshwari, H., Shekhar, S., Saxena, A., & Chhaya, N. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.151 · Cited by 2 · see citing works
Yano, K., & Utsumi, A. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.178 · Cited by 2 · see citing works
Chatgpt Inaccuracy Mitigation During Technical Report Understanding: Are we There Yet?
Tamanna, S. B., Uddin, G., Wang, S., Xia, L., & Zhang, L. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00145 · Cited by 2 · see citing works
Towards Learning Causal Representations of Technical Word Embeddings for Smart Troubleshooting
Trilla, A., Mijatovic, N., & Vilasis-Cardona, X. (2022) · International Journal of Prognostics and Health Management · doi:10.36001/ijphm.2022.v13i2.3127 · Cited by 2 · see citing works
Lin, H., Yang, B., Yao, L., Liu, D., Zhang, H., Xie, J., Zhang, M., & Su, J. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.200 · Cited by 2 · see citing works
Quantifying Exposure Bias for Open-ended Language GenerationPreprint
He, T., Zhang, J., Zhou, Z., & Glass, J. R. (2021) · arXiv (Cornell University) · Cited by 2 · see citing works
- Authors
- Wang, C., & Sennrich, R.
- Affiliations
- University of Edinburgh (GB)
- University of Zurich (CH)
- Corresponding author
- Sennrich, Rico
- Venue
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Published
- 2020
- Pages
- 3544-3552
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W3035214886
- Cited by
- 118
- References
- 36
- Citations by year
- 2026: 7 · 2025: 14 · 2024: 11 · 2023: 31 · 2022: 22 · 2021: 28 · 2020: 5
- FWCI
- 8.21
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- Zurich Open Repository and Archive (University of Zurich) (submittedVersion, cc-by)
- Funders
- National Science Foundation
- Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung (176727)
- Samsung
- Engineering and Physical Sciences Research Council (EP/S001271/1)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- exposure bias, hallucinations, domain shift, neural machine translation, beam search, spot size, model robustness, training algorithms
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
