Detecting Hallucinated Content in Conditional Neural Sequence Generation
Open accessAbstract
Neural sequence models can generate highly fluent sentences, but recent studies have also shown that they are also prone to hallucinate additional content not supported by the input.These variety of fluent but wrong outputs are particularly problematic, as it will not be possible for users to tell they are being presented incorrect content.To detect these errors, we propose a task to predict whether each token in the output sequence is hallucinated (not contained in the input) and collect new manually annotated evaluation sets for this task.We also introduce a method for learning to detect hallucinations using pretrained language models fine tuned on synthetic data that includes automatically inserted hallucinations.Experiments on machine translation (MT) and abstractive summarization demonstrate that our proposed approach consistently outperforms strong baselines on all benchmark datasets.We further demonstrate how to use the token-level hallucination labels to define a fine-grained loss over the target sequence in low-resource MT and achieve significant improvements over strong baseline methods.We also apply our method to word-level quality estimation for MT and show its effectiveness in both supervised and unsupervised settings 1 .
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Within NLP, the definition of hallucination as generated content that is nonsensical or unfaithful to the provided source content is the most inclusive and standard.
“Within the context of NLP, the above definition of hallucination, the generated content that is nonsensical or unfaithful to the provided source content [72, 177, 198, 325], is the most inclusive and standard.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [325]
Section 2 (page not supplied) defines hallucinated content as a generated span unsupported by the source input, supporting the unfaithfulness part of the claim. It treats incomprehensible output separately from output containing hallucinations, and does not establish that the citing sentence’s broader definition is the most inclusive or standard in NLP. (LLM, conf=0.86)
Experimental — accuracy not yet measuredEven if a summary contains a large amount of hallucinatory content, it can achieve a high ROUGE score.
“In addition, Zhou et al. [325] mention that even if a summary contains a large amount of hallucinatory content, it can achieve a high ROUGE [154] score.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as Zhou et al. [325]
The Introduction (page unavailable) says ROUGE does not correlate well with the faithfulness of model outputs, supporting the general point that a ROUGE score can be misleading. Table 2 (page unavailable) reports substantial hallucinated-token percentages in summarization outputs, but our copy does not show that those outputs achieved high ROUGE scores. (LLM, conf=0.83)
Experimental — accuracy not yet measuredZhou et al.'s method shows higher correlations with human factual consistency evaluation than the baselines.
“This method shows higher correlations with human factual consistency evaluation than the baselines.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [325]
The passage discussing Table 2 reports that the method’s predictions correlate with human hallucination assessments (Pearson r = 0.986). Section 5.2 and Table 1 show better token-level F1 scores than the baselines, but they do not compare the methods’ correlations with human evaluations. The claimed higher correlation than the baselines is therefore unsupported. (LLM, conf=0.85)
Experimental — accuracy not yet measuredExperimental results show both a translation quality improvement in terms of BLEU and also a large reduction in hallucination.
“Experimental results show both a translation quality improvement in terms of BLEU and also a large reduction in hallucination.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [325]
The token loss truncation method shows good results in the low-resource languages scenario.
“The token loss truncation method shows good results in the low-resource languages scenario.”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [325]
Recent language models have a tendency to generate false statements.
“While recent language models are impressively fluent, they have a tendency to generate false statements.”
From TruthfulQA: Measuring How Models Mimic Human Falsehoods (2022) · cites this paper as Zhou et al., 2021
The appearance of hallucinations is found not to affect the BLEU score of the translated text.
“Most detrimentally, the appearance of hallucinations is found not to affect the BLEU score of the translated text [251, 325].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [325]
Section 6.1, Table 4, reports BLEU scores alongside percentages of hallucinated tokens, and Results and Analysis says the best method improved both BLEU and hallucination reduction. Neither passage establishes whether the appearance of hallucinations affects BLEU; the reported changes occur across different translation methods. (LLM, conf=0.73)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 59 · Cited by 122
Research works printed but not linked: 43
- Antonios Anastasopoulos, Alessandro Cattelan, Zi- Yi Dou, Marcello Federico, Christian Federman, Dmitriy Genzel, Francisco Guzmán, Junjie Hu, Mac- duff Hughes, Philipp Koehn, et al. 2020. Tico-19: the translation initiative for covid-19. arXiv preprint arXiv:2007.01788. · arXiv
- Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150. · arXiv
- Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. · arXiv
- Vishrav Chaudhary, Yuqing Tang, Francisco Guzmán, Holger Schwenk, and Philipp Koehn. 2019. Low- resource corpus filtering using multilingual sentence embeddings. In Proceedings of the Fourth Confer- ence on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 261-266.
- Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, Online.
- Erick Fonseca, Lisa Yankovskaya, André F. T. Martins, Mark Fishel, and Christian Federmann. 2019. Find- ings of the WMT 2019 shared tasks on quality esti- mation. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 1-10, Florence, Italy. Association for Computational Linguistics. · DOI
- Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc'Aurelio Ranzato. 2019. The flores evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 6100- 6113.
- Junxian He, Jiatao Gu, Jiajun Shen, and Marc'Aurelio Ranzato. 2020. Revisiting self-training for neural sequence generation. aInternational Conference on Learning Representations (ICLR).
- Marcin Junczys-Dowmunt. 2018. Dual conditional cross-entropy filtering of noisy parallel corpora. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 888-895.
- Daniel Kang and Tatsunori Hashimoto. 2020. Im- proved natural language generation via loss trunca- tion. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, On- line.
- Fabio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André FT Martins. 2019. Openkiwi: An open source framework for quality estimation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 117-122.
- Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451. · arXiv
- Philipp Koehn, Francisco Guzmán, Vishrav Chaud- hary, and Juan Pino. 2019. Findings of the wmt 2019 shared task on parallel corpus filtering for low-resource conditions. In Proceedings of the Fourth Conference on Machine Translation (Volume 3: Shared Task Papers, Day 2), pages 54-72.
- Philipp Koehn and Rebecca Knowles. 2017. Six chal- lenges for neural machine translation. In Proceed- ings of the First Workshop on Neural Machine Trans- lation, pages 28-39.
- Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. Has machine translation achieved human parity? a case for document-level evaluation. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4791-4796, Brussels, Belgium. Association for Computational Linguistics. · DOI
- Chin-Yew Lin and Eduard Hovy. 2004. Rouge: A pack- age for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
- Ruixuan Luo, Jingjing Xu, Yi Zhang, Xuancheng Ren, and Xu Sun. 2019. Pkuseg: A toolkit for multi-domain chinese word segmentation. CoRR, abs/1906.11455. · arXiv
- Gary Marcus and Ernest Davis. 2020. Gpt-3, bloviator: Openai's language generator has no idea what it's talking about. MIT Technology Review.
- Marianna Martindale, Marine Carpuat, Kevin Duh, and Paul McNamee. 2019. Identifying fluently inade- quate output in neural and statistical machine transla- tion. In Proceedings of Machine Translation Summit XVII Volume 1: Research Track, pages 233-243.
- George A Miller. 1998. WordNet: An electronic lexical database. MIT press.
- Mathias Müller, Annette Rios, and Rico Sennrich. 2019. Domain robustness in neural machine trans- lation. arXiv preprint arXiv:1911.03109. · arXiv
- Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations.
- Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186- 191, Belgium, Brussels. Association for Computa- tional Linguistics.
- Ratish Puduppully, Li Dong, and Mirella Lapata. 2019. Data-to-text generation with content selection and planning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6908-6915.
- Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020. Leveraging pre-trained checkpoints for se- quence generation tasks. Transactions of the Asso- ciation for Computational Linguistics, 8:264-280.
- Masoud Jalili Sabet, Philipp Dufter, and Hinrich Schütze. 2020. Simalign: High quality word align- ments without parallel training data using static and contextualized embeddings. arXiv preprint arXiv:2004.08728. · arXiv
- H Scudder. 1965. Probability of error of some adap- tive pattern-recognition machines. IEEE Transac- tions on Information Theory, 11(3):363-371.
- Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer- generator networks. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073- 1083.
- Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text gener- ation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, On- line.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715- 1725, Berlin, Germany. Association for Computa- tional Linguistics. · DOI · Paper page
- Lucia Specia, Frédéric Blain, Varvara Logacheva, Ramón Astudillo, and André Martins. 2018. Find- ings of the wmt 2018 shared task on quality estima- tion. Association for Computational Linguistics.
- Lucia Specia, Fr éd éric Blain, Marina Fomicheva, Erick Fonseca, Vishrav Chaudhary, Francisco Guzm án, and Andr é F. T. Martins. 2020. Find- ings of the wmt 2020 shared task on quality estima- tion. In Proceedings of the Fifth Conference on Ma- chine Translation, pages 743-764, Online. Associa- tion for Computational Linguistics.
- Lucia Specia, Najeh Hajlaoui, Catalina Hallett, and Wilker Aziz. 2011. Predicting machine translation adequacy. In Machine Translation Summit, vol- ume 13, pages 19-23.
- Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P Parikh. 2019. Sticking to the facts: Con- fident decoding for faithful data-to-text generation. arXiv preprint arXiv:1910.08684. · arXiv
- Kristina Toutanova, Dan Klein, Christopher D Man- ning, and Yoram Singer. 2003. Feature-rich part-of- speech tagging with a cyclic dependency network. In Proceedings of the 2003 conference of the North American chapter of the association for computa- tional linguistics on human language technology- volume 1, pages 173-180. Association for Compu- tational Linguistics.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information pro- cessing systems, pages 5998-6008.
- Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020a. Asking and answering questions to evaluate the fac- tual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, Online.
- Yong Wang, Longyue Wang, Shuming Shi, Victor OK Li, and Zhaopeng Tu. 2020b. Go from the gen- eral to the particular: Multi-domain translation with domain transformation networks. In Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI), New York, USA.
- Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Di- nan, Kyunghyun Cho, and Jason Weston. 2019. Neu- ral text generation with unlikelihood training. In International Conference on Learning Representa- tions.
- Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between hu- man and machine translation. arXiv preprint arXiv:1609.08144. · arXiv
- Boliang Zhang, Ajay Nagesh, and Kevin Knight. 2020. Parallel corpus filtering via pre-trained language models. arXiv preprint arXiv:2005.06166. · arXiv
- Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020. Extractive summarization as text matching. In Proceedings of the 58th Annual Meeting of the Association for Com- putational Linguistics, Online.
- Chunting Zhou, Xuezhe Ma, Di Wang, and Graham Neubig. 2019. Density matching for bilingual word embedding. In Meeting of the North American Chapter of the Association for Computational Lin- guistics (NAACL), Minneapolis, USA.
- Authors
- Zhou, C., Neubig, G., Gu, J., Diab, M., Guzmán, F., Zettlemoyer, L., & Ghazvininejad, M.
- Affiliations
- Meta (United States) (US)
- Carnegie Mellon University (US)
- Venue
- Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021
- Published
- 2021
- Pages
- 1393-1404
- OpenAlex
- W3173343821
- Cited by
- 122
- References
- 59
- Citations by year
- 2026: 6 · 2025: 14 · 2024: 26 · 2023: 35 · 2022: 25 · 2021: 16
- FWCI
- 11.78
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Findings of the Association for Computational Linguistics: ACL (Findings of ACL) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- hallucination detection, pre-trained language model fine-tuning, machine translation, abstractive summarization, low-resource translation, word-level quality estimation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
