Do Language Models Know When They’re Hallucinating References?
Open accessAbstract
State-of-the-art language models (LMs) are notoriously susceptible to generating hallucinated information. Such inaccurate outputs not only undermine the reliability of these models but also limit their use and raise serious concerns about misinformation and propaganda. In this work, we focus on hallucinated book and article references and present them as the "model organism" of language model hallucination research, due to their frequent and easy-to-discern nature. We posit that if a language model cites a particular reference in its output, then it should ideally possess sufficient information about its authors and content, among other relevant details. Using this basic insight, we illustrate that one can identify hallucinated references without ever consulting any external resources, by asking a set of direct or indirect queries to the language model about the references. These queries can be considered as "consistency checks." Our findings highlight that while LMs, including GPT-4, often produce inconsistent author lists for hallucinated references, they also often accurately recall the authors of real references. In this sense, the LM can be said to "know" when it is hallucinating references. Furthermore, these findings show how hallucinated references can be dissected to shed light on their nature. Replication code and results can be found at https://github.com/microsoft/hallucinated-references.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Language models know when they are hallucinating.
“Another line of work that sheds light on the nature of hallucinations shows that LMs know when they’re hallucinating [1,3,18,20].”
From Calibrated Language Models Must Hallucinate (2024) · cites this paper as [1]
The Introduction describes testing whether LMs can recognize their own hallucinated references. The Conclusions report that self-detection methods are only partially reliable and leave their generalizability to other kinds of hallucination open. Thus, the paper supports a narrower finding than the unqualified claim that LMs know when they are hallucinating. Page numbers are not provided in our copy. (LLM, conf=0.88)
Experimental — accuracy not yet measuredGPT-3.5 (ChatGPT) produces false citations more often than not when asked to provide the authors of a given study.
“One recent study shows that GPT-3.5 (ChatGPT) produces false citations more often than not when asked to provide the authors of a given study (Agrawal et al., 2023).”
From Fine-tuning Language Models for Factuality (2023) · cites this paper as Agrawal et al., 2023
Section 6.1 (Table 1; page not available in our copy) reports that 59.6% of ChatGPT-generated reference titles were classified as hallucinations, supporting the ‘more often than not’ portion. Section 4.2 says the authors also asked ChatGPT for the authors of generated references, but Table 1 does not report a false-citation rate for those author-list requests. The claim conflates the two tasks. (LLM, conf=0.88)
Experimental — accuracy not yet measuredReference-free querying is rooted in the theory that hallucinations are more likely to originate in flat probability distributions with higher next-token uncertainties, whereas factual answers should always have a high probability of being the generated response given a prompt.
“This approach is rooted in the theory that hallucinations are more likely to originate in flat probability distributions with higher next-token uncertainties, whereas factual answers should always have a high probability of being the generated response given a prompt.”
From Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models (2024) · cites this paper as Agrawal et al. 2023
Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.
Experimental — accuracy not yet measuredLarge language models are prone to confidently generating incorrect claims.
“However, despite these large language models' exposure to diverse datasets, they are prone to confidently generating incorrect claims.”
From Fine-tuning Language Models for Factuality (2023) · cites this paper as Agrawal et al., 2023
Agrawal et al. advocated for the use of indirect queries, which often pose open-ended questions to elicit specific information, and by employing these indirect queries, consistency across multiple model generations can be better evaluated.
“Agrawal et al. [3], inspired by investigative interviews, advocated for the use of indirect queries. Unlike direct ones, these indirect counterparts often pose open-ended questions to elicit specific information. By employing these indirect queries, consistency across multiple model generations can be better evaluated.”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 25
- Sai Anirudh Athaluri, Sandeep Varma Manthena, V S R Krishna Manoj Kesapragada, Vineel Yarlagadda, Tirth Dave, and Rama Tulasi Siri Duddumpudi. 2023. Exploring the Boundaries of Reality: Investigating the Phenomenon of Artificial Intelligence Hallucina- tion in Scientific Writing Through ChatGPT Refer- ences. Cureus. · DOI · Paper page
- Amos Azaria and Tom Mitchell. 2023. The In- ternal State of an LLM Knows When its Lying. ArXiv:2304.13734 [cs. · DOI · Paper page
- Sébastien Bubeck, Varun Chandrasekaran, Ronen El- dan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lund- berg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of Artificial General Intelligence: Early experiments with GPT-4. ArXiv:2303.12712 [cs. · DOI · Paper page
- Elizabeth R DeLong, David M DeLong, and Daniel L Clarke-Pearson. 1988. Comparing the areas under two or more correlated receiver operating character- istic curves: a nonparametric approach. Biometrics, pages 837-845.
- Owain Evans, Owen Cotton-Barratt, Lukas Finnve- den, Adam Bales, Avital Balwit, Peter Wills, Luca Righetti, and William Saunders. 2021. Truthful AI: Developing and governing AI that does not lie. ArXiv:2110.06674 [cs]. · DOI · Paper page
- Tom Fawcett. 2006. An introduction to roc analysis. Pattern recognition letters, 27(8):861-874.
- Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pages 3816-3830.
- Zhijiang Guo, Michael Schlichtkrull, and Andreas Vla- chos. 2022. A Survey on Automated Fact-Checking. Transactions of the Association for Computational Linguistics, 10:178-206. · DOI · Paper page
- Kelvin Guu, Albert Webson, Ellie Pavlick, Lucas Dixon, Ian Tenney, and Tolga Bolukbasi. 2023. Simfluence: Modeling the Influence of Individual Training Examples by Simulating Training Runs. ArXiv:2303.08114 [cs]. · DOI
- Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hal- lucination in Natural Language Generation. ACM Computing Surveys, 55(12):1-38. · DOI · Paper page
- Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
- Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, Deep Ganguli, Danny Hernandez, Josh Jacobson, Jackson Kernion, Shauna Kravec, Liane Lovitt, Ka- mal Ndousse, Catherine Olsson, Sam Ringer, Dario Amodei, Tom Brown, Jack Clark, Nicholas Joseph, Ben Mann, Sam McCandlish, Chris Olah, and Jared Kaplan. 2022. Language Models (Mostly) Know What They Know. ArXiv:2207.05221 [cs. · DOI · Paper page
- Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Teaching Models to Express Their Uncertainty in Words. ArXiv:2205.14334 [cs. · DOI · Paper page
- Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 8086-8098.
- Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Lan- guage Models. ArXiv:2303.08896 [cs]. · arXiv
- Mary L McHugh. 2012. Interrater reliability: the kappa statistic. Biochemia medica, 22(3):276-282.
- OpenAI. 2023. GPT-4 Technical Report. ArXiv:2303.08774 · DOI · Paper page
- Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. ArXiv:2203.02155 [cs]. · DOI · Paper page
- Bernard Rous. 2012. Major update to ACM's Comput- ing Classification System. Communications of the ACM, 55(11):12. · DOI
- Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with au- tomatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235.
- Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, ...(421- others), and Ziyi Wu. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. · DOI · Paper page
- Nathaniel Swinger, Maria De-Arteaga, Neil Thomas Heffernan IV, Mark DM Leiserson, and Adam Tau- man Kalai. 2019. What are the biases in my word embedding? In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 305- 311.
- Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. · arXiv
- Annelies Vredeveldt, Peter J. van Koppen, and Pär An- ders Granhag. 2014. The Inconsistent Suspect: A Systematic Review of Different Types of Consistency in Truth Tellers and Liars. In Ray Bull, editor, Inves- tigative Interviewing, pages 183-207. Springer, New York, NY. · DOI
- Ludwig Wittgenstein. 1953. Philosophical Investiga- tions. Wiley-Blackwell, New York, NY, USA.
Cited by
Sorted by citations
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
Dahl, M., Magesh, V., Suzgun, M., & Ho, D. E. (2024) · Journal of Legal Analysis · doi:10.1093/jla/laae003 · Cited by 288 · see citing works
Self-citationChain-of-Verification Reduces Hallucination in Large Language Models
Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., & Weston, J. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.212 · Cited by 282 · see citing works
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516 · see citing works
Hallucination‐Free? Assessing the Reliability of Leading <scp>AI</scp> Legal Research Tools
Magesh, V., Surani, F., Dahl, M., Suzgun, M., Manning, C. D., & Ho, D. E. (2025) · Journal of Empirical Legal Studies · doi:10.1111/jels.12413 · Cited by 182 · see citing works
Self-citationTowards trustworthy LLMs: a review on debiasing and dehallucinating in large language models
Lin, Z., Guan, S., Zhang, W., Zhang, H., Li, Y., & Zhang, H. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10896-y · Cited by 141 · see citing works
Calibrated Language Models Must Hallucinate
Kalai, A. T., & Vempala, S. S. (2024) · Proceedings of the 56th Annual ACM Symposium on Theory of Computing · doi:10.1145/3618260.3649777 · Cited by 91 · see citing works
Self-citationThe Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models
Li, J., Chen, J., Ren, R., Cheng, X., Zhao, X., Nie, J.-Y., & Wen, J.-R. (2024) · Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2024.acl-long.586 · Cited by 70 · see citing works
Cognitive Mirage: A Review of Hallucinations in Large Language Models
Ye, H., Liu, T., Zhang, A., Hua, W., & Jia, W. (2023) · arXiv · doi:10.48550/arxiv.2309.06794 · Cited by 61 · see citing works
Chain-of-Verification Reduces Hallucination in Large Language Models
Dhuliawala, S., Komeili, M., Xu, J., Raileanu, R., Li, X., Celikyilmaz, A., & Weston, J. (2023) · arXiv · doi:10.48550/arxiv.2309.11495 · Cited by 42 · see citing works
Language models for materials discovery and sustainability: Progress, challenges, and opportunities
Pei, Z., Yin, J., & Zhang, J. (2025) · Progress in Materials Science · doi:10.1016/j.pmatsci.2025.101495 · Cited by 30 · see citing works
ExpertQA: Expert-Curated Questions and Attributed Answers
Malaviya, C., Lee, S., Chen, S., Sieber, E., Yatskar, M., & Roth, D. (2024) · Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) · doi:10.18653/v1/2024.naacl-long.167 · Cited by 28 · see citing works
Measuring short-form factuality in large language models
Wei, J., Karina, N., Chung, H. W., Jiao, Y. J., Papay, S., Glaese, A., Schulman, J., & Fedus, W. (2024) · arXiv · doi:10.48550/arxiv.2411.04368 · Cited by 14 · see citing works
“Other work has attempted to improve the calibration of language models (Lin et al., 2022b; Agrawal et al., 2024), which could lead to increased adoption of language models in real-world settings.”
BackgroundFine-tuning Language Models for Factuality
Tian, K., Mitchell, E., Yao, H., Manning, C. D., & Finn, C. (2023) · arXiv · doi:10.48550/arxiv.2311.08401 · Cited by 14 · see citing works
“5 (ChatGPT) produces false citations more often than not when asked to provide the authors of a given study (Agrawal et al., 2023).”
Marey, A., Serdysnki, K. C., Killeen, B. D., Unberath, M., & Umair, M. (2024) · BJR|Artificial Intelligence · doi:10.1093/bjrai/ubae008 · Cited by 7 · see citing works
van Rensburg, L. J. J. (2025) · arXiv · doi:10.48550/arxiv.2511.04683 · Cited by 1
Phantom References: Hallucinated Citations That Survive Peer Review at Top-Tier Conferences
Russinovich, M., Kumar, R. S. S., & Salem, A. (2026) · arXiv · doi:10.48550/arxiv.2607.00738
“Agrawal et al. [18] further show that models can often signal when a reference they produced is fabricated, and Rao and Callison-Burch [8] show that pairing models with deterministic retrieval sharply improves BibTex generation.”
HalluHard: A Hard Multi-Turn Hallucination Benchmark
Fan, D., Delsad, S., Flammarion, N., & Andriushchenko, M. (2026) · arXiv · doi:10.48550/arxiv.2602.01031
“In-parameter hallucination is commonly evaluated using short-form factual prompts (e.g., QA-style queries) (Lin et al., 2022; Wei et al., 2024a; Agrawal et al., 2024; Pandit et al., 2025), and also via long-form generation where responses are broken into atomic facts that can be verified one by one…”
Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2411.15594
From Illusion to Insight: A Taxonomic Survey of Hallucination Mitigation Techniques in LLMs
Kazlaris, I., Antoniou, E., Diamantaras, K., & Bratsas, C. (2025) · doi:10.20944/preprints202508.1942.v1 · Cited by 13 · see citing works
- Authors
- Agrawal, A., Suzgun, M., Mackey, L., & Kalai, A. T.
- Affiliations
- Microsoft (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- Pages
- 912-928
- arXiv
- 2305.18248
- OpenAlex
- W7126447459
- Cited by
- 37
- References
- 0
- Citations by year
- 2026: 14 · 2025: 12 · 2024: 8 · 2023: 3
- FWCI
- 10.03
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Findings of the Association for Computational Linguistics: EACL (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- consistency checking, GPT-4, misinformation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
