The DOI matches its Crossref record; 115 of the 115 references it cites were found in a registry or an index.
Factuality challenges in the era of large language models and opportunities for fact-checking
Open accessPublished version of a preprint
Abstract
The emergence of tools based on large language models (LLMs), such as OpenAI’s ChatGPT and Google’s Gemini, has garnered immense public attention owing to their advanced natural language generation capabilities. These remarkably natural-sounding tools have the potential to be highly useful for various tasks. However, they also tend to produce false, erroneous or misleading content—commonly referred to as hallucinations. Moreover, LLMs can be misused to generate convincing, yet false, content and profiles on a large scale, posing a substantial societal challenge by potentially deceiving users and spreading inaccurate information. This makes fact-checking increasingly important. Despite their issues with factual accuracy, LLMs have shown proficiency in various subtasks that support fact-checking, which is essential to ensure factually accurate responses. In light of these concerns, we explore issues related to factuality in LLMs and their impact on fact-checking. We identify key challenges, imminent threats and possible solutions to these factuality issues. We also thoroughly examine these challenges, existing solutions and potential prospects for fact-checking. By analysing the factuality constraints within LLMs and their impact on fact-checking, we aim to contribute to a path towards maintaining accuracy at a time of confluence of generative artificial intelligence and misinformation. Large language models (LLMs) present challenges, including a tendency to produce false or misleading content and the potential to create misinformation or disinformation. Augenstein and colleagues explore issues related to factuality in LLMs and their impact on fact-checking.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The text we found for this paper looks like another document, so the self-check wasn’t run.
Citations
Cites
A Mathematical Theory of Communication
Shannon, C. E. (1948) · Bell System Technical Journal · doi:10.1002/j.1538-7305.1948.tb01338.x · Cited by 88,486
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“As a result, LLMs are rapidly gaining popularity among professionals in a wide range of technical fields and everyday users with limited or no technical knowledge.”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 1×
“Despite the quantum leap in generating natural language, LLMs tend to generate hallucinations including nonsensical output, incoherent content and factual inaccuracies 41 .”
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · cited 1×
“In addition to being one of the earliest testbeds for studying communication between AI and humans 35 , chatbots also represent the most notable and extensive use of LLMs, providing an interface for a broader audience to solve a multitude of tasks.”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093 · cited 2×
“Despite having achieved impressive capabilities in memorizing and reasoning about natural language, and even though they may Perspective https://doi.org/10.1038/s42256-024-00881-z of inauthentic or misleading content at scale 24 .”
InfluentialA Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.18223 · Cited by 2,090 · cited 2×
“Despite having achieved impressive capabilities in memorizing and reasoning about natural language, and even though they may Perspective https://doi.org/10.1038/s42256-024-00881-z of inauthentic or misleading content at scale 24 .”
InfluentialThe halo effect: Evidence for unconscious alteration of judgments.
Nisbett, R. E., & Wilson, T. D. (1977) · Journal of Personality and Social Psychology · doi:10.1037/0022-3514.35.4.250 · Cited by 2,021 · cited 1×
“The proficiency of a model in addressing one topic might lead users to believe that it can excel in any open-domain conversation, an example of a cognitive bias known as the halo effect 52 .”
Research works printed but not linked: 18
- OpenAI. GPT-4 technical report. Preprint at https://arxiv.org/abs/2303.08774 (2023). · arXiv
- Iftikhar, L. et al. DocGPT: impact of ChatGPT-3 on health services as a virtual doctor. EC Paediatri. 12, 45–55 (2023).
- DeVerna, M. R., Yan, H. Y., Yang, K.-C. & Menczer, F. Fact-checking information generated by a large language model can decrease news discernment. Preprint at https://arxiv.org/abs/2308.10800 (2023). · arXiv
- Zhang, H. et al. R-Tuning: teaching large language models to refuse unknown questions. In Proc. 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Vol. 1 (eds Duh, K. et al.) 7113–7139 (ACL, 2024).
- Ji, Z. et al. Survey of hallucination in natural language generation. ACM Comput. Surv. https://doi.org/10.1145/3571730 (2023). · DOI · Paper page
- Qin, J. et al. Why does new knowledge create messy ripple effects in llms? Preprint at https://arxiv.org/abs/2407.12828 (2024). · arXiv
- Zhang, Y. et al. Knowledge overshadowing causes amalgamated hallucination in large language models: analysis and solution. Preprint at https://arxiv.org/abs/2407.08039v1 (2024). · arXiv
- Golchin, S. & Surdeanu, M. Time travel in LLMs: tracing data contamination in large language models. In Proc. 12th International Conference on Learning Representations (2024).
- Guu, K., Lee, K., Tung, Z., Pasupat, P. & Chang, M.-W. REALM: retrieval-augmented language model pre-training. In Proc. 37th International Conference on Machine Learning 3929–3938 (JMLR, 2020).
- Yu, P. & Ji, H. Self information update for large language models through mitigating exposure bias. Preprint at https://arxiv.org/abs/2305.18582 (2023). · arXiv
- Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. BERTScore: evaluating text generation with BERT. In Proc. 8th International Conference on Learning Representations (2020).
- Su, J., Zhuo, T. Y., Mansurov, J., Wang, D. & Nakov, P. Fake news detectors are biased against texts generated by large language models. Preprint at https://arxiv.org/abs/2309.08674 (2023). · arXiv
- Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W. & Feizi, S. Can AI-generated text be reliably detected? Preprint at https://arxiv.org/abs/2303.11156 (2023). · arXiv
- Zhang, B., Ding, D. & Jing, L. How would stance detection techniques evolve after the launch of ChatGPT? Preprint at https://arxiv.org/abs/2212.14548 (2022). · arXiv
- Wang, Y., Wang, M. & Nakov, P. Rethinking STS and NLI in large language models. In Findings of the Association for Computational Linguistics: EACL 2024 965–982 (ACL, 2024).
- Yang, K.-C. & Menczer, F. Large language models can rate news outlet credibility. Preprint at https://arxiv.org/abs/2304.00228 (2023). · arXiv
- Lu, S., Bigoulaeva, I., Sachdeva, R., Madabushi, H. T. & Gurevych, I. Are emergent abilities in large language models just in-context learning? Preprint at https://arxiv.org/abs/2309.01809 (2023). · arXiv
- Gupta, G., Rastegarpanah, B., Iyer, A., Rubin, J. & Kenthapadi, K. Measuring distributional shifts in text: the advantage of language model-based embeddings. Preprint at https://arxiv.org/abs/2312.02337 (2023). · arXiv
Cited by
Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models
Ferrara, E. (2023) · arXiv · doi:10.48550/arxiv.2304.03738 · Cited by 282
Self-citationShould ChatGPT be biased? Challenges and risks of bias in large language models
Ferrara, E. (2023) · First Monday · doi:10.5210/fm.v28i11.13346 · Cited by 282
Self-citationKnowledge Graphs, Large Language Models, and Hallucinations: An NLP Perspective
Lavrinovics, E., Biswas, R., Bjerva, J., & Hose, K. (2024) · Journal of Web Semantics · doi:10.1016/j.websem.2024.100844 · Cited by 180
“A major flaw that prevents widespread deployment of LLMs is factual inconsistencies, also referred to as hallucinations, which impair trust in AI systems and even pose societal risks in the form of generating convincing false information (Augenstein et al., 2024, Puccetti et al., 2024).”
Theodorakopoulos, L., Theodoropoulou, A., & Klavdianos, C. (2025) · Journal of theoretical and applied electronic commerce research · doi:10.3390/jtaer20020115 · Cited by 130
Fang, L., Yu, X., Cai, J., Chen, Y., Wu, S., Liu, Z., Yang, Z., Lu, H., Gong, X., Liu, Y., et al. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11423-3 · Cited by 124
Metadata
- Authors
- Augenstein, I., Baldwin, T., Cha, M., Chakraborty, T., Ciampaglia, G. L., Corney, D., DiResta, R., Ferrara, E., Hale, S., Halevy, A., Hovy, E., Ji, H., Menczer, F., Miguez, R., Nakov, P., Scheufele, D., Sharma, S., & Zagni, G.
- Affiliations
- University of Copenhagen (DK)
- Mohamed bin Zayed University of Artificial Intelligence (AE)
- Max Planck Institute for Security and Privacy (DE)
- Indian Institute of Technology Delhi (IN)
- University of Maryland, College Park (US)
- Full Fact (GB)
- University of Southern California (US)
- University of Oxford (GB)
- Meta (United States) (US)
- Carnegie Mellon University (US)
- University of Illinois Urbana-Champaign (US)
- Indiana University Bloomington (US)
- University of Wisconsin–Madison (US)
- Corresponding author
- Chakraborty, Tanmoy
- ORCID
- Venue
- Nature Machine Intelligence
- Published
- 2024-08-22
- Volume
- 6
- Issue
- 8
- Pages
- 852-863
- ISSN
- 2522-5839 (ISSN-L)
- OpenAlex
- W4401815218
- Cited by
- 187
- References
- 96
- Citations by year
- 2026: 99 · 2025: 74 · 2024: 14
- FWCI
- 134.13
- Type
- journal article
- Language
- en
- Open access
- Licence
- cc-by
- www.springernature.com/gp/researchers/text-and-data-mining (tdm)
- www.springernature.com/gp/researchers/text-and-data-mining (vor)
- Locations
- Nature Machine Intelligence (publishedVersion)
- Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by)
- Funders
- National Science Foundation (2239194, 2229885)
- John S. and James L. Knight Foundation
- Craig Newmark Philanthropies
- European Commission
- National Research Foundation (RS-2022-00165347, IBS-R029-C2)
- National Research Foundation of Korea (RS-2022-00165347)
- Institute for Basic Science (IBS-R029-C2)
- Defense Advanced Research Projects Agency (HR001121C0169, HR001120C0123)
- Economic and Social Research Council (ES/X502352/1)
- HORIZON EUROPE European Research Council (101077481)
- Topics
- Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- large language models, fact-checking, factual accuracy, hallucinations, misinformation, disinformation, generative AI, ChatGPT, Gemini
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Factuality challenges in the era of large language models and opportunities for fact-checking. https://app.cochat.ai/papers/10.1038/s42256-024-00881-z
- The paper
- Augenstein, I., Baldwin, T., Cha, M., Chakraborty, T., Ciampaglia, G. L., Corney, D., DiResta, R., Ferrara, E., Hale, S., Halevy, A., Hovy, E., Ji, H., Menczer, F., Miguez, R., Nakov, P., Scheufele, D., Sharma, S., & Zagni, G. (2024). Factuality challenges in the era of large language models and opportunities for fact-checking. Nature Machine Intelligence. https://doi.org/10.1038/s42256-024-00881-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.