Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 115 of the 115 references it cites were found in a registry or an index.

Factuality challenges in the era of large language models and opportunities for fact-checking

Open access

Published version of a preprint

Abstract

The emergence of tools based on large language models (LLMs), such as OpenAI’s ChatGPT and Google’s Gemini, has garnered immense public attention owing to their advanced natural language generation capabilities. These remarkably natural-sounding tools have the potential to be highly useful for various tasks. However, they also tend to produce false, erroneous or misleading content—commonly referred to as hallucinations. Moreover, LLMs can be misused to generate convincing, yet false, content and profiles on a large scale, posing a substantial societal challenge by potentially deceiving users and spreading inaccurate information. This makes fact-checking increasingly important. Despite their issues with factual accuracy, LLMs have shown proficiency in various subtasks that support fact-checking, which is essential to ensure factually accurate responses. In light of these concerns, we explore issues related to factuality in LLMs and their impact on fact-checking. We identify key challenges, imminent threats and possible solutions to these factuality issues. We also thoroughly examine these challenges, existing solutions and potential prospects for fact-checking. By analysing the factuality constraints within LLMs and their impact on fact-checking, we aim to contribute to a path towards maintaining accuracy at a time of confluence of generative artificial intelligence and misinformation. Large language models (LLMs) present challenges, including a tendency to produce false or misleading content and the potential to create misinformation or disinformation. Augenstein and colleagues explore issues related to factuality in LLMs and their impact on fact-checking.

Overview

Integrity signals

Registry

Crossref · 10.1038/s42256-024-00881-z

Cites

115 of 115 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

The text we found for this paper looks like another document, so its claims and self-check weren’t run. The registry and citation checks still apply.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The text we found for this paper looks like another document, so the self-check wasn’t run.

Citations

Cites

Show all 96 references

Research works printed but not linked: 18

  1. OpenAI. GPT-4 technical report. Preprint at https://arxiv.org/abs/2303.08774 (2023). · arXiv
  2. Iftikhar, L. et al. DocGPT: impact of ChatGPT-3 on health services as a virtual doctor. EC Paediatri. 12, 45–55 (2023).
  3. DeVerna, M. R., Yan, H. Y., Yang, K.-C. & Menczer, F. Fact-checking information generated by a large language model can decrease news discernment. Preprint at https://arxiv.org/abs/2308.10800 (2023). · arXiv
  4. Zhang, H. et al. R-Tuning: teaching large language models to refuse unknown questions. In Proc. 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Vol. 1 (eds Duh, K. et al.) 7113–7139 (ACL, 2024).
  5. Ji, Z. et al. Survey of hallucination in natural language generation. ACM Comput. Surv. https://doi.org/10.1145/3571730 (2023). · DOI · Paper page
  6. Qin, J. et al. Why does new knowledge create messy ripple effects in llms? Preprint at https://arxiv.org/abs/2407.12828 (2024). · arXiv
  7. Zhang, Y. et al. Knowledge overshadowing causes amalgamated hallucination in large language models: analysis and solution. Preprint at https://arxiv.org/abs/2407.08039v1 (2024). · arXiv
  8. Golchin, S. & Surdeanu, M. Time travel in LLMs: tracing data contamination in large language models. In Proc. 12th International Conference on Learning Representations (2024).
  9. Guu, K., Lee, K., Tung, Z., Pasupat, P. & Chang, M.-W. REALM: retrieval-augmented language model pre-training. In Proc. 37th International Conference on Machine Learning 3929–3938 (JMLR, 2020).
  10. Yu, P. & Ji, H. Self information update for large language models through mitigating exposure bias. Preprint at https://arxiv.org/abs/2305.18582 (2023). · arXiv
  11. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. BERTScore: evaluating text generation with BERT. In Proc. 8th International Conference on Learning Representations (2020).
  12. Su, J., Zhuo, T. Y., Mansurov, J., Wang, D. & Nakov, P. Fake news detectors are biased against texts generated by large language models. Preprint at https://arxiv.org/abs/2309.08674 (2023). · arXiv
  13. Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W. & Feizi, S. Can AI-generated text be reliably detected? Preprint at https://arxiv.org/abs/2303.11156 (2023). · arXiv
  14. Zhang, B., Ding, D. & Jing, L. How would stance detection techniques evolve after the launch of ChatGPT? Preprint at https://arxiv.org/abs/2212.14548 (2022). · arXiv
  15. Wang, Y., Wang, M. & Nakov, P. Rethinking STS and NLI in large language models. In Findings of the Association for Computational Linguistics: EACL 2024 965–982 (ACL, 2024).
  16. Yang, K.-C. & Menczer, F. Large language models can rate news outlet credibility. Preprint at https://arxiv.org/abs/2304.00228 (2023). · arXiv
  17. Lu, S., Bigoulaeva, I., Sachdeva, R., Madabushi, H. T. & Gurevych, I. Are emergent abilities in large language models just in-context learning? Preprint at https://arxiv.org/abs/2309.01809 (2023). · arXiv
  18. Gupta, G., Rastegarpanah, B., Iyer, A., Rubin, J. & Kenthapadi, K. Measuring distributional shifts in text: the advantage of language model-based embeddings. Preprint at https://arxiv.org/abs/2312.02337 (2023). · arXiv

Cited by

See all 194

Metadata

Authors
Augenstein, I., Baldwin, T., Cha, M., Chakraborty, T., Ciampaglia, G. L., Corney, D., DiResta, R., Ferrara, E., Hale, S., Halevy, A., Hovy, E., Ji, H., Menczer, F., Miguez, R., Nakov, P., Scheufele, D., Sharma, S., & Zagni, G.
Affiliations
  • University of Copenhagen (DK)
  • Mohamed bin Zayed University of Artificial Intelligence (AE)
  • Max Planck Institute for Security and Privacy (DE)
  • Indian Institute of Technology Delhi (IN)
  • University of Maryland, College Park (US)
  • Full Fact (GB)
  • University of Southern California (US)
  • University of Oxford (GB)
  • Meta (United States) (US)
  • Carnegie Mellon University (US)
  • University of Illinois Urbana-Champaign (US)
  • Indiana University Bloomington (US)
  • University of Wisconsin–Madison (US)
Corresponding author
Chakraborty, Tanmoy
Venue
Nature Machine Intelligence
Published
2024-08-22
Volume
6
Issue
8
Pages
852-863
ISSN
2522-5839 (ISSN-L)
OpenAlex
W4401815218
Cited by
187
References
96
Citations by year
2026: 99 · 2025: 74 · 2024: 14
FWCI
134.13
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Machine Intelligence (publishedVersion)
  • Research at the University of Copenhagen (University of Copenhagen) (publishedVersion, cc-by)
Funders
  • National Science Foundation (2239194, 2229885)
  • John S. and James L. Knight Foundation
  • Craig Newmark Philanthropies
  • European Commission
  • National Research Foundation (RS-2022-00165347, IBS-R029-C2)
  • National Research Foundation of Korea (RS-2022-00165347)
  • Institute for Basic Science (IBS-R029-C2)
  • Defense Advanced Research Projects Agency (HR001121C0169, HR001120C0123)
  • Economic and Social Research Council (ES/X502352/1)
  • HORIZON EUROPE European Research Council (101077481)
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
large language models, fact-checking, factual accuracy, hallucinations, misinformation, disinformation, generative AI, ChatGPT, Gemini

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Factuality challenges in the era of large language models and opportunities for fact-checking. https://app.cochat.ai/papers/10.1038/s42256-024-00881-z
The paper
Augenstein, I., Baldwin, T., Cha, M., Chakraborty, T., Ciampaglia, G. L., Corney, D., DiResta, R., Ferrara, E., Hale, S., Halevy, A., Hovy, E., Ji, H., Menczer, F., Miguez, R., Nakov, P., Scheufele, D., Sharma, S., & Zagni, G. (2024). Factuality challenges in the era of large language models and opportunities for fact-checking. Nature Machine Intelligence. https://doi.org/10.1038/s42256-024-00881-z

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.