CoChat
Sign Up
PreprintCited by 38Misinformation and Its ImpactsEthics and Social Impacts of AITopic Modeling

Factuality Challenges in the Era of Large Language Models

Open access
Augenstein, Isabelle, Baldwin, Timothy, Cha, Meeyoung, Chakraborty, Tanmoy, Ciampaglia, Giovanni Luca, Corney, David, DiResta, Renee, Ferrara, Emilio +10 more·2023·arXiv
DOI 10.48550/arxiv.2310.05189arXiv 2310.05189

Preprint — published as 10.1038/s42256-024-00881-z · Paper page

Open this paper in CoChat — free accountOpen PDF

Abstract

The emergence of tools based on Large Language Models (LLMs), such as OpenAI's ChatGPT, Microsoft's Bing Chat, and Google's Bard, has garnered immense public attention. These incredibly useful, natural-sounding tools mark significant advances in natural language generation, yet they exhibit a propensity to generate false, erroneous, or misleading content -- commonly referred to as "hallucinations." Moreover, LLMs can be exploited for malicious applications, such as generating false but credible-sounding content and profiles at scale. This poses a significant challenge to society in terms of the potential deception of users and the increasing dissemination of inaccurate information. In light of these risks, we explore the kinds of technological innovations, regulatory reforms, and AI literacy initiatives needed from fact-checkers, news organizations, and the broader research and policy communities. By identifying the risks, the imminent threats, and some viable solutions, we seek to shed light on navigating various aspects of veracity in the era of generative AI.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2310.05189

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Misinformation infiltrates generative model outputs through two primary pathways: memorization of inaccuracies from the source dataset and generation of high-likelihood sequences containing clear factual errors.

    “This analysis brings us to two primary pathways through which misinformation infiltrates the outputs of generative models: the memorization of inaccuracies from the source dataset and the generation of high-likelihood sequences that, despite the model's prediction, contain clear factual errors (Augenstein et al. 2023).”

    From Epistemic Injustice in Generative AI (2024) · cites this paper as Augenstein et al. 2023

    The Introduction and §2.1 Factuality Challenges say LLMs can generate factually false content, and the Introduction notes that training datasets can introduce biases. But our copy does not identify memorized dataset inaccuracies and high-likelihood yet erroneous sequences as two primary pathways for misinformation; searches for memorization and likelihood did not establish those mechanisms. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Google Search can be leveraged to obtain dynamic reference answers for each fact in a response with the best tradeoff among accuracy, accessibility, and timeliness.

    “Google Search can be leveraged to obtain dynamic reference answers for each fact in a response with the best tradeoff among accuracy, accessibility, and timeliness (Augenstein et al., 2023).”

    From Long-form factuality in large language models (2024) · cites this paper as Augenstein et al., 2023

    The Introduction (page unavailable) says search engines such as Google are more reliable than ChatGPT because they clearly indicate their sources. It does not say Google Search provides dynamic reference answers for each fact or offers the best tradeoff among accuracy, accessibility, and timeliness. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Google Search holds a large amount of information about many subjects that allows it to serve as a strong proxy for factual accuracy.

    “Google Search holds a large amount of information about many subjects that allows it to serve as a strong proxy for factual accuracy (Augenstein et al., 2023).”

    From Long-form factuality in large language models (2024) · cites this paper as Augenstein et al., 2023

    The Introduction (page unavailable) describes search engines such as Google as historically reliable gateways to authoritative sources, while noting that they have sometimes amplified disinformation. It does not say that Google Search's breadth of information makes it a strong proxy for factual accuracy; the Unreliable Evaluation section instead describes maintaining factuality ground truth as an open problem. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Still being checked.

Part of this paper’s reference list could not be split into single entries.

Research works printed but not linked: 58

  1. Shannon, C. E. A mathematical theory of communication. The Bell Syst. Tech. J. 27, 379-423, DOI: 10.1002/j.1538-7305.1948.tb01338.x (1948). · DOI · Paper page
  2. Huang, J. & Chang, K. C.-C. Towards Reasoning in Large Language Models: A Survey. In Findings of the Associ- ation for Computational Linguistics: ACL 2023, 1049-1065, DOI: 10.18653/v1/2023.findings-acl.67 (Association for Computational Linguistics, Toronto, Canada, 2023). · DOI · Paper page
  3. OpenAI. GPT-4 technical report. arXiv:2303.08774 (2023). · arXiv
  4. Touvron, H. et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023). · arXiv
  5. Thoppilan, R. et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239 (2022). · arXiv
  6. Sengupta, N. et al. Jais and Jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint 2308.16149 (2023). 2308.16149. · arXiv
  7. Radford, A. et al. Language models are unsupervised multitask learners. OpenAI blog 1, 9 (2019).
  8. Brown, T. et al. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, vol. 33, 1877-1901 (
  9. Zhao, W. X. et al. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023). · arXiv
  10. Li, S. et al. Open-domain hierarchical event schema induction by incremental prompting and verification. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 5677-5697, DOI: 10.18653/v1/2023.acl-long.312 (Association for Computational Linguistics, Toronto, Canada, 2023). · DOI
  11. Wei, J. et al. Emergent abilities of large language models. Trans. Mach. Learn. Res. 2022 (2022).
  12. Bang, Y. et al. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Inter- activity. Preprint 2302.04023, arXiv (2023). DOI: 10.48550/arXiv.2302.04023. · DOI · Paper page
  13. Sison, A. J. G., Daza, M. T., Gozalo-Brizuela, R. & Garrido-Merchán, E. C. ChatGPT: More Than a "Weapon of Mass Deception" -Ethical Challenges and Responses from the Human-Centered Artificial Intelligence (HCAI) Perspective. Int. J. Human-Computer Interact. 0, 1-20, DOI: 10.1080/10447318.2023.2225931 (2023). https://doi.org/10.1080/10447318.2023.2225931. · DOI · Paper page
  14. Chin, H. et al. User-chatbot conversations during the COVID-19 pandemic: Study based on topic modeling and senti- ment analysis. J. Med. Internet Res. 25, e40922, DOI: 10.2196/40922 (2023). · DOI
  15. Palanica, A., Flaschner, P., Thommandram, A., Li, M. & Fossat, Y. Physicians' perceptions of chatbots in health care: Cross-sectional web-based survey. J Med Internet Res 21, e12887, DOI: 10.2196/12887 (2019). · DOI · Paper page
  16. Peskoff, D. & Stewart, B. Credible without credit: Domain experts assess generative language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 427-438, DOI: 10.18653/v1/2023.acl-short.37 (Association for Computational Linguistics, Toronto, Canada, 2023). · DOI
  17. Srivastava, B. Did chatbots miss their "Apollo moment"? Potential, gaps, and lessons from using collaboration assistants during COVID-19. Patterns 2, 100308, DOI: https://doi.org/10.1016/j.patter.2021.100308 (2021). · DOI
  18. Menczer, F., Crandall, D., Ahn, Y.-Y. & Kapadia, A. Addressing the harms of AI-generated inauthentic content. Nat. Mach. Intell. 5, 678-680, DOI: 10.1038/s42256-023-00690-w (2023). · DOI · Paper page
  19. Yang, K.-C. & Menczer, F. Anatomy of an AI-powered malicious social botnet. Preprint 2307.16336, arXiv (2023). DOI: 10.48550/arXiv.2307.16336. · DOI
  20. DeVerna, M. R., Yan, H. Y., Yang, K.-C. & Menczer, F. Artificial intelligence is ineffective and potentially harmful for fact checking. Preprint 2308.10800, arXiv (2023). DOI: 10.48550/arXiv.2308.10800. · DOI
  21. Ferrara, E. The history of digital spam. Commun. ACM 62, 82-91, DOI: 10.1145/3299768 (2019). · DOI
  22. Pan, Y. et al. On the risk of misinformation pollution with large language models. arXiv preprint 2305.13661 (2023). 2305.13661. · arXiv
  23. Mirsky, Y. & Lee, W. The Creation and Detection of Deepfakes: A Survey. ACM Comput. Surv. 54, DOI: 10.1145/3425780 (2021). · DOI · Paper page
  24. Ji, Z. et al. Survey of hallucination in natural language generation. ACM Comput. Surv. 55, DOI: 10.1145/3571730 (2023). · DOI · Paper page
  25. Kusunose, K., Kashima, S. & Sata, M. Evaluation of the accuracy of ChatGPT in answering clinical questions on the Japanese Society of Hypertension Guidelines. Circ. J. 87, 1030-1033, DOI: 10.1253/circj.CJ-23-0308 (2023). · DOI
  26. Antaki, F., Touma, S., Milad, D., El-Khoury, J. & Duval, R. Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of Its Successes and Shortcomings. Ophthalmol. Sci. 3, 100324, DOI: https://doi.org/10.1016/j.xops.2023.100324 (2023). · DOI · Paper page
  27. Bai, H., Voelkel, J. G., Eichstaedt, j. C. & Willer, R. Artificial intelligence can persuade humans on political issues, DOI: 10.31219/osf.io/stakv (2023). · DOI
  28. Brashier, N. M. & Marsh, E. J. Judging truth. Annu. review psychology 71, 499-515 (2020).
  29. Srivastava, A. et al. Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models. Transactions on Mach. Learn. Res. (2023).
  30. Wang, A. et al. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceed- ings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 353-355, DOI: 10.18653/v1/W18-5446 (Association for Computational Linguistics, Brussels, Belgium, 2018). · DOI · Paper page
  31. Wang, A. et al. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In Pro- ceedings of the 33rd International Conference on Neural Information Processing Systems, 294 (Curran Associates Inc., Red Hook, NY, USA, 2019).
  32. Lin, S., Hilton, J. & Evans, O. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 3214-3252, DOI: 10.18653/v1/2022.acl-long.229 (Association for Computational Linguistics, Dublin, Ireland, 2022). · DOI · Paper page
  33. Min, S. et al. FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation. arXiv preprint 2305.14251 (2023). 2305.14251. · arXiv
  34. Lee, N., Bang, Y., Madotto, A. & Fung, P. Towards few-shot fact-checking via perplexity. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1971-1981, DOI: 10.18653/v1/2021.naacl-main.158 (Association for Computational Linguistics, Online, 2021). · DOI
  35. Truong, B. T., Lou, X., Flammini, A. & Menczer, F. Vulnerabilities of the online public square to manipulation. Preprint 1907.06130, arXiv (2023). DOI: 10.48550/arXiv.1907.06130. · DOI
  36. Avram, M., Micallef, N., Patil, S. & Menczer, F. Exposure to social engagement metrics increases vulnerability to misinformation. HKS Misinformation Rev. 1, DOI: 10.37016/mr-2020-033 (2020). · DOI · Paper page
  37. Pierri, F. et al. Online misinformation is linked to early COVID-19 vaccination hesitancy and refusal. Sci. Reports 12, 5966, DOI: 10.1038/s41598-022-10070-w (2022). · DOI · Paper page
  38. Wang, Y., Li, H., Han, X., Nakov, P. & Baldwin, T. Do-not-answer: A dataset for evaluating safeguards in llms. arXiv preprint 2308.13387 (2023). 2308.13387. · arXiv
  39. Yu, W. et al. A survey of knowledge-enhanced text generation. ACM Comput. Surv. 54, DOI: 10.1145/3512467 (2022). · DOI · Paper page
  40. Guu, K., Lee, K., Tung, Z., Pasupat, P. & Chang, M.-W. Realm: retrieval-augmented language model pre-training. ArXiv (2020).
  41. Filippova, K. Controlled hallucinations: learning to generate faithfully from noisy data. In Findings of the Association for Computational Linguistics: EMNLP 2020, 864-870, DOI: 10.18653/v1/2020.findings-emnlp.76 (Association for Computational Linguistics, Online, 2020). · DOI · Paper page
  42. Gou, Z. et al. Critic: large language models can self-correct with tool-interactive critiquing. In arxiv (2023).
  43. Cohen, R., Hamri, M., Geva, M. & Globerson, A. Lm vs lm: detecting factual errors via cross examination. In arxiv (2023).
  44. Du, Y., Li, S., Torralba, A., Tenenbaum, J. B. & Mordatch, I. Improving factuality and reasoning in language models through multiagent debate. In arxiv (2023).
  45. Dziri, N., Madotto, A., Zaïane, O. & Bose, A. J. Neural path hunter: reducing hallucination in dialogue systems via path grounding. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2197- 2214, DOI: 10.18653/v1/2021.emnlp-main.168 (Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2021). · DOI · Paper page
  46. De Cao, N., Aziz, W. & Titov, I. Editing factual knowledge in language models. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, 6491-6506, DOI: 10.18653/v1/2021.emnlp-main.522 (Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 2021). · DOI · Paper page
  47. Mitchell, E., Lin, C., Bosselut, A., Finn, C. & Manning, C. D. Fast model editing at scale. arXiv preprint arXiv:2110.11309 (2021). · arXiv
  48. Mitchell, E., Lin, C., Bosselut, A., Manning, C. D. & Finn, C. Memory-based model editing at scale. In International Conference on Machine Learning, 15817-15831 (PMLR, 2022).
  49. Meng, K., Bau, D., Andonian, A. & Belinkov, Y. Locating and editing factual associations in gpt. Adv. Neural Inf. Process. Syst. 35, 17359-17372 (2022).
  50. Meng, K., Sharma, A. S., Andonian, A., Belinkov, Y. & Bau, D. Mass-editing memory in a transformer. arXiv preprint arXiv:2210.07229 (2022). · arXiv
  51. Schmidt, F. Generalization in Generation: A closer look at Exposure Bias. In Proceedings of the 3rd Workshop on Neural Generation and Translation, 157-167, DOI: 10.18653/v1/D19-5616 (Association for Computational Linguistics, Hong Kong, 2019). · DOI
  52. Yu, P. & Ji, H. Self information update for large language models through mitigating exposure bias. arXiv preprint arXiv:2305.18582 (2023). · arXiv
  53. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q. & Artzi, Y. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 (OpenReview.net, 2020).
  54. Zhao, W. et al. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 563-578, DOI: 10.18653/v1/D19-1053 (Association for Computational Linguistics, Hong Kong, China, 2019). · DOI · Paper page
  55. Sebastian, G. Privacy and data protection in chatgpt and other ai chatbots: Strategies for securing user information. Int. J. Secur. Priv. Pervasive Comput. 15, 1-14, DOI: 10.4018/IJSPPC.325475 (2023). · DOI
  56. Wang, Y. et al. M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection. arXiv:2305.14902 (2023). · arXiv
  57. Fung, Y. et al. Infosurgeon: cross-media fine-grained information consistency checking for fake news detection. In Proc. The Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2021) (2021).
  58. Huang, K.-H., McKeown, K., Nakov, P., Choi, Y. & Ji, H. Faking fake news for real fake news detection: propaganda- loaded training data generation. In Proc. The 61st Annual Meeting of the Association for Computational Linguistics (ACL2023) Findings (2023).

Cited by

Sorted by citations

Related

Authors
Augenstein, I., Baldwin, T., Cha, M., Chakraborty, T., Ciampaglia, G. L., Corney, D., DiResta, R., Ferrara, E., Hale, S., Halevy, A., Hovy, E., Ji, H., Menczer, F., Miguez, R., Nakov, P., Scheufele, D., Sharma, S., & Zagni, G.
Affiliations
  • University of Copenhagen (DK)
  • Mohamed bin Zayed University of Artificial Intelligence (AE)
  • Indian Institute of Technology Delhi (IN)
  • University of Maryland, College Park (US)
  • Full Fact (GB)
  • University of Southern California (US)
  • University of Oxford (GB)
  • Meta (United States) (US)
  • Carnegie Mellon University (US)
  • University of Illinois Urbana-Champaign (US)
  • Indiana University Bloomington (US)
  • University of Wisconsin–Madison (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2310.05189
arXiv
2310.05189
OpenAlex
W4387559778
Cited by
32
References
0
Citations by year
2026: 1 · 2025: 3 · 2024: 25 · 2023: 3
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • National Science Foundation (2239194)
  • John S. and James L. Knight Foundation
  • Craig Newmark Philanthropies
  • European Commission
  • National Research Foundation (RS-2022-00165347, IBS-R029-C2)
  • National Research Foundation of Korea (RS-2022-00165347)
  • Institute for Basic Science (IBS-R029-C2)
  • Defense Advanced Research Projects Agency (HR001121C0169, HR001120C0123)
  • HORIZON EUROPE European Research Council (101077481)
Topics
  • Social Sciences › Sociology and Political Science › Misinformation and Its Impacts
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
large language models, hallucinations, factual accuracy, misinformation, generative AI, AI-generated content, AI literacy, fact-checking, AI regulation, deception, ChatGPT, Bing Chat, Google Bard

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review