Why Does ChatGPT Fall Short in Providing Truthful Answers?
Open accessAbstract
Recent advancements in large language models, such as ChatGPT, have demonstrated significant potential to impact various aspects of human life. However, ChatGPT still faces challenges in providing reliable and accurate answers to user questions. To better understand the model's particular weaknesses in providing truthful answers, we embark an in-depth exploration of open-domain question answering. Specifically, we undertake a detailed examination of ChatGPT's failures, categorized into: comprehension, factuality, specificity, and inference. We further pinpoint factuality as the most contributing failure and identify two critical abilities associated with factuality: knowledge memorization and knowledge recall. Through experiments focusing on factuality, we propose several potential enhancement strategies. Our findings suggest that augmenting the model with granular external knowledge and cues for knowledge recall can enhance the model's factuality in answering questions.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most existing work focuses on investigating the causes of hallucination for specific tasks and small language models.
“Most exiting work mainly focuses on investigating the causes of hallucination for specific tasks and small language models (Cao et al., 2022;Zheng et al., 2023;Das et al., 2023).”
From HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models (2023) · cites this paper as Zheng et al., 2023
The Related Work section says prior ChatGPT studies predominantly categorize and identify problems, with limited in-depth investigation of the underlying deficiencies causing failures—the opposite of the claim that most existing work investigates causes. Our copy does not establish that most such work concerns small language models. Page numbers were not provided. (LLM, conf=0.84)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe paper uses hallucination patterns from prior work: four QA patterns (comprehension, factualness, specificity, inference), three dialogue patterns (extrinsic-soft, extrinsic-hard, extrinsic-grouped), and three summarization patterns (factual, non-factual, intrinsic).
“Specifically, we consider four types of hallucination patterns for question answering (i.e., comprehension, factualness, specificity, and inference) (Zheng et al., 2023), three types of hallucination patterns for knowledgegrounded dialogue (i.e., extrinsic-soft, extrinsichard, and extrinsic-grouped) (Das et al., 2023), and three types of hallucination patterns for text summarization (i.e., factual”
From HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models (2023) · cites this paper as Zheng et al., 2023
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
Cited by 40
Research works printed but not linked: 16
- The process of question answering -a computer simulation of cognition. American Journal of Computational Linguistics, 6(3-4), 1980.
- D. Adiwardana, M.-T. Luong, D. R. So, J. Hall, N. Fiedel, R. Thoppilan, Z. Yang, A. Kul- shreshtha, G. Nemade, Y. Lu, et al. Towards a human-like open-domain chatbot. ArXiv preprint, abs/2001.09977, 2020. · arXiv
- S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. ArXiv preprint, abs/2303.12712, 2023. · arXiv
- C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924-2936, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-1300. · DOI
- J. Huang and K. C.-C. Chang. Towards reasoning in large language models: A survey. In Find- ings of the Association for Computational Linguistics: ACL 2023. Association for Computational Linguistics, 2023.
- J. Huang, H. Shao, and K. C.-C. Chang. Are large pre-trained language models leaking your personal information? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2038-2047, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics.
- J. Huang, K. C.-C. Chang, J. Xiong, and W.-m. Hwu. Can language models be specific? how? In Findings of the Association for Computational Linguistics: ACL 2023. Association for Computa- tional Linguistics, 2023a.
- J. Huang, W. Ping, P. Xu, M. Shoeybi, K. C.-C. Chang, and B. Catanzaro. Raven: In-context learning with retrieval augmented encoder-decoder language models. arXiv preprint arXiv:2308.07922, 2023b. · arXiv
- G. Izacard, P. Lewis, M. Lomeli, L. Hosseini, F. Petroni, T. Schick, J. Dwivedi-Yu, A. Joulin, S. Riedel, and E. Grave. Atlas: Few-shot learning with retrieval augmented language models. arXiv preprint arXiv, 2208, 2022.
- M. Komeili, K. Shuster, and J. Weston. Internet-augmented dialogue generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8460-8478, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.579. · DOI · Paper page
- M. McHugh. Interrater reliability: The kappa statistic. Biochemia medica : časopis Hrvatskoga društva medicinskih biokemičara / HDMB, 22:276-82, 2012. doi: 10.11613/BM.2012.031. · DOI · Paper page
- F. Petroni, T. Rocktäschel, S. Riedel, P. Lewis, A. Bakhtin, Y. Wu, and A. Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1250. · DOI · Paper page
- P. P. Ray. Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems, 2023.
- W. Shi, S. Min, M. Yasunaga, M. Seo, R. James, M. Lewis, L. Zettlemoyer, and W.-t. Yih. Replug: Retrieval-augmented black-box language models. ArXiv preprint, abs/2301.12652, 2023. · arXiv
- Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1259. · DOI · Paper page
- T. Y. Zhuo, Y. Huang, C. Chen, and Z. Xing. Exploring ai ethics of chatgpt: A diagnostic analysis, 2023. This figure "fig_code.png" is available in "png" format from: http://arxiv.org/ps/2304.10513v3
- Authors
- Zheng, S., Huang, J., & Chang, K. C.-C.
- Affiliations
- University of Illinois Urbana-Champaign (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2304.10513
- OpenAlex
- W4366736258
- Cited by
- 40
- References
- 0
- Citations by year
- 2026: 2 · 2025: 11 · 2024: 14 · 2023: 13
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- ChatGPT, large language models, factual accuracy, open-domain question answering, knowledge memorization, knowledge recall, inference error
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
