A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
Open accessAbstract
Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, Pascale Fung. Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLM-generated content inevitably contains hallucinations.
“While LLMgenerated content can match the fluency of professional writing, it inevitably contains hallucinations [1,4]-misleading information that appears credible but lacks factual accuracy.”
From Beyond Binary: Towards Fine-Grained LLM-Generated Text Detection via Role Recognition and Involvement Measurement (2025) · cites this paper as [4]
Section 4, “Factuality and Hallucination,” says LLMs are susceptible to generating untruthful information and reports hallucinations in several ChatGPT tasks. It does not say hallucinations are inevitable in all LLM-generated content; it also notes that some extrinsic hallucinations are factual. (LLM, conf=0.88)
Experimental — accuracy not yet measuredUpdating existing innovations with state-of-the-art LLMs may further reduce the amount of manual effort required for fine-tuning and achieve similar performances.
“Updating existing innovations with state-of-the-art LLMs may further reduce the amount of manual effort required for fine-turning and achieve similar performances (Bang et al., 2023).”
From Practical and ethical challenges of large language models in education: A systematic scoping review (2024) · cites this paper as Bang et al., 2023
The Introduction reports that zero-shot ChatGPT was on par with or better than fully fine-tuned task-specific models on some NLP tasks, supporting the possibility of similar performance without task-specific fine-tuning. The paper does not evaluate updating existing innovations or measure whether doing so reduces manual effort; its Multitask Generalization section also says fine-tuned models outperform ChatGPT on most tasks. Page numbers are unavailable in our copy. (LLM, conf=0.86)
Experimental — accuracy not yet measuredLLM results are impressive but highlight inherent limitations, particularly flexibility in transferring knowledge from one set of data to another, which brings about artificial hallucinations.
“Results are impressive, but they also highlight inherent limitations, particularly the flexibility in transferring knowledge from one set of data to another, which also bring about states of ‘artificial hallucinations’”
From Active inference goes to school: the importance of active learning in the age of large language models (2024) · cites this paper as [68]
The Abstract reports strong performance alongside hallucination problems, and §4 (Factuality and Hallucination) documents hallucinations across tasks. Neither passage says that flexibility in transferring knowledge between datasets causes those hallucinations. (LLM, conf=0.85)
Experimental — accuracy not yet measuredState-of-the-art LLMs could potentially achieve similar performance with a zero-shot approach.
“Whereas, the stateof-the-art LLMs could potentially achieve similar performance with a zero-shot approach (Bang et al., 2023).”
From Practical and ethical challenges of large language models in education: A systematic scoping review (2024) · cites this paper as Bang et al., 2023
LLMs are prone to hallucination, generating plausible yet nonfactual content.
“Nevertheless, in tandem with these remarkable advancements, concerns have arisen about the tendency of LLMs to generate hallucinations [15, 105], resulting in seemingly plausible yet factually unsupported content.”
From A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2025) · cites this paper as [15]
Many LLMs can generate nonfactual and untruthful information that cannot be verified from the inputs across tasks.
“Previous works (Ji et al., 2023;Bang et al., 2023) point out that many LLMs can generate nonfactual and untruthful information that cannot be verified from the inputs across tasks.”
From Recursively summarizing enables long-term dialogue memory in large language models (2025) · cites this paper as Bang et al., 2023
Hallucination issues in healthcare chatbots arise when responses appear factually accurate but lack validity.
“Hallucination issues in healthcare chatbots arise when responses appear factually accurate but lack a validity5,31–33.”
From Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI (2024) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 70 · Cited by 682
Research works printed but not linked: 27
- Paul Christiano, Jan Leike, Tom B. Brown, Miljan Mar- tic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. · DOI · Paper page
- Aidan Gilson, Conrad Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, and David Chartash. 2022. How well does chatgpt do when taking the medical licensing exams? the im- plications of large language models for medical edu- cation and knowledge assessment. medRxiv, pages 2022-12.
- James Hawthorne. 2021. Inductive Logic. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philoso- phy, Spring 2021 edition. Metaphysics Research Lab, Stanford University.
- Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022a. Survey of halluci- nation in natural language generation. ACM Comput. Surv. Just Accepted. · DOI · Paper page
- Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study. · DOI · Paper page
- Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282-6293, Online. Association for Computational Linguistics. · DOI · Paper page
- Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Huang. 2023a. A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets. In Findings of the Association for Com- putational Linguistics: ACL 2023, pages 431-469, Toronto, Canada. Association for Computational Lin- guistics. · DOI
- Anton E Lawson. 2005. What is the role of induction and deduction in reasoning and scientific inquiry? Journal of Research in Science Teaching, 42(6):716- 740.
- Nayeon Lee, Yejin Bang, Andrea Madotto, and Pascale Fung. 2021. Towards few-shot fact-checking via perplexity. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1971-1981, Online. Association for Computational Linguistics. · DOI
- Nayeon Lee, Wei Ping, Peng Xu, Mostofa Patwary, Pas- cale Fung, Mohammad Shoeybi, and Bryan Catan- zaro. 2022. Factuality enhanced language models for open-ended text generation. In Advances in Neural Information Processing Systems.
- Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics. · DOI · Paper page
- Vaibhav Mavi, Anubhav Jangra, and Adam Jatowt. 2022. A survey on multi-hop question answering and gen- eration. arXiv preprint arXiv:2204.09140. · arXiv
- Pasquale Minervini, Sebastian Riedel, Pontus Stenetorp, Edward Grefenstette, and Tim Rocktäschel. 2020. Learning reasoning strategies in end-to-end differen- tiable proving. In Proceedings of the 37th Interna- tional Conference on Machine Learning, ICML'20. JMLR.org.
- Roshanak Mirzaee and Parisa Kordjamshidi. 2022. Transfer learning with synthetic corpora for spatial role labeling and reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 6148-6165, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
- Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjamshidi. 2021. SPARTQA: A textual question answering benchmark for spatial reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 4582-4598, Online. Association for Computational Linguistics. · DOI
- Seungwhan Moon, Pararth Shah, Anuj Kumar, and Ra- jen Subba. 2019. Opendialkg: Explainable conver- sational reasoning with attention-based walks over knowledge graphs. In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pages 845-854.
- Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When being un- seen from mBERT is just the beginning: Handling new languages with multilingual language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 448-462, Online. Association for Computa- tional Linguistics. · DOI
- Oded Nov, Nina Singh, and Devin M Mann. 2023. Putting chatgpt's medical advice to the (turing) test. medRxiv, pages 2023-01.
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages 8748-8763. PMLR.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical rea- soning abilities of neural models. In International Conference on Learning Representations.
- Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. 2022b. StepGame: A new benchmark for robust multi-hop spatial reasoning in texts. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11321- 11329. · DOI
- Dan Su, Tiezheng Yu, and Pascale Fung. 2021. Improve query focused abstractive summarization by incorpo- rating answer relevance. In Findings of the Associ- ation for Computational Linguistics: ACL-IJCNLP 2021, pages 3124-3131.
- Douglas Walton. 2014. Abductive reasoning. Univer- sity of Alabama Press.
- Ada Wan. 2022. Fairness in representation for multi- lingual NLP: Insights from controlled experiments on conditional language modeling. In International Conference on Learning Representations.
- Jason Weston, Antoine Bordes, Sumit Chopra, and Tomás Mikolov. 2016a. Towards ai-complete ques- tion answering: A set of prerequisite toy tasks. In 4th International Conference on Learning Represen- tations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
- BigScience Workshop, :, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luc- cioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Vil- lanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Belt- agy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pe- dro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Lev- kovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Gi- ada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Al- mubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Lu- dovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Šaško, diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369-2380.
- Authors
- Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., Do, Q. V., Xu, Y., & Fung, P.
- Affiliations
- Hong Kong University of Science and Technology (HK)
- ORCID
- Venue
- Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2023
- Pages
- 675-718
- OpenAlex
- W4392669753
- Cited by
- 682
- References
- 70
- Citations by year
- 2026: 95 · 2025: 229 · 2024: 266 · 2023: 92
- FWCI
- 120.84
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) (publishedVersion, cc-by)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- ChatGPT, large language models, reasoning, hallucinations, interactivity, multilingual evaluation, multimodal assessment, chatbots
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
