The DOI matches its DataCite record; 118 of the 151 references it cites were found in a registry or an index.
A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
Open accessAbstract
This paper proposes a framework for quantitatively evaluating interactive LLMs such as ChatGPT using publicly available data sets. We carry out an extensive technical evaluation of ChatGPT using 23 data sets covering 8 different common NLP application tasks. We evaluate the multitask, multilingual and multi-modal aspects of ChatGPT based on these data sets and a newly designed multimodal dataset. We find that ChatGPT outperforms LLMs with zero-shot learning on most tasks and even outperforms fine-tuned models on some tasks. We find that it is better at understanding non-Latin script languages than generating them. It is able to generate multimodal content from textual prompts, via an intermediate code generation step. Moreover, we find that ChatGPT is 63.41% accurate on average in 10 different reasoning categories under logical reasoning, non-textual reasoning, and commonsense reasoning, hence making it an unreliable reasoner. It is, for example, better at deductive than inductive reasoning. ChatGPT suffers from hallucination problems like other LLMs and it generates more extrinsic hallucinations from its parametric memory as it does not have access to an external knowledge base. Finally, the interactive feature of ChatGPT enables human collaboration with the underlying LLM to improve its performance, i.e, 8% ROUGE-1 on summarization and 2% ChrF++ on machine translation, in a multi-turn "prompt engineering" fashion. We also release codebase for evaluation set extraction.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“ChatGPT generally performs poorly on commonsense reasoning tasks, but relatively better than non-text semantic reasoning [6].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models (LLMs), such as Chat-GPT, are prone to generate hallucinations, i.e., content that conflicts with the source or cannot be verified by the factual knowledge.”
From HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models (2023) · cites this paper as Bang et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“On symbolic reasoning tasks, ChatGPT is mostly worse than GPT-3.5, which may be because ChatGPT is prone to uncertain responses, leading to poor performance [6].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“On symbolic reasoning tasks, ChatGPT is mostly worse than GPT-3.5, which may be because ChatGPT is prone to uncertain responses, leading to poor performance [6].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Fine-tuned models, such as Vícuna and ChatGPT, exhibit exceptional performance with near-perfect scores, surpassing models that lack supervised fine-tuning by a significant margin [5, 6].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The results indicated that these LLMs perform poorly when it came to non-Latin languages and languages with limited resources.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs also achieve satisfying performance across several natural language processing tasks, including machine translation [6, 130, 208], text generation [20], and question answering [102, 114].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“LLMs are prone to hallucination (Bang et al., 2023), where they generate plausible but factually incorrect or unverifiable information (Ji et al., 2023; Huang et al., 2025).”
From MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models (2025) · cites this paper as Bang et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“However, its proficiency in mathematical reasoning still requires improvement [6, 45, 265].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“Specifically, ChatGPT demonstrates comparable zero-shot performance to the text-davinci-002 [6], but performs worse than GPT-3.5 [159].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“Meanwhile, ChatGPT also lacks spatial reasoning ability, but exhibits better temporal reasoning.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“Research by Bang et al. [6] underscores that fully fine-tuned models tailored for specific tasks surpass ChatGPT in both task-oriented and knowledge-based dialogue contexts.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“The findings from Bang et al. [6] indicated that ChatGPT performs X → Eng translation well, but it still lacks the ability to perform Eng → X translation.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“In linguistic contexts featuring non-Latin scripts and limited resources, LLMs manifest suboptimal performance [2, 6, 100, 250].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“As the evaluation by Bang et al. (2023) found, ChatGPT is only 63.41% accurate on average in ten different reasoning categories under logical reasoning, non-textual reasoning, and common-sense reasoning, making it an unreliable reasoner.”
From AI and ethics: Investigating the first policy responses of higher education institutions to the challenge of generative AI (2024) · cites this paper as Bang et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“However, LLMs sometimes suffer from confabulations (or hallucinations), which can result in them making plausible but incorrect statements1,2.”
From Mathematical discoveries from program search with large language models (2024) · cites this paper as [1]
In the citing paper
“In low-resource learning environments, LLMs exhibit significant advantages over small language models [251], but the ability of ChatGPT to understand low-resource languages is limited [6].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
In the citing paper
“Furthermore, Bang et al. [6] highlighted that ChatGPT still faces a limitation in translating sentences written in non-Latin script languages with rich linguistic resources.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
In the citing paper
“they exhibit noteworthy proficiency in temporal reasoning [6].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
In the citing paper
“However, like other LLMs, ChatGPT suffers from hallucination problems. As it does not have access to an external knowledge base, more extrinsic hallucinations are generated [7].”
From Artificial Intelligence in Medical Education: Comparative Analysis of ChatGPT, Bing, and Medical Students in Germany (2023) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302 · cited 2×
“…is a purely text-prompted language model, it is unlikely to explore its multimodal capabilities with visual inputs like contemporary vision-language works (Rombach et al., 2022; Ramesh et al., 2021; Yu et al., 2021a; Radford et al., 2021; Dai et al., 2022; Lovenia et al., 2022; Dai et al., 2023a).”
BackgroundLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1×
“…is a purely text-prompted language model, it is unlikely to explore its multimodal capabilities with visual inputs like contemporary vision-language works (Rombach et al., 2022; Ramesh et al., 2021; Yu et al., 2021a; Radford et al., 2021; Dai et al., 2022; Lovenia et al., 2022; Dai et al., 2023a).”
BackgroundLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · cited 4×
“The large version of Bart (Lewis et al., 2020b) model fine-tuned on both datasets is conducted for comparison.”
MethodInfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855 · cited 1×
“We test on 100 samples from two common summarization datasets: half from SAMSum (Gliwa et al., 2019), a dialogue summarization dataset, and another half from CNN/DM (Hermann et al., 2015; Nallapati et al., 2016), news summarization datasets.”
MethodPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 2×
“Training data size affects language understanding and generation ability of LMs (Raffel et al., 2022; Cahyawijaya et al., 2021; Rae et al., 2021; Work-shop et al., 2022; Chowdhery et al., 2022; Hoff-mann et al., 2022).”
BackgroundResultZero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417 · cited 2×
“…is a purely text-prompted language model, it is unlikely to explore its multimodal capabilities with visual inputs like contemporary vision-language works (Rombach et al., 2022; Ramesh et al., 2021; Yu et al., 2021a; Radford et al., 2021; Dai et al., 2022; Lovenia et al., 2022; Dai et al., 2023a).”
Background
Research works printed but not linked: 22
- Li Du, Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. 2022. e-CARE: a new dataset for exploring explain- able causal reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 432-446, Dublin, Ireland. Association for Computational Lin- guistics. · DOI
- Simon Frieder, Luca Pinchetti, Ryan-Rhys Grif- fiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023. Mathematical capabilities of chatgpt. · DOI · Paper page
- Aidan Gilson, Conrad Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, and David Chartash. 2022. How well does chatgpt do when taking the medical licensing exams? the im- plications of large language models for medical edu- cation and knowledge assessment. medRxiv, pages 2022-12.
- James Hawthorne. 2021. Inductive Logic. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philoso- phy, Spring 2021 edition. Metaphysics Research Lab, Stanford University.
- Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Si- monyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models. · DOI · Paper page
- Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022a. Survey of halluci- nation in natural language generation. ACM Comput. Surv. Just Accepted. · DOI · Paper page
- Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study. · DOI · Paper page
- Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Huang. 2023a. A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets. In Findings of the Association for Com- putational Linguistics: ACL 2023, pages 431-469, Toronto, Canada. Association for Computational Lin- guistics. · DOI
- Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020b. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880.
- Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Man- ning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Lad- hak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic eval- uation of language models. · DOI · Paper page
- Hongyuan Lu, Haoyang Huang, Shuming Ma, Dong- dong Zhang, Wai Lam, and Furu Wei. 2022. Trip: Triangular document-level pre-training for multilingual language models. arXiv preprint arXiv:2212.07752. · arXiv
- Pasquale Minervini, Sebastian Riedel, Pontus Stenetorp, Edward Grefenstette, and Tim Rocktäschel. 2020. Learning reasoning strategies in end-to-end differen- tiable proving. In Proceedings of the 37th Interna- tional Conference on Machine Learning, ICML'20. JMLR.org.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Partha Pratim Ray. 2023. Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems.
- Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. 2022b. StepGame: A new benchmark for robust multi-hop spatial reasoning in texts. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11321- 11329. · DOI
- Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, and Jason Weston. 2022. Blenderbot 3: a deployed conversational agent that continually learns to respon- sibly engage. · DOI · Paper page
- Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2022. Recitation-augmented language models. In The Eleventh International Conference on Learning Representations.
- Ada Wan. 2022. Fairness in representation for multi- lingual NLP: Insights from controlled experiments on conditional language modeling. In International Conference on Learning Representations.
- Peter Cathcart Wason and Philip Nicholas Johnson- Laird. 1972. Psychology of reasoning: Structure and content, volume 86. Harvard University Press.
- Jason Weston, Antoine Bordes, Sumit Chopra, and Tomás Mikolov. 2016a. Towards ai-complete ques- tion answering: A set of prerequisite toy tasks. In 4th International Conference on Learning Represen- tations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
- PMLR.
- Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Good- man. 2022. Star: Bootstrapping reasoning with rea- soning. In Advances in Neural Information Process- ing Systems.
Cited by
Sorted by citations
Items 201–300 of 379
Enhancing In-Context Learning with Answer Feedback for Multi-span Question Answering
Huang, Z., Zhou, J., Xiao, G., & Cheng, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44696-2_58 · Cited by 15
InstructTODS: Large Language Models for End-to-End Task-Oriented Dialogue Systems
Chung, W., Cahyawijaya, S., Wilie, B., Lovenia, H., & Fung, P. (2023) · doi:10.18653/v1/2023.nlint-1.1 · Cited by 36
Self-citationTyphoon: Thai Large Language Models
Pipatanakul, K., Jirabovonvisut, P., Manakul, P., Sripaisarnmongkol, S., Patomwong, R., Chokchainant, P., & Tharnpipitchai, K. (2023) · arXiv · doi:10.48550/arxiv.2312.13951 · Cited by 9
Language-Models-as-a-Service: Overview of a New Paradigm and its Challenges
La Malfa, E., Petrov, A., Frieder, S., Weinhuber, C., Burnell, R., Nazar, R., Cohn, A., Shadbolt, N., & Wooldridge, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15865 · Cited by 9
“…that humans correctly classify (Berglund et al., 2023; Hao et al., 2023; Kocoń et al., 2023), exhibit implicit linguistic biases (Ahia et al., 2023; Bang et al., 2023; Petrov et al., 2023b) or are brittle to adversarial prompts (Kocoń et al., 2023; Shen et al., 2023; Schlarmann and Hein, 2023; Zou…”
BackgroundFaltings, F., Galley, M., Brantley, K., Peng, B., Cai, W., Zhang, Y., Gao, J., & Dolan, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.270 · Cited by 11
MindMap: Constructing Evidence Chains for Multi-Step Reasoning in Large Language Models
Wu, Y., Han, X., Song, W., Cheng, M., & Li, F. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29896 · Cited by 8
GenAIPABench: A Benchmark for Generative AI-based Privacy Assistants
Hamid, A., Samidi, H. R., Pappachan, P., Finin, T., & Yus, R. (2024) · Proceedings on Privacy Enhancing Technologies · doi:10.56553/popets-2024-0081 · Cited by 21
‘So, we have occupied TikTok’: Ukrainian women in #ParticipativeWar
Kalnes, Ø., & Bjørge, N. M. (2025) · Media War & Conflict · doi:10.1177/17506352241307010 · Cited by 10
“AI has weaknesses as translation tools, especially with low-resource languages and non-Latin scripts (Bang et al., 2023; Lai et al., 2023; Zhu et al., 2023).”
Bresler, T. E., Wilson, T. W., Makaryan, T., Pandya, S. V., Palmer, K., Meyer, R., Htway, Z., & Fujita, M. (2025) · Journal of Surgical Oncology · doi:10.1002/jso.70005 · Cited by 8
“One of the main criticisms and dangers of LLMs is their propensity to generate Hallucinations ; additional, sometimes false information generated in an attempt to provide a complete and logical answer [25].”
Javahery, P., Alpat, M. F., & Kamali, J. (2025) · Discover Computing · doi:10.1007/s10791-025-09720-0 · Cited by 12
“Moreover, ChatGPT helps teachers create suitable activities through its ability to generate various types of content in response to given prompts [7].”
Agile Processes in Software Engineering and Extreme Programming – Workshops
Kruchten, P. B., & Gregory, P. (2023) · Lecture notes in business information processing · doi:10.1007/978-3-031-48550-3 · Cited by 9
Guiding Computational Stance Detection with Expanded Stance Triangle Framework
Liu, Z., Yap, Y. K., Chieu, H. L., & Chen, N. F. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.220 · Cited by 9
Gubelmann, R., Kalouli, A.-L., Niklaus, C., & Handschuh, S. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.4 · Cited by 9
“However, our findings agree with Bang et al. (2023), who find that ChatGPT performs reasonably well on deductive and abductive inference tasks.”
ResultDEĞERLİ, A. H., & Tatlısu, N. B. (2023) · Journal of Tourism and Gastronomy Studies · doi:10.21325/jotags.2024.1312 · Cited by 8
Hurd, T. C., Payton, F. C., & Hood, D. B. (2024) · Milbank Quarterly · doi:10.1111/1468-0009.12712 · Cited by 9
“56,57 Despite these limitations, it still outperforms crowdsourcing in zero-shot learning, a type of AI in which a model is trained to recognize/categorize objects or concepts that it has not seen before and may supplant current crowdsourcing and data codification approaches.”
Perils and Opportunities in Using Large Language Models in Psychological ResearchPreprint
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J. P., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/d695y · Cited by 8
BRAINTEASER: Lateral Thinking Puzzles for Large Language Models
Jiang, Y., Ilievski, F., Ma, K., & Sourati, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.885 · Cited by 29
Zizka, L. (2025) · Journal of Hospitality & Tourism Education · doi:10.1080/10963758.2025.2496663 · Cited by 12
Building Stereotype Repositories with Complementary Approaches for Scale and Depth
Dev, S., Jha, A., Goyal, J., Tewari, D., Dave, S., & Prabhakaran, V. (2023) · Workshop on Cross-Cultural Considerations in NLP (C3NLP) · doi:10.18653/v1/2023.c3nlp-1.9 · Cited by 8
Generative Models For Indic Languages: Evaluating Content Generation Capabilities
Bhat, S., Varma, V., IIIT Hyderabad Gachibowli, H. 5., & Pedanekar, N. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_021 · Cited by 7
Evaluating Generative Models for Graph-to-Text Generation
Yuan, S., & Färber, M. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_133 · Cited by 21
Arefin, S. E., Heya, T. A., Al-Qudah, H. S. S., Ineza, Y., & Serwadda, A. (2024) · International Conference on Agents and Artificial Intelligence (ICAART) · doi:10.5220/0012467100003636 · Cited by 8
“Example 3: Constraints: Input: nums = [1,1,1,3,3,4,3,2,4,2] 1 <= nums.”
BackgroundInfluentialBuilding a literature knowledge base towards transparent biomedical AIPreprint
Huang, Y., Han, Z., Luo, X., Luo, X., Gao, Y., Zhao, M., Tang, F., Wang, Y., Chen, J., Li, C., et al. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.09.22.614323 · Cited by 6
“Firstly, LLMs sometimes hallucinate in the sense that they generate fabricated results and sources in their answers [11, 25].”
BackgroundDebate Helps Supervise Unreliable Experts
Michael, J., Mahdi, S., Rein, D., Petty, J., Dirani, J., Padmakumar, V., & Bowman, S. R. (2023) · arXiv · doi:10.48550/arxiv.2311.08702 · Cited by 8
Bertaglia, T., Heisig, L., Kaushal, R., & Iamnitchi, A. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31303 · Cited by 8
Does ChatGPT generate fake results? Challenges in retrieving content through ChatGPT
Safdar, M., Siddique, N., Gulzar, A., Yasin, H., & Khan, M. A. (2024) · Digital Library Perspectives · doi:10.1108/dlp-01-2024-0006 · Cited by 10
Luna, D. S., Broer, I., Bilandzić, H., Taddicken, M., Schuller, B. W., & Bürger, M. (2025) · Public Understanding of Science · doi:10.1177/09636625251328854 · Cited by 15
“For example, studies have evaluated the factuality of models such as ChatGPT, exposing issues like ‘hallucinations’, plausible-sounding but factually incorrect outputs (e.g. Bang et al., 2023).”
Investigating the Utility of ChatGPT in the Issue Tracking System: An Exploratory Study
Das, J. K., Mondal, S., & Roy, C. K. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3645083 · Cited by 6
“Given ChatGPT’s reported tendency to generate hallucinated code [10, 11], this question attempts to understand the extent to which developers incorporate Chat-GPT’s code in their actual coding projects versus primarily using it for other purposes.”
BackgroundCan ChatGPT Pass a Physics Test?
Rēvalde, G., Zholdakhmet, M., Abola, A., & Murzagaliyeva, A. (2025) · Technology Knowledge and Learning · doi:10.1007/s10758-025-09814-0 · Cited by 9
Artificial intelligence in academic Research: Contributor, constructivist or cheat?
Scott‐Kennel, J., 张, 蓉., & Scott, J. M. (2025) · The Journal of Marketing Theory and Practice · doi:10.1080/10696679.2025.2457672 · Cited by 12
“…Pro and Claude 3 Opus included 2 While Chatbots such as ChatGPT are known to produce “extrinsic hallucinations” from its parametric memory (Bang et al., 2023, p. 2), the authors have carefully checked references and output to ensure their authenticity, addressing concerns about…”
Fu, Y., Inoue, K., Chu, C., & Kawahara, T. (2023) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/2023.sigdial-1.60 · Cited by 13
Is ChatGPT a friend or foe in the war on misinformation?: A South African perspective
Senekal, B., & Brokensha, S. (2023) · Communicare: Journal for Communication Studies in Africa · doi:10.36615/jcsa.v42i2.2437 · Cited by 7
“As discussed in the literature review on LLMs, a significant drawback of using these networks is that they may generate false information (Lin, Hilton & Evans, 2021), and ChatGPT is no exception (Bang et al., 2023; Guo et al., 2023).”
BackgroundModels See Hallucinations: Evaluating the Factuality in Video Captioning
Liu, H., & Wan, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.723 · Cited by 6
Massaro, A., & Samo, G. (2023) · Verbum · doi:10.15388/verb.42 · Cited by 6
Dabran-Zivan, S., Klein-Avraham, I., & Baram-Tsabari, A. (2026) · PLOS One · doi:10.1371/journal.pone.0345300 · Cited by 6
Gao, Y., Arava, S. K., Li, Y., & WSnyder, J. (2024) · International Journal on Cybernetics & Informatics · doi:10.5121/ijci.2024.130202 · Cited by 9
“Even for use cases that focus on natural language, LLMs often suffer from the well documented hallucination problem [16-18], especially in cases where lengthy or domain-specific context is important.”
BackgroundJo, E., Yoo, H., Kim, J.-H., Kim, Y.-M., Song, S., & Joo, H. J. (2024) · JMIR Formative Research · doi:10.2196/47814 · Cited by 5
“ChatGPT gave hallucinated answers relating to clinics that were not actual locations, like “K Dental Clinic” [19].”
BackgroundInfluentialAlsofyani, M., & Wang, L. (2025) · The Journal of Supercomputing · doi:10.1007/s11227-025-07237-3 · Cited by 5
“Largelanguagemodels(LLMs)havedemonstratedtransformativepotentialacrossvar-iousdomains,includingsoftwareengineering,mathematics[19–21],high-performance computing(HPC)[22,23],andnaturallanguageprocessing(NLP)[24,25].”
Laskar, M. T. R., Rahman, M., Jahan, I., Hoque, E., & Huang, J. X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.686 · Cited by 6
“…received a lot of attention recently due to their impressive language generation capability – ensuring high fluency, coherence, and grammatical correctness on the generated texts (Laskar et al., 2023a; Qin et al., 2023; Bang et al., 2023; Yang et al., 2023; Wang et al., 2023; Koco´n et al., 2023).”
BackgroundWeber, L., Bruni, E., & Hupkes, D. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.20 · Cited by 25
What Else Do I Need to Know? The Effect of Background Information on Users’ Reliance on QA Systems
Goyal, N., Briakou, E., Liu, A., Baumler, C., Bonial, C., Micher, J. C., Voss, C. R., Carpuat, M. J., & Daumé, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.201 · Cited by 12
“A vast host of existing and ongoing work in NLP examines the deficiencies of these language models, ranging from hallucinated generations (Bang et al., 2023; Ji et al., 2023), lack of transparency (Weld and Bansal, 2019; Lipton, 2018), reasoning gaps (Bang et al., 2023; Press et al., 2022), and…”
BackgroundLarge Language Models and Multimodal Retrieval for Visual Word Sense Disambiguation
Kritharoula, A., Lymperaiou, M., & Stamou, G. B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.807 · Cited by 27
“…technique faces the risks associated with hallucinations and untruthful generation, which cannot be easily detected or resolved, according to related state-of-the-art research (Bang et al., 2023; Mündler et al., 2023; Azaria and Mitchell, 2023; Zheng et al., 2023; Zhang et al., 2023a).”
BackgroundIizuka, S., Mochizuki, S., Ohashi, A., Yamashita, S., Guo, A., & Higashinaka, R. (2024) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v2i1.27668 · Cited by 6
Image Content Generation with Causal Reasoning
Li, X., Fan, B., Zhang, R., Jin, L., Wang, D., Guo, Z., Zhao, Y., & Li, R. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i12.29269 · Cited by 11
Comrie, D. (2023) · medRxiv · doi:10.1101/2023.04.17.23288681 · Cited by 5
Position Papers of the 18th Conference on Computer Science and Intelligence Systems
(2023) · Annals of Computer Science and Information Systems · doi:10.15439/978-83-969601-1-5 · Cited by 5
Mind the Gap Between Conversations for Improved Long-Term Dialogue Generation
Zhang, Q., Naradowsky, J., & Miyao, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.720 · Cited by 11
Learning Tools Using ChatGPT in the Biochemistry Class: Creating Notes and Performance on Exams
Roman, A., Simaitis, M., Sheely, K., Roth, Y. M., Tansey, J. T., & Cogan, J. (2025) · Biochemistry and Molecular Biology Education · doi:10.1002/bmb.21904 · Cited by 6
“It has been shown that ChatGPT suffers from hallucinations, seemingly producing convincing but inaccurate responses, and that on reasoning tasks its accuracy is relatively low [4].”
Resolving Elliptical Compounds in German Medical Text
Kammer, N., Borchert, F., Winkler, S., de Melo, G., & Schapranow, M.-P. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.26 · Cited by 5
“However, the low zero-shot performance relative to carefully tuned supervised models is not unique to our problem (Jimenez Gutierrez et al., 2022; Yuan et al., 2023; Bang et al., 2023; Hu et al., 2023), in particular for languages other than English (Lai et al., 2023).”
BackgroundEnhancing text comprehension for Question Answering with Contrastive Learning
Lee, S., & Lee, M. (2023) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2023.repl4nlp-1.7 · Cited by 4
“It is clear that ChatGPT has led to tremendous development and success, but it has a fatal problem called hallucination (Alkaissi and McFarlane, 2023; Bang et al., 2023).”
BackgroundDisentangling Extraction and Reasoning in Multi-hop Spatial Reasoning
Mirzaee, R., & Kordjamshidi, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.221 · Cited by 7
Does the English Matter? Elicit Cross-lingual Abilities of Large Language Models
Ranaldi, L., & Pucci, G. (2023) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2023.mrl-1.14 · Cited by 6
“Thus, performance is usually lower for non-English languages, especially for low-resource ones (Huang et al., 2023; Bang et al., 2023).”
BackgroundUniversity Students’ Perspective on ChatGPT and Technology Literacies
Dawa, T., Dhendup, S., Tashi, S., & Rosso, M. A. (2024) · Educational innovation and practice · doi:10.17102/eip.7.2024.10 · Cited by 4
Towards Automated Regulation Analysis for Effective Privacy Compliance
Manandhar, S., Singh, K., & Nadkarni, A. (2024) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.24650 · Cited by 5
“…Models (LLMS) : While models such as ChatGPT [54] have demonstrated exceptional performance in a variety of NLP tasks, researchers are still grappling to understand the risks and reliability [11] of using these models in important use-cases, particularly as they may suffer from hallucination [10].”
MethodAdarkwah, M. A., Islam, A. Y. M. A., Schneider, K., Luckin, R., Thomas, M., & Spector, J. M. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2364140 · Cited by 6
LLM-enhanced Self-training for Cross-domain Constituency Parsing
Li, J., Zhang, M., Guo, P., Zhang, M., & Zhang, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.508 · Cited by 5
Contextual Interaction for Argument Post Quality Assessment
Wang, Y., Chen, X., He, B., & Sun, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.645 · Cited by 4
Wang, J., Cheng, Y., Lin, D., Leong, C. T., & Li, W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.72 · Cited by 11
Research on the Application of ChatGPT in the Interdisciplinars of Higher Education
Li, X., Ju, P., Zhang, J., & Zhao, S. (2024) · Journal of Artificial Intelligence Practice · doi:10.23977/jaip.2024.070107 · Cited by 4
Deciphering Compatibility Relationships with Textual Descriptions via Extraction and Explanation
Wang, Y., He, Z., He, Z., Xu, H., & McAuley, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i8.28764 · Cited by 6
Walker, J., Koutsiana, E., Nwachukwu, M., Meroño-Peñuela, A., & Simperl, E. P. B. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650844 · Cited by 4
“One interviewee (5) suggested that we need to be more “open-minded” in order to advance the field of quality assessment using LLMs.”
InfluentialTowards semantically enriched embeddings for knowledge graph completion
Alam, M., van Harmelen, F., & Acosta, M. (2024) · Neurosymbolic Artificial Intelligence · doi:10.3233/nai-240731 · Cited by 7
Proceedings of The ACM Collective Intelligence Conference
(2023) · doi:10.1145/3582269 · Cited by 4
Paolo, L. D. D., White, B. P., GUÉNIN--CARLUT, A., et al. (2024) · doi:10.31219/osf.io/zwa83 · Cited by 4
Self-Consistent Narrative Prompts on Abductive Natural Language Inference
Chan, C., Liu, X., Chan, T. H., Cheng, J., Song, Y., Wong, G. Y., & See, S. C. W. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.67 · Cited by 15
Vervoort, L., & Nikolaev, V. (2025) · Journal for General Philosophy of Science · doi:10.1007/s10838-025-09735-7 · Cited by 4
Assessing Logical Puzzle Solving in Large Language Models: Insights from a Minesweeper Case Study
Li, Y., Wang, H., & Zhang, C. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.4 · Cited by 11
SCREEN: A Benchmark for Situated Conversational Recommendation
Lin, D., Wang, J., Leong, C. T., & Li, W. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681651 · Cited by 3
“Despite efforts to increase variability through controlled settings, LLMs occasionally generate responses with hallucinations [1].”
BackgroundResearch on Miniaturization Trend of ChatGPT Technology Model
Shi, X., Abisado, M., & Song, X. (2023) · Journal of Artificial Intelligence and Technology · doi:10.37965/jait.2023.0298 · Cited by 3
Mapeamento do Perfil das Mulheres Brasileiras em Processamento de Linguagem Natural
Caseli, H. D. M., de Amorim, E. C. F., Schneider, E. T. R., et al. (2023) · doi:10.5753/wit.2023.229504 · Cited by 3
“Embora a parte aplicada seja a que está mais presente no nosso dia-a-dia, o lado teórico é imprescindı́vel para o estudo dos fenômenos linguı́sticos da maneira apropriada e o desenvolvimento adequado de produtos precisos [Bang et al. 2023].”
BackgroundUna aproximación a ChatGPT como herramienta jurídica: sesgos, capacidades y utilidades futuras
Dolmestch, R. N., & Fuentes-Loureiro, M.-Á. (2023) · IDP Revista de Internet Derecho y Política · doi:10.7238/idp.v0i39.417024 · Cited by 13
ChatGPT’s Information Seeking Strategy: Insights from the 20-Questions Game
Bertolazzi, L., Mazzaccara, D., Merlo, F., & Bernardi, R. (2023) · International Conference on Natural Language Generation · doi:10.18653/v1/2023.inlg-main.11 · Cited by 8
“Moreover, Bang et al. (2023) evaluate ChatGPT reasoning skills: though it lacks inducting reasoning skills, it performs well on deductive reasoning in clean settings.”
BackgroundInfluentialIntegrating ChatGPT into English Language Teaching and Learning: Strengths and Weaknesses
Juan, W., Qing, S., Yunus, M. M., & Rafiq, K. R. M. (2023) · International Journal of Academic Research in Business and Social Sciences · doi:10.6007/ijarbss/v13-i12/20251 · Cited by 12
DEMASQ: Unmasking the ChatGPT Wordsmith
Kumari, K., Pegoraro, A., Fereidooni, H., & Sadeghi, A. (2024) · Network and Distributed System Security Symposium · doi:10.14722/ndss.2024.241190 · Cited by 10
Shi, D., You, C., Huang, J., Li, T., & Xiong, D. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29861 · Cited by 7
Evaluating the Effectiveness of AI Source Disclosure in Human–AI Communication
Bródy, G., Rouillard, V., Asherov, D., & Aravind, A. (2024) · doi:10.21428/e4baedd9.3f5bb369 · Cited by 3
Artificial Intelligence - Intelligent Art?
Voigts, E., Auer, R. M., Elflein, D., Kunas, S., Röhnert, J., & Seelinger, C. (2024) · Digitale Gesellschaft · doi:10.1515/9783839469224 · Cited by 14
Dong, J., Luo, X., & Zhu, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5501-1_23 · Cited by 3
Joseph, T., & Kenneth, M. H. (2024) · East African Journal of Information Technology · doi:10.37284/eajit.7.1.2111 · Cited by 4
An empirical evaluation of arabic text formality transfer: a comparative study
Abdu, F. J., Mughaus, R., Abudalfa, S. I., Ahmed, M., & Abdelalí, A. (2025) · Language Resources and Evaluation · doi:10.1007/s10579-025-09873-w · Cited by 3
Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkers
Wang, Y., Reddy, R. G., Mujahid, Z. M., Arora, A., Rubashevskii, A., Geng, J., Afzal, O. M., Pan, L., Borenstein, N., Pillai, A., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.09000 · Cited by 3
Kadegaonkar, M., & Karim, K. (2025) · Proceedings of the AAAI Symposium Series · doi:10.1609/aaaiss.v6i1.36059 · Cited by 3
Yan, D. “., Kasuma, S. A. A., & Amini, M. (2026) · Springer eBooks · doi:10.1007/978-981-95-4871-2_14 · Cited by 2
The Inverse Data Law: Market Imperatives, Data, and Quality in AI Supported Care
Ameen, S., Wong, M. C., Yee, K. C., Nøhr, C. G., & Turner, P. A. (2023) · Studies in health technology and informatics · doi:10.3233/shti230166 · Cited by 4
“For instance, generative large language models (LLMs) [25] that have a capacity to (a) authoritatively “hallucinate” misinformation, (b) produce different outputs at each time step, and (c) that may not be privy to the nuances of language across CALD groups; are likely to have different evaluation parameters when comp…”
BackgroundDepWiGNN: A Depth-wise Graph Neural Network for Multi-hop Spatial Reasoning in Text
Li, S., Deng, Y., & Lam, W. P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.428 · Cited by 3
Mögliche Einsatzgebiete von Künstlicher Intelligenz im Verlagswesen
Klotz, D. (2024) · Springer eBooks · doi:10.1007/978-3-658-43037-5_1 · Cited by 2
Can You Translate for Me? Code-Switched Machine Translation with Large Language Models
Khatri, J., Srivastava, V. P., & Vig, L. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-short.10 · Cited by 4
“Recently, there have been several works exploring the performance of machine translation for various language-pairs us-ing LLMs (Lin et al., 2022; Jiao et al., 2023; Lyu et al., 2023; Bang et al., 2023; Agrawal et al., 2022; Zhang et al., 2023; Moslem et al., 2023).”
BackgroundYang, C., Cao, B., & Fan, J. (2024) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v18i1.31419 · Cited by 5
“In the realm of text clustering, a prevalent practice is to deploy a classical clustering model, e.g. K-Means (MacQueen 1965; Steinbach, Karypis, and Kumar 2000), directly on the representations generated by embedding models (Muennighoff et al. 2023; Wang et al. 2022; Su et al. 2023).”
BackgroundMethodResultInfluentialRoy, A., & Roy, S. S. (2025) · Information systems engineering and management · doi:10.1007/978-3-031-76710-4_6 · Cited by 2
Abrantes, J. (2025) · Cureus · doi:10.7759/cureus.84165 · Cited by 2
“Performance can vary widely depending on language resource levels and representation in training data [11,12].”
A Benchmark for Cross-Domain Argumentative Stance Classification on Social Media
Yuan, J., Xi, R., & Singh, M. P. (2025) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v19i1.35927 · Cited by 4
Knowledge Engineering in the Era of Artificial Intelligence
Hose, K. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-42914-9_1 · Cited by 2
Coordination within Conversational Agents with Multiple Sources
Matos, V. B., Grava, R. S., Tavares, R., José, M. M., Pirozelli, P., Brandão, A. A. F., Peres, S. M., & Cozman, F. G. (2023) · doi:10.5753/eniac.2023.234533 · Cited by 2
“Even though such systems are currently very effective and have the ability to act in open domain tasks, they do not provide guarantees of correctness and coherence in their responses [3].”
BackgroundMadusanka, T., Zahid, I., Li, H., Pratt-Hartmann, I. E., & Batista-Navarro, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.536 · Cited by 3
“ChatGPT Due to the recent success of ChatGPT in solving many natural language tasks in a zero-shot setting (Bang et al., 2023), we employ it in a similar context.”
MethodCross-Document Event Coreference Resolution: Instruct Humans or Instruct GPT?
Zhao, J., Xue, N., & Min, B. (2023) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2023.conll-1.38 · Cited by 3
“Bang et al. (2023) conducted a comprehensive evaluation of ChatGPT, demonstrating its superior performance on various NLP tasks over other LLMs while highlighting its potential hallucination issues in reasoning tasks.”
BackgroundExpNote: Black-box Large Language Models are better Task Solvers with Experience Notebook
Sun, W., Yu, X., He, S., Zhao, J., & Liu, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1034 · Cited by 4
Paek, E. S., Fan, T., Finch, J. D., & Choi, J. D. (2024) · Information · doi:10.3390/info15090580 · Cited by 2
“Bang et al. [27] and Chung et al. [28] adapted GPT to build end-to-end task-oriented dialogue systems, surpassing prior systems in task execution.”
Metadata
- Authors
- Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., Do, Q. V., Xu, Y., & Fung, P.
- Affiliations
- Hong Kong University of Science and Technology (HK)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2302.04023
- OpenAlex
- W4319793302
- Cited by
- 337
- References
- 151
- Citations by year
- 2026: 14 · 2025: 38 · 2024: 101 · 2023: 184
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- ChatGPT, LLM evaluation, hallucinations, reasoning, multilingual NLP, multimodal generation, zero-shot learning, commonsense reasoning, deductive reasoning, inductive reasoning, human-AI collaboration, prompt engineering, summarization, machine translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
