The DOI matches its DataCite record; 118 of the 151 references it cites were found in a registry or an index.
A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
Open accessAbstract
This paper proposes a framework for quantitatively evaluating interactive LLMs such as ChatGPT using publicly available data sets. We carry out an extensive technical evaluation of ChatGPT using 23 data sets covering 8 different common NLP application tasks. We evaluate the multitask, multilingual and multi-modal aspects of ChatGPT based on these data sets and a newly designed multimodal dataset. We find that ChatGPT outperforms LLMs with zero-shot learning on most tasks and even outperforms fine-tuned models on some tasks. We find that it is better at understanding non-Latin script languages than generating them. It is able to generate multimodal content from textual prompts, via an intermediate code generation step. Moreover, we find that ChatGPT is 63.41% accurate on average in 10 different reasoning categories under logical reasoning, non-textual reasoning, and commonsense reasoning, hence making it an unreliable reasoner. It is, for example, better at deductive than inductive reasoning. ChatGPT suffers from hallucination problems like other LLMs and it generates more extrinsic hallucinations from its parametric memory as it does not have access to an external knowledge base. Finally, the interactive feature of ChatGPT enables human collaboration with the underlying LLM to improve its performance, i.e, 8% ROUGE-1 on summarization and 2% ChrF++ on machine translation, in a multi-turn "prompt engineering" fashion. We also release codebase for evaluation set extraction.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“ChatGPT generally performs poorly on commonsense reasoning tasks, but relatively better than non-text semantic reasoning [6].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Large language models (LLMs), such as Chat-GPT, are prone to generate hallucinations, i.e., content that conflicts with the source or cannot be verified by the factual knowledge.”
From HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models (2023) · cites this paper as Bang et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“On symbolic reasoning tasks, ChatGPT is mostly worse than GPT-3.5, which may be because ChatGPT is prone to uncertain responses, leading to poor performance [6].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Fine-tuned models, such as Vícuna and ChatGPT, exhibit exceptional performance with near-perfect scores, surpassing models that lack supervised fine-tuning by a significant margin [5, 6].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The results indicated that these LLMs perform poorly when it came to non-Latin languages and languages with limited resources.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMs also achieve satisfying performance across several natural language processing tasks, including machine translation [6, 130, 208], text generation [20], and question answering [102, 114].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“LLMs are prone to hallucination (Bang et al., 2023), where they generate plausible but factually incorrect or unverifiable information (Ji et al., 2023; Huang et al., 2025).”
From MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models (2025) · cites this paper as Bang et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“However, its proficiency in mathematical reasoning still requires improvement [6, 45, 265].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“Specifically, ChatGPT demonstrates comparable zero-shot performance to the text-davinci-002 [6], but performs worse than GPT-3.5 [159].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“Meanwhile, ChatGPT also lacks spatial reasoning ability, but exhibits better temporal reasoning.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“Research by Bang et al. [6] underscores that fully fine-tuned models tailored for specific tasks surpass ChatGPT in both task-oriented and knowledge-based dialogue contexts.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“The findings from Bang et al. [6] indicated that ChatGPT performs X → Eng translation well, but it still lacks the ability to perform Eng → X translation.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“In linguistic contexts featuring non-Latin scripts and limited resources, LLMs manifest suboptimal performance [2, 6, 100, 250].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
The cited source says what the claim attributes to it.
In the citing paper
“As the evaluation by Bang et al. (2023) found, ChatGPT is only 63.41% accurate on average in ten different reasoning categories under logical reasoning, non-textual reasoning, and common-sense reasoning, making it an unreliable reasoner.”
From AI and ethics: Investigating the first policy responses of higher education institutions to the challenge of generative AI (2024) · cites this paper as Bang et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“However, LLMs sometimes suffer from confabulations (or hallucinations), which can result in them making plausible but incorrect statements1,2.”
From Mathematical discoveries from program search with large language models (2024) · cites this paper as [1]
In the citing paper
“In low-resource learning environments, LLMs exhibit significant advantages over small language models [251], but the ability of ChatGPT to understand low-resource languages is limited [6].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
In the citing paper
“On symbolic reasoning tasks, ChatGPT is mostly worse than GPT-3.5, which may be because ChatGPT is prone to uncertain responses, leading to poor performance [6].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
In the citing paper
“Furthermore, Bang et al. [6] highlighted that ChatGPT still faces a limitation in translating sentences written in non-Latin script languages with rich linguistic resources.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
In the citing paper
“they exhibit noteworthy proficiency in temporal reasoning [6].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
In the citing paper
“However, like other LLMs, ChatGPT suffers from hallucination problems. As it does not have access to an external knowledge base, more extrinsic hallucinations are generated [7].”
From Artificial Intelligence in Medical Education: Comparative Analysis of ChatGPT, Bing, and Medical Students in Germany (2023) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302 · cited 2×
“…is a purely text-prompted language model, it is unlikely to explore its multimodal capabilities with visual inputs like contemporary vision-language works (Rombach et al., 2022; Ramesh et al., 2021; Yu et al., 2021a; Radford et al., 2021; Dai et al., 2022; Lovenia et al., 2022; Dai et al., 2023a).”
BackgroundLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1×
“…is a purely text-prompted language model, it is unlikely to explore its multimodal capabilities with visual inputs like contemporary vision-language works (Rombach et al., 2022; Ramesh et al., 2021; Yu et al., 2021a; Radford et al., 2021; Dai et al., 2022; Lovenia et al., 2022; Dai et al., 2023a).”
BackgroundLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · cited 4×
“The large version of Bart (Lewis et al., 2020b) model fine-tuned on both datasets is conducted for comparison.”
MethodInfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865
Teaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,855 · cited 1×
“We test on 100 samples from two common summarization datasets: half from SAMSum (Gliwa et al., 2019), a dialogue summarization dataset, and another half from CNN/DM (Hermann et al., 2015; Nallapati et al., 2016), news summarization datasets.”
MethodPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 2×
“Training data size affects language understanding and generation ability of LMs (Raffel et al., 2022; Cahyawijaya et al., 2021; Rae et al., 2021; Work-shop et al., 2022; Chowdhery et al., 2022; Hoff-mann et al., 2022).”
BackgroundResultZero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417 · cited 2×
“…is a purely text-prompted language model, it is unlikely to explore its multimodal capabilities with visual inputs like contemporary vision-language works (Rombach et al., 2022; Ramesh et al., 2021; Yu et al., 2021a; Radford et al., 2021; Dai et al., 2022; Lovenia et al., 2022; Dai et al., 2023a).”
Background
Flagged first, then by citations
33 of the 151 references couldn’t be matched to a known paper.
Research works printed but not linked: 22
- Li Du, Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. 2022. e-CARE: a new dataset for exploring explain- able causal reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 432-446, Dublin, Ireland. Association for Computational Lin- guistics. · DOI
- Simon Frieder, Luca Pinchetti, Ryan-Rhys Grif- fiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023. Mathematical capabilities of chatgpt. · DOI · Paper page
- Aidan Gilson, Conrad Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, and David Chartash. 2022. How well does chatgpt do when taking the medical licensing exams? the im- plications of large language models for medical edu- cation and knowledge assessment. medRxiv, pages 2022-12.
- James Hawthorne. 2021. Inductive Logic. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philoso- phy, Spring 2021 edition. Metaphysics Research Lab, Stanford University.
- Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Si- monyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models. · DOI · Paper page
- Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022a. Survey of halluci- nation in natural language generation. ACM Comput. Surv. Just Accepted. · DOI · Paper page
- Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study. · DOI · Paper page
- Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Huang. 2023a. A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets. In Findings of the Association for Com- putational Linguistics: ACL 2023, pages 431-469, Toronto, Canada. Association for Computational Lin- guistics. · DOI
- Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020b. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880.
- Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Man- ning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Lad- hak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic eval- uation of language models. · DOI · Paper page
- Hongyuan Lu, Haoyang Huang, Shuming Ma, Dong- dong Zhang, Wai Lam, and Furu Wei. 2022. Trip: Triangular document-level pre-training for multilingual language models. arXiv preprint arXiv:2212.07752. · arXiv
- Pasquale Minervini, Sebastian Riedel, Pontus Stenetorp, Edward Grefenstette, and Tim Rocktäschel. 2020. Learning reasoning strategies in end-to-end differen- tiable proving. In Proceedings of the 37th Interna- tional Conference on Machine Learning, ICML'20. JMLR.org.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Partha Pratim Ray. 2023. Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems.
- Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. 2022b. StepGame: A new benchmark for robust multi-hop spatial reasoning in texts. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11321- 11329. · DOI
- Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, and Jason Weston. 2022. Blenderbot 3: a deployed conversational agent that continually learns to respon- sibly engage. · DOI · Paper page
- Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2022. Recitation-augmented language models. In The Eleventh International Conference on Learning Representations.
- Ada Wan. 2022. Fairness in representation for multi- lingual NLP: Insights from controlled experiments on conditional language modeling. In International Conference on Learning Representations.
- Peter Cathcart Wason and Philip Nicholas Johnson- Laird. 1972. Psychology of reasoning: Structure and content, volume 86. Harvard University Press.
- Jason Weston, Antoine Bordes, Sumit Chopra, and Tomás Mikolov. 2016a. Towards ai-complete ques- tion answering: A set of prerequisite toy tasks. In 4th International Conference on Learning Represen- tations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
- PMLR.
- Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Good- man. 2022. Star: Bootstrapping reasoning with rea- soning. In Advances in Neural Information Process- ing Systems.
Cited by
Sorted by citations
Items 101–200 of 378
Humans are still better than ChatGPT: Case of the IEEEXtreme competition
Koubâa, A., Qureshi, B., Ammar, A., Khan, Z., Boulila, W., & Ghouti, L. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e21624 · Cited by 46
Hou, C., Zhu, G., Zheng, J., Zhang, L., Huang, X., Zhong, T., Li, S., Du, H., & Ker, C. L. (2024) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3636555.3636910 · Cited by 44
Dictionaries and lexicography in the AI era
Lew, R. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-02889-7 · Cited by 34
“Admittedly, GPT and similar Large Language Models demonstrate remarkable pro fi ciency in English, but their capabilities in other languages, especially minor or less-documented ones, are not nearly as impressive (Bang et al. 2023, Lai et al. 2023).”
Sottana, A., Liang, B., Zou, K., & Yuan, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.543 · Cited by 76
Cho, Y., Rai, S., Ungar, L., Sedoc, J., & Guntuku, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.698 · Cited by 33
Alzboon, M. S., Qawasmeh, S., Alqaraleh, M. K. S., Abuashour, A., Bader, A. F., & Al-Batah, M. S. (2023) · International Conference on Emerging Smart Technologies and Applications (eSmarTA) · doi:10.1109/esmarta59349.2023.10293294 · Cited by 32
“It achieved 44.4%, 42.4%, 64.4.4%, and 57.8% accuracy on USMLE Step 1 and Step 2 questions [31].”
BackgroundInfluentialUPRISE: Universal Prompt Retrieval for Improving Zero-Shot Evaluation
Cheng, D., Huang, S., Bi, J., Zhan, Y., Liu, J., Wang, Y., Sun, H., Wei, F., Deng, W., & Zhang, Q. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.758 · Cited by 32
Uncovering the Causes of Emotions in Software Developer Communication Using Zero-shot LLMs
Imran, M. M., Chatterjee, P., & Damevski, K. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639223 · Cited by 39
“We also observed a number of hallucinations in the zero-shot LLMs output [43], where the models generated responses that were outside of what was asked.”
MethodResultXu, S., Pang, L., Shen, H., Cheng, X., & Chua, T. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645363 · Cited by 91
ChatGPT Improves Creative Problem-Solving Performance in University Students: An Experimental StudyPreprint
Urban, M., Děchtěrenko, F., Lukavský, J., et al. (2023) · doi:10.31234/osf.io/9z2tc · Cited by 31
Large Knowledge Model: Perspectives and Challenges
Chen, H. (2024) · Data Intelligence · doi:10.3724/2096-7004.di.2024.0001 · Cited by 56
“Despite this, its performance still falls short of the state-of-the-art (SOTA) achievements of existing small models in this domain [56, 57, 58].”
BackgroundThe Consequences of Generative AI for UGC and Online Community EngagementPreprint
Burtch, G., Lee, D., & Chen, Z. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4521754 · Cited by 33
“This point is notable, given many well-known LLMs are known to engage in ‘hallucination’ (Bang et al. 2023), i.e., providing factually incorrect information with an air of confidence.”
BackgroundKoLA: Carefully Benchmarking World Knowledge of Large Language Models
Yu, J., Wang, X., Tu, S., Cao, S., Zhang-Li, D., Lv, X., Peng, H., Yao, Z., Zhang, X., Li, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.09296 · Cited by 29
Generative artificial intelligence: implications for biomedical and health professions education
Hersh, W. (2025) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-103123-094756 · Cited by 28
Artificial intelligence—friend or foe in fake news campaigns
Węcel, K., Sawiński, M., Stróżyna, M., Lewoniewski, W., Księżniak, E., Stolarski, P., & Abramowicz, W. (2023) · Economics and Business Review · doi:10.18559/ebr.2023.2.736 · Cited by 28
“…et al., 2023), poetry (Kirmani, 2022), songs (McGee, 2023), humour (Kirmani, 2022), memes (Yang et al., 2023), etc. ChatGPT can also present rational skills such as counting (Frieder et al., 2023), analogies (Bang et al., 2023), concept blending, forecasting (Lopez‐ Lira & Tang, 2023), etc.”
BackgroundDolmestch, R. N. (2023) · Derecho PUCP · doi:10.18800/derechopucp.202302.007 · Cited by 69
Wang, L., He, J., Li, S., Liu, N., & Lim, E. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-53302-0_3 · Cited by 38
Feedback seeking abilities of L2 writers using ChatGPT: a mixed method multiple case study
Yan, D. “. (2024) · Kybernetes · doi:10.1108/k-09-2023-1933 · Cited by 33
TARJAMAT: Evaluation of Bard and ChatGPT on Machine Translation of Ten Arabic Varieties
Kadaoui, K., Magdy, S. M., Waheed, A., Khondaker, M. T. I., El-Shangiti, A. O., Nagoudi, E. M. B., & Abdul-Mageed, M. (2023) · Arabic Natural Language Processing Conference (ArabicNLP) · doi:10.18653/v1/2023.arabicnlp-1.6 · Cited by 46
Yong, Z., Zhang, R., Forde, J. Z., Wang, S., Subramonian, A., Lovenia, H., Cahyawijaya, S., Winata, G. I., Sutawika, L., Cruz, J. C. B., et al. (2023) · Workshop on Computational Approaches to Code Switching · doi:10.18653/v1/2023.calcs-1.5 · Cited by 48
Self-citationLaskar, M. T. R., Fu, X.-Y., Chen, C., & Shashi Bhushan, T. N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.33 · Cited by 46
Berti, A., Kourani, H., Häfke, H., Li, C.-Y., & Schuster, D. (2024) · Lecture notes in business information processing · doi:10.1007/978-3-031-61007-3_2 · Cited by 38
“Their potential in PM lies in embedded domain knowledge useful for generating database queries and insights [24], logical and temporal reasoning capabilities [2,19], inference abilities over structured data [14].”
BackgroundLai, V. D., Nguyen, C., Ngo, N., Nguyen, T., Dernoncourt, F., Rossi, R. A., & Nguyen, T. H. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.28 · Cited by 51
Li, F., Hogg, D., & Cohn, A. G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29811 · Cited by 30
Pu, D., & Demberg, V. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-srw.1 · Cited by 56
Lion: Adversarial Distillation of Proprietary Large Language Models
Jiang, Y., Chan, C., Chen, M., & Wang, W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.189 · Cited by 93
“…of instruction-following large language models such as ChatGPT (Ope-nAI, 2022) and GPT-4 (OpenAI, 2023), the techniques of instruction tuning (Wei et al., 2022b) have attracted a lot of attention (Wei et al., 2022c; Bubeck et al., 2023; Bang et al., 2023; Kocon et al., 2023; Chan et al., 2023a).”
MethodLemke, C., Kirchner, K., Anandarajah, L., & Herfurth, F. (2023) · European Conference on e-Learning · doi:10.34190/ecel.22.1.1828 · Cited by 39
“On the other hand, these tools tend to generate speculative content (Bang et al, 2023), emphasizing the urgency to train students to check the correctness of content for meaningful everyday pedagogical practice.”
BackgroundMultilingual Large Language Models Are Not (Yet) Code-Switchers
Zhang, R., Cahyawijaya, S., Cruz, J. C. B., Winata, G. I., & Aji, A. F. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.774 · Cited by 55
Self-citationA New Benchmark and Reverse Validation Method for Passage-level Hallucination Detection
Yang, S., Sun, R., & Wan, X. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.256 · Cited by 46
Bridging large language model disparities: skill tagging of multilingual educational content
Kwak, Y., & Pardos, Z. A. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13465 · Cited by 25
Low Resource Summarization using Pre-trained Language Models
Munaf, M., Afzal, H., Mahmood, K., & Iltaf, N. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3675780 · Cited by 34
“Summarization by ChatGPT in zero-shot settings without fine-tuning and language understanding of low resource language also remains a challenge with low evaluation results [46].”
BackgroundFagbohun, O., Harrison, R. M., & Dereventsov, A. (2023) · Journal of Artificial Intelligence Machine Learning and Data Science · doi:10.51219/jaimld/oluwole-fagbohun/15 · Cited by 61
“With the advent of more sophisticated LLMs like ChatGPT 24 , the complexity of tasks that prompts are intended to address has also increased.”
AlAfnan, M. A. (2024) · Journal of Artificial Intelligence and Technology · doi:10.37965/jait.2024.0549 · Cited by 45
Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus
Zhang, T., Qiu, L., Guo, Q., Deng, C., Zhang, Y., Zhang, Z., Zhou, C., Wang, X., & Fu, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.58 · Cited by 71
Isiaku, L., Kwala, A. F., Sambo, K. U., & Isiaku, H. H. (2024) · Journal of University Teaching and Learning Practice · doi:10.53761/7egat807 · Cited by 21
“Malik et al. (2023) pointed out its limitations, noting deficiencies in common sense reasoning tasks and raising concerns about its language coverage (Bang et al., 2023).”
BackgroundEvaluating ChatGPT and Bard AI on Arabic Sentiment Analysis
Al-Thubaity, A. O., Alkhereyf, S., Murayshid, H. S., Alshalawi, N., Omirah, M., Alateeq, R., Almutairi, R. E., Alsuwailem, R. I., Alhassoun, M., & Alkhanen, I. (2023) · Arabic Natural Language Processing Conference (ArabicNLP) · doi:10.18653/v1/2023.arabicnlp-1.27 · Cited by 26
“Other studies have evaluated LLMs for their performance on other low-resource languages (Ahuja et al., 2023; Bang et al., 2023; Lai et al., 2023).”
BackgroundLin, H., Luo, Z., Ma, J., & Chen, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.611 · Cited by 40
“” As we clarify the ground-truth harmfulness label in the observed attributes of the prompt, the hallu-cination issue (Bang et al., 2023) of LLMs could be effectively alleviated.”
BackgroundJiao, W., Huang, J.-T., Wang, W., He, Z., Liang, T., Wang, X., Shi, S., & Tu, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1001 · Cited by 61
Large Language Models are Complex Table Parsers
Zhao, B., Ji, C., Zhang, Y., He, W., Wang, Y., Wang, Q., Feng, R., & Zhang, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.914 · Cited by 35
“…has showcased remarkable capabilities in generation (Maddigan and Susnjak, 2023; Dong et al., 2023; Liu et al., 2023a), contextual understanding (Bang et al., 2023; Gao et al., 2023b; Amin et al., 2023), and reasoning (Liu et al., 2023b; Gao et al., 2023a), has profoundly impacted on the field…”
BackgroundMartin, P. P., & Graulich, N. (2024) · Journal of Science Education and Technology · doi:10.1007/s10956-023-10087-4 · Cited by 26
Seo, J., Kamath, S. S., Zeidieh, A., Venkatesh, S., & McCurry, S. (2024) · The 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675660 · Cited by 25
“For example, the hallucination efect [8, 17] in data visualization descriptions may become even more critical for BLV users, potentially misleading their informed decision process.”
Empowering Agile-Based Generative Software Development through Human-AI Teamwork
Zhang, S., Xing, Z., Guo, R., Xu, F., Chen, L., Zhang, Z., Zhang, X., Feng, Z., & Zhuang, Z. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3702987 · Cited by 43
Ghanadian, H., Nejadgholi, I., & Osman, H. A. (2023) · Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA) · doi:10.18653/v1/2023.wassa-1.16 · Cited by 26
Pei, J., & Jurgens, D. (2023) · Linguistic Annotation Workshop · doi:10.18653/v1/2023.law-1.25 · Cited by 37
Chen, Z., Lu, Y., & Wang, W. Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.284 · Cited by 36
Systematic Assessment of Factual Knowledge in Large Language Models
Luo, L., Vu, T.-T., Phung, D., & Haf, R. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.885 · Cited by 29
Med-HALT: Medical Domain Hallucination Test for Large Language Models
Pal, A., Umapathi, L. K., & Sankarasubbu, M. (2023) · arXiv · doi:10.48550/arxiv.2307.15343 · Cited by 152
Natural language processing for social science research: A comprehensive review
Hou, Y., & Huang, J. (2025) · Chinese Journal of Sociology · doi:10.1177/2057150x241306780 · Cited by 35
“…demonstrate impressive ability in various complex tasks, including natural language understanding, translation, text summarization, question answering, and even reasoning tasks, even reaching a higher score than humans in some benchmarks (Bang et al., 2023; Street et al., 2024; Qin et al., 2023).”
Hallucination Detection for Generative Large Language Models by Bayesian Sequential Estimation
Wang, X., Yan, Y.-L., Huang, L., Zheng, X., & Huang, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.949 · Cited by 19
Computational Argumentation-based Chatbots: A Survey
Castagna, F., Kökciyan, N., Sassoon, I. K., Parsons, S., & Sklar, E. I. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15407 · Cited by 32
“That being said, LLMs and the chatbots based on them also suffer from a number of downsides (Frieder et al., 2023; Bang et al., 2023; Wei et al., 2022a; Ji et al., 2023; Zhuo et al., 2023) including: faulty reasoning, inexplicable appearance of previously unknown abilities (a phenomenon denoted as…”
BackgroundZhaı, X., Nyaaba, M., & Ma, W. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4451722 · Cited by 17
Michail, A., Konstantinou, S., & Clematide, S. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.140 · Cited by 18
“Preliminary studies (Jiao et al., 2023; Bang et al., 2023) suggest that ChatGPT has a competitive performance against established commercial translation systems, demonstrating its ability to generate multilingual content.”
BackgroundRaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models
Hoshi, Y., Miyashita, D., Ng, Y., Tatsuno, K., Morioka, Y., Torii, O., & Deguchi, J. (2023) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2023.emnlp-demo.4 · Cited by 18
Wang, K., Duan, F., Wang, S., Li, P., Xian, Y., Yin, C., Rong, W., & Xiong, Z. (2023) · arXiv · doi:10.48550/arxiv.2308.13259 · Cited by 16
Winder, P. D., Hildebrand, C., & Hartmann, J. (2025) · PLoS ONE · doi:10.1371/journal.pone.0325459 · Cited by 25
“LLMs have demonstrated exceptional nat-ural language understanding capabilities [16–18] and the ability to generate such personalized advice [19], potentially addressing key limitations of traditional advisory systems (see also [20] for a recent meta-analysis and increased adoption rates of AI…”
Successfully Using ChatGPT in Logistics: Are We There Yet?
Voß, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-43612-3_1 · Cited by 15
DEVELOPMENT OF A QUESTION ANSWERING CHATBOT FOR BLOCKCHAIN DOMAIN
Mansurova, A., Nugumanova, A., & Makhambetova, Z. (2023) · Scientific Journal of Astana IT University · doi:10.37943/15xndz6667 · Cited by 21
“In factual text generation, a challenging problem is the generation of hallu - cinations [23] - confident but misleading information.”
BackgroundYu, S. C. L., He, J., Gutiérrez-Basulto, V., & Pan, J. Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.594 · Cited by 24
Out-of-Distribution Generalization in Natural Language Processing: Past, Present, and Future
Yang, L., Song, Y., Ren, X., Lyu, C., Wang, Y., Zhuo, J., Liu, L., Wang, J., Foster, J., & Zhang, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.276 · Cited by 21
“OOD Robustness Previous studies have extensively examined various aspects of ChatGPT in the domains of law (Choi et al., 2023), ethics (Shen et al., 2023), reasoning (Bang et al., 2023) and planning (Yao et al., 2023).”
BackgroundFreeAL: Towards Human-Free Active Learning in the Era of Large Language Models
Xiao, R., Dong, Y., Zhao, J., Wu, R., Lin, M., Chen, G., & Wang, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.896 · Cited by 36
ChatGPT y educación universitaria. Posibilidades y límites de ChatGPT como herramienta docente
Ribera, M., & Díaz, O. (2024) · doi:10.36006/15224-1 · Cited by 15
Truth-O-Meter: Handling Multiple Inconsistent Sources Repairing LLM Hallucinations
Galitsky, B. A., Chernyavskiy, A., & Ilvovsky, D. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657679 · Cited by 16
“However, despite its powerful capabilities, several studies have consistently demonstrated significant flaws in LLMs, such as errors in factuality, reasoning, math, coding, biases, etc. [1, 2].”
Global and Local Hierarchy-aware Contrastive Framework for Implicit Discourse Relation Recognition
Jiang, Y., Zhang, L., & Wang, W. (2023) · doi:10.18653/v1/2023.findings-acl.510 · Cited by 15
Unleashing the Power of ChatGPT in Finance Research: Opportunities and ChallengesPreprint
Feng, Z., Hu, G., & Li, B. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4424979 · Cited by 13
Exploring the Impacts of ChatGPT on Future Scientific WorkPreprint
Irons, J. L., Mason, C., Cooper, P. S., Sidra, S., Reeson, A. F., & Paris, C. L. (2023) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/j2u9x · Cited by 16
Toward non-human-centered design: designing an academic article with ChatGPT
LİVBERBER, T. (2023) · El Profesional de la Informacion · doi:10.3145/epi.2023.sep.12 · Cited by 18
Knowledge-Augmented Language Model Verification
Baek, J., Jeong, S., Kang, M., Park, J. C., & Hwang, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.107 · Cited by 33
Learning to Describe for Predicting Zero-shot Drug-Drug Interactions
Zhu, F., Zhang, Y., Chen, L., Qin, B., & Xu, R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.918 · Cited by 19
Advancing plant biology through deep learning-powered natural language processing
Peng, S., & Rajjou, L. (2024) · Plant Cell Reports · doi:10.1007/s00299-024-03294-9 · Cited by 14
“Additionally, the stability of Large Language Models (LLMs) in generating accurate results and mitigating issues is a subject demanding meticulous attention and refinement (Bang et al. 2023).”
Impacts of generative AI on user contributions: evidence from a coding Q &A platform
Li, X., & Kim, K. (2024) · Marketing Letters · doi:10.1007/s11002-024-09747-1 · Cited by 23
“Recent studies find concerning evidence on LLM hallucination, where false statements are erroneously assumed to be true by AI, resulting in the generation of falsehoods (Bang et al., 2023).”
Tian, S., Wang, D., Wang, J., & Zhong, W. (2025) · The Interpreter and Translator Trainer · doi:10.1080/1750399x.2025.2534269 · Cited by 16
“…(multi-turn, dialogic engagement in translation tasks), personalisation (generating translations tailored to users’ stylistic preferences and genres), and contextualisation (integrating additional context through prompts to improve translation relevance) (Bang et al. 2023; Lyu, Xu, and Wang 2023).”
La enseñanza del español mediada por tecnología
Muñoz–Basols, J., Gutiérrez, M. F., & Cerezo, L. (2023) · Routledge eBooks · doi:10.4324/9781003146391 · Cited by 28
Cahyawijaya, S., Lovenia, H., Koto, F., Adhista, D., Dave, E., Oktavianti, S., Akbar, S. M., Lee, J., Shadieq, N., Cenggoro, T. W., et al. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.60 · Cited by 28
Self-citationNaing, S. Z. S., & Udomwong, P. (2024) · Data Intelligence · doi:10.1162/dint_a_00250 · Cited by 15
KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detection
Choi, S.-H., Fang, T., Wang, Z., & Song, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.867 · Cited by 46
ChatGPT, an Excellent Liar: How Conversational Agents’ Hallucinations Impact Learning and Teaching
Salamin, A.-D., Russo, D., & Rueger, D. (2023) · The Proceedings of the International Conference on Teaching, Learning and Education. · doi:10.33422/6th.iacetl.2023.11.100 · Cited by 15
User Preferences on a Generative AI User Interface Through a Choice Experiment
Yeon, J., Jung, Y., Baek, Y., Lee, D., Shin, J., & Chung, W. Y. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2400379 · Cited by 14
Self-citationBeyond Information: Is ChatGPT Empathetic Enough?
Université du Québec à Montréal, C., Belkhir, A., Sadat, F., & Université du Québec à Montréal, C. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_018 · Cited by 13
“For instance, Bang et al. (2023) evaluates the multilingual performance of ChatGPT on three tasks of language identification, sentiment analysis, and machine translation.”
BackgroundChatGPT and computational-based research: benefits, drawbacks, and machine learning applications
Atkinson, C. F. (2023) · Discover Artificial Intelligence · doi:10.1007/s44163-023-00091-3 · Cited by 18
Proceedings of the 4th New Frontiers in Summarization Workshop
(2023) · doi:10.18653/v1/2023.newsum-1 · Cited by 12
Di Paolo, L. D., White, B., Guénin-Carlut, A., Constant, A., & Clark, A. (2024) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2023.0148 · Cited by 14
Applying ChatGPT as a new business strategy: A great power comes with great responsibility
Limna, P., Kraiwanit, T., Jangjarat, K., & Shaengchart, Y. (2023) · Corporate & Business Strategy Review · doi:10.22495/cbsrv4i4siart2 · Cited by 8
Bhat, S., & Varma, V. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.8 · Cited by 12
“Similar studies for misinformation in (Bang et al., 2023) and hate speech in (Huang et al., 2023) have considered ChatGPT for annotations.”
BackgroundYou only need to change your direction: A look at the potential impact of ChatGPT on education
Kohler, K. (2024) · Technology in Language Teaching & Learning · doi:10.29140/tltl.v6n1.1103 · Cited by 19
Mapping (A)Ideology: A Taxonomy of European Parties Using Generative LLMs as Zero-Shot Learners
Leo, R. D., Zeng, C., Dinas, E., & Tamtam, R. (2025) · Political Analysis · doi:10.1017/pan.2025.7 · Cited by 13
Xu, S., Liu, S., Culhane, T., Pertseva, E., Wu, M.-H., Semnani, S. J., & Lam, M. S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.353 · Cited by 26
“However, LLMs are prone to halluci-nate (Bang et al., 2023), while using a convincing and confident tone.”
BackgroundA Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry
Huang, Y., Tang, K., Chen, M., & Wang, B. (2024) · arXiv · doi:10.48550/arxiv.2404.15777 · Cited by 14
Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
Ye, J., Wang, Y., Huang, Y., Chen, D., Zhang, Q., Moniz, N., Gao, T., Geyer, W., Huang, C., Chen, P.-Y., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.02736 · Cited by 11
Revisiting the Knowledge Injection Frameworks
Fu, P., Zhang, Y., Wang, H., Qiu, W., & Zhao, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.677 · Cited by 14
Bouyamourn, A. Z. A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.192 · Cited by 13
Casals-Farre, O., Baskaran, R., Singh, A. P., Kaur, H., Hoque, T. U., de Almeida, A., Coffey, M. J., & Hassoulas, A. (2025) · Scientific Reports · doi:10.1038/s41598-025-97327-2 · Cited by 12
Can ChatGPT Understand Causal Language in Science Claims?
Kim, Y., Guo, L., Yu, B., & Li, Y. (2023) · Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA) · doi:10.18653/v1/2023.wassa-1.33 · Cited by 10
“Previous studies have evaluated ChatGPT on various classification tasks (Qin et al., 2023; Bang et al., 2023; Huang et al., 2023; Koco´n et al., 2023), using various prompt designs.”
Method(2023) · International Research Journal of Modernization in Engineering Technology and Science · doi:10.56726/irjmets45426 · Cited by 10
CAR: Conceptualization-Augmented Reasoner for Zero-Shot Commonsense Question Answering
Wang, W., Fang, T., Ding, W., Xu, B., Liu, X., Song, Y., & Bosselut, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.902 · Cited by 25
Opportunities for Artificial Intelligence (AI) in language teaching and learning
Muñoz–Basols, J., & Gutiérrez, M. F. (2023) · Routledge eBooks · doi:10.4324/9781003146391-18 · Cited by 11
Perils and Opportunities in Using Large Language Models in Psychological ResearchPreprint
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J. P., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/tg79n · Cited by 10
An Adaptive Prompt Generation Framework for Task-oriented Dialogue System
Gao, J. Y., Xiang, L., Wu, H., Zhao, H. X., Tong, Y., & He, Z. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.76 · Cited by 9
“As a result, it may lead to the suboptimal performance of LLMs and hallucinatory outputs (Bang et al., 2023).”
BackgroundMethodInfluentialFailures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulation
Yang, Z., Li, P., & Liu, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.109 · Cited by 24
Contrasting rule and machine learning based digital self triage systems in the USA
Naved, B. A., & Luo, Y. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01367-3 · Cited by 15
“ChatGPT was found to be 63% accurate on average in 10 different reasoning categories 56 .”
Enhancing In-Context Learning with Answer Feedback for Multi-span Question Answering
Huang, Z., Zhou, J., Xiao, G., & Cheng, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44696-2_58 · Cited by 15
Metadata
- Authors
- Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., Do, Q. V., Xu, Y., & Fung, P.
- Affiliations
- Hong Kong University of Science and Technology (HK)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2302.04023
- OpenAlex
- W4319793302
- Cited by
- 337
- References
- 151
- Citations by year
- 2026: 14 · 2025: 38 · 2024: 101 · 2023: 184
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- ChatGPT, LLM evaluation, hallucinations, reasoning, multilingual NLP, multimodal generation, zero-shot learning, commonsense reasoning, deductive reasoning, inductive reasoning, human-AI collaboration, prompt engineering, summarization, machine translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
