A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
Open accessAbstract
This paper proposes a framework for quantitatively evaluating interactive LLMs such as ChatGPT using publicly available data sets. We carry out an extensive technical evaluation of ChatGPT using 23 data sets covering 8 different common NLP application tasks. We evaluate the multitask, multilingual and multi-modal aspects of ChatGPT based on these data sets and a newly designed multimodal dataset. We find that ChatGPT outperforms LLMs with zero-shot learning on most tasks and even outperforms fine-tuned models on some tasks. We find that it is better at understanding non-Latin script languages than generating them. It is able to generate multimodal content from textual prompts, via an intermediate code generation step. Moreover, we find that ChatGPT is 63.41% accurate on average in 10 different reasoning categories under logical reasoning, non-textual reasoning, and commonsense reasoning, hence making it an unreliable reasoner. It is, for example, better at deductive than inductive reasoning. ChatGPT suffers from hallucination problems like other LLMs and it generates more extrinsic hallucinations from its parametric memory as it does not have access to an external knowledge base. Finally, the interactive feature of ChatGPT enables human collaboration with the underlying LLM to improve its performance, i.e, 8% ROUGE-1 on summarization and 2% ChrF++ on machine translation, in a multi-turn "prompt engineering" fashion. We also release codebase for evaluation set extraction.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Large language models are prone to generate hallucinated content that conflicts with the source or cannot be verified by factual knowledge.
“Large language models (LLMs), such as Chat-GPT, are prone to generate hallucinations, i.e., content that conflicts with the source or cannot be verified by the factual knowledge.”
From HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models (2023) · cites this paper as Bang et al., 2023
Section 4 says LLMs are susceptible to hallucination, and Appendix G defines intrinsic hallucinations as contradicting the source and extrinsic hallucinations as unverifiable from the source or input. The claim substitutes unverifiability by factual knowledge for unverifiability from the source. Appendix G also reports factual extrinsic hallucinations, so not every example is unverifiable by factual knowledge. (LLM, conf=0.86)
Experimental — accuracy not yet measuredLLMs are prone to hallucination, generating plausible but factually incorrect or unverifiable information.
“LLMs are prone to hallucination (Bang et al., 2023), where they generate plausible but factually incorrect or unverifiable information (Ji et al., 2023; Huang et al., 2025).”
From MedHallu: A Comprehensive Benchmark for Detecting Medical Hallucinations in Large Language Models (2025) · cites this paper as Bang et al., 2023
ChatGPT is only 63.41% accurate on average across ten reasoning categories (logical reasoning, non-textual reasoning and common-sense reasoning), making it an unreliable reasoner.
“As the evaluation by Bang et al. (2023) found, ChatGPT is only 63.41% accurate on average in ten different reasoning categories under logical reasoning, non-textual reasoning, and common-sense reasoning, making it an unreliable reasoner.”
From AI and ethics: Investigating the first policy responses of higher education institutions to the challenge of generative AI (2024) · cites this paper as Bang et al. (2023)
LLMs sometimes suffer from confabulations (hallucinations), which can result in them making plausible but incorrect statements.
“However, LLMs sometimes suffer from confabulations (or hallucinations), which can result in them making plausible but incorrect statements1,2.”
From Mathematical discoveries from program search with large language models (2024) · cites this paper as [1]
ChatGPT demonstrates comparable zero-shot summarization performance to text-davinci-002 but performs worse than GPT-3.5.
“Specifically, ChatGPT demonstrates comparable zero-shot performance to the text-davinci-002 [6], but performs worse than GPT-3.5 [159].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
ChatGPT generally performs poorly on commonsense reasoning tasks, but relatively better than non-text semantic reasoning.
“ChatGPT generally performs poorly on commonsense reasoning tasks, but relatively better than non-text semantic reasoning [6].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
ChatGPT's poor symbolic reasoning performance may be because it is prone to uncertain responses.
“On symbolic reasoning tasks, ChatGPT is mostly worse than GPT-3.5, which may be because ChatGPT is prone to uncertain responses, leading to poor performance [6].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
ChatGPT lacks spatial reasoning ability but exhibits better temporal reasoning.
“Meanwhile, ChatGPT also lacks spatial reasoning ability, but exhibits better temporal reasoning.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
Fully fine-tuned task-specific models surpass ChatGPT in both task-oriented and knowledge-based dialogue contexts.
“Research by Bang et al. [6] underscores that fully fine-tuned models tailored for specific tasks surpass ChatGPT in both task-oriented and knowledge-based dialogue contexts.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
ChatGPT performs X-to-English translation well but lacks the ability to perform English-to-X translation.
“The findings from Bang et al. [6] indicated that ChatGPT performs X → Eng translation well, but it still lacks the ability to perform Eng → X translation.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
Fine-tuned models such as Vícuna and ChatGPT exhibit near-perfect scores, surpassing models that lack supervised fine-tuning by a significant margin.
“Fine-tuned models, such as Vícuna and ChatGPT, exhibit exceptional performance with near-perfect scores, surpassing models that lack supervised fine-tuning by a significant margin [5, 6].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [6]
ChatGPT suffers from hallucination problems; as it does not have access to an external knowledge base, more extrinsic hallucinations are generated.
“However, like other LLMs, ChatGPT suffers from hallucination problems. As it does not have access to an external knowledge base, more extrinsic hallucinations are generated [7].”
From Artificial Intelligence in Medical Education: Comparative Analysis of ChatGPT, Bing, and Medical Students in Germany (2023) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 15,880 · cited 2× · see citing works
“…is a purely text-prompted language model, it is unlikely to explore its multimodal capabilities with visual inputs like contemporary vision-language works (Rombach et al., 2022; Ramesh et al., 2021; Yu et al., 2021a; Radford et al., 2021; Dai et al., 2022; Lovenia et al., 2022; Dai et al., 2023a).”
BackgroundLearning Transferable Visual Models From Natural Language Supervision
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2103.00020 · Cited by 15,293 · cited 1× · see citing works
“…is a purely text-prompted language model, it is unlikely to explore its multimodal capabilities with visual inputs like contemporary vision-language works (Rombach et al., 2022; Ramesh et al., 2021; Yu et al., 2021a; Radford et al., 2021; Dai et al., 2022; Lovenia et al., 2022; Dai et al., 2023a).”
BackgroundLewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · cited 4× · see citing works
“The large version of Bart (Lewis et al., 2020b) model fine-tuned on both datasets is conducted for comparison.”
MethodInfluentialChain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · see citing works
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · see citing works
Training Language Models to Follow Instructions with Human Feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2011 · Cited by 3,117 · see citing works
BERTScore: Evaluating Text Generation with BERT
Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019) · arXiv · doi:10.48550/arxiv.1904.09675 · Cited by 2,865 · see citing works
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1421 · Cited by 2,805 · cited 2× · see citing works
“Based on our evaluation with CommonsenseQA (Talmor et al., 2018), PiQA (Bisk et al., 2020) and Pep-3k (Wang et al., 2018b), ChatGPT shows surprisingly good commonsense reasoning capability, perhaps due to its large parametric memory.”
MethodTeaching Machines to Read and ComprehendPreprint
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,779 · cited 1× · see citing works
“We test on 100 samples from two common summarization datasets: half from SAMSum (Gliwa et al., 2019), a dialogue summarization dataset, and another half from CNN/DM (Hermann et al., 2015; Nallapati et al., 2016), news summarization datasets.”
MethodPaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 2× · see citing works
“Training data size affects language understanding and generation ability of LMs (Raffel et al., 2022; Cahyawijaya et al., 2021; Rae et al., 2021; Work-shop et al., 2022; Chowdhery et al., 2022; Hoff-mann et al., 2022).”
BackgroundResultZero-shot text-to-image generation
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., & Sutskever, I. (2021) · arXiv · doi:10.48550/arxiv.2102.12092 · Cited by 2,417 · cited 2× · see citing works
“…is a purely text-prompted language model, it is unlikely to explore its multimodal capabilities with visual inputs like contemporary vision-language works (Rombach et al., 2022; Ramesh et al., 2021; Yu et al., 2021a; Radford et al., 2021; Dai et al., 2022; Lovenia et al., 2022; Dai et al., 2023a).”
BackgroundAbstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., & Xiang, B. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1028 · Cited by 2,361 · cited 2× · see citing works
“We test on 100 samples from two common summarization datasets: half from SAMSum (Gliwa et al., 2019), a dialogue summarization dataset, and another half from CNN/DM (Hermann et al., 2015; Nallapati et al., 2016), news summarization datasets.”
MethodNo Language Left Behind: Scaling Human-Centered Machine TranslationPreprint
Costa-jussà, M., Cross, J., Çelebi, O., Elbayad, M., Heafield, K., Heffernan, K., Kalbassi, E., Lam, J., Licht, D., Maillard, J., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2207.04672 · Cited by 2,176 · cited 4× · see citing works
“We experiment with 6 languages: French, Chinese, Indonesian, Korean, Javanese, and Sundanese from the FLORES-200 dataset (Team et al., 2022; Goyal et al., 2021).”
MethodInfluentialA Call for Clarity in Reporting BLEU Scores
Post, M. (2018) · Proceedings of the Third Conference on Machine Translation: Research Papers · doi:10.18653/v1/w18-6319 · Cited by 2,112 · cited 2× · see citing works
“Evaluation: 1) human-targeted translation error rate (HTER) 8 , SacreBLEU (Post, 2018) and METEOR (Banerjee and Lavie, 2005) between the Marathi generated sentence compared to the human post-edited sentence, 2) HTER, Sacre-BLEU, METEOR, and semantic similarity score, i.e., BERTScore (Zhang* et al., 2020), between the…”
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · cited 1× · see citing works
“To refine the summary, we simply input another prompt – “Please make the summary shorter” after the first response.”
MethodChatGPT is fun, but not an author
Thorp, H. H. (2023) · Science · doi:10.1126/science.adg7879 · Cited by 1,437 · cited 1× · see citing works
“…(Guo et al., 2023; Davis, 2023b); it hallu-cinates with human-like fluency and eloquence on things that are not based on truth (Shen et al., 2023; Thorp, 2023; Smith, 2023); and as a general-purpose language model trained from everything on the web, its language coverage is questionable (Lu et…”
BackgroundGorp, A. V. (2012) · Psychology of Adjustment: The Search for Meaningful Balance · doi:10.1080/0046760x.2011.630029 · Cited by 1,375 · cited 2×
“Training data size affects language understanding and generation ability of LMs (Raffel et al., 2022; Cahyawijaya et al., 2021; Rae et al., 2021; Work-shop et al., 2022; Chowdhery et al., 2022; Hoff-mann et al., 2022).”
BackgroundResultEmergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264 · see citing works
Deep reinforcement learning from human preferences
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017) · arXiv · doi:10.48550/arxiv.1706.03741 · Cited by 1,058 · cited 7× · see citing works
“ChatGPT is a successor of the large language model (LLM) InstructGPT (Ouyang et al., 2022) with a dialog interface that is fine-tuned using the Reinforcement Learning with Human Feedback (RLHF) (Christiano et al., 2017) approach.”
MethodInfluentialchrF: character n-gram F-score for automatic MT evaluation
Popović, M. (2015) · Conference on Machine Translation (WMT) · doi:10.18653/v1/w15-3049 · Cited by 1,026 · cited 2× · see citing works
“We evaluate the machine translation ability of Chat-GPT on both high-resource and low-resource languages using the ChrF++ metric (Popovi´c, 2015).”
BackgroundMethodChatGPT and other large language models are double-edged swords
Shen, Y., Heacock, L., Elias, J., Hentel, K. D., Reig, B., Shih, G., & Moy, L. (2023) · Radiology · doi:10.1148/radiol.230163 · Cited by 1,008 · cited 2× · see citing works
“On the other hand, there has also been as much sharing of its failures in simple tasks (Gilson et al., 2022; Choi et al., 2023; Shen et al., 2023).”
BackgroundTruthfulQA: Measuring How Models Mimic Human Falsehoods
Lin, S., Hilton, J., & Evans, O. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.229 · Cited by 904 · see citing works
Self-Consistency Improves Chain of Thought Reasoning in Language Models
Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2203.11171 · Cited by 829 · cited 1× · see citing works
“Reasoning is one of the most actively discussed and debated abilities of LLMs as scaling the model parameter size also increases the implicit knowledge in LLMs (Wei et al., 2022a; Wang et al., 2022; Huang and Chang, 2022).”
BackgroundLaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · cited 1× · see citing works
“To ensure factuality, it is possible to build LLMs with an interface to an external knowledge source, like Blenderbot 3.0 (Shuster et al., 2022) and LaMDa (Thoppilan et al., 2022).”
MethodInfluentialSelf-citationThe State and Fate of Linguistic Diversity and Inclusion in the NLP World
Joshi, P., Santy, S., Budhiraja, A., Bali, K., & Choudhury, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.560 · Cited by 766 · cited 1× · see citing works
“These raise the concern of language diversity and inclusivity in ChatGPT (Joshi et al., 2020; Aji et al., 2022).”
BackgroundChatGPT: Jack of all trades, master of none
Kocoń, J., Cichecki, I., Kaszyca, O., Kochanek, M., Szydło, D., Baran, J., Bielaniewicz, J., Gruza, M., Janz, A., Kanclerz, K., et al. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101861 · Cited by 758 · see citing works
The role of AI in drug discovery: challenges, opportunities, and strategies
Blanco-González, A., Cabezón, A., Seco-González, A., Conde-Torres, D., Antelo-Riveiro, P., Piñeiro, Á., & Garcia-Fandino, R. (2023) · Pharmaceuticals · doi:10.3390/ph16060891 · Cited by 690 · see citing works
Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks
Weston, J., Bordes, A., Chopra, S., Rush, A. M., van Merriënboer, B., Joulin, A., & Mikolov, T. (2015) · arXiv · doi:10.48550/arxiv.1502.05698 · Cited by 670 · cited 1× · see citing works
“We first investigate basic reasoning skills with bAbI tasks (Weston et al., 2016b), 30 examples each from task 15 (inductive) and task 16 (deductive).”
MethodBeyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.04615 · Cited by 569 · cited 3× · see citing works
“Mao et al. (2023) a tool’s usefulness to society by evaluating ChatGPT’s capabilities on 25 diverse analytical NLP tasks, most of them subjective even to humans.”
BackgroundMethodSelf-citationChatGPT makes medicine easy to swallow: an exploratory case study on simplified radiology reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B. O., Ricke, J., et al. (2023) · European Radiology · doi:10.1007/s00330-023-10213-1 · Cited by 549 · cited 3× · see citing works
“Responsible Generative AI Previous works have discussed the ethical implications or concerns associated with ChatGPT (and other LLMs) (Jabotinsky and Sarel, 2022; Susnjak, 2022; Blanco-Gonzalez et al., 2022; Aydın and Karaarslan, 2022; Jeblick et al., 2022).”
BackgroundIndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding
Wilie, B., Vincentio, K., Winata, G. I., Cahyawijaya, S., Li, X., Lim, Z. Y., Soleman, S., Mahendra, R., Fung, P., Bahar, S., et al. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.85 · Cited by 540 · see citing works
Self-citationIs ChatGPT a General-Purpose Natural Language Processing Task Solver?
Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., & Yang, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.85 · Cited by 519 · cited 1× · see citing works
“…robust complex reasoning capabilities, which should also be facilitated by the creation of more comprehensive benchmarks for assessing these abilities, such as works by Laskar et al. (2023b); Qin et al. (2023); Davis (2023a), particularly when multiple abilities are required to complete the tasks.”
BackgroundThink you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462 · cited 1× · see citing works
“We provide the options to ChatGPT as: Choose from: left, right, above, below, lower-left, lower-right, upper-left, upper-right . we only use the QA part of EntailmentBank, that the authors took from ARC dataset (Clark et al., 2018), as we aim to test for reasoning capability.”
MethodReasoning with Language Model Prompting: A SurveyPreprint
Qiao, S.-F., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S.-M., Tan, C.-Q., Huang, F., & Chen, H.-J. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.09597 · Cited by 456 · cited 5× · see citing works
“Other works focus on more specific abilities such as mathematical skills (Davis, 2023b), reasoning (Webb et al., 2022a; Qiao et al., 2022).”
BackgroundMethodInfluentialTowards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2022) · arXiv · doi:10.48550/arxiv.2212.10403 · Cited by 428 · see citing works
SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization
Gliwa, B., Mochol, I., Biesek, M., & Wawer, A. (2019) · Proceedings of the 2nd Workshop on New Frontiers in Summarization · doi:10.18653/v1/d19-5409 · Cited by 424 · cited 2× · see citing works
“We test on 100 samples from two common summarization datasets: half from SAMSum (Gliwa et al., 2019), a dialogue summarization dataset, and another half from CNN/DM (Hermann et al., 2015; Nallapati et al., 2016), news summarization datasets.”
MethodEmergent analogical reasoning in large language models
Webb, T., Holyoak, K. J., & Lu, H. (2023) · Nature Human Behaviour · doi:10.1038/s41562-023-01659-w · Cited by 404 · cited 2× · see citing works
“Other works focus on more specific abilities such as mathematical skills (Davis, 2023b), reasoning (Webb et al., 2022a; Qiao et al., 2022).”
BackgroundOpenAI ChatGPT Generated Literature Review: Digital Twin in Healthcare
Aydın, Ö., & Karaarslan, E. (2022) · doi:10.2139/ssrn.4308687 · Cited by 391 · cited 1× · see citing works
“Responsible Generative AI Previous works have discussed the ethical implications or concerns associated with ChatGPT (and other LLMs) (Jabotinsky and Sarel, 2022; Susnjak, 2022; Blanco-Gonzalez et al., 2022; Aydın and Karaarslan, 2022; Jeblick et al., 2022).”
BackgroundChatGPT: The End of Online Exam Integrity?
Susnjak, T. (2022) · arXiv · doi:10.48550/arxiv.2212.09292 · Cited by 370 · cited 2× · see citing works
“Responsible Generative AI Previous works have discussed the ethical implications or concerns associated with ChatGPT (and other LLMs) (Jabotinsky and Sarel, 2022; Susnjak, 2022; Blanco-Gonzalez et al., 2022; Aydın and Karaarslan, 2022; Jeblick et al., 2022).”
BackgroundDissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345 · cited 1× · see citing works
“Mahowald et al. (2023) thoroughly studies LLMs capabilities by distinguishing formal and functional linguistic competence with reference to cognitive science, psychology, and NLP to clarify the discourse surrounding LLMs’ potential.”
Abductive Commonsense ReasoningPreprint
Bhagavatula, C., Bras, R. J. L., Malaviya, C., Sakaguchi, K., Holtzman, A., Rashkin, H., Downey, D., Yih, S., & Choi, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.05739 · Cited by 331 · cited 1× · see citing works
“We test ChatGPT’s language-based abductive reasoning ability with 30 samples from α NLI dataset (Bhagavatula et al., 2020), which requires the model to select the most plausible explanation given the conclusion.”
MethodFortes, G. (2021) · A Logical Theory of Causality · doi:10.4135/9781483346663.n2 · Cited by 321 · cited 1×
“Inductive reasoning is based on “observations or evidence” while deductive is based on “truth of the premises” (i.e., necessarily true inference) (Douven, 2017).”
BackgroundHow Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection
Guo, B., Zhang, X., Wang, Z., Jiang, M., Nie, J., Ding, Y., Yue, J., & Wu, Y. (2023) · arXiv · doi:10.48550/arxiv.2301.07597 · Cited by 318 · cited 3× · see citing works
“Guo et al. (2023) conducts linguistic analyses of ChatGPT’s writing against human experts and found that ChatGPT responses are strictly focused on the given question, more formal, objective, and less emotional.”
BackgroundResultOpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs
Moon, S., Shah, P., Kumar, A., & Subba, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1081 · Cited by 306 · cited 1× · see citing works
“To quantitatively measure ChatGPT’s performance on knowledge-grounded dialogue, we utilize 50 samples from the test set of OpenDialKG (Moon et al., 2019), which contains open-ended dialogues grounded on a knowledge path.”
MethodCommonsenseQA: A Question Answering Challenge Targeting Commonsense KnowledgePreprint
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00937 · Cited by 296 · see citing works
oLMpics-On What Language Model Pre-training Captures
Talmor, A., Elazar, Y., Goldberg, Y., & Berant, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00342 · Cited by 267 · see citing works
S oloist : BuildingTask Bots at Scale with Transfer Learning and Machine Teaching
Peng, B., Li, C., Li, J., Shayandeh, S., Lidén, L., & Gao, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00399 · Cited by 264 · see citing works
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark DatasetsPreprint
Laskar, M. T. R., Bari, M. S., Rahman, M., Bhuiyan, M. A. H., Joty, S. R., & Huang, X.-J. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2305.18486 · Cited by 260 · cited 1× · see citing works
“There has been recent automatic or human-in-the-loop evaluations such as (Laskar et al., 2023a) Nevertheless, many of the benchmarks are not necessarily interpretable to laypeople for general purposes, such as named entity recognition and etc.”
BackgroundChatGPT: The End of Online Exam Integrity?
Sušnjak, T., & McIntosh, T. R. (2024) · Education Sciences · doi:10.3390/educsci14060656 · Cited by 245 · cited 2× · see citing works
“Responsible Generative AI Previous works have discussed the ethical implications or concerns associated with ChatGPT (and other LLMs) (Jabotinsky and Sarel, 2022; Susnjak, 2022; Blanco-Gonzalez et al., 2022; Aydın and Karaarslan, 2022; Jeblick et al., 2022).”
BackgroundChatGPT is not all you need. A state-of-the-art review of large generative AI models
Gozalo-Brizuela, R., & Garrido-Merchan, E. C. (2023) · arXiv · doi:10.48550/arxiv.2301.04655 · Cited by 245 · see citing works
Analysing Mathematical Reasoning Abilities of Neural Models
Saxton, D., Grefenstette, E., Hill, F., & Kohli, P. (2019) · arXiv · doi:10.48550/arxiv.1904.01557 · Cited by 241 · cited 2× · see citing works
“Not surprisingly, it could only score 23.33% (7/30) for the MATH dataset (Saxton et al., 2019), which tests mathematical reasoning.”
BackgroundInfluentialScaling Language Models: Methods, Analysis & Insights from Training Gopher
Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.11446 · Cited by 235 · cited 1× · see citing works
“Training data size affects language understanding and generation ability of LMs (Raffel et al., 2022; Cahyawijaya et al., 2021; Rae et al., 2021; Work-shop et al., 2022; Chowdhery et al., 2022; Hoff-mann et al., 2022).”
BackgroundLai, V. D., Ngo, N. T., Veyseh, A. P. B., Man, H., Dernoncourt, F., Bui, T., & Nguyen, T. H. (2023) · arXiv · doi:10.48550/arxiv.2304.05613 · Cited by 233 · cited 1× · see citing works
“Nevertheless, ChatGPT still lacks the ability to understand and generate sentences in low-resource languages (§2.2), which is also supported by Lai et al. (2023).”
BackgroundInfluentialThe Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation
Goyal, N., Gao, C., Chaudhary, V., Chen, P., Wenzek, G., Ju, D., Krishnan, S., Ranzato, M., Guzmán, F., & Fan, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00474 · Cited by 209 · cited 2× · see citing works
“We experiment with 6 languages: French, Chinese, Indonesian, Korean, Javanese, and Sundanese from the FLORES-200 dataset (Team et al., 2022; Goyal et al., 2021).”
BackgroundMethodCarnap, R. (1945) · Philosophia Scientiæ · doi:10.1086/286851 · Cited by 203 · cited 2× · see citing works
“Inductive and deductive reasoning are common forms of logical reasoning that are categorized by “a degree to which the premise supports the conclusion” based on logic and philosophy (Qiao et al 2022; Rogers et al., 2022; Hawthorne, 2021).”
BackgroundKolko, J. (2017) · doi:10.1007/1-4020-3907-7 · Cited by 200 · cited 1× · see citing works
“Another way to categorize is based on the “direction of reasoning” – deductive is from premise to conclusion while abductive is from conclusion to the most probable premise that supports the conclusion (Walton, 2014).”
Logic and artificial intelligence
Nilsson, N. J. (1991) · Artificial Intelligence · doi:10.1016/0004-3702(91)90049-p · Cited by 198 · see citing works
Zang, X., Rastogi, A., & Chen, J. (2020) · Workshop on NLP for Conversational AI (NLP4ConvAI) · doi:10.18653/v1/2020.nlp4convai-1.13 · Cited by 195 · cited 1× · see citing works
“2 (Zang et al., 2020).”
GPTEval: A Survey on Assessments of ChatGPT and GPT-4Preprint
Mao, R., Chen, G., Zhang, X., et al. (2023) · International Conference on Language Resources and Evaluation · doi:10.48550/arxiv.2308.12488 · Cited by 178 · cited 1× · see citing works
“Mao et al. (2023) a tool’s usefulness to society by evaluating ChatGPT’s capabilities on 25 diverse analytical NLP tasks, most of them subjective even to humans.”
BackgroundQA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
Rogers, A., Gardner, M., & Augenstein, I. (2022) · ACM Computing Surveys · doi:10.1145/3560260 · Cited by 177 · cited 4× · see citing works
“Inductive and deductive reasoning are common forms of logical reasoning that are categorized by “a degree to which the premise supports the conclusion” based on logic and philosophy (Qiao et al 2022; Rogers et al., 2022; Hawthorne, 2021).”
BackgroundInfluentialNews Summarization and Evaluation in the Era of GPT-3
Goyal, T., Li, J. J., & Durrett, G. (2022) · arXiv · doi:10.48550/arxiv.2209.12356 · Cited by 175 · cited 1× · see citing works
“Average of performances for CNN and DM from Goyal et al. (2022).”
One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia
Aji, A. F., Winata, G. I., Koto, F., et al. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2203.13357 · Cited by 172 · cited 4×
“We explore the sentiment analysis ability of ChatGPT through 4 languages with diverse amounts of resources in NusaX (Winata et al., 2022): English (eng), Indonesian (ind), Javanese (jav), and Buginese (bug).”
BackgroundInfluentialSelf-citationGilson, A., Safranek, C., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2022) · doi:10.1101/2022.12.23.22283901 · Cited by 172 · see citing works
Knowledge-Grounded Dialogue Generation with Pre-trained Language Models
Zhao, X., Wu, W., Xu, C., Tao, C., Zhao, D., & Yan, R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.272 · Cited by 168 · cited 1× · see citing works
“To boost informativeness, these systems leverage external knowledge, including structured knowledge such as knowledge graphs (Zhao et al., 2020; Ji et al., 2022b) and unstructured knowledge such as free text (Xu et al., 2022, 2023).”
BackgroundRed teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity
Zhuo, T. Y., Huang, Y., Chen, C., & Xing, Z. (2023) · arXiv · doi:10.48550/arxiv.2301.12867 · Cited by 164 · see citing works
UBAR: Towards Fully End-to-End Task-Oriented Dialog System with GPT-2
Yang, Y., Li, Y., & Quan, X. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17674 · Cited by 163 · see citing works
NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages
Winata, G. I., Aji, A. F., Cahyawijaya, S., et al. (2022) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2205.15960 · Cited by 159 · see citing works
Self-citationWhat is the role of induction and deduction in reasoning and scientific inquiry?
Lawson, A. E. (2005) · Journal of Research in Science Teaching · doi:10.1002/tea.20067 · Cited by 124 · cited 1× · see citing works
“On the contrary, inductive reasoning involves specific observation of patterns , processing them on increasingly abstract cycles of hypothetico-deductive reasoning to draw a more general conclusion (Lawson, 2005).”
BackgroundStepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts
Shi, Z., Zhang, Q., & Lipani, A. (2022) · AAAI Conference on Artificial Intelligence · doi:10.48550/arxiv.2204.08292 · Cited by 123 · cited 1×
“…physical commonsense reasoning: given a sentence such as “When boiling butter, when it’s ready, you can " , the goal is to fill in the blank with one of two answer options, “Pour it onto a plate" and “Pour Prompt Please judge if this predicate is (likely) plausible or implausible: “knife chop rock””
BackgroundIndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation
Cahyawijaya, S., Winata, G. I., Wilie, B., Vincentio, K., Li, X., Kuncoro, A., Ruder, S., Lim, Z. Y., Bahar, S., Khodra, M. L., et al. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.699 · Cited by 119 · cited 1× · see citing works
“Training data size affects language understanding and generation ability of LMs (Raffel et al., 2022; Cahyawijaya et al., 2021; Rae et al., 2021; Work-shop et al., 2022; Chowdhery et al., 2022; Hoff-mann et al., 2022).”
BackgroundSelf-citationSelf-Instruct: Aligning Language Models with Self-Generated Instructions
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., & Hajishirzi, H. (2022) · arXiv · doi:10.48550/arxiv.2212.10560 · Cited by 118 · see citing works
Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation
Dai, W., Hou, L., Shang, L.-F., et al. (2022) · Findings · doi:10.48550/arxiv.2203.06386 · Cited by 115 · cited 2×
“…is a purely text-prompted language model, it is unlikely to explore its multimodal capabilities with visual inputs like contemporary vision-language works (Rombach et al., 2022; Ramesh et al., 2021; Yu et al., 2021a; Radford et al., 2021; Dai et al., 2022; Lovenia et al., 2022; Dai et al., 2023a).”
BackgroundSelf-citationCLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text
Sinha, K., Sodhani, S., Dong, J., Pineau, J., & Hamilton, W. L. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1458 · Cited by 110 · cited 3× · see citing works
“An interesting finding along with CLUTRR was that Chat-GPT can’t differentiate son and grandson but can differentiate daughter and granddaughter when it induces the logical rules governing kinship relationships.”
Learning Reasoning Strategies in End-to-End Differentiable Proving
Minervini, P., Riedel, S., Stenetorp, P., et al. (2020) · International Conference on Machine Learning · doi:10.3233/faia210359 · Cited by 108
NusaCrowd: Open Source Initiative for Indonesian NLP Resources
Cahyawijaya, S., Lovenia, H., Aji, A. F., et al. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.09648 · Cited by 99 · cited 1×
“Furthermore, ChatGPT achieves near-perfect scores on three tasks, i.e., bAbI task 15, EntailmentBank, and Pep-3k.”
MethodSelf-citationChatGPT Makes Medicine Easy to Swallow: An Exploratory Case Study on Simplified Radiology Reports
Jeblick, K., Schachtner, B., Dexl, J., Mittermeier, A., Stüber, A. T., Topalis, J., Weber, T., Wesp, P., Sabel, B., Ricke, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.14882 · Cited by 97 · see citing works
Explaining Answers with Entailment Trees
Dalvi, B. B., Jansen, P. A., Tafjord, O., Xie, Z., Smith, H., Pipatanangkura, L., & Clark, P. E. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.585 · Cited by 96 · cited 1× · see citing works
“When we repeat the analysis on advanced tasks, specifically on CLUTRR (Sinha et al., 2019) for induction and En-tailmentBank for deduction (Dalvi et al., 2021), the same conclusion holds based on our experiment.”
Müller, B., Anastasopoulos, A., Sagot, B., & Seddah, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.38 · Cited by 96 · see citing works
Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingPreprint
Dai, W., Liu, Z., Ji, Z., et al. (2022) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2210.07688 · Cited by 91 · cited 1× · see citing works
“LLMs are known to be susceptible to generating nonfactual, untruthful information, which is referred to as hallucination (Lee et al., 2022; Ji et al., 2022a,b; Su et al., 2022; Dai et al., 2023b; Xu et al., 2023).”
BackgroundSelf-citationRHO ($ρ$): Reducing Hallucination in Open-domain Dialogues with Knowledge GroundingPreprint
Ji, Z., Liu, Z., Lee, N., et al. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2212.01588 · Cited by 87 · cited 1× · see citing works
“To boost informativeness, these systems leverage external knowledge, including structured knowledge such as knowledge graphs (Zhao et al., 2020; Ji et al., 2022b) and unstructured knowledge such as free text (Xu et al., 2022, 2023).”
BackgroundSelf-citationUNKs Everywhere: Adapting Multilingual Language Models to New Scripts
Pfeiffer, J., Vulić, I., Gurevych, I., & Ruder, S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.800 · Cited by 86 · see citing works
Benchmarks for Automated Commonsense Reasoning: A Survey
Davis, E. (2023) · ACM Computing Surveys · doi:10.1145/3615355 · Cited by 84 · cited 1× · see citing works
“…robust complex reasoning capabilities, which should also be facilitated by the creation of more comprehensive benchmarks for assessing these abilities, such as works by Laskar et al. (2023b); Qin et al. (2023); Davis (2023a), particularly when multiple abilities are required to complete the tasks.”
BackgroundPutting ChatGPT's Medical Advice to the (Turing) Test
Nov, O., Singh, N., & Mann, D. M. (2023) · medRxiv · doi:10.48550/arxiv.2301.10035 · Cited by 81 · cited 1×
“Nov et al. (2023) also studies ChatGPT’s generated medical advice if it passes the Turing test.”
BackgroundDissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2023) · arXiv · doi:10.48550/arxiv.2301.06627 · Cited by 80 · see citing works
Towards Few-shot Fact-Checking via Perplexity
Lee, N., Bang, Y., Madotto, A., & Fung, P. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.158 · Cited by 79 · cited 1× · see citing works
“We test ChatGPT’s ability to detect misinformation with the test sets that consist of scientific and social claims related to COVID-19 (Lee et al., 2021) with 100 samples.”
MethodSelf-citationMavi, V., Jangra, A., & Jatowt, A. (2024) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000102 · Cited by 79 · cited 1× · see citing works
“Multi-hop Reasoning To be able to reason over a larger context, a system has to perform multi-hop reasoning over more than one piece of information to arrive at the answer (Mavi et al., 2022).”
BackgroundBenchmarks for Automated Commonsense Reasoning: A Survey
Davis, E. (2023) · arXiv · doi:10.48550/arxiv.2302.04752 · Cited by 77 · see citing works
AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization
Yu, T., Liu, Z., & Fung, P. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.471 · Cited by 75 · cited 1× · see citing works
“Interactivity on Summarization Summarization models aim to extract essential information from documents and to generate short, concise, and readable text (Yu et al., 2021b; Su et al., 2021).”
BackgroundSelf-citationVision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization
Yu, T., Dai, W., Liu, Z., & Fung, P. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.326 · Cited by 75 · cited 1× · see citing works
“…is a purely text-prompted language model, it is unlikely to explore its multimodal capabilities with visual inputs like contemporary vision-language works (Rombach et al., 2022; Ramesh et al., 2021; Yu et al., 2021a; Radford et al., 2021; Dai et al., 2022; Lovenia et al., 2022; Dai et al., 2023a).”
BackgroundSelf-citationChatGPT for (Finance) Research: The Bananarama ConjecturePreprint
Dowling, M., & Lucey, B. M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4322651 · Cited by 73 · cited 1× · see citing works
“There are many investigations of ChatGPT’s understanding and potential applications in different fields such as law (Choi et al., 2023), medical domain (Blanco-Gonzalez et al., 2022; Jeblick et al., 2022) and finance (Birch, 2022; Dowling and Lucey, 2023).”
BackgroundDescription-Driven Task-Oriented Dialog ModelingPreprint
Zhao, J., Gupta, R., Cao, Y., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2201.08904 · Cited by 73 · see citing works
Factuality Enhanced Language Models for Open-Ended Text Generation
Lee, N., Ping, W., Xu, P., Patwary, M., Fung, P., Shoeybi, M., & Catanzaro, B. (2022) · arXiv · doi:10.48550/arxiv.2206.04624 · Cited by 68 · cited 2× · see citing works
“LLMs are known to be susceptible to generating nonfactual, untruthful information, which is referred to as hallucination (Lee et al., 2022; Ji et al., 2022a,b; Su et al., 2022; Dai et al., 2023b; Xu et al., 2023).”
BackgroundSelf-citationFew-Shot Bot: Prompt-Based Learning for Dialogue SystemsPreprint
Madotto, A., Lin, Z., Winata, G. I., & Fung, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08118 · Cited by 66 · cited 1× · see citing works
“In detail, we ask the model to provide the belief state as domain-intent: [slot1, value1], . . . in the prompt following previous zero-shot (Lin et al., 2021) and few-shot (Madotto et al., 2021) approaches, and provide an exhaustive list of domain-intent-slot-value for the given dialogue.”
BackgroundSelf-citationTIMEDIAL: Temporal Commonsense Reasoning in Dialog
Qin, L., Gupta, A., Upadhyay, S., He, L., Choi, Y., & Faruqui, M. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.549 · Cited by 59 · cited 1× · see citing works
“For this category, we conduct experiments on the dataset TimeDial (Qin et al., 2021), which solely requires temporal reasoning.”
MethodIlmania, A., Abdurrahman, F., Cahyawijaya, S., et al. (2018) · International Conference on Asian Language Processing · doi:10.1109/ialp.2018.8629181 · Cited by 55 · cited 1× · see citing works
“Sentiment analysis has been widely explored for both high-resource and low-resource languages (Wang et al., 2018a; Wilie et al., 2020; Ilmania et al., 2018).”
BackgroundSelf-citationCommonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey
Bhargava, P., & Ng, V. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21496 · Cited by 54 · cited 1× · see citing works
“Recent works show that LLMs perform impressively well on commonsense reasoning benchmarks (Qiao et al., 2022; Huang and Chang, 2022; Bhargava and Ng, 2022).”
BackgroundModeling Semantic Plausibility by Injecting World Knowledge
Wang, S., Durrett, G., & Erk, K. (2018) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n18-2049 · Cited by 54 · cited 4× · see citing works
“We use Pep-3k (Wang et al., 2018b), which requires the model to recognize plausible but possibly novel events, such as “man swallow paintball" .”
BackgroundMethodInfluentialZero-Shot Dialogue State Tracking via Cross-Task Transfer
Lin, Z., Liu, B., Madotto, A., Moon, S., Zhou, Z., Crook, P., Wang, Z., Zhou, Y., Cho, E., Subba, R., et al. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.622 · Cited by 52 · cited 1× · see citing works
“In detail, we ask the model to provide the belief state as domain-intent: [slot1, value1], . . . in the prompt following previous zero-shot (Lin et al., 2021) and few-shot (Madotto et al., 2021) approaches, and provide an exhaustive list of domain-intent-slot-value for the given dialogue.”
BackgroundSelf-citationThoughtSource: A central hub for large language model reasoning data
Ott, S., Hebenstreit, K., Liévin, V., Hother, C., Moradi, M., Mayrhauser, M., Praas, R., Winther, O., & Samwald, M. (2023) · Scientific Data · doi:10.1038/s41597-023-02433-3 · Cited by 52 · cited 1× · see citing works
“This aligns with the findings that LLMs are also limited in several ways, and fail to produce accurate predictions due to their inability to accomplish complex reasoning, such as solving tasks that require multi-hop reasoning (Ott et al., 2023).”
BackgroundTransfer Learning with Synthetic Corpora for Spatial Role Labeling and Reasoning
Mirzaee, R., & Kordjamshidi, P. (2022) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2210.16952 · Cited by 51
SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning
Mirzaee, R., Faghihi, H. R., Qiang, N., & Kordjamshidi, P. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.364 · Cited by 44 · cited 1× · see citing works
“…physical commonsense reasoning: given a sentence such as “When boiling butter, when it’s ready, you can " , the goal is to fill in the blank with one of two answer options, “Pour it onto a plate" and “Pour Prompt Please judge if this predicate is (likely) plausible or implausible: “knife chop rock””
BackgroundMathematics, word problems, common sense, and artificial intelligencePreprint
Davis, E. (2023) · Bulletin of the American Mathematical Society · doi:10.48550/arxiv.2301.09723 · Cited by 44 · see citing works
PIQA: Reasoning about Physical Commonsense in Natural Language
Bisk, Y., Zellers, R., Le bras, R., Gao, J., & Choi, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6239 · Cited by 37 · cited 3× · see citing works
“Based on our evaluation with CommonsenseQA (Talmor et al., 2018), PiQA (Bisk et al., 2020) and Pep-3k (Wang et al., 2018b), ChatGPT shows surprisingly good commonsense reasoning capability, perhaps due to its large parametric memory.”
MethodRead before Generate! Faithful Long Form Question Answering with Machine Reading
Su, D., Li, X., Zhang, J., Shang, L., Jiang, X., Liu, Q., & Fung, P. (2022) · arXiv · doi:10.48550/arxiv.2203.00343 · Cited by 37 · cited 1× · see citing works
“LLMs are known to be susceptible to generating nonfactual, untruthful information, which is referred to as hallucination (Lee et al., 2022; Ji et al., 2022a,b; Su et al., 2022; Dai et al., 2023b; Xu et al., 2023).”
BackgroundSelf-citationShades of BLEU, Flavours of Success: The Case of MultiWOZ
Nekvinda, T., & Dušek, O. (2021) · Workshop on Generation, Evaluation and Metrics (GEM) · doi:10.18653/v1/2021.gem-1.4 · Cited by 29 · see citing works
Crosslingual Generalization through Multitask Finetuning
Muennighoff, N., Wang, T., Sutawika, L., Roberts, A., Biderman, S., Scao, T. L., Bari, M. S., Shen, S., Yong, Z.-X., Schoelkopf, H., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.01786 · Cited by 29 · see citing works
Retrieval-Free Knowledge-Grounded Dialogue Response Generation with Adapters
Xu, Y., Ishii, E., Cahyawijaya, S., Liu, Z., Winata, G. I., Madotto, A., Su, D., & Fung, P. (2022) · DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc) · doi:10.18653/v1/2022.dialdoc-1.10 · Cited by 28 · see citing works
Self-citationImprove Query Focused Abstractive Summarization by Incorporating Answer Relevance
Su, D., Yu, T., & Fung, P. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.275 · Cited by 25 · cited 1× · see citing works
“Interactivity on Summarization Summarization models aim to extract essential information from documents and to generate short, concise, and readable text (Yu et al., 2021b; Su et al., 2021).”
BackgroundSelf-citationEmergent Analogical Reasoning in Large Language ModelsPreprint
Webb, T. W., Holyoak, K. J., & Lu, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.09196 · Cited by 21 · see citing works
Moeller, P. D. (2005) · Educational Media Reviews Online · doi:10.59236/emro.v7i10a2680 · Cited by 18
Findings of the WMT 2022 Shared Task on Automatic Post-Editing
Bhattacharyya, P., Chatterjee, R., Freitag, M., et al. (2022) · Conference on Machine Translation · doi:10.18653/v1/2022.wmt-1.5 · Cited by 18
Specializing Multilingual Language Models: An Empirical Study
Chau, E. C., & Smith, N. A. (2021) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2021.mrl-1.5 · Cited by 15 · see citing works
Modeling Event Plausibility with Consistent Conceptual Abstraction
Porada, I., Suleman, K., Trischler, A., et al. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.138 · Cited by 11 · see citing works
Diverse and Faithful Knowledge-Grounded Dialogue Generation via Sequential Posterior Inference
Xu, Y., Kong, D., Xu, D., et al. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2306.01153 · Cited by 10 · cited 1×
“LLMs are known to be susceptible to generating nonfactual, untruthful information, which is referred to as hallucination (Lee et al., 2022; Ji et al., 2022a,b; Su et al., 2022; Dai et al., 2023b; Xu et al., 2023).”
BackgroundSelf-citationEvery picture tells a story: Image-grounded controllable stylistic story generation
Lovenia, H., Wilie, B., Barraud, R., et al. (2022) · LATECHCLFL · doi:10.48550/arxiv.2209.01638 · Cited by 10 · cited 1×
“…is a purely text-prompted language model, it is unlikely to explore its multimodal capabilities with visual inputs like contemporary vision-language works (Rombach et al., 2022; Ramesh et al., 2021; Yu et al., 2021a; Radford et al., 2021; Dai et al., 2022; Lovenia et al., 2022; Dai et al., 2023a).”
BackgroundSelf-citationModeling Semantic Plausibility by Injecting World KnowledgePreprint
Wang, S., Durrett, G., & Erk, K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.00619 · Cited by 6 · see citing works
Is ChatGPT a General-Purpose Natural Language Processing Task Solver?
Qin, C., Zhang, A., Zhang, Z., Chen, J., Yasunaga, M., & Yang, D. (2023) · arXiv · doi:10.48550/arxiv.2302.06476
ThoughtSource: A central hub for large language model reasoning data
Ott, S., Hebenstreit, K., Liévin, V., Hother, C. E., Moradi, M., Mayrhauser, M., Praas, R., Winther, O., & Samwald, M. (2023) · arXiv · doi:10.48550/arxiv.2301.11596
Flagged first, then by citations
33 of the 151 references couldn’t be matched to a known paper.
Research works printed but not linked: 21
- Li Du, Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. 2022. e-CARE: a new dataset for exploring explain- able causal reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 432-446, Dublin, Ireland. Association for Computational Lin- guistics. · DOI
- Simon Frieder, Luca Pinchetti, Ryan-Rhys Grif- fiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. 2023. Mathematical capabilities of chatgpt. · DOI · Paper page
- Aidan Gilson, Conrad Safranek, Thomas Huang, Vimig Socrates, Ling Chi, Richard Andrew Taylor, and David Chartash. 2022. How well does chatgpt do when taking the medical licensing exams? the im- plications of large language models for medical edu- cation and knowledge assessment. medRxiv, pages 2022-12.
- James Hawthorne. 2021. Inductive Logic. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philoso- phy, Spring 2021 edition. Metaphysics Research Lab, Stanford University.
- Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Si- monyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optimal large language models. · DOI · Paper page
- Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022a. Survey of halluci- nation in natural language generation. ACM Comput. Surv. Just Accepted. · DOI · Paper page
- Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study. · DOI · Paper page
- Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Huang. 2023a. A systematic study and comprehensive evaluation of ChatGPT on benchmark datasets. In Findings of the Association for Com- putational Linguistics: ACL 2023, pages 431-469, Toronto, Canada. Association for Computational Lin- guistics. · DOI
- Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020b. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880.
- Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Ku- mar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Man- ning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Lad- hak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2022. Holistic eval- uation of language models. · DOI · Paper page
- Hongyuan Lu, Haoyang Huang, Shuming Ma, Dong- dong Zhang, Wai Lam, and Furu Wei. 2022. Trip: Triangular document-level pre-training for multilingual language models. arXiv preprint arXiv:2212.07752. · arXiv
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Partha Pratim Ray. 2023. Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems.
- Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. 2022b. StepGame: A new benchmark for robust multi-hop spatial reasoning in texts. Proceedings of the AAAI Conference on Artificial Intelligence, 36(10):11321- 11329. · DOI
- Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, and Jason Weston. 2022. Blenderbot 3: a deployed conversational agent that continually learns to respon- sibly engage. · DOI · Paper page
- Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. 2022. Recitation-augmented language models. In The Eleventh International Conference on Learning Representations.
- Ada Wan. 2022. Fairness in representation for multi- lingual NLP: Insights from controlled experiments on conditional language modeling. In International Conference on Learning Representations.
- Peter Cathcart Wason and Philip Nicholas Johnson- Laird. 1972. Psychology of reasoning: Structure and content, volume 86. Harvard University Press.
- Jason Weston, Antoine Bordes, Sumit Chopra, and Tomás Mikolov. 2016a. Towards ai-complete ques- tion answering: A set of prerequisite toy tasks. In 4th International Conference on Learning Represen- tations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
- PMLR.
- Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Good- man. 2022. Star: Bootstrapping reasoning with rea- soning. In Advances in Neural Information Process- ing Systems.
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
“, 2023) ✓ ✓ (Bang et al., 2023) ✓ ✓ ✓ ✓ ✓ ✓ ✓ (Bai et al.”
BackgroundInfluentialHuang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 2,142 · see citing works
Unifying Large Language Models and Knowledge Graphs: A Roadmap
Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3352100 · Cited by 1,376 · see citing works
Practical and ethical challenges of large language models in education: A systematic scoping review
Yan, L., Sha, L., Zhao, L., Li, Y., Martinez‐Maldonado, R., Chen, G., Li, X., Jin, Y., & Gašević, D. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13370 · Cited by 935 · see citing works
How is ChatGPT's behavior changing over time?
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09009 · Cited by 507 · see citing works
Can Large Language Models Transform Computational Social Science?
Ziems, C., Held, W., Shaikh, O., Chen, J., Zhang, Z., & Yang, D. (2024) · Computational Linguistics · doi:10.1162/coli_a_00502 · Cited by 585 · see citing works
Evaluating Object Hallucination in Large Vision-Language Models
Li, Y., Du, Y., Zhou, K., Wang, J., Xin, Z., & Wen, J.-R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.20 · Cited by 580 · see citing works
Mathematical discoveries from program search with large language models
Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S., Wang, P., Fawzi, O., et al. (2024) · Nature · doi:10.1038/s41586-023-06924-6 · Cited by 544 · see citing works
The impact of ChatGPT on higher education
Dempere, J., Modugu, K., Hesham, A., & Ramasamy, L. K. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1206936 · Cited by 519 · see citing works
Towards Reasoning in Large Language Models: A Survey
Huang, J., & Chang, K. C.-C. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.67 · Cited by 428 · see citing works
Benchmarking Large Language Models in Retrieval-Augmented Generation
Chen, J., Lin, H., Han, X., & Sun, L. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29728 · Cited by 454 · see citing works
Use chat gpt to solve programming bugs
Surameery, N. M. S., & Shakor, M. Y. (2023) · International Journal of Information technology and Computer Engineering · doi:10.55529/ijitc.31.17.22 · Cited by 382 · see citing works
“A novel ramework for evaluating interactive language models like ChatGPT using publicly available datasets is proposed in [8] .”
MethodGender bias and stereotypes in Large Language Models
Kotek, H., Dockum, R., & Sun, D. (2023) · Proceedings of The ACM Collective Intelligence Conference · doi:10.1145/3582269.3615599 · Cited by 381 · see citing works
Improving large language models for clinical named entity recognition via prompt engineering
Hu, Y., Chen, Q., Du, J., Peng, X., Keloth, V. K., Zuo, X., Zhou, Y., Li, Z., Jiang, X., Lu, Z., et al. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad259 · Cited by 314 · see citing works
HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
Li, J., Cheng, X., Zhao, X., Nie, J.-Y., & Wen, J.-R. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.397 · Cited by 343 · see citing works
Query Rewriting in Retrieval-Augmented Large Language Models
Ma, X., Gong, Y., He, P., Zhao, H., & Duan, N. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.322 · Cited by 391 · see citing works
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972 · see citing works
Towards Mitigating LLM Hallucination via Self Reflection
Ji, Z., Yu, T., Xu, Y., Lee, N., Ishii, E., & Fung, P. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.findings-emnlp.123 · Cited by 563 · see citing works
Self-citationThe Challenges and Opportunities of AI-Assisted Writing: Developing AI Literacy for the AI Age
Cardon, P. W., Fleischmann, C., Aritz, J., Logemann, M., & Heidewald, J. (2023) · Business and Professional Communication Quarterly · doi:10.1177/23294906231176517 · Cited by 245 · see citing works
“Bang et al. (2023) explored the ability of ChatGPT to engage in 10 different reasoning categories.”
BackgroundIs chatGPT a good nlg evaluator? A preliminary study
Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., & Zhou, J. (2023) · Proceedings of the 4th New Frontiers in Summarization Workshop · doi:10.18653/v1/2023.newsum-1.1 · Cited by 235 · see citing works
Shen, X., Chen, Z., Backes, M., Shen, Y., & Zhang, Y. (2024) · Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3658644.3670388 · Cited by 441 · see citing works
Lai, V. D., Ngo, N. T., Veyseh, A. P. B., Man, H., Dernoncourt, F., Bui, T., & Nguyen, T. H. (2023) · arXiv · doi:10.48550/arxiv.2304.05613 · Cited by 233 · see citing works
Korzynski, P., Mazurek, G., Krzypkowska, P., & Kurasinski, A. (2023) · Entrepreneurial Business and Economics Review · doi:10.15678/eber.2023.110302 · Cited by 235 · see citing works
“Bang et al. (2023) found that similarly to other LLMs, ChatGPT suffers from hallucination problems that refer to the phenomenon of LLMs generating plausible‐sounding but incorrect or unsupported information in their responses.”
BackgroundInfluentialLee, U., Jung, H., Jeon, Y., Sohn, Y., Hwang, W., Moon, J., & Kim, H. (2024) · Education and Information Technologies · doi:10.1007/s10639-023-12249-8 · Cited by 217 · see citing works
“ChatGPT, as identified in several studies (Bang et al., 2023; Alkaissi & McFarlane, 2023; Rudolph et al., 2023; Ali et al., 2022), can produce a ’hallucination’ effect or plausible but incorrect answers, particularly for complex tasks.”
Lai, V., Ngo, N., Pouran Ben Veyseh, A., Man, H., Dernoncourt, F., Bui, T., & Nguyen, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.878 · Cited by 233 · see citing works
Dabis, A., & Csáki, C. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03526-z · Cited by 181 · see citing works
Let’s Have a Chat! A Conversation with ChatGPT: Technology, Applications, and Limitations
Shahriar, S., & Hayawi, K. (2023) · Artificial Intelligence and Applications · doi:10.47852/bonviewaia3202939 · Cited by 167 · see citing works
“ChatGPT can produce code representations of visual images based on text prompts, but its drawing skills are somewhat limited currently [38].”
BackgroundMethodInfluentialEvaluating Large Language Models in Class-Level Code Generation
Du, X.-Y., Liu, M.-W., Wang, K.-X., Wang, H.-L., Liu, J.-W., Chen, Y.-X., Feng, J.-Y., Sha, C.-F., Peng, X., & Lou, Y. (2024) · International Conference on Software Engineering · doi:10.1145/3597503.3639219 · Cited by 377 · see citing works
“To date, the evaluation for LLMs has covered a wide range [20], encompassing not only traditional NLP tasks ( e.g., sentiment analysis [17], question answering [16], and reasoning [19]) but also some specific downstream domains ( e.g., medicine [23], agent [34] and recommendation system [28]).”
BackgroundHolderried, F., Stegemann–Philipps, C., Herschbach, L., Moldt, J.-A., Nevins, A., Griewatz, J., Holderried, M., Herrmann-Werner, A., Festl-Wietek, T., & Mahling, M. (2024) · JMIR Medical Education · doi:10.2196/53961 · Cited by 156 · see citing works
“5 is known to perform reasonably well in both abductive and commonsense reasoning tasks [46,47]; our data confirmed these observations.”
ResultTowards Interpretable Mental Health Analysis with Large Language Models
Yang, K., Ji, S., Zhang, T., Xie, Q., Kuang, Z., & Ananiadou, S. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.370 · Cited by 155 · see citing works
“Most recently, many efforts have evaluated the performance of LLMs such as ChatGPT and GPT-4 on various NLP tasks (Bang et al., 2023; Qin et al., 2023), such as machine translation (Jiao et al., 2023), text generation and evaluation (Benoit, 2023; Luo et al., 2023), language inference (Zhong et…”
BackgroundAlkhalaf, M., Yu, P., Yin, M., & Deng, C. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104662 · Cited by 156 · see citing works
Toxicity in chatgpt: Analyzing persona-assigned language models
Deshpande, A., Murahari, V., Rajpurohit, T., Kalyan, A., & Narasimhan, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.88 · Cited by 178 · see citing works
Detecting and Preventing Hallucinations in Large Vision Language Models
Gunjal, A., Yin, J., & Bas, E. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29771 · Cited by 161 · see citing works
Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study
Cao, Y., Zhou, L., Lee, S., Cabello, L., Chen, M., & Hershcovich, D. (2023) · Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP) · doi:10.18653/v1/2023.c3nlp-1.7 · Cited by 163 · see citing works
Oniani, D., Hilsman, J., Peng, Y., Poropatich, R. K., Pamplin, J. C., Legault, G. L., & Wang, Y. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00965-x · Cited by 143 · see citing works
Med-HALT: Medical Domain Hallucination Test for Large Language Models
Pal, A., Umapathi, L. K., & Sankarasubbu, M. (2023) · Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL) · doi:10.18653/v1/2023.conll-1.21 · Cited by 152 · see citing works
Exintaris, B., Karunaratne, N., & Yuriev, E. (2023) · Journal of Chemical Education · doi:10.1021/acs.jchemed.3c00481 · Cited by 123 · see citing works
Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
Liu, Y., Han, T., Ma, S., Zhang, J., Yang, Y., Tian, J., He, H., Li, A., He, M., Liu, Z., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.01852
“Reference [3] evaluates the technical performance of ChatGPT in multitask, multilingual, and multimodal aspects based on 23 standard public datasets and newly designed multimodal datasets, including eight different common natural language processing application tasks.”
BackgroundInfluentialPerils and opportunities in using large language models in psychological research
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae245 · Cited by 111 · see citing works
“Many studies of state-of-the-art LLMs have concluded that their outputs are highly “human-like” (24 – 26).”
A Survey of Generative AI Applications
Gozalo-Brizuela, R., & Merchan, E. E. G. (2024) · Journal of Computer Science · doi:10.3844/jcssp.2024.801.818 · Cited by 101 · see citing works
Roos, J., Kasapovic, A., Jansen, T., & Kaczmarczyk, R. (2023) · JMIR Medical Education · doi:10.2196/46482 · Cited by 98 · see citing works
“As it does not have access to an external knowledge base, more extrinsic hallucinations are generated [7].”
BackgroundUnifying Large Language Models and Knowledge Graphs: A RoadmapPreprint
Pan, S., Luo, L., Wang, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.08302 · Cited by 97 · see citing works
“However, further studies reveal that LLMs are not able to recall facts and often experience hallucinations by generating statements that are factually incorrect [15], [28].”
BackgroundZhai, X.-M., Nyaaba, M., & Ma, W. (2024) · Science Education · doi:10.1007/s11191-024-00496-1 · Cited by 102 · see citing works
A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets
Laskar, M. T. R., Bari, M. S., Rahman, M., Bhuiyan, M. A. H., Joty, S., & Huang, J. X. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.29 · Cited by 117 · see citing works
Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models
Wang, X., Tang, X., Zhao, X., Wang, J., & Wen, J.-R. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.621 · Cited by 116 · see citing works
Performance of two large language models for data extraction in evidence synthesis
Konet, A., Thomas, I., Gartlehner, G., Kahwati, L., Hilscher, R., Kugley, S., Crotty, K., Viswanathan, M., & Chew, R. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1732 · Cited by 79 · see citing works
Human-AI Collaboration in Thematic Analysis using ChatGPT: A User Study and Design Recommendations
Yan, L., Echeverría, V., Fernandez-Nieto, G. M., Jin, Y., Swiecki, Z., Zhao, L., Gašević, D., & Martínez‐Maldonado, R. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650732 · Cited by 102 · see citing works
“P8 cautioned: "First of all is that, for instance, it might be inaccurate in the sense that the codes might be made up.”
BackgroundDr ChatGPT tell me what I want to hear: How different prompts impact health answer correctness
Koopman, B., & Zuccon, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.928 · Cited by 73 · see citing works
“Besides the decision to treat Unsure as Incorrect, we did not perform an in depth analysis of these explanations (e.g., to verify whether the claims made in the explanation were true or were hallucinations (Ji et al., 2022; Bang et al., 2023)), because we lack the required medical expertise.”
Li, X., Chan, S., Zhu, X., Pei, Y., Ma, Z., Liu, X., & Shah, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.39 · Cited by 86 · see citing works
“…(Christiano et al., 2023) and masked language model objectives, the most recent models such as ChatGPT 1 and GPT-4 2 have demonstrated exceptional capabilities in a wide range of natural language processing (NLP) tasks (Bang et al., 2023a; Liu et al., 2023; Omar et al., 2023; Khoury et al., 2023).”
Ha, J., Jeon, H., Han, D. S., Seo, J., & Oh, C. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642472 · Cited by 76 · see citing works
Huang, H., Tang, T., Zhang, D., Zhao, X., Song, T., Xia, Y., & Wei, F. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.826 · Cited by 75 · see citing works
ChatGPT and assistive AI in structured radiology reporting: A systematic review
Sacoransky, E., Kwan, B. Y. M., & Soboleski, D. (2024) · Current Problems in Diagnostic Radiology · doi:10.1067/j.cpradiol.2024.07.007 · Cited by 65 · see citing works
Rethinking homework in the age of artificial intelligence
Ibrahim, H., Asim, R., Zaffar, F., Rahwan, T., & Zaki, Y. (2023) · IEEE Intelligent Systems · doi:10.1109/mis.2023.3255599 · Cited by 65 · see citing works
The perils and promises of fact-checking with large language models
Quelle, D., & Bovet, A. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1341697 · Cited by 65 · see citing works
Retrieving Supporting Evidence for Generative Question Answering
Huo, S., Arabzadeh, N., & Clarke, C. L. A. (2023) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3624918.3625336 · Cited by 65 · see citing works
“However, despite the excitement created by these improvements, the LLMs may confidently and convincingly generate hallucinated results [3, 17].”
BackgroundExploring the future of mathematics teaching: Insight with ChatGPT
Supriyadi, E., & Kuncoro, K. S. (2023) · UNION Jurnal Ilmiah Pendidikan Matematika · doi:10.30738/union.v11i2.14898 · Cited by 60 · see citing works
“, 2023), assessing its efficacy in various natural language processing tasks (Bang et al., 2023), and discussing its ethical implications (Mattas, 2023).”
BackgroundResultGraph Neural Prompting with Large Language Models
Tian, Y., Song, H., Wang, Z., Wang, H., Hu, Z., Wang, F., Chawla, N. V., & Xu, P. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29875 · Cited by 75 · see citing works
Nayak, A., Alkaitis, M. S., Nayak, K., Nikolov, M. C., Weinfurt, K. P., & Schulman, K. A. (2023) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2023.2561 · Cited by 57 · see citing works
ChatGPT Informed Graph Neural Network for Stock Movement PredictionPreprint
Chen, Z., Zheng, L., Cheng, L., Yuan, J., & Zhu, D. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4464002 · Cited by 56 · see citing works
“While not initially designed for financial data processing, LLMs have demonstrated their capability to excel in a broad spectrum of Natural Language Processing (NLP) tasks, ranging from language translation to text summarization, question answering, sentiment analysis, and text generation [12].”
BackgroundZero-shot Temporal Relation Extraction with ChatGPT
Yuan, C., Xie, Q., & Ananiadou, S. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.7 · Cited by 59 · see citing works
GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP
Khondaker, M. T. I., Waheed, A., Nagoudi, E. M. B., & Abdul-Mageed, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.16 · Cited by 79 · see citing works
Towards LLM-based Fact Verification on News Claims with a Hierarchical Step-by-Step Prompting Method
Zhang, X., & Gao, W. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.64 · Cited by 90 · see citing works
Battle of the Wordsmiths: Comparing ChatGPT, GPT-4, Claude, and BardPreprint
Borji, A., & Mohammadian, M. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4476855 · Cited by 53 · see citing works
Towards a Catalog of Prompt Patterns to Enhance the Discipline of Prompt Engineering
Schmidt, D. C., Spencer-Smith, J. B., Fu, Q., & White, J. (2024) · ACM SIGAda Ada Letters · doi:10.1145/3672359.3672364 · Cited by 59 · see citing works
“Large language models (LLMs) [1,2] with conversational interfaces, such as ChatGPT [3], are generating and reasoning about art, music, essays and computer programs.”
BackgroundChatgpt: Jack of All Trades, Master of NonePreprint
Kocoń, J., Cichecki, I., Kaszyca, O., et al. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4372889 · Cited by 52 · see citing works
Exploring ChatGPT Capabilities and Limitations: A Critical Review of the NLP Game ChangerPreprint
Koubâa, A., Boulila, W., Ghouti, L., Alzahem, A., & Latif, S. (2023) · Preprints.org · doi:10.20944/preprints202303.0438.v1 · Cited by 52 · see citing works
Liu, X., Wu, J., Shao, A., Shen, W., Ye, P., Wang, Y., Ye, J., Jin, K., & Yang, J. (2024) · Journal of Medical Internet Research · doi:10.2196/51926 · Cited by 52 · see citing works
Deng, Y., Liao, L., Chen, L., Wang, H., Lei, W., & Chua, T. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.711 · Cited by 55 · see citing works
The imitation game: Detecting human and AI-generated texts in the era of ChatGPT and BARD
Hayawi, K., Shahriar, S., & Mathew, S. S. (2024) · Journal of Information Science · doi:10.1177/01655515241227531 · Cited by 55 · see citing works
“They can occasionally generate nonsensical or inappropriate content and are highly sensitive to input phrasing [10–12].”
Natural Language Reasoning, A Survey
Yu, F., Zhang, H., Tiwari, P., & Wang, B. (2024) · ACM Computing Surveys · doi:10.1145/3664194 · Cited by 78 · see citing works
“Such knowledge may be obvious to people but non-trivial to machines since they are difficult to retrieve from the web due to reporting bias—e.g., “when people are hungry, they would like to eat something,” 304:24 F. Yu et al.”
Gubelmann, R. (2023) · Grazer Philosophische Studien · doi:10.1163/18756735-00000182 · Cited by 47 · see citing works
“27 For an early pre-print (without peer-review) of a comprehensive study of ChatGPT’s abilities, see Bang et al. (2023).”
BackgroundSynergi: A Mixed-Initiative System for Scholarly Synthesis and Sensemaking
Kang, H. B., Wu, T., Chang, J. C., & Kittur, A. (2023) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3586183.3606759 · Cited by 45 · see citing works
“Synergi uses Loopy Belief Propagation (LBP) and a novel message weighting scheme on a local citation graph to fnd papers most likely to be important in reviewing related literature, exploring new grounds beyond prior application of LBP in sensemaking over citation graphs [11].”
BackgroundMethodLi, M., Shi, T., Ziems, C., Kan, M., Chen, N. F., Liu, Z., & Yang, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.92 · Cited by 58 · see citing works
“LLMs struggle with named entity recognition (Ding et al., 2022; Qin et al., 2023), relational reasoning (Bang et al., 2023), affective tasks (Koco´n et al., 2023; Amin et al., 2023) and semantic similarity tasks (Kocmi and Federmann, 2023; Wang et al., 2023).”
BackgroundChatGPT MT: Competitive for High- (but Not Low-) Resource Languages
Robinson, N., Ogayo, P., Mortensen, D. R., & Neubig, G. (2023) · Conference on Machine Translation (WMT) · doi:10.18653/v1/2023.wmt-1.40 · Cited by 58 · see citing works
Dubinski, D., Won, S.-Y., Trnovec, S., Behmanesh, B., Baumgarten, P., Dinc, N., Konczalla, J., Chan, A., Bernstock, J. D., Freiman, T. M., et al. (2024) · Acta Neurochirurgica · doi:10.1007/s00701-024-05908-3 · Cited by 45 · see citing works
“ChatGPT and other large language models are prone to hallucination, generating fac-tually incorrect but grammatically fluent content underscoring the need for safeguard in critical use cases [2].”
BackgroundStrengths, Weaknesses, Opportunities, and Threats of Using ChatGPT in Scientific Research
Giray, L., Jacob, J., & Gumalin, D. L. (2024) · International Journal of Technology in Education · doi:10.46328/ijte.618 · Cited by 58 · see citing works
“Studies, such as the research conducted by Bang et al. (2023), have demonstrated the effectiveness of ChatGPT in translating simple text, including non-Latin script languages.”
MethodLiu, M. X., Wu, T., Chen, T., Li, F. M., Kittur, A., & Myers, B. A. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642149 · Cited by 56 · see citing works
“However, LLMs face well-known challenges like hallucination and falsehood [7, 97], which could make their outputs uncertain and less trustworthy, often requiring manual inspection and verification before use [71].”
Exploring prompt pattern for generative artificial intelligence in automatic question generation
Wang, L., Song, R., Guo, W., & Yang, H. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2024.2412082 · Cited by 53 · see citing works
“” This is primarily attributed to ChatGPT ’ s tendency to generate “ hal-lucination ” effects or seemingly plausible but incorrect answers, especially for complex tasks (Bang et al., 2023).”
Towards Autonomous Testing Agents via Conversational Large Language Models
Feldt, R., Kang, S., Yoon, J., & Yoo, S. (2023) · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00148 · Cited by 49 · see citing works
LLMs for Financial Advisement: A Fairness and Efficacy Study in Personal Decision Making
Lakkaraju, K., Jones, S. E., Vuruma, S. K. R., Pallagani, V., Muppasani, B. C., & Srivastava, B. (2023) · 4th ACM International Conference on AI in Finance · doi:10.1145/3604237.3626867 · Cited by 42 · see citing works
Semnani, S. J., Yao, V., Zhang, H., & Lam, M. S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.157 · Cited by 52 · see citing works
Large language models meet text-centric multimodal sentiment analysis: a survey
Yang, H., Zhao, Y., Wu, Y., Wang, S., Zheng, T., Zhang, H., Ma, Z., Che, W., Wang, S., Wei, S., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4593-8 · Cited by 88 · see citing works
“While there have been some attempts to apply LLMs in text-based sentiment analysis [6, 10–13], there is a lack of systematic and comprehensive analysis regarding the application of LLMs and LMMs in multimodal sentiment analysis.”
Professional Certification Benchmark Dataset: The First 500 Jobs for Large Language Models
Noever, D. A., & Ciolino, M. (2023) · doi:10.5121/csit.2023.131211 · Cited by 38 · see citing works
AI and narrative embeddings detect PTSD following childbirth via birth stories
Bartal, A., Jagodnik, K. M., Chan, S. J., & Dekel, S. (2024) · Scientific Reports · doi:10.1038/s41598-024-54242-2 · Cited by 45 · see citing works
“In addition to its conversational abilities, ChatGPT has demonstrated remarkable performance on various other NLP tasks, including question-answering 14 , even in zero-or few-shot learning scenarios 4 .”
A Primer for Evaluating Large Language Models in Social-Science Research
Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251325174 · Cited by 51 · see citing works
“…LLMs to solve a set of tasks and reporting average performance or comparing their responses with human samples can lead to conclusions about general and “human-like” capabilities (e.g., see Bang et al., 2023; Coda-Forno et al., 2023; Dillion et al., 2023; Horton, 2023; H. Liu et al., 2023).”
Zero-Shot Cross-Lingual Summarization via Large Language Models
Wang, J., Liang, Y., Meng, F., Zou, B., Li, Z., Qu, J., & Zhou, J. (2023) · Workshop on New Frontiers in Summarization (NewSum) · doi:10.18653/v1/2023.newsum-1.2 · Cited by 62 · see citing works
“To further exploit the potentiality of conversational LLMs, inspired by Bang et al. (2023), after prompting with Dir, TS or ST prompt, we adopt an interactive ( Inta ) prompt to make the preliminarily generated summary more concise: Please make the Chinese summary shorter. and the whole process is…”
BackgroundMethodFighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation
Lucas, J., Uchendu, A., Yamashita, M., Lee, J., Rohatgi, S., & Lee, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.883 · Cited by 37 · see citing works
Jahan, I., Laskar, M. T. R., Peng, C., & Huang, J. X. (2023) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2023.bionlp-1.30 · Cited by 45 · see citing works
Artificial Intelligence Co-Piloted AuditingPreprint
Gu, H., Schreyer, M., Moffitt, K. C., et al. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4444763 · Cited by 36 · see citing works
Hamed, E. S., Eid, A. M., & Alberry, M. (2023) · Cureus · doi:10.7759/cureus.38784 · Cited by 36 · see citing works
“However, it is important to note that ChatGPT may generate incomplete, inconsistent, or irrelevant information that does not match user intentions or expectations [5,6].”
BackgroundUnveiling the New Frontier: ChatGPT-3 Powered Translation for Arabic-English Language Pairs
Alkhawaja, L. A. (2024) · Theory and Practice in Language Studies · doi:10.17507/tpls.1402.05 · Cited by 41 · see citing works
Huang, T. Y., Safranek, C. W., Socrates, V., Chartash, D., Wright, D., Dilip, M., Sangal, R. B., & Taylor, R. A. (2024) · Journal of Medical Internet Research · doi:10.2196/60336 · Cited by 37 · see citing works
“…could look toward expanding the scope of the use of LLMs in discharge instructions, such as integrating multicultural and multilingual personalization into discharge instructions as well as expanding the amount of teaching and patient education embedded into the discharge instructions [41,42].”
Proposed Framework for Human-like Language Processing of ChatGPT in Academic Writing
Mahyoob, M., Algaraady, J., & Alblwi, A. (2023) · International Journal of Emerging Technologies in Learning (iJET) · doi:10.3991/ijet.v18i14.41725 · Cited by 42 · see citing works
Juhász, L., Mooney, P., Hochmair, H. H., & Guan, B. (2023) · doi:10.31223/x5hq1p · Cited by 46 · see citing works
Cue-CoT: Chain-of-thought Prompting for Responding to In-depth Dialogue Questions with LLMs
Wang, H., Wang, R., Mi, F., Deng, Y., Wang, Z., Liang, B., Xu, R., & Wong, K.-F. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.806 · Cited by 38 · see citing works
“, 2023), especially after the appearance of ChatGPT1, recently revolutionize the paradigm of various natural language processing (NLP) tasks, including dialogue response generation tasks (Bang et al., 2023).”
BackgroundHumans are still better than ChatGPT: Case of the IEEEXtreme competition
Koubâa, A., Qureshi, B., Ammar, A., Khan, Z., Boulila, W., & Ghouti, L. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e21624 · Cited by 36 · see citing works
Hou, C., Zhu, G., Zheng, J., Zhang, L., Huang, X., Zhong, T., Li, S., Du, H., & Ker, C. L. (2024) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3636555.3636910 · Cited by 39 · see citing works
Dictionaries and lexicography in the AI era
Lew, R. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-02889-7 · Cited by 34 · see citing works
“Admittedly, GPT and similar Large Language Models demonstrate remarkable pro fi ciency in English, but their capabilities in other languages, especially minor or less-documented ones, are not nearly as impressive (Bang et al. 2023, Lai et al. 2023).”
Sottana, A., Liang, B., Zou, K., & Yuan, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.543 · Cited by 51 · see citing works
- Authors
- Bang, Y., Cahyawijaya, S., Lee, N., Dai, W., Su, D., Wilie, B., Lovenia, H., Ji, Z., Yu, T., Chung, W., Do, Q. V., Xu, Y., & Fung, P.
- Affiliations
- Hong Kong University of Science and Technology (HK)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2302.04023
- OpenAlex
- W4319793302
- Cited by
- 337
- References
- 151
- Citations by year
- 2026: 14 · 2025: 38 · 2024: 101 · 2023: 184
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- ChatGPT, LLM evaluation, hallucinations, reasoning, multilingual NLP, multimodal generation, zero-shot learning, commonsense reasoning, deductive reasoning, inductive reasoning, human-AI collaboration, prompt engineering, summarization, machine translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
