The DOI matches its DataCite record; the references it cites are not checked yet.
Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4
Open accessAbstract
Harnessing logical reasoning ability is a comprehensive natural language understanding endeavor. With the release of Generative Pretrained Transformer 4 (GPT-4), highlighted as "advanced" at reasoning tasks, we are eager to learn the GPT-4 performance on various logical reasoning tasks. This report analyses multiple logical reasoning datasets, with popular benchmarks like LogiQA and ReClor, and newly-released datasets like AR-LSAT. We test the multi-choice reading comprehension and natural language inference tasks with benchmarks requiring logical reasoning. We further construct a logical reasoning out-of-distribution dataset to investigate the robustness of ChatGPT and GPT-4. We also make a performance comparison between ChatGPT and GPT-4. Experiment results show that ChatGPT performs significantly better than the RoBERTa fine-tuning method on most logical reasoning benchmarks. With early access to the GPT-4 API we are able to conduct intense experiments on the GPT-4 model. The results show GPT-4 yields even higher performance on most logical reasoning datasets. Among benchmarks, ChatGPT and GPT-4 do relatively well on well-known datasets like LogiQA and ReClor. However, the performance drops significantly when handling newly released and out-of-distribution datasets. Logical reasoning remains challenging for ChatGPT and GPT-4, especially on out-of-distribution and natural language inference datasets. We release the prompt-style logical reasoning datasets as a benchmark suite and name it LogiEval.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMsdemonstrate robust arithmetic reasoning capabilities [159] and excel in logical reasoning [124].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [124]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Despite the advances of LLMs, they still struggle with complex logical reasoning problems (Liu et al., 2023b).”
From Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning (2023) · cites this paper as Liu et al., 2023b
In the citing paper
“However, both models face challenges when handling new and out-of-distribution data.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [124]
In the citing paper
“In logical reasoning, Liu et al. [124] indicated that ChatGPT and GPT-4 outperform traditional fine-tuning methods on most benchmarks, demonstrating their superiority in logical reasoning.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [124]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 16
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc- Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learn- ers. CoRR, abs/2005.14165. · arXiv
- Maxwell John Cresswell. 1973. Logics and languages (1st ed.). Routledge.
- Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Proc. of ICML.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language under- standing. CoRR.
- Pratik Joshi, Somak Aditya, Aalok Sathe, and Mono- jit Choudhury. 2020. Taxinli: Taking a ride up the NLU hill. CoRR.
- Hanmeng Liu, Leyang Cui, Jian Liu, and Yue Zhang. 2020. Natural language inference in context -inves- tigating contextual reasoning over long texts. CoRR.
- Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 2019. Roberta: A robustly optimized bert pretrain- ing approach. arXiv.
- Bill MacCartney and Christopher D. Manning. 2007a. Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entail- ment and Paraphrasing.
- Bill MacCartney and Christopher D Manning. 2007b. Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entail- ment and Paraphrasing. OpenAI. 2023. Gpt-4 technical report.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text trans- former. CoRR.
- Swarnadeep Saha, Yixin Nie, and Mohit Bansal. 2020. Conjnli: Natural language inference over conjunc- tive sentences. CoRR.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR.
- Alex Wang, Amanpreet Singh, Julian Michael, Fe- lix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Pro- ceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Net- works for NLP.
- Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. 2022. From lsat: The progress and challenges of complex reasoning. IEEE/ACM Transactions on Au- dio, Speech, and Language Processing.
- Adina Williams, Nikita Nangia, and Samuel Bowman. 2018a. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proc. of NAACL.
- Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Ken- taro Inui, Satoshi Sekine, , and Johan Bos. 2019a. Help: A dataset for identifying shortcomings of neu- ral models in monotonicity reasoning. In Proceed- ings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM2019).
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
Self-citationHow is ChatGPT's behavior changing over time?
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09009 · Cited by 507
ChatGPT: The End of Online Exam Integrity?
Susnjak, T. (2022) · arXiv · doi:10.48550/arxiv.2212.09292 · Cited by 370
“It is also true that LLMs still encounter persistent difficulties across complex reasoning tasks [30] from which it is argued and claimed that due to this, they are again unsuitable for answering sophisticated exam questions with reliability [31,32].”
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
LLMs for knowledge graph construction and reasoning: recent capabilities and future opportunities
Zhu, Y., Wang, X., Chen, J., Qiao, S., Ou, Y., Yao, Y., Deng, S., Chen, H., & Zhang, N. (2024) · World Wide Web · doi:10.1007/s11280-024-01297-w · Cited by 294
“By comprehending the fundamental capabilities of LLMs, our study further delves into potential future directions in the field.”
Yang, L., Chen, H., Li, Z., Ding, X., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3360454 · Cited by 238
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Pan, L., Albalak, A., Wang, X., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2305.12295 · Cited by 144
“…with access to external tools and resources, such as the information retriever (Nakano et al., 2021; Shi et al., 2023; Lazaridou et al., 2022), calculator (Cobbe et al., 2021), code interpreter (Wang et al., 2022), planner (Liu et al., 2023a), and other pre-trained models (Shen et al., 2023).”
BackgroundChristou, P. (2023) · The Qualitative Report · doi:10.46743/2160-3715/2023.6406 · Cited by 226
Perils and opportunities in using large language models in psychological research
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae245 · Cited by 111
“…using LLMs to study human behavior need to move beyond methods that simulate single responses — such as merely predicting group averages or simulating an individual’s response across various tasks (e.g. (24, 25, 27)) — and instead develop robust methods to emulate the complexity of human samples.”
E, K., S, P., R, G., R, K. L., A, B., M, G., T, O., S, R., V, R., H, M., et al. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04752-w · Cited by 89
Li, X., Chan, S., Zhu, X., Pei, Y., Ma, Z., Liu, X., & Shah, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.39 · Cited by 158
Information Retrieval meets Large Language Models: A strategic report from Chinese IR community
Ai, Q., Bai, T., Cao, Z., Chang, Y., Chen, J., Chen, Z., Cheng, Z., Dong, S., Dou, Z., Feng, F., et al. (2023) · AI Open · doi:10.1016/j.aiopen.2023.08.001 · Cited by 115
Kayalı, B., Yavuz, M., Balat, Ş., & Çalışan, M. (2023) · Australasian Journal of Educational Technology · doi:10.14742/ajet.8915 · Cited by 110
Yan, C., Grabowska, M. E., Dickson, A. L., Li, B., Wen, Z., Roden, D. M., Michael Stein, C., Embí, P. J., Peterson, J. F., Feng, Q., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01038-3 · Cited by 73
How should the advancement of large language models affect the practice of science?
Binz, M., Alaniz, S., Roskies, A., Aczel, B., Bergstrom, C. T., Allen, C., Schad, D., Wulff, D., West, J. D., Zhang, Q., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2401227121 · Cited by 74
Liu, X., Wu, J., Shao, A., Shen, W., Ye, P., Wang, Y., Ye, J., Jin, K., & Yang, J. (2024) · Journal of Medical Internet Research · doi:10.2196/51926 · Cited by 52
Impacts of generative artificial intelligence on the future of labor market: a systematic review
Salari, N., Beiromvand, M., Hosseinian-Far, A., Habibi, J., Babajani, F., & Mohammadi, M. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100652 · Cited by 49
Lü, J., Tian, X., Zhang, C., Zhao, Y., Zhang, J., Zhang, W., Feng, C., He, J., Wang, J., & He, F. (2024) · Energy and Built Environment · doi:10.1016/j.enbenv.2024.03.010 · Cited by 55
From Transcripts to Insights: Uncovering Corporate Risks Using Generative AIPreprint
Kim, A. G., Muhn, M., & Nikolaev, V. V. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4593660 · Cited by 45
Park, J., Fang, Y., Ta, C., Zhang, G., Idnay, B., Chen, F., Feng, D. D., Shyu, R., Gordon, E. R., Spotnitz, M., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104649 · Cited by 42
Generative AI in Marketing: Promises, Perils, and Public Policy Implications
Kumar, V., Kotler, P., Gupta, S., & Rajan, B. (2024) · Journal of Public Policy & Marketing · doi:10.1177/07439156241286499 · Cited by 77
Professional Certification Benchmark Dataset: The First 500 Jobs for Large Language Models
Noever, D. A., & Ciolino, M. (2023) · doi:10.5121/csit.2023.131211 · Cited by 41
A Primer for Evaluating Large Language Models in Social-Science Research
Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251325174 · Cited by 66
Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation
Lucas, J., Uchendu, A., Yamashita, M., Lee, J., Rohatgi, S., & Lee, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.883 · Cited by 37
Large Language Models are Temporal and Causal Reasoners for Video Question Answering
Ko, D., Lee, J. E., Kang, W., Roh, B., & Kim, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.261 · Cited by 66
Gubelmann, R. (2024) · Philosophy & Technology · doi:10.1007/s13347-024-00696-1 · Cited by 36
Humans are still better than ChatGPT: Case of the IEEEXtreme competition
Koubâa, A., Qureshi, B., Ammar, A., Khan, Z., Boulila, W., & Ghouti, L. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e21624 · Cited by 46
Hein, D., Christie, A. L., Holcomb, M. J., Xie, B., Jain, A., Vento, J. A., Rakheja, N., Shakur, A. H., Christley, S., Cowell, L. G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01686-z · Cited by 39
Zhao, Z., Zhou, G., Xie, Z., Wu, L., & Huang, J. X. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3658673 · Cited by 29
Qualitative Research in the Era of AI: A Return to Positivism or a New Paradigm?
Chatzichristos, G. (2025) · International Journal of Qualitative Methods · doi:10.1177/16094069251337583 · Cited by 34
Berti, A., Kourani, H., Häfke, H., Li, C.-Y., & Schuster, D. (2024) · Lecture notes in business information processing · doi:10.1007/978-3-031-61007-3_2 · Cited by 38
Arkoudas, K. (2023) · doi:10.20944/preprints202308.0148.v1 · Cited by 26
Text2VRScene: Exploring the Framework of Automated Text-driven Generation System for VR Experience
Yin, Z., Wang, Y., Papatheodorou, T., & Hui, P. (2024) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr58804.2024.00090 · Cited by 29
Hardware–Software Co-Design Enabling Static and Dynamic Sparse Attention Mechanisms
Zhao, J., Zeng, P., Shen, G., Chen, Q., & Guo, M. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3373592 · Cited by 25
Development and evaluation of a large language model of ophthalmology in Chinese
Zheng, C., Ye, H., Guo, J., Yang, J., Fei, P., Yuan, Y., Huang, D., Huang, Y., Peng, J., Xie, X., et al. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-324526 · Cited by 24
Exploring the Boundaries of GPT-4 in Radiology
Liu, Q., Hyland, S. L., Bannur, S., Bouzid, K., Castro, D. C., Wetscherek, M. T., Tinn, R., Sharma, H., Pérez‐García, F., Schwaighofer, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.891 · Cited by 42
Koroleva, D. O., & Jogezai, N. A. (2024) · Information and Learning Sciences · doi:10.1108/ils-10-2023-0137 · Cited by 27
Distortions in Judged Spatial Relations in Large Language Models
Fulman, N., Memduhoğlu, A., & Zipf, A. (2024) · The Professional Geographer · doi:10.1080/00330124.2024.2372792 · Cited by 24
Chauhan, A., Khaliq, F., & Nayak, K. R. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00471-z · Cited by 26
Kim, S., Kim, K., & Wonjeong Jo, C. (2024) · Preventive Medicine Reports · doi:10.1016/j.pmedr.2024.102723 · Cited by 18
LogiCoT: Logical Chain-of-Thought Instruction Tuning
Liu, H., Teng, Z., Cui, L., Zhang, C., Zhou, Q., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.191 · Cited by 37
Self-citationBioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology
O’Donoghue, O., Shtedritski, A., Ginger, J., Abboud, R., Ghareeb, A. E., & Rodriques, S. G. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.162 · Cited by 51
Evaluating ChatGPT-4 Vision on Brazil's National Undergraduate Computer Science Exam
Mendonça, N. C. (2024) · ACM Transactions on Computing Education · doi:10.1145/3674149 · Cited by 22
Uncertain about ChatGPT: enabling the uncertainty evaluation of large language models
Jousselme, A.-L., de Villiers, J. P., de Freitas, A., Blasch, E., Dragos, V., Pavlin, G., Costa, P., Laskey, K. B., & Laudy, C. (2023) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion52260.2023.10224086 · Cited by 16
Lu, M., Gao, F., Tang, X., & Chen, L. (2024) · iScience · doi:10.1016/j.isci.2024.109451 · Cited by 19
Identification and Description of Emotions by Current Large Language ModelsPreprint
Patel, S., & Fan, J. (2023) · Research Square · doi:10.1101/2023.07.17.549421 · Cited by 18
Predicting Learning Performance with Large Language Models: A Study in Adult Literacy
Zhang, L., Lin, J., Borchers, C., Sabatini, J., Hollander, J., Cao, M., & Hu, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60609-0_24 · Cited by 18
Integrating GPT as an Assistant for Low-Cost Virtual Reality Escape-Room Games
Rychert, A., Ganuza, M. L., & Selzer, M. N. (2024) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2024.3426314 · Cited by 16
The bewitching AI: The Illusion of Communication with Large Language Models
Bottazzi Grifoni, E., & Ferrario, R. (2025) · Philosophy & Technology · doi:10.1007/s13347-025-00893-6 · Cited by 15
Structured knowledge-based causal discovery: Agentic streams of thought
Meier, S., Raut, P. N., Mahr, F., Thielen, N., Franke, J. E., & Risch, F. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104202 · Cited by 14
The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics
Leiter, C., Opitz, J., Deutsch, D., Gao, Y., Dror, R., & Eger, S. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.10 · Cited by 19
Wu, T., & Chang, M. (2023) · International Conference on Engineering and Emerging Technologies (ICEET) · doi:10.1109/iceet60227.2023.10525933 · Cited by 14
The Impact of LLM Hallucinations on Motor Skill Learning: A Case Study in Badminton
Qiu, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3444783 · Cited by 17
Exploring the challenges faced by Dutch truck drivers in the era of technological advancement
de Winter, J., Driessen, T., Dodou, D., & Cannoo, A. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1352979 · Cited by 15
Bitcoin research with a transaction graph dataset
Schnoering, H., & Vazirgiannis, M. (2025) · Scientific Data · doi:10.1038/s41597-025-04684-8 · Cited by 19
Xu, A. Y., Singh, M., Balmaceno-Criss, M., Oh, A., Leigh, D., Daher, M., Alsoof, D., McDonald, C. L., Diebo, B. G., & Daniels, A. H. (2025) · Journal of orthopaedic surgery · doi:10.1177/10225536241268789 · Cited by 12
Teaching Information Literacy and Critical Thinking Skills in Chat GPT Time
Bušelić, V. (2023) · International Conference on Computing, Networking, Telecommunications & Engineering Sciences Applications (CoNTESA) · doi:10.1109/contesa61248.2023.10384921 · Cited by 14
Perils and Opportunities in Using Large Language Models in Psychological ResearchPreprint
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J. P., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/tg79n · Cited by 10
MindMap: Constructing Evidence Chains for Multi-Step Reasoning in Large Language Models
Wu, Y., Han, X., Song, W., Cheng, M., & Li, F. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29896 · Cited by 8
Abouzeid, E., Wassef, R., Jawwad, A., & Harris, P. (2025) · JMIR Medical Education · doi:10.2196/69521 · Cited by 12
Perils and Opportunities in Using Large Language Models in Psychological ResearchPreprint
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J. P., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/d695y · Cited by 8
Ma, S., Chen, Q., Wang, X., Zheng, C., Peng, Z., Yin, M., & Ma, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4772689 · Cited by 8
S2AF : An action framework to self-check the Understanding Self-Consistency of Large Language Models
Shao, H., Wang, F., & Xie, Z. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107365 · Cited by 7
Sotardi, V. A. (2024) · Journal of Adolescent Research · doi:10.1177/07435584241291165 · Cited by 8
Anderson, R., Scala, C., Samuel, J., Kumar, V., & Jain, P. H. (2024) · Journal of Big Data and Artificial Intelligence · doi:10.54116/jbdai.v2i1.30 · Cited by 6
Diversity Measures: Domain-Independent Proxies for Failure in Language Model Queries
Ngu, N., Lee, N., & Shakarian, P. (2024) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc59802.2024.00034 · Cited by 6
Joh, H. C., Kim, M.-H., Ko, J. Y., Kim, J. S., & Jue, M. (2024) · Indian Journal of Dermatology · doi:10.4103/ijd.ijd_1050_23 · Cited by 11
LLM-Driven Knowledge Extraction in Temporal and Description Logics
Duranti, D., Giorgini, P., Mazzullo, A., Robol, M., & Roveri, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77792-9_12 · Cited by 7
Hiltmann, T. (2024) · De Gruyter eBooks · doi:10.1515/9783111351490-014 · Cited by 5
Schweitzer, S., Conrads, M., & Naeve, J. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.09.406 · Cited by 7
Ansarinejad, M., Gaweesh, S. M., & Ahmed, M. M. (2025) · Accident Analysis & Prevention · doi:10.1016/j.aap.2025.108178 · Cited by 8
Anderson, R., Scala, C., Samuel, J., Kumar, V., & Jain, P. H. (2023) · Preprints.org · doi:10.20944/preprints202308.1003.v1 · Cited by 5
Hein, D., Christie, A., Holcomb, M. J., Xie, B., Jain, A., Vento, J., Rakheja, N., Shakur, A. H., Christley, S., Cowell, L. G., et al. (2025) · medRxiv · doi:10.1101/2025.02.11.25322107 · Cited by 6
Vervoort, L., & Nikolaev, V. (2025) · Journal for General Philosophy of Science · doi:10.1007/s10838-025-09735-7 · Cited by 4
Rothenhäusler, A. (2024) · De Gruyter eBooks · doi:10.1515/9783111351490-013 · Cited by 4
Prompt suffix-attack against text-to-image diffusion models
Xiong, S., Du, Y., Wang, Z., & Sun, P. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129659 · Cited by 4
Avaliando a habilidade do ChatGPT de realizar provas de Dedução Natural em Lógica Proposicional
Martins, F. L. B., de Oliveira, A. C. A., de Vasconcelos, D. R., & de Menezes, M. V. (2023) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2023.234658 · Cited by 4
GPT-4 Can't ReasonPreprint
Arkoudas, K. (2023) · Preprints.org · doi:10.20944/preprints202308.0148.v2 · Cited by 5
How should the advent of large language models affect the practice of science?Preprint
Binz, M., Alaniz, S., Roskies, A. L., Aczél, B., Bergstrom, C. T., Allen, C., Schad, D. J., Wulff, D. U., West, J. D., Zhang, Q., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/yr9xb · Cited by 16
Large Language Model Reasoning Failures
Song, P., Han, P., & Goodman, N. (2026) · arXiv · doi:10.48550/arxiv.2602.06176 · Cited by 3
Assessing Logical Puzzle Solving in Large Language Models: Insights from a Minesweeper Case Study
Li, Y., Wang, H., & Zhang, C. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.4 · Cited by 11
Self-Rationalization in the Wild: A Large-scale Out-of-Distribution Evaluation on NLI-related tasks
Yang, J., Glockner, M., Rocha, A., & Gurevych, I. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00741 · Cited by 4
Java coding using artificial intelligence
Уандыкова, М., Baitenova, L. M., Мухамеджанова, Г. С., Yeleukulova, A., & Миркасимова, Т. (2024) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1473870 · Cited by 6
Knowledge Trap in Augmented Intelligence: Human Experts' Overfocus on Details Hurts Task PerformancePreprint
Wang, W., Liu, X., Zhang, X., & Hong, Y. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4395858 · Cited by 3
Joh, H. C., Kim, M.-H., Ko, J. Y., Kim, J. S., & Jue, M. (2023) · Research Square · doi:10.21203/rs.3.rs-3241164/v1 · Cited by 3
Is the Answer in the Text? Challenging ChatGPT with Evidence Retrieval from Instructive Text
Henning, S., Anthonio, T., Zhou, W. L., Adel, H., Mesgar, M., & Friedrich, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.949 · Cited by 2
VEglue: Testing Visual Entailment Systems via Object-Aligned Joint Erasing
Chang, Z., Li, M., Wang, J., Li, C., & Wang, Q. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3731244 · Cited by 3
Martins, F. L. B., de Oliveira, A. C. A., de Vasconcelos, D. R., & de Menezes, M. V. (2025) · Revista Brasileira de Informática na Educação · doi:10.5753/rbie.2025.4500 · Cited by 2
Evaluating Large Language Models in Psychological Research: A Guide for ReviewersPreprint
Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/ag7hy · Cited by 3
Evaluating Logical Reasoning Ability of Large Language Models
Chan, E., & Chan, A. C.-F. (2026) · doi:10.1109/icnlp69856.2026.11527900 · Cited by 1
Toward Enhancing Reasoning Capabilities of LLMs: An Approach via Synthetic Logic Corpus
Morishita, T., Morio, G., Yamaguchi, A., & Sogawa, Y. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.520 · Cited by 1
Understanding User Prompting Behavior in Generative AI : A Component Analysis
Jin, Z., Meng, G., Wang, X., Wang, J., Liu, C. X., & Zhang, J. (2025) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.1318 · Cited by 1
Characterizing and Verifying Scientific Claims: Qualitative Causal Structure is All You Need
Wu, J., Chao, W., Zhou, X., & Luo, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.828 · Cited by 1
Lievens, J. (2024) · INTED proceedings · doi:10.21125/inted.2024.0394 · Cited by 2
Integrating Language Models with Symbolic Formulas for First-Order Logic Reasoning
Yu, S., Li, L., Wang, Y., & Zeng, D. D. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446308 · Cited by 1
The Accuracy of Domain Specific and Descriptive Analysis Generated by Large Language Models
Otieno, D. O., Abri, F., Siami‐Namini, S., & Namin, A. S. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00274 · Cited by 2
Limitations of and Lessons from the Learning of Large Language ModelsPreprint
Oldenburg, R. (2023) · Qeios · doi:10.32388/9fh6ad · Cited by 1
Xu, F., Zhou, T., Nguyen, T., Bao, H., Lin, C., & Du, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4805664 · Cited by 1
PAACE: A Plan-Aware Automated Agent Context Engineering Framework
Yuksel, K. A. (2025) · arXiv · doi:10.48550/arxiv.2512.16970
Metadata
- Authors
- Liu, H., Ning, R., Teng, Z., Liu, J., Zhou, Q., & Zhang, Y.
- Affiliations
- Westlake University (CN)
- Zhejiang University (CN)
- Nanyang Technological University (SG)
- Fudan University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2304.03439
- OpenAlex
- W4363671827
- Cited by
- 100
- References
- 0
- Citations by year
- 2026: 4 · 2025: 20 · 2024: 39 · 2023: 36
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- logical reasoning, ChatGPT, GPT-4, natural language inference, reading comprehension, benchmark dataset, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
