Evaluating the logical reasoning ability of chatgpt and GPT-4
Open accessAbstract
Harnessing logical reasoning ability is a comprehensive natural language understanding endeavor. With the release of Generative Pretrained Transformer 4 (GPT-4), highlighted as "advanced" at reasoning tasks, we are eager to learn the GPT-4 performance on various logical reasoning tasks. This report analyses multiple logical reasoning datasets, with popular benchmarks like LogiQA and ReClor, and newly-released datasets like AR-LSAT. We test the multi-choice reading comprehension and natural language inference tasks with benchmarks requiring logical reasoning. We further construct a logical reasoning out-of-distribution dataset to investigate the robustness of ChatGPT and GPT-4. We also make a performance comparison between ChatGPT and GPT-4. Experiment results show that ChatGPT performs significantly better than the RoBERTa fine-tuning method on most logical reasoning benchmarks. With early access to the GPT-4 API we are able to conduct intense experiments on the GPT-4 model. The results show GPT-4 yields even higher performance on most logical reasoning datasets. Among benchmarks, ChatGPT and GPT-4 do relatively well on well-known datasets like LogiQA and ReClor. However, the performance drops significantly when handling newly released and out-of-distribution datasets. Logical reasoning remains challenging for ChatGPT and GPT-4, especially on out-of-distribution and natural language inference datasets. We release the prompt-style logical reasoning datasets as a benchmark suite and name it LogiEval.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
LLMs demonstrate robust arithmetic reasoning capabilities and excel in logical reasoning.
“LLMsdemonstrate robust arithmetic reasoning capabilities [159] and excel in logical reasoning [124].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [124]
The citation to this paper is attached to the logical-reasoning clause, not the arithmetic-reasoning clause. The Conclusion reports that ChatGPT and GPT-4 outperform a traditional fine-tuning method on most logical-reasoning benchmarks, but also says they struggle with new and out-of-distribution data and do not perform especially well on logical-reasoning NLI tasks. Thus, 'excel' overstates the paper's qualified finding. (LLM, conf=0.88)
Experimental — accuracy not yet measuredLLMs still struggle with complex logical reasoning problems.
“Despite the advances of LLMs, they still struggle with complex logical reasoning problems (Liu et al., 2023b).”
From Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning (2023) · cites this paper as Liu et al., 2023b
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 16
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc- Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learn- ers. CoRR, abs/2005.14165. · arXiv
- Maxwell John Cresswell. 1973. Logics and languages (1st ed.). Routledge.
- Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Proc. of ICML.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language under- standing. CoRR.
- Pratik Joshi, Somak Aditya, Aalok Sathe, and Mono- jit Choudhury. 2020. Taxinli: Taking a ride up the NLU hill. CoRR.
- Hanmeng Liu, Leyang Cui, Jian Liu, and Yue Zhang. 2020. Natural language inference in context -inves- tigating contextual reasoning over long texts. CoRR.
- Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 2019. Roberta: A robustly optimized bert pretrain- ing approach. arXiv.
- Bill MacCartney and Christopher D. Manning. 2007a. Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entail- ment and Paraphrasing.
- Bill MacCartney and Christopher D Manning. 2007b. Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entail- ment and Paraphrasing. OpenAI. 2023. Gpt-4 technical report.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text trans- former. CoRR.
- Swarnadeep Saha, Yixin Nie, and Mohit Bansal. 2020. Conjnli: Natural language inference over conjunc- tive sentences. CoRR.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR.
- Alex Wang, Amanpreet Singh, Julian Michael, Fe- lix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Pro- ceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Net- works for NLP.
- Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. 2022. From lsat: The progress and challenges of complex reasoning. IEEE/ACM Transactions on Au- dio, Speech, and Language Processing.
- Adina Williams, Nikita Nangia, and Samuel Bowman. 2018a. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proc. of NAACL.
- Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Ken- taro Inui, Satoshi Sekine, , and Johan Bos. 2019a. Help: A dataset for identifying shortcomings of neu- ral models in monotonicity reasoning. In Proceed- ings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM2019).
Cited by
Sorted by citations
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 2,842 · see citing works
Self-citationHow is ChatGPT's behavior changing over time?
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09009 · Cited by 507 · see citing works
ChatGPT: The End of Online Exam Integrity?
Susnjak, T. (2022) · arXiv · doi:10.48550/arxiv.2212.09292 · Cited by 370 · see citing works
“It is also true that LLMs still encounter persistent difficulties across complex reasoning tasks [30] from which it is argued and claimed that due to this, they are again unsuitable for answering sophisticated exam questions with reliability [31,32].”
A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335 · see citing works
LLMs for knowledge graph construction and reasoning: recent capabilities and future opportunities
Zhu, Y., Wang, X., Chen, J., Qiao, S., Ou, Y., Yao, Y., Deng, S., Chen, H., & Zhang, N. (2024) · World Wide Web · doi:10.1007/s11280-024-01297-w · Cited by 294 · see citing works
“By comprehending the fundamental capabilities of LLMs, our study further delves into potential future directions in the field.”
Yang, L., Chen, H., Li, Z., Ding, X., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3360454 · Cited by 238 · see citing works
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213 · see citing works
Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
Pan, L., Albalak, A., Wang, X., & Wang, W. Y. (2023) · arXiv · doi:10.48550/arxiv.2305.12295 · Cited by 144 · see citing works
“…with access to external tools and resources, such as the information retriever (Nakano et al., 2021; Shi et al., 2023; Lazaridou et al., 2022), calculator (Cobbe et al., 2021), code interpreter (Wang et al., 2022), planner (Liu et al., 2023a), and other pre-trained models (Shen et al., 2023).”
BackgroundChristou, P. (2023) · The Qualitative Report · doi:10.46743/2160-3715/2023.6406 · Cited by 192 · see citing works
Perils and opportunities in using large language models in psychological research
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2024) · PNAS Nexus · doi:10.1093/pnasnexus/pgae245 · Cited by 111 · see citing works
“…using LLMs to study human behavior need to move beyond methods that simulate single responses — such as merely predicting group averages or simulating an individual’s response across various tasks (e.g. (24, 25, 27)) — and instead develop robust methods to emulate the complexity of human samples.”
E, K., S, P., R, G., R, K. L., A, B., M, G., T, O., S, R., V, R., H, M., et al. (2023) · BMC Medical Education · doi:10.1186/s12909-023-04752-w · Cited by 89 · see citing works
Li, X., Chan, S., Zhu, X., Pei, Y., Ma, Z., Liu, X., & Shah, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.39 · Cited by 86 · see citing works
Information Retrieval meets Large Language Models: A strategic report from Chinese IR community
Ai, Q., Bai, T., Cao, Z., Chang, Y., Chen, J., Chen, Z., Cheng, Z., Dong, S., Dou, Z., Feng, F., et al. (2023) · AI Open · doi:10.1016/j.aiopen.2023.08.001 · Cited by 100 · see citing works
Kayalı, B., Yavuz, M., Balat, Ş., & Çalışan, M. (2023) · Australasian Journal of Educational Technology · doi:10.14742/ajet.8915 · Cited by 103 · see citing works
Yan, C., Grabowska, M. E., Dickson, A. L., Li, B., Wen, Z., Roden, D. M., Michael Stein, C., Embí, P. J., Peterson, J. F., Feng, Q., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01038-3 · Cited by 73 · see citing works
How should the advancement of large language models affect the practice of science?
Binz, M., Alaniz, S., Roskies, A., Aczel, B., Bergstrom, C. T., Allen, C., Schad, D., Wulff, D., West, J. D., Zhang, Q., et al. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2401227121 · Cited by 74 · see citing works
Liu, X., Wu, J., Shao, A., Shen, W., Ye, P., Wang, Y., Ye, J., Jin, K., & Yang, J. (2024) · Journal of Medical Internet Research · doi:10.2196/51926 · Cited by 52 · see citing works
Impacts of generative artificial intelligence on the future of labor market: a systematic review
Salari, N., Beiromvand, M., Hosseinian-Far, A., Habibi, J., Babajani, F., & Mohammadi, M. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100652 · Cited by 49 · see citing works
Lü, J., Tian, X., Zhang, C., Zhao, Y., Zhang, J., Zhang, W., Feng, C., He, J., Wang, J., & He, F. (2024) · Energy and Built Environment · doi:10.1016/j.enbenv.2024.03.010 · Cited by 48 · see citing works
From Transcripts to Insights: Uncovering Corporate Risks Using Generative AIPreprint
Kim, A. G., Muhn, M., & Nikolaev, V. V. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4593660 · Cited by 45 · see citing works
Park, J., Fang, Y., Ta, C., Zhang, G., Idnay, B., Chen, F., Feng, D. D., Shyu, R., Gordon, E. R., Spotnitz, M., et al. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104649 · Cited by 42 · see citing works
Generative AI in Marketing: Promises, Perils, and Public Policy Implications
Kumar, V., Kotler, P., Gupta, S., & Rajan, B. (2024) · Journal of Public Policy & Marketing · doi:10.1177/07439156241286499 · Cited by 61 · see citing works
Professional Certification Benchmark Dataset: The First 500 Jobs for Large Language Models
Noever, D. A., & Ciolino, M. (2023) · doi:10.5121/csit.2023.131211 · Cited by 38 · see citing works
A Primer for Evaluating Large Language Models in Social-Science Research
Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459251325174 · Cited by 51 · see citing works
Fighting Fire with Fire: The Dual Role of LLMs in Crafting and Detecting Elusive Disinformation
Lucas, J., Uchendu, A., Yamashita, M., Lee, J., Rohatgi, S., & Lee, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.883 · Cited by 37 · see citing works
Large Language Models are Temporal and Causal Reasoners for Video Question Answering
Ko, D., Lee, J. E., Kang, W., Roh, B., & Kim, H. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.261 · Cited by 48 · see citing works
Gubelmann, R. (2024) · Philosophy & Technology · doi:10.1007/s13347-024-00696-1 · Cited by 36 · see citing works
Humans are still better than ChatGPT: Case of the IEEEXtreme competition
Koubâa, A., Qureshi, B., Ammar, A., Khan, Z., Boulila, W., & Ghouti, L. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e21624 · Cited by 36 · see citing works
Hein, D., Christie, A. L., Holcomb, M. J., Xie, B., Jain, A., Vento, J. A., Rakheja, N., Shakur, A. H., Christley, S., Cowell, L. G., et al. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01686-z · Cited by 30 · see citing works
Qualitative Research in the Era of AI: A Return to Positivism or a New Paradigm?
Chatzichristos, G. (2025) · International Journal of Qualitative Methods · doi:10.1177/16094069251337583 · Cited by 34 · see citing works
Zhao, Z., Zhou, G., Xie, Z., Wu, L., & Huang, J. X. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3658673 · Cited by 29 · see citing works
Berti, A., Kourani, H., Häfke, H., Li, C.-Y., & Schuster, D. (2024) · Lecture notes in business information processing · doi:10.1007/978-3-031-61007-3_2 · Cited by 31 · see citing works
Arkoudas, K. (2023) · doi:10.20944/preprints202308.0148.v1 · Cited by 26 · see citing works
Text2VRScene: Exploring the Framework of Automated Text-driven Generation System for VR Experience
Yin, Z., Wang, Y., Papatheodorou, T., & Hui, P. (2024) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr58804.2024.00090 · Cited by 25 · see citing works
Hardware–Software Co-Design Enabling Static and Dynamic Sparse Attention Mechanisms
Zhao, J., Zeng, P., Shen, G., Chen, Q., & Guo, M. (2024) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2024.3373592 · Cited by 25 · see citing works
Development and evaluation of a large language model of ophthalmology in Chinese
Zheng, C., Ye, H., Guo, J., Yang, J., Fei, P., Yuan, Y., Huang, D., Huang, Y., Peng, J., Xie, X., et al. (2024) · British Journal of Ophthalmology · doi:10.1136/bjo-2023-324526 · Cited by 24 · see citing works
Exploring the Boundaries of GPT-4 in Radiology
Liu, Q., Hyland, S. L., Bannur, S., Bouzid, K., Castro, D. C., Wetscherek, M. T., Tinn, R., Sharma, H., Pérez‐García, F., Schwaighofer, A., et al. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.891 · Cited by 29 · see citing works
Koroleva, D. O., & Jogezai, N. A. (2024) · Information and Learning Sciences · doi:10.1108/ils-10-2023-0137 · Cited by 26 · see citing works
Distortions in Judged Spatial Relations in Large Language Models
Fulman, N., Memduhoğlu, A., & Zipf, A. (2024) · The Professional Geographer · doi:10.1080/00330124.2024.2372792 · Cited by 20 · see citing works
Chauhan, A., Khaliq, F., & Nayak, K. R. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00471-z · Cited by 23 · see citing works
Kim, S., Kim, K., & Wonjeong Jo, C. (2024) · Preventive Medicine Reports · doi:10.1016/j.pmedr.2024.102723 · Cited by 18 · see citing works
LogiCoT: Logical Chain-of-Thought Instruction Tuning
Liu, H., Teng, Z., Cui, L., Zhang, C., Zhou, Q., & Zhang, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.191 · Cited by 21 · see citing works
Self-citationBioPlanner: Automatic Evaluation of LLMs on Protocol Planning in Biology
O’Donoghue, O., Shtedritski, A., Ginger, J., Abboud, R., Ghareeb, A. E., & Rodriques, S. G. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.162 · Cited by 26 · see citing works
Evaluating ChatGPT-4 Vision on Brazil's National Undergraduate Computer Science Exam
Mendonça, N. C. (2024) · ACM Transactions on Computing Education · doi:10.1145/3674149 · Cited by 17 · see citing works
Uncertain about ChatGPT: enabling the uncertainty evaluation of large language models
Jousselme, A.-L., de Villiers, J. P., de Freitas, A., Blasch, E., Dragos, V., Pavlin, G., Costa, P., Laskey, K. B., & Laudy, C. (2023) · International Conference on Information Fusion (FUSION) · doi:10.23919/fusion52260.2023.10224086 · Cited by 16 · see citing works
Lu, M., Gao, F., Tang, X., & Chen, L. (2024) · iScience · doi:10.1016/j.isci.2024.109451 · Cited by 17 · see citing works
Identification and Description of Emotions by Current Large Language ModelsPreprint
Patel, S., & Fan, J. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.07.17.549421 · Cited by 17 · see citing works
Predicting Learning Performance with Large Language Models: A Study in Adult Literacy
Zhang, L., Lin, J., Borchers, C., Sabatini, J., Hollander, J., Cao, M., & Hu, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60609-0_24 · Cited by 15 · see citing works
Integrating GPT as an Assistant for Low-Cost Virtual Reality Escape-Room Games
Rychert, A., Ganuza, M. L., & Selzer, M. N. (2024) · IEEE Computer Graphics and Applications · doi:10.1109/mcg.2024.3426314 · Cited by 15 · see citing works
Structured knowledge-based causal discovery: Agentic streams of thought
Meier, S., Raut, P. N., Mahr, F., Thielen, N., Franke, J. E., & Risch, F. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104202 · Cited by 13 · see citing works
The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics
Leiter, C., Opitz, J., Deutsch, D., Gao, Y., Dror, R., & Eger, S. (2023) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2023.eval4nlp-1.10 · Cited by 15 · see citing works
Wu, T., & Chang, M. (2023) · International Conference on Engineering and Emerging Technologies (ICEET) · doi:10.1109/iceet60227.2023.10525933 · Cited by 12 · see citing works
The Impact of LLM Hallucinations on Motor Skill Learning: A Case Study in Badminton
Qiu, Y. (2024) · IEEE Access · doi:10.1109/access.2024.3444783 · Cited by 16 · see citing works
Exploring the challenges faced by Dutch truck drivers in the era of technological advancement
de Winter, J., Driessen, T., Dodou, D., & Cannoo, A. (2024) · Frontiers in Public Health · doi:10.3389/fpubh.2024.1352979 · Cited by 12 · see citing works
Bitcoin research with a transaction graph dataset
Schnoering, H., & Vazirgiannis, M. (2025) · Scientific Data · doi:10.1038/s41597-025-04684-8 · Cited by 12 · see citing works
Xu, A. Y., Singh, M., Balmaceno-Criss, M., Oh, A., Leigh, D., Daher, M., Alsoof, D., McDonald, C. L., Diebo, B. G., & Daniels, A. H. (2025) · Journal of orthopaedic surgery · doi:10.1177/10225536241268789 · Cited by 11 · see citing works
Teaching Information Literacy and Critical Thinking Skills in Chat GPT Time
Bušelić, V. (2023) · International Conference on Computing, Networking, Telecommunications & Engineering Sciences Applications (CoNTESA) · doi:10.1109/contesa61248.2023.10384921 · Cited by 9 · see citing works
Perils and Opportunities in Using Large Language Models in Psychological ResearchPreprint
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J. P., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2023) · doi:10.31219/osf.io/tg79n · Cited by 8 · see citing works
MindMap: Constructing Evidence Chains for Multi-Step Reasoning in Large Language Models
Wu, Y., Han, X., Song, W., Cheng, M., & Li, F. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29896 · Cited by 8 · see citing works
Abouzeid, E., Wassef, R., Jawwad, A., & Harris, P. (2025) · JMIR Medical Education · doi:10.2196/69521 · Cited by 12 · see citing works
Perils and Opportunities in Using Large Language Models in Psychological ResearchPreprint
Abdurahman, S., Atari, M., Karimi-Malekabadi, F., Xue, M. J., Trager, J. P., Park, P. S., Golazizian, P., Omrani, A., & Dehghani, M. (2023) · doi:10.31234/osf.io/d695y · Cited by 8 · see citing works
Ma, S., Chen, Q., Wang, X., Zheng, C., Peng, Z., Yin, M., & Ma, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4772689 · Cited by 8 · see citing works
S2AF : An action framework to self-check the Understanding Self-Consistency of Large Language Models
Shao, H., Wang, F., & Xie, Z. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107365 · Cited by 7 · see citing works
Sotardi, V. A. (2024) · Journal of Adolescent Research · doi:10.1177/07435584241291165 · Cited by 8 · see citing works
Anderson, R., Scala, C., Samuel, J., Kumar, V., & Jain, P. H. (2024) · Journal of Big Data and Artificial Intelligence · doi:10.54116/jbdai.v2i1.30 · Cited by 6 · see citing works
Diversity Measures: Domain-Independent Proxies for Failure in Language Model Queries
Ngu, N., Lee, N., & Shakarian, P. (2024) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc59802.2024.00034 · Cited by 6 · see citing works
Joh, H. C., Kim, M.-H., Ko, J. Y., Kim, J. S., & Jue, M. (2024) · Indian Journal of Dermatology · doi:10.4103/ijd.ijd_1050_23 · Cited by 6 · see citing works
LLM-Driven Knowledge Extraction in Temporal and Description Logics
Duranti, D., Giorgini, P., Mazzullo, A., Robol, M., & Roveri, M. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-77792-9_12 · Cited by 6 · see citing works
Hiltmann, T. (2024) · De Gruyter eBooks · doi:10.1515/9783111351490-014 · Cited by 5 · see citing works
Schweitzer, S., Conrads, M., & Naeve, J. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.09.406 · Cited by 7 · see citing works
Ansarinejad, M., Gaweesh, S. M., & Ahmed, M. M. (2025) · Accident Analysis & Prevention · doi:10.1016/j.aap.2025.108178 · Cited by 5 · see citing works
Anderson, R., Scala, C., Samuel, J., Kumar, V., & Jain, P. H. (2023) · Preprints.org · doi:10.20944/preprints202308.1003.v1 · Cited by 5 · see citing works
Hein, D., Christie, A., Holcomb, M. J., Xie, B., Jain, A., Vento, J., Rakheja, N., Shakur, A. H., Christley, S., Cowell, L. G., et al. (2025) · medRxiv · doi:10.1101/2025.02.11.25322107 · Cited by 6 · see citing works
Vervoort, L., & Nikolaev, V. (2025) · Journal for General Philosophy of Science · doi:10.1007/s10838-025-09735-7 · Cited by 4 · see citing works
Rothenhäusler, A. (2024) · De Gruyter eBooks · doi:10.1515/9783111351490-013 · Cited by 4 · see citing works
Prompt suffix-attack against text-to-image diffusion models
Xiong, S., Du, Y., Wang, Z., & Sun, P. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.129659 · Cited by 4 · see citing works
Avaliando a habilidade do ChatGPT de realizar provas de Dedução Natural em Lógica Proposicional
Martins, F. L. B., de Oliveira, A. C. A., de Vasconcelos, D. R., & de Menezes, M. V. (2023) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2023.234658 · Cited by 4 · see citing works
GPT-4 Can't ReasonPreprint
Arkoudas, K. (2023) · Preprints.org · doi:10.20944/preprints202308.0148.v2 · Cited by 4 · see citing works
How should the advent of large language models affect the practice of science?Preprint
Binz, M., Alaniz, S., Roskies, A. L., Aczél, B., Bergstrom, C. T., Allen, C., Schad, D. J., Wulff, D. U., West, J. D., Zhang, Q., et al. (2023) · doi:10.31219/osf.io/yr9xb · Cited by 8 · see citing works
Large Language Model Reasoning Failures
Song, P., Han, P., & Goodman, N. (2026) · arXiv · doi:10.48550/arxiv.2602.06176 · Cited by 3 · see citing works
Assessing Logical Puzzle Solving in Large Language Models: Insights from a Minesweeper Case Study
Li, Y., Wang, H., & Zhang, C. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.4 · Cited by 5 · see citing works
Self-Rationalization in the Wild: A Large-scale Out-of-Distribution Evaluation on NLI-related tasks
Yang, J., Glockner, M., Rocha, A., & Gurevych, I. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00741 · Cited by 4 · see citing works
Java coding using artificial intelligence
Уандыкова, М., Baitenova, L. M., Мухамеджанова, Г. С., Yeleukulova, A., & Миркасимова, Т. (2024) · Frontiers in Computer Science · doi:10.3389/fcomp.2024.1473870 · Cited by 3 · see citing works
Knowledge Trap in Augmented Intelligence: Human Experts' Overfocus on Details Hurts Task PerformancePreprint
Wang, W., Liu, X., Zhang, X., & Hong, Y. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4395858 · Cited by 3 · see citing works
Joh, H. C., Kim, M.-H., Ko, J. Y., Kim, J. S., & Jue, M. (2023) · Research Square · doi:10.21203/rs.3.rs-3241164/v1 · Cited by 3 · see citing works
Is the Answer in the Text? Challenging ChatGPT with Evidence Retrieval from Instructive Text
Henning, S., Anthonio, T., Zhou, W. L., Adel, H., Mesgar, M., & Friedrich, A. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.949 · Cited by 2 · see citing works
VEglue: Testing Visual Entailment Systems via Object-Aligned Joint Erasing
Chang, Z., Li, M., Wang, J., Li, C., & Wang, Q. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3731244 · Cited by 2 · see citing works
Martins, F. L. B., de Oliveira, A. C. A., de Vasconcelos, D. R., & de Menezes, M. V. (2025) · Revista Brasileira de Informática na Educação · doi:10.5753/rbie.2025.4500 · Cited by 2 · see citing works
Evaluating Large Language Models in Psychological Research: A Guide for ReviewersPreprint
Abdurahman, S., Ziabari, A. S., Moore, A. K., Bartels, D. M., & Dehghani, M. (2024) · doi:10.31234/osf.io/ag7hy · Cited by 2 · see citing works
Evaluating Logical Reasoning Ability of Large Language Models
Chan, E., & Chan, A. C.-F. (2026) · doi:10.1109/icnlp69856.2026.11527900
Toward Enhancing Reasoning Capabilities of LLMs: An Approach via Synthetic Logic Corpus
Morishita, T., Morio, G., Yamaguchi, A., & Sogawa, Y. (2025) · Journal of Natural Language Processing · doi:10.5715/jnlp.32.520 · Cited by 1 · see citing works
Understanding User Prompting Behavior in Generative AI : A Component Analysis
Jin, Z., Meng, G., Wang, X., Wang, J., Liu, C. X., & Zhang, J. (2025) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.1318 · Cited by 1 · see citing works
Characterizing and Verifying Scientific Claims: Qualitative Causal Structure is All You Need
Wu, J., Chao, W., Zhou, X., & Luo, Z. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.828 · Cited by 1 · see citing works
Lievens, J. (2024) · INTED proceedings · doi:10.21125/inted.2024.0394 · Cited by 1 · see citing works
Integrating Language Models with Symbolic Formulas for First-Order Logic Reasoning
Yu, S., Li, L., Wang, Y., & Zeng, D. D. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10446308 · Cited by 1 · see citing works
The Accuracy of Domain Specific and Descriptive Analysis Generated by Large Language Models
Otieno, D. O., Abri, F., Siami‐Namini, S., & Namin, A. S. (2024) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00274 · Cited by 1 · see citing works
Limitations of and Lessons from the Learning of Large Language ModelsPreprint
Oldenburg, R. (2023) · Qeios · doi:10.32388/9fh6ad · Cited by 1 · see citing works
Xu, F., Zhou, T., Nguyen, T., Bao, H., Lin, C., & Du, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4805664 · Cited by 1 · see citing works
PAACE: A Plan-Aware Automated Agent Context Engineering Framework
Yuksel, K. A. (2025) · arXiv · doi:10.48550/arxiv.2512.16970
A Framework for Automated Worker Evaluation Based on Free-Text Responses with No Ground TruthPreprint
Geva, T., Goldstein, A., & Yahav, I. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4288950
- Authors
- Liu, H., Ning, R., Teng, Z., Liu, J., Zhou, Q., & Zhang, Y.
- Affiliations
- Westlake University (CN)
- Zhejiang University (CN)
- Nanyang Technological University (SG)
- Fudan University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2304.03439
- OpenAlex
- W4363671827
- Cited by
- 100
- References
- 0
- Citations by year
- 2026: 4 · 2025: 20 · 2024: 39 · 2023: 36 · 2022: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- logical reasoning, ChatGPT, GPT-4, natural language inference, reading comprehension, benchmark dataset, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
