The DOI matches its DataCite record; the references it cites are not checked yet.
Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4
Open accessAbstract
Harnessing logical reasoning ability is a comprehensive natural language understanding endeavor. With the release of Generative Pretrained Transformer 4 (GPT-4), highlighted as "advanced" at reasoning tasks, we are eager to learn the GPT-4 performance on various logical reasoning tasks. This report analyses multiple logical reasoning datasets, with popular benchmarks like LogiQA and ReClor, and newly-released datasets like AR-LSAT. We test the multi-choice reading comprehension and natural language inference tasks with benchmarks requiring logical reasoning. We further construct a logical reasoning out-of-distribution dataset to investigate the robustness of ChatGPT and GPT-4. We also make a performance comparison between ChatGPT and GPT-4. Experiment results show that ChatGPT performs significantly better than the RoBERTa fine-tuning method on most logical reasoning benchmarks. With early access to the GPT-4 API we are able to conduct intense experiments on the GPT-4 model. The results show GPT-4 yields even higher performance on most logical reasoning datasets. Among benchmarks, ChatGPT and GPT-4 do relatively well on well-known datasets like LogiQA and ReClor. However, the performance drops significantly when handling newly released and out-of-distribution datasets. Logical reasoning remains challenging for ChatGPT and GPT-4, especially on out-of-distribution and natural language inference datasets. We release the prompt-style logical reasoning datasets as a benchmark suite and name it LogiEval.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LLMsdemonstrate robust arithmetic reasoning capabilities [159] and excel in logical reasoning [124].”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [124]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Despite the advances of LLMs, they still struggle with complex logical reasoning problems (Liu et al., 2023b).”
From Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning (2023) · cites this paper as Liu et al., 2023b
In the citing paper
“However, both models face challenges when handling new and out-of-distribution data.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [124]
In the citing paper
“In logical reasoning, Liu et al. [124] indicated that ChatGPT and GPT-4 outperform traditional fine-tuning methods on most benchmarks, demonstrating their superiority in logical reasoning.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [124]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The publisher hasn’t deposited this paper’s reference list.
Research works printed but not linked: 16
- Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc- Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learn- ers. CoRR, abs/2005.14165. · arXiv
- Maxwell John Cresswell. 1973. Logics and languages (1st ed.). Routledge.
- Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Proc. of ICML.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language under- standing. CoRR.
- Pratik Joshi, Somak Aditya, Aalok Sathe, and Mono- jit Choudhury. 2020. Taxinli: Taking a ride up the NLU hill. CoRR.
- Hanmeng Liu, Leyang Cui, Jian Liu, and Yue Zhang. 2020. Natural language inference in context -inves- tigating contextual reasoning over long texts. CoRR.
- Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov. 2019. Roberta: A robustly optimized bert pretrain- ing approach. arXiv.
- Bill MacCartney and Christopher D. Manning. 2007a. Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entail- ment and Paraphrasing.
- Bill MacCartney and Christopher D Manning. 2007b. Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entail- ment and Paraphrasing. OpenAI. 2023. Gpt-4 technical report.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text trans- former. CoRR.
- Swarnadeep Saha, Yixin Nie, and Mohit Bansal. 2020. Conjnli: Natural language inference over conjunc- tive sentences. CoRR.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR.
- Alex Wang, Amanpreet Singh, Julian Michael, Fe- lix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Pro- ceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Net- works for NLP.
- Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. 2022. From lsat: The progress and challenges of complex reasoning. IEEE/ACM Transactions on Au- dio, Speech, and Language Processing.
- Adina Williams, Nikita Nangia, and Samuel Bowman. 2018a. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proc. of NAACL.
- Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Ken- taro Inui, Satoshi Sekine, , and Johan Bos. 2019a. Help: A dataset for identifying shortcomings of neu- ral models in monotonicity reasoning. In Proceed- ings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM2019).
Cited by
Sorted by citations
Items 101–111 of 111
Rayadurgam, V. C., & Afshan, N. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4474326
Lu, M., Gao, F., Tang, X., & Chen, L. (2023) · ChemRxiv · doi:10.26434/chemrxiv-2023-gxmxr
Findings of the 1st Shared Task on Multi-lingual Multi-task Information Retrieval at MRL 2023
Tinner, F., Adelani, D. I., Emezue, C. C., Hajili, M., Goldman, O., Adilazuarda, M. F., Kautsar, M. D. A., Mirsaidova, A., Kural, M., Klenner, D. M. M., et al. (2023) · Workshop on Multilingual Representation Learning (MRL) · doi:10.18653/v1/2023.mrl-1.24
Multi-dimensional Knowledge Graph Embedding with Entity Type Information
Park, J.-H., & Gim, J. (2023) · The Journal of Korean Institute of Information Technology · doi:10.14801/jkiit.2023.21.11.63
AI REACTION TO WORLD ENGLISHES: A STUDY OF FICTION
Lebedeva, E. S. (2024) · The Humanities and Social Studies in the Far East · doi:10.31079/1992-2868-2024-21-1-18-22
Guan, I. Y. N., Zhang, G., Liu, X., Zhao, E., & Wu, J. (2024) · International Conference on Electrical Engineering, Control and Robotics (EECR) · doi:10.1109/eecr60807.2024.10607324
Complementary advantages exhibited by ChatGPT and human readers in text reading inference
Zhou, T., Zhang, Y., Cao, S., Li, Y., Wang, T., & Yang, Y. (2025) · Current Psychology · doi:10.1007/s12144-025-08010-8
Self-citationAhn, S., Moon, B., Park, S., Kim, S., & Kim, K. (2025) · The Journal of Korean Institute of Information Technology · doi:10.14801/jkiit.2025.23.6.31
Jung, Y., & Govindarazan, K. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3798763
Transferring Strategic Thinking From Large to Small Language Models for Multi-Agent Systems
Lorè, N., Ilami, S., & Heydari, B. (2026) · ASME Open Journal of Engineering · doi:10.1115/1.4072118
Signs of machine generation in student works on higher mathematics
Kostrova, Y. S. (2026) · Pedagogy Theory & Practice · doi:10.30853/ped20260166
Metadata
- Authors
- Liu, H., Ning, R., Teng, Z., Liu, J., Zhou, Q., & Zhang, Y.
- Affiliations
- Westlake University (CN)
- Zhejiang University (CN)
- Nanyang Technological University (SG)
- Fudan University (CN)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2304.03439
- OpenAlex
- W4363671827
- Cited by
- 100
- References
- 0
- Citations by year
- 2026: 4 · 2025: 20 · 2024: 39 · 2023: 36
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- logical reasoning, ChatGPT, GPT-4, natural language inference, reading comprehension, benchmark dataset, large language models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
