Mind meets machine: Unravelling GPT-4’s cognitive psychology
Open accessThe DOI matches its Crossref record; 19 of the 19 references it cites were found in a registry or an index.
Abstract
Cognitive psychology delves on understanding perception, attention, memory, language, problem-solving, decision-making, and reasoning. Large Language Models (LLMs) are emerging as potent tools increasingly capable of performing human-level tasks. The recent development in the form of Generative Pre-trained Transformer 4 (GPT-4) and its demonstrated success in tasks complex to humans exam and complex problems has led to an increased confidence in the LLMs to become perfect instruments of intelligence. Although GPT-4 report has shown performance on some cognitive psychology tasks, a comprehensive assessment of GPT-4, via the existing well-established datasets is required. In this study, we focus on the evaluation of GPT-4’s performance on a set of cognitive psychology datasets such as CommonsenseQA, SuperGLUE, MATH and HANS. In doing so, we understand how GPT-4 processes and integrates cognitive psychology with contextual information, providing insight into the underlying cognitive processes that enable its ability to generate the responses. We show that GPT-4 exhibits a high level of accuracy in cognitive psychology tasks relative to the prior state-of-the-art models. Our results strengthen the already available assessments and confidence on GPT-4’s cognitive psychology abilities. It has significant potential to revolutionize the field of Artificial Intelligence (AI), by enabling machines to bridge the gap between human and machine reasoning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference
McCoy, R. T., Pavlick, E., & Linzen, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1334 · Cited by 944
The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation
Brundage, M., Avin, S., Clark, J., Toner, H., Eckersley, P., Garfinkel, B., Dafoe, A., Scharre, P., Zeitzoff, T., Filar, B., et al. (2018) · arXiv · doi:10.48550/arxiv.1802.07228 · Cited by 849
The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation
Brundage, M., Avin, S., Clark, J., Toner, H., Eckersley, P., Garfinkel, B., Dafoe, A., Scharre, P., Zeitzoff, T., Filar, B., et al. (2018) · Apollo - University of Cambridge Repository · doi:10.17863/cam.22520 · Cited by 849
HellaSwag: Can a Machine Really Finish Your Sentence?
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., & Choi, Y. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1472 · Cited by 830
Using cognitive psychology to understand GPT-3
Binz, M., & Schulz, E. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2218523120 · Cited by 559
Harrer, S. (2023) · eBioMedicine · doi:10.1016/j.ebiom.2023.104512 · Cited by 480
WinoGrande: an adversarial winograd schema challenge at scale
Sakaguchi, K., Bras, R. L., Bhagavatula, C., & Choi, Y. (2021) · Communications of the ACM · doi:10.1145/3474381 · Cited by 356
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2024) · Patterns · doi:10.1016/j.patter.2024.100943 · Cited by 343
Can large language models reason about medical questions?
Liévin, V., Hother, C. E., Motzfeldt, A. G., & Winther, O. (2022) · arXiv · doi:10.48550/arxiv.2207.08143 · Cited by 343
Measuring Mathematical Problem Solving With the MATH Dataset
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., & Steinhardt, J. (2021) · arXiv · doi:10.48550/arxiv.2103.03874 · Cited by 306
Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies
Aher, G., Arriaga, R. I., & Kalai, A. T. (2022) · arXiv · doi:10.48550/arxiv.2208.10264 · Cited by 161
Singh, H., Tyagi, S., Kumar, P., Gill, S. S., & Buyya, R. (2021) · Simulation Modelling Practice and Theory · doi:10.1016/j.simpat.2021.102353 · Cited by 104
Self-citationProbing the psychology of AI models
Shiffrin, R., & Mitchell, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2300963120 · Cited by 70
Bansal, K., Mittal, K., Ahuja, G., Singh, A., & Gill, S. S. (2020) · Internet Technology Letters · doi:10.1002/itl2.156 · Cited by 70
Self-citationThe journey of qos-aware autonomic cloud computing
Singh, S., Chana, I., & Singh, M. (2017) · IT Professional · doi:10.1109/mitp.2017.26 · Cited by 43
Self-citationText and Patterns: For Effective Chain of Thought, It Takes Two to Tango
Madaan, A., & Yazdanbakhsh, A. (2022) · arXiv · doi:10.48550/arxiv.2209.07686 · Cited by 24
Cited by
The Rise and Potential of Large Language Model Based Agents: A Survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2023) · arXiv · doi:10.48550/arxiv.2309.07864 · Cited by 972
The rise and potential of large language model based agents: a survey
Xi, Z., Chen, W., Guo, X., He, W., Ding, Y., Hong, B., Zhang, M., Wang, J., Jin, S., Zhou, E., et al. (2025) · Science China Information Sciences · doi:10.1007/s11432-024-4222-0 · Cited by 970
Transformative effects of ChatGPT on modern education: Emerging Era of AI Chatbots
Gill, S. S., Xu, M., Patros, P., Wu, H., Kaur, R., Kaur, K., Fuller, S., Singh, M., Arora, P., Parlikad, A. K., et al. (2024) · Internet of Things and Cyber-Physical Systems · doi:10.1016/j.iotcps.2023.06.002 · Cited by 488
Self-citationLarge language models empowered agent-based modeling and simulation: a survey and perspectives
Gao, C., Lan, X., Li, N., Yuan, Y., Ding, J., Zhou, Z., Xu, F., & Li, Y. (2024) · Humanities and Social Sciences Communications · doi:10.1057/s41599-024-03611-3 · Cited by 294
LLMs for knowledge graph construction and reasoning: recent capabilities and future opportunities
Zhu, Y., Wang, X., Chen, J., Qiao, S., Ou, Y., Yao, Y., Deng, S., Chen, H., & Zhang, N. (2024) · World Wide Web · doi:10.1007/s11280-024-01297-w · Cited by 294
Metadata
- Authors
- Dhingra, S., Singh, M., S.B., V., Malviya, N., & Gill, S. S.
- Venue
- BenchCouncil Transactions on Benchmarks, Standards and Evaluations
- Publisher
- Elsevier BV
- Published
- 2023-09
- Volume
- 3
- Issue
- 3
- Pages
- 100139
- ISSN
- 2772-4859 (ISSN-L)
- OpenAlex
- W4386384991
- Cited by
- 42
- References
- 19
- FWCI
- 3.69
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.