Human-like problem-solving abilities in large language models using ChatGPT
Open accessAbstract
Backgrounds: The field of Artificial Intelligence (AI) has seen a major shift in recent years due to the development of new Machine Learning (ML) models such as Generative Pre-trained Transformer (GPT). GPT has achieved previously unheard-of levels of accuracy in most computerized language processing tasks and their chat-based variations. Aim: The aim of this study was to investigate the problem-solving abilities of ChatGPT using two sets of verbal insight problems, with a known performance level established by a sample of human participants. Materials and methods: A total of 30 problems labeled as “ practice problems ” and “ transfer problems ” were administered to ChatGPT. ChatGPT's answers received a score of “0” for each incorrectly answered problem and a score of “1” for each correct response. The highest possible score for both the practice and transfer problems was 15 out of 15. The solution rate for each problem (based on a sample of 20 subjects) was used to assess and compare the performance of ChatGPT with that of human subjects. Results: The study highlighted that ChatGPT can be trained in out-of-the-box thinking and demonstrated potential in solving verbal insight problems. The global performance of ChatGPT equalled the most probable outcome for the human sample in both practice problems and transfer problems as well as upon their combination. Additionally, ChatGPT answer combinations were among the 5% of most probable outcomes for the human sample both when considering practice problems and pooled problem sets. These findings demonstrate that ChatGPT performance on both set of problems was in line with the mean rate of success of human subjects, indicating that it performed reasonably well. Conclusions: The use of transformer architecture and self-attention in ChatGPT may have helped to prioritize inputs while predicting, contributing to its potential in verbal insight problem-solving. ChatGPT has shown potential in solving insight problems, thus highlighting the importance of incorporating AI into psychological research. However, it is acknowledged that there are still open challenges. Indeed, further research is required to fully understand AI's capabilities and limitations in verbal problem-solving.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ChatGPT's performance on verbal insight problems is comparable to that of human participants.
“Orrù et al. [147] demonstrated the potential of ChatGPT for solving verbal insight problems, as ChatGPT's performance was comparable to that of human participants.”
From A Survey on Evaluation of Large Language Models (2024) · cites this paper as [147]
Experiments are particularly effective in isolating variables and testing specific interventions, such as the inclusion of AI tools.
“Experiments are particularly effective in isolating variables and testing specific interventions, such as the inclusion of AI tools [38].”
From Exploring the Impact of AI Tools on Cognitive Skills: A Comparative Analysis (2025) · cites this paper as [38]
The Introduction describes an evaluation of ChatGPT’s verbal problem-solving, and Materials and Methods describes the tasks administered to it. Neither passage addresses whether experiments are particularly effective at isolating variables or testing interventions such as adding AI tools. Searches for those methodological claims found no supporting passage in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 29 · Cited by 170
Research works printed but not linked: 5
- Newell A. Simon H. A. (1972). Human Problem Solving (Vol. 104, No. 9). Englewood Cliffs, NJ: Prentice-Hall.
- Parikh A. Täckström O. Das D. Uszkoreit J. (2016). “A decomposable attention model,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (Austin, TX: Association of Computational Linguistics), 2249– 2255.
- Polya G. (2004). How to Solve It: A New Aspect of Mathematical Method (No. 246). Princeton university press.
- Vaswani A. Shazeer N. Parmar N. Uszkoreit J. Jones L. Gomez A. N.. (2017). Attention is all you need. Adv. Neural Inform. Process. Syst. 30, 1– 11.
- Wallas G. (1926). The Art of Thought (Vol. 10). Harcourt, Brace.
- Authors
- Orrù, G., Piarulli, A., Conversano, C., & Gemignani, A.
- Affiliations
- University of Pisa (IT)
- Corresponding author
- Orrù, Graziella
- Venue
- Frontiers in Artificial Intelligence
- Publisher
- Frontiers Media SA
- Published
- 2023-05-24
- Volume
- 6
- Pages
- 1199350-1199350
- ISSN
- 2624-8212 (ISSN-L)
- PMC
- PMC10244637
- OpenAlex
- W4377942506
- Cited by
- 170
- References
- 29
- Citations by year
- 2026: 27 · 2025: 67 · 2024: 62 · 2023: 14
- FWCI
- 15.96
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Frontiers in Artificial Intelligence (publishedVersion, cc-by)
- Preprints.org (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- CINECA IRIS Institutial research information system (University of Pisa) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- ChatGPT, large language models, problem solving, transfer problem, human-AI comparison, out-of-the-box thinking, self-attention, transformer models
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
