Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index; 7 of the 85 sources its claims cite were checked for support.

Emergent analogical reasoning in large language models

Open access
Webb, Taylor, Holyoak, Keith J., Lu, Hongjing · 2023 · Nature Human Behaviour

Abstract

The recent advent of large language models has reinvigorated debate over whether human cognitive capacities might emerge in such generic models given sufficient training data. Of particular interest is the ability of these models to reason about novel problems zero-shot, without any direct training. In human cognition, this capacity is closely tied to an ability to reason by analogy. Here we performed a direct comparison between human reasoners and a large language model (the text-davinci-003 variant of Generative Pre-trained Transformer (GPT)-3) on a range of analogical tasks, including a non-visual matrix reasoning task based on the rule structure of Raven's Standard Progressive Matrices. We found that GPT-3 displayed a surprisingly strong capacity for abstract pattern induction, matching or even surpassing human capabilities in most settings; preliminary tests of GPT-4 indicated even better performance. Our results indicate that large language models such as GPT-3 have acquired an emergent ability to find zero-shot solutions to a broad range of analogy problems.

Overview

CoChat Summary

TL;DR

GPT-3 (text-davinci-003) showed a surprisingly strong zero-shot ability to solve text-based analogical reasoning tasks, matching or surpassing human participants on matrix reasoning, letter-string analogies, and several verbal analogy datasets. However, humans clearly outperformed GPT-3 on story analogies requiring sensitivity to higher-order causal relations, and GPT-3’s analogical problem-solving was constrained by lack of physical understanding and long-term memory.

Key findings

  • On the novel text-based Digit Matrices task modeled on Raven’s SPM, GPT-3 surpassed average human performance for generative answers (OR = 1.88, p = 0.005) and multiple-choice answers (OR = 6.27, p = 2.3 × 10⁻⁸), while showing human-like error patterns across problem subtypes (r(30) = 0.39, p = 0.027).
  • On letter-string analogies, GPT-3 outperformed humans overall (OR = 1.76, p = 6.3 × 10⁻⁵), especially on zero-generalization problems, and matched human error patterns across transformation and generalization types (r(39) = 0.7, p = 3.6 × 10⁻⁷).
  • On four-term verbal analogies, GPT-3 performed as well or better than humans on the UCLA VAT and Sternberg & Nigro datasets, surpassed estimated SAT student performance, and was in the same range though numerically weaker on Jones et al.; it also showed human-like sensitivity to semantic distance (OR = 3.24, p = 0.0165).
  • On story analogies, humans outperformed GPT-3 (OR = 0.37, p = 0.0003); GPT-3 was significant in the near analogy condition (p = 0.0039) but not in the far analogy condition (p = 0.065), whereas humans were significant in both. Preliminary GPT-4 tests showed stronger story-analogy performance.

Methods

  • GPT-3 (text-davinci-003) was queried through the OpenAI API with temperature 0 and evaluated zero-shot on four task domains: text-based Digit Matrices, letter-string analogies, four-term verbal analogies, and story analogies; additional qualitative tests examined analogical problem-solving and preliminary GPT-4 performance.
  • Human comparisons used UCLA undergraduates for Digit Matrices (N = 40), letter-string analogies (N = 57), UCLA VAT (N = 57), and story analogies (N = 54), plus published human data for other verbal analogy datasets (N = 20, N = 241, and SAT estimates).
  • Analyses used logistic regression at the trial level, correlation analyses across problem subtypes, binomial tests for GPT-3 story analogies, and one-sample t-tests for human story analogies.

Limitations

  • Stated by the paper: GPT-3 lacks long-term memory for specific episodes, cannot retrieve prior analogs once its context is cleared, lacks physical understanding of the world, and is limited in evaluating causal relations in cross-domain far analogies; it also cannot be considered a model of the acquisition or evolution of human analogical reasoning.
  • Stated by the paper: the Digit Matrices are text-based and do not test visual processing from pixel-level inputs, and GPT-3’s performance on letter-string analogies was sensitive to prompt and formatting.
  • Evident from the text: human samples were mostly UCLA undergraduates, GPT-4 testing was preliminary and limited by API access, and some story-analogy and problem-solving results are qualitative or based on small numbers of trials.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41562-023-01659-w

Cites

61 of 61 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

1 Valid · 9 Not valid · 43 Not checked · 111 Own results and interpretations

78 more claims · 1 with their cited sources checked

Create a free account to see all 164 claims and to verify the 78 references still open.

See all claims — free account
  • Probably not a substantial claim. 42
  • It may state two separate claims as one. 47
  • The quote only partly states this claim. 3

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 55 references

Research works printed but not linked: 11

  1. Snow, R. E., Kyllonen, P. C. & Marshalek, B. et al. The topography of ability and learning correlations. Adv. Psychol. Hum. Intell. 2, 103 (1984).
  2. Barrett, D., Hill, F., Santoro, A., Morcos, A. & Lillicrap, T. in International Conference on Machine Learning (eds Dy, J. & Krause, A.) 511–520 (PMLR, 2018).
  3. Zhang, C., Gao, F., Jia, B., Zhu, Y. & Zhu, S.-C. in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition (eds Gupta, A. et al.) 5317–5327 (IEEE, 2019).
  4. Hill, F., Santoro, A., Barrett, D. G. T., Morcos, A. S. & Lillicrap, T. P. Learning to make analogies by contrasting abstract relational structure. in 7th International Conference on Learning Representations, ICLR https://openreview.net/forum?id=SylLYsCcFm (2019).
  5. Subhra Mondal, S., Webb, T. W. & Cohen, J. D. Learning to reason over visual objects. in 11th International Conference on Learning Representations, ICLR https://openreview.net/forum?id=uR6x8Be7o_M (2023).
  6. Hofstadter, D. R. & Mitchell, M. in Advances in Connectionist and Neural Computation Theory Vol. 2 (eds Holyoak, K. J. & Barnden, J. A.) 31–112 (Ablex, 1994).
  7. Turney, P. D., Littman, M. L., Bigham, J. & Shnayder, V. in Proc. International Conference on Recent Advances in Natural Language Processing (eds Angelova, G. at al.) 482–489 (RANLP, 2003).
  8. Kroger, J. K., Holyoak, K. J. & Hummel, J. E. Varieties of sameness: the impact of relational complexity on perceptual comparisons. Cogn. Sci. 28, 335–358 (2004).
  9. Mitchell, M. Analogy-Making as Perception: A Computer Model (MIT Press, 1993).
  10. Webb, T. W. et al. in International Conference on Machine Learning (eds Daumé, H. III & Singh, A.) 10136–10146 (PMLR, 2020).
  11. Webb, T. W., Sinha, I. & Cohen, J. D. Emergent symbols through binding in external memory. in 9th International Conference on Learning Representations, ICLR https://openreview.net/forum?id=LSFCEb3GYU7 (2021).

Cited by

See all 415

Metadata

Authors
Webb, T., Holyoak, K. J., & Lu, H.
Affiliations
University of California, Los Angeles (US)
Corresponding author
Webb, Taylor
Venue
Nature Human Behaviour
Published
2023-07-31
Volume
7
Issue
9
Pages
1526-1541
ISSN
2397-3374 (ISSN-L)
OpenAlex
W4385430086
Cited by
404
References
55
Citations by year
2026: 81 · 2025: 158 · 2024: 117 · 2023: 48
FWCI
38.08
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Human Behaviour (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
NSF | Directorate for Computer & Information Science & Engineering | Division of Information and Intelligent Systems (IIS-195644)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
  • Physical Sciences › Computer Science › Artificial Intelligence › AI-based Problem Solving and Planning
Keywords
analogical reasoning, large language models, GPT-3, zero-shot reasoning, emergent capabilities, human-computer comparison, Raven's Standard Progressive Matrices, matrix reasoning, GPT-4
MeSH terms
Cognition, Humans, Language, Problem Solving

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-05, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-05 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash and gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Emergent analogical reasoning in large language models. https://app.cochat.ai/papers/10.1038/s41562-023-01659-w
The paper
Webb, T., Holyoak, K. J., & Lu, H. (2023). Emergent analogical reasoning in large language models. Nature Human Behaviour. https://doi.org/10.1038/s41562-023-01659-w

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.