Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index; 10 of the 16 sources its claims cite were checked for support.

Using cognitive psychology to understand GPT-3

Open access
Binz, Marcel, Schulz, Eric · 2023 · Proceedings of the National Academy of Sciences

Abstract

We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3’s decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3’s behavior is impressive: It solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multiarmed bandit task, and shows signatures of model-based reinforcement learning. Yet, we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed exploration, and that it fails miserably in a causal reasoning task. Taken together, these results enrich our understanding of current large language models and pave the way for future investigations using tools from cognitive psychology to study increasingly capable and opaque artificial agents.

Overview

CoChat Summary

## TL;DR

The paper treats GPT-3 like a participant in cognitive psychology experiments and finds it can solve many classical reasoning tasks reasonably well. But its behavior is brittle to prompt changes and it lacks some human-like features, especially directed exploration and robust causal reasoning.

## Key findings

  • Across 12 vignette-based problems, GPT-3 answered 6 correctly, and all 6 responses were described as human-like.
  • In the decision-making benchmark, only Davinci performed above chance (t(29134) = −16.85, P < .001), but it still did not reach human performance (t(29134) = −11.50, P < .001).
  • In the horizon / information-search task, GPT-3 showed random exploration but not directed exploration: reward difference mattered (β = 0.18 ± 0.01, z = 14.48, P < .001), while the horizon interaction did not (β = −0.02 ± 0.02, z = −1.47, P = .14); human-like directed exploration was not observed (β = −0.15 ± 0.27, z = −0.56, P = .58).
  • In the two-step deliberation task, GPT-3 showed signatures consistent with model-based reinforcement learning: rare-transition reward affected stay/switch behavior (χ2(1, N = 1984) = 36.53, P < .001).
  • On causal reasoning, GPT-3 succeeded in some vignette cases, but in the task-based causal test it failed to use causal structure properly; the authors conclude its causal inference was “neither the correct nor human-like.”

## Methods

  • The authors used the public OpenAI API and primarily the Davinci GPT-3 model, with temperature set to 0 for deterministic responses unless otherwise noted.
  • They ran 12 canonical vignette-based prompts from cognitive psychology and also conducted task-based, procedurally generated experiments: a decision-making benchmark with over 13,000 problems; 3,200 simulations of the horizon task; 200 simulations of the two-step task; and a causal reasoning task.
  • Analyses included choice-probability contrasts, logistic regression, and chi-square tests; for the decision task they also report regret relative to human subjects.

## Limitations

  • The paper states that vignette results are hard to interpret because GPT-3 may have encountered similar or identical scenarios in training data, and because small prompt changes can drastically alter responses.
  • It also states that the vignette tasks can be adversarially modified, causing GPT-3 to respond very differently, which makes the findings brittle.
  • Evident from the text: several results rely on prompt phrasing, cover stories, and API settings (for example, temperature and choice labels), so the findings may not generalize to other prompting setups.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1073/pnas.2218523120

Cites

55 of 55 references verified0 citation errors · 0 not checkable

Statistics

12 tests recomputed · 0 with the reported and recomputed p on opposite sides of .05

Figures

Fig. 1. Vignette-based tasks. (A) Example prompt of a hypothetical scenario, in this case, the famous Linda problem, as submitted to GPT-3. (B) Results. While in 12 out 12 standard vignettes, GPT-3 answers either correctly or makes human-like mistakes, it makes mistakes that are not human-like when given the adversarial vignettes.
Fig. 1. Vignette-based tasks. (A) Example prompt of a hypothetical scenario, in this case, the famous Linda problem, as submitted to GPT-3. (B) Results. While in 12 out 12 standard vignettes, GPT-3 answers either correctly or makes human-like mistakes, it makes mistakes that are not human-like when given the adversarial vignettes.CC BY-NC-ND · PubMed Central
Fig. 2. Decisions from descriptions. (A) Example prompt of a problem provided to GPT-3. (B) Example prompt of a problem provided to GPT-3. (C) Mean regret averaged over all 13,000 problems taken from Peterson et al. (23). Lower regret means better performance. Error bars indicate the SE of the mean. (D) Log-odds ratios of different contrasts used to test for cognitive biases. Positive values indicate that the given bias is present in humans (circle) or GPT-3 (triangle). Human data adapted from Ruggeri et al. (24).
Fig. 2. Decisions from descriptions. (A) Example prompt of a problem provided to GPT-3. (B) Example prompt of a problem provided to GPT-3. (C) Mean regret averaged over all 13,000 problems taken from Peterson et al. (23). Lower regret means better performance. Error bars indicate the SE of the mean. (D) Log-odds ratios of different contrasts used to test for cognitive biases. Positive values indicate that the given bias is present in humans (circle) or GPT-3 (triangle). Human data adapted from Ruggeri et al. (24).CC BY-NC-ND · PubMed Central
Fig. 3. Horizon task. (A) Visual overview of the horizon task paradigm. Each column pair corresponds to one example task. (B) Example prompt for one trial as submitted to GPT-3. (C) Mean regret for GPT-3 and human subjects by horizon condition. Lower regret means better performance. Error bars indicate the SE of the mean. Human data taken from Zaller et al. (29).
Fig. 3. Horizon task. (A) Visual overview of the horizon task paradigm. Each column pair corresponds to one example task. (B) Example prompt for one trial as submitted to GPT-3. (C) Mean regret for GPT-3 and human subjects by horizon condition. Lower regret means better performance. Error bars indicate the SE of the mean. Human data taken from Zaller et al. (29).CC BY-NC-ND · PubMed Central
Fig. 4. Two-step task. (A) Visual overview of the two-step task paradigm. (B) Example prompt of one trial in the canonical two-step task as submitted to GPT-3. (C) Model-free learning in dependency of rewarded and unrewarded as well as common and rare transitions. (D) Model-based learning in dependency of rewarded and unrewarded as well as common and rare transitions. (E) Human behavior in dependency of rewarded and unrewarded as well as common and rare transitions. Human data adapted from Daw et al. (30). (F) GPT-3’s behavior in dependency of rewarded and unrewarded as well as common and rare transitions. Error bars indicate the SE of the mean.
Fig. 4. Two-step task. (A) Visual overview of the two-step task paradigm. (B) Example prompt of one trial in the canonical two-step task as submitted to GPT-3. (C) Model-free learning in dependency of rewarded and unrewarded as well as common and rare transitions. (D) Model-based learning in dependency of rewarded and unrewarded as well as common and rare transitions. (E) Human behavior in dependency of rewarded and unrewarded as well as common and rare transitions. Human data adapted from Daw et al. (30). (F) GPT-3’s behavior in dependency of rewarded and unrewarded as well as common and rare transitions. Error bars indicate the SE of the mean.CC BY-NC-ND · PubMed Central
Fig. 5. Causal reasoning. (A) Example prompt for the causal reasoning task adapted from Waldmann and Hagmayer (31). (B) GPT-3’s responses alongside responses of people and an ideal agent in the common-cause condition. (C) GPT-3’s responses alongside responses of people and an ideal agent in the causal-chain condition.
Fig. 5. Causal reasoning. (A) Example prompt for the causal reasoning task adapted from Waldmann and Hagmayer (31). (B) GPT-3’s responses alongside responses of people and an ideal agent in the common-cause condition. (C) GPT-3’s responses alongside responses of people and an ideal agent in the causal-chain condition.CC BY-NC-ND · PubMed Central
Fig. 6. Prompt variations. (A) Performance for different prompt variations in the decisions from the descriptions paradigm. (B) Performance for different prompt variations in the horizon task. (C) Effect of random exploration for different prompt variations in the horizon task. (D) Effect of directed exploration for different prompt variations in the horizon task. (E) GPT-3’s behavior in dependency of rewarded and unrewarded as well as common and rare transitions for the alien cover story (reproduced from Fig. 4F). (F). GPT-3’s behavior in dependency of rewarded and unrewarded as well as common and rare transitions for the magical carpet cover story. Error bars indicate the SE of the mean.
Fig. 6. Prompt variations. (A) Performance for different prompt variations in the decisions from the descriptions paradigm. (B) Performance for different prompt variations in the horizon task. (C) Effect of random exploration for different prompt variations in the horizon task. (D) Effect of directed exploration for different prompt variations in the horizon task. (E) GPT-3’s behavior in dependency of rewarded and unrewarded as well as common and rare transitions for the alien cover story (reproduced from Fig. 4F). (F). GPT-3’s behavior in dependency of rewarded and unrewarded as well as common and rare transitions for the magical carpet cover story. Error bars indicate the SE of the mean.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

9 Valid · 1 Not valid · 5 Not checked · 69 Own results and interpretations

49 more claims

Create a free account to see all 84 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 9
  • It may state two separate claims as one. 23
  • The quote only partly states this claim. 1

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

10 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

1 of 1

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upt(5550) = -0.75reported p = .23 · recomputed p = .453

    “t(5550)= − 0.75, p = .23”

  • Consistentt(29134) = -11.5reported p < .001 · recomputed p < .001

    “t(29134)= − 11.50, p < .001”

  • Consistentt(5566) = -0.043reported p = .97 · recomputed p = .966

    “t(5566)= − 0.043, p = .97”

  • Consistentt(5550) = -4.07reported p < .001 · recomputed p < .001

    “t(5550)= − 4.07, p < .001”

  • Consistentt(38878) = -5.03reported p < .001 · recomputed p < .001

    “t(38878)= − 5.03, p < .001”

5 of 12 · the rest aren’t loaded

7 more recomputed tests

Create a free account to see every test the Stats check recomputed.

See all tests — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Self-check results appear here once an independent check has confirmed them.

Citations

Cites

Show all 47 references

Research works printed but not linked: 3

  1. M. Chen et al . Evaluating large language models trained on code. arXiv [Preprint] (2021).http://arxiv.org/abs/2107.03374 (Accessed 20 January 2023).
  2. G. Marcus, E. Davis, GPT-3, bloviator: Openai’s language generator has no idea what it’s talking about (Technol, Rev, 2020).
  3. C. R. Jones et al . “Distributional semantics still can’t account for affordances” in Proceedings of the Annual Meeting of the Cognitive Science Society (2022) vol. 44.

Cited by

See all 559

Metadata

Authors
Binz, M., & Schulz, E.
Affiliations
Max Planck Institute for Biological Cybernetics (DE)
Corresponding author
Binz, Marcel
Venue
Proceedings of the National Academy of Sciences
Published
2023-02-02
Volume
120
Issue
6
Pages
e2218523120-e2218523120
ISSN
  • 0027-8424 (ISSN-L)
  • 1091-6490
OpenAlex
W4318919287
Cited by
559
References
47
Citations by year
2026: 146 · 2025: 162 · 2024: 151 · 2023: 92
FWCI
51.80
Type
journal article
Language
en
Open access
Locations
  • Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
  • PsyArXiv (OSF Preprints) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • Volkswagen Foundation (VW98569)
  • Deutsche Forschungsgemeinschaft (EXC2064/1-390727645)
  • Max-Planck-Gesellschaft (-)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
GPT-3, large language models, cognitive psychology, decision-making, information search, deliberation, causal reasoning, model-based reinforcement learning, multi-armed bandits, directed exploration
MeSH terms
Cognitive Psychology, Decision Making, Humans, Learning, Problem Solving, Reinforcement, Psychology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Using cognitive psychology to understand GPT-3. https://app.cochat.ai/papers/10.1073/pnas.2218523120
The paper
Binz, M., & Schulz, E. (2023). Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2218523120

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.