The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index; 10 of the 16 sources its claims cite were checked for support.
Using cognitive psychology to understand GPT-3
Open accessAbstract
We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3’s decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3’s behavior is impressive: It solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multiarmed bandit task, and shows signatures of model-based reinforcement learning. Yet, we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed exploration, and that it fails miserably in a causal reasoning task. Taken together, these results enrich our understanding of current large language models and pave the way for future investigations using tools from cognitive psychology to study increasingly capable and opaque artificial agents.
Overview
CoChat Summary
## TL;DR
The paper treats GPT-3 like a participant in cognitive psychology experiments and finds it can solve many classical reasoning tasks reasonably well. But its behavior is brittle to prompt changes and it lacks some human-like features, especially directed exploration and robust causal reasoning.
## Key findings
- Across 12 vignette-based problems, GPT-3 answered 6 correctly, and all 6 responses were described as human-like.
- In the decision-making benchmark, only Davinci performed above chance (t(29134) = −16.85, P < .001), but it still did not reach human performance (t(29134) = −11.50, P < .001).
- In the horizon / information-search task, GPT-3 showed random exploration but not directed exploration: reward difference mattered (β = 0.18 ± 0.01, z = 14.48, P < .001), while the horizon interaction did not (β = −0.02 ± 0.02, z = −1.47, P = .14); human-like directed exploration was not observed (β = −0.15 ± 0.27, z = −0.56, P = .58).
- In the two-step deliberation task, GPT-3 showed signatures consistent with model-based reinforcement learning: rare-transition reward affected stay/switch behavior (χ2(1, N = 1984) = 36.53, P < .001).
- On causal reasoning, GPT-3 succeeded in some vignette cases, but in the task-based causal test it failed to use causal structure properly; the authors conclude its causal inference was “neither the correct nor human-like.”
## Methods
- The authors used the public OpenAI API and primarily the Davinci GPT-3 model, with temperature set to 0 for deterministic responses unless otherwise noted.
- They ran 12 canonical vignette-based prompts from cognitive psychology and also conducted task-based, procedurally generated experiments: a decision-making benchmark with over 13,000 problems; 3,200 simulations of the horizon task; 200 simulations of the two-step task; and a causal reasoning task.
- Analyses included choice-probability contrasts, logistic regression, and chi-square tests; for the decision task they also report regret relative to human subjects.
## Limitations
- The paper states that vignette results are hard to interpret because GPT-3 may have encountered similar or identical scenarios in training data, and because small prompt changes can drastically alter responses.
- It also states that the vignette tasks can be adversarially modified, causing GPT-3 to respond very differently, which makes the findings brittle.
- Evident from the text: several results rely on prompt phrasing, cover stories, and API settings (for example, temperature and choice labels), so the findings may not generalize to other prompting setups.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures






Verification
Claims the paper makes
9 Valid · 1 Not valid · 5 Not checked · 69 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“For all three items of the CRT, GPT-3 responded with the intuitive but incorrect answer, as has been observed in earlier work (15).”
§ Using cognitive psychology to understand GPT-3 > Results > Vignette-Based Investigations.
What the sources say
- Partly supportedcites [15] sourceExperimental — accuracy not yet measured
The Introduction (p. 2) shows GPT-3 giving the intuitive but incorrect answer to the bat-and-ball CRT item. Appendix A (p. 14) discusses additional CRT questions but says GPT-3 makes the same mistakes as humans “often (but not always)”; the text does not establish that it gave that answer on all three original items. (LLM, conf=0.83)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“While humans show such an effect (27), we did not find it in GPT-3 (β = −0.15 ± 0.27, z = −0.56, P = .58), which demonstrates that the model did not employ directed exploration.”
§ Using cognitive psychology to understand GPT-3 > Results > Task-Based Investigations.
What the sources say
- Supportedcites [27] source
The Results report that increasing the horizon from 1 to 6 increased the probability of choosing the more informative option in the unequal-information condition. The model fit also found a significant increase in the information bonus, consistent with the positive horizon effect on directed exploration attributed to humans. (LLM, conf=0.93)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“First, the latest generation of language models is already able to perform above the human level in the majority of tasks from standard benchmark datasets (11, 12), making purely performance-based evaluation less meaningful as time progresses.”
§ Using cognitive psychology to understand GPT-3 > A Cognitive Psychology View on GPT-3.
What the sources say
- Partly supportedcites [11] sourceExperimental — accuracy not yet measured
The Introduction reports that PaLM exceeded the average human-rater score on 65% of BIG-Bench tasks, supporting the majority-of-tasks point for that benchmark. It does not establish the claim across standard benchmark datasets generally. Section 8.1 also cautions that the human-rater scores are not fully representative of human performance. (LLM, conf=0.87)
- Partly supportedcites [12] sourceExperimental — accuracy not yet measured
Table 1 (Introduction, p. 2) shows Flan-PaLM exceeding the average human rater’s aggregate MMLU score, but not the average human expert’s. The paper does not establish that models exceed human performance on a majority of benchmark tasks; its Evaluation protocol (p. 5) describes the selected challenging benchmarks as ones on which current models remain well below expert human raters. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“People tend to solve this task using a combination of model-free and model-based learning (30) as shown in Fig. 4E.”
§ Using cognitive psychology to understand GPT-3 > Results > Task-Based Investigations.
What the sources say
- Supportedcites [30] source
The Results > Behavior section says the two-step task exercised both model-free and model-based learning strategies, with individual variation in how much subjects used each. It also reports that a hybrid model combining the two fit choices better than either strategy alone. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“It is well-known that large language models can be highly sensitive to specific prompts (34).”
§ Using cognitive psychology to understand GPT-3 > Results > Task-Based Investigations.
What the sources say
- Supportedcites [34] source
The abstract (p. 1) says that prompt templates with different wording choices lead to significant accuracy differences. Section 2.1 (p. 2) likewise states that downstream performance is closely tied to prompt wording, supporting the claim that large language models can be highly sensitive to specific prompts. (LLM, conf=0.94)
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
49 more claims
Create a free account to see all 84 claims and every source verdict.
- Probably not a substantial claim. 9
- It may state two separate claims as one. 23
- The quote only partly states this claim. 1
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“cognitive psychology of LLM treats LLM as an agent whose input-output correlation between prompts (stimulus) and generations (responses) can be probed under controlled conditions to infer the agent’s internal representations.”
From Understanding large language models demands distinguishing human projection from machine cognition (2026) · cites this paper as [36]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These models are trained on large amounts of text, for example, obtained from the internet, and can answer questions, provide summaries or translations and create stories or poems (Fig. 1a)3,4.”
From The future landscape of large language models in medicine (2023) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“as misconceived prompts may result in biases or misinformation with potentially serious consequences4.”
From The future landscape of large language models in medicine (2023) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These results add to the growing body of work characterizing the emergent capabilities of LLMs, 24-28 and suggest that the most sophisticated LLMs may already possess an emergent capacity to reason by analogy.”
From Emergent analogical reasoning in large language models (2023) · cites this paper as [28]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In contrast, the task-based cognitive approaches discussed earlier in this paper36,38,40–43 do not commit this error. They use established experimental paradigms to empirically test whether human and LLM cognition converge or diverge on specific dimensions, without presupposing cognitive equivalence as a starting point.”
From Understanding large language models demands distinguishing human projection from machine cognition (2026) · cites this paper as [36]
10 more claims about this paper
Create a free account to see every claim other papers make about this one.
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“t(5550)= − 0.75, P = .23”
§ Using cognitive psychology to understand GPT-3 > Results > Task-Based Investigations.
Reported p = .23 · recomputed p = .453
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upt(5550) = -0.75reported p = .23 · recomputed p = .453
“t(5550)= − 0.75, p = .23”
- Consistentt(29134) = -11.5reported p < .001 · recomputed p < .001
“t(29134)= − 11.50, p < .001”
- Consistentt(5566) = -0.043reported p = .97 · recomputed p = .966
“t(5566)= − 0.043, p = .97”
- Consistentt(5550) = -4.07reported p < .001 · recomputed p < .001
“t(5550)= − 4.07, p < .001”
- Consistentt(38878) = -5.03reported p < .001 · recomputed p < .001
“t(38878)= − 5.03, p < .001”
7 more recomputed tests
Create a free account to see every test the Stats check recomputed.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Self-check results appear here once an independent check has confirmed them.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Cognitive Reflection and Decision Making
Frederick, S. (2005) · Journal of Economic Perspectives · doi:10.1257/089533005775196732 · Cited by 5,256
Subjective probability: A judgment of representativeness
Kahneman, D., & Tversky, A. (1972) · Cognitive Psychology · doi:10.1016/0010-0285(72)90016-3 · Cited by 4,851 · cited 1×
“Finally, we asked GPT-3 to provide an answer to the “hospital problem” 17 (Hospital, see SI Appendix), in which participants are asked which of two hospitals, a smaller or a larger one, is more likely to report more days on which more than 60% of all born children were boys.”
Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment.
Tversky, A., & Kahneman, D. (1983) · Psychological Review · doi:10.1037/0033-295x.90.4.293 · Cited by 3,991 · cited 1×
“We began our investigations of GPT-3’s decision-making by prompting the canonical “Linda problem” 15 (Linda, see Figure 1A).”
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Research works printed but not linked: 3
- M. Chen et al . Evaluating large language models trained on code. arXiv [Preprint] (2021).http://arxiv.org/abs/2107.03374 (Accessed 20 January 2023).
- G. Marcus, E. Davis, GPT-3, bloviator: Openai’s language generator has no idea what it’s talking about (Technol, Rev, 2020).
- C. R. Jones et al . “Distributional semantics still can’t account for affordances” in Proceedings of the Annual Meeting of the Cognitive Science Society (2022) vol. 44.
Cited by
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187
“For example, starting with only a single user-specified notion that one agent wants to throw a Valentine’s Day party, the agents autonomously spread invitations to the party over the next two”
BackgroundMethodChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · arXiv · doi:10.48550/arxiv.2303.15056 · Cited by 1,207
ChatGPT outperforms crowd workers for text-annotation tasks
Gilardi, F., Alizadeh, M., & Kubli, M. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2305016120 · Cited by 1,197
“…very well for a wide range of purposes, including ideological scaling (Wu et al., 2023), the classification of legislative proposals (Nay, 2023), the resolution of cognitive psychology tasks, (Binz and Schulz, 2023), and the simulation of human samples for survey research (Argyle et al., 2023).”
BackgroundThe future landscape of large language models in medicine
Clusmann, J., Kolbinger, F. R., Muti, H. S., Carrero, Z. I., Eckardt, J.-N., Laleh, N. G., Löffler, C. M. L., Schwarzkopf, S.-C., Unger, M., Veldhuizen, G. P., et al. (2023) · Communications Medicine · doi:10.1038/s43856-023-00370-1 · Cited by 1,045
“It is particularly important students learn appropriate prompt engineering, i.e., appropriate and precise phrasing of an appropriate input to achieve the desired output 62 , as misconceived prompts may result in biases or misinformation with potentially serious consequences 4 .”
Background
Metadata
- Authors
- Binz, M., & Schulz, E.
- Affiliations
- Max Planck Institute for Biological Cybernetics (DE)
- Corresponding author
- Binz, Marcel
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2023-02-02
- Volume
- 120
- Issue
- 6
- Pages
- e2218523120-e2218523120
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC9963545
- OpenAlex
- W4318919287
- Cited by
- 559
- References
- 47
- Citations by year
- 2026: 146 · 2025: 162 · 2024: 151 · 2023: 92
- FWCI
- 51.80
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- PsyArXiv (OSF Preprints) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Volkswagen Foundation (VW98569)
- Deutsche Forschungsgemeinschaft (EXC2064/1-390727645)
- Max-Planck-Gesellschaft (-)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- GPT-3, large language models, cognitive psychology, decision-making, information search, deliberation, causal reasoning, model-based reinforcement learning, multi-armed bandits, directed exploration
- MeSH terms
- Cognitive Psychology, Decision Making, Humans, Learning, Problem Solving, Reinforcement, Psychology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Using cognitive psychology to understand GPT-3. https://app.cochat.ai/papers/10.1073/pnas.2218523120
- The paper
- Binz, M., & Schulz, E. (2023). Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences. https://doi.org/10.1073/pnas.2218523120
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.