The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Testing theory of mind in large language models and humans
Open accessPublished version of a preprint
Abstract
At the core of what defines us as humans is the concept of theory of mind: the ability to track other people’s mental states. The recent development of large language models (LLMs) such as ChatGPT has led to intense debate about the possibility that these models exhibit behaviour that is indistinguishable from human behaviour in theory of mind tasks. Here we compare human and LLM performance on a comprehensive battery of measurements that aim to measure different theory of mind abilities, from understanding false beliefs to interpreting indirect requests and recognizing irony and faux pas. We tested two families of LLMs (GPT and LLaMA2) repeatedly against these measures and compared their performance with those from a sample of 1,907 human participants. Across the battery of theory of mind tests, we found that GPT-4 models performed at, or even sometimes above, human levels at identifying indirect requests, false beliefs and misdirection, but struggled with detecting faux pas. Faux pas, however, was the only test where LLaMA2 outperformed humans. Follow-up manipulations of the belief likelihood revealed that the superiority of LLaMA2 was illusory, possibly reflecting a bias towards attributing ignorance. By contrast, the poor performance of GPT originated from a hyperconservative approach towards committing to conclusions rather than from a genuine failure of inference. These findings not only demonstrate that LLMs exhibit behaviour that is consistent with the outputs of mentalistic inference in humans but also highlight the importance of systematic testing to ensure a non-superficial comparison between human and artificial intelligences.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent studies have shown that Large Language Models (LLMs), including GPT-3 and GPT-4, can excel in tasks that seemingly require ToM (Kosinski, 2023; van Duijn et al., 2023; Strachan et al., 2024; Street et al., 2024)”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Strachan et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This is amplified by the ability of LLMs to infer as well—or better—than humans the beliefs and intentions of an interlocutor from text (26).”
From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [26]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“later models demonstrate an increasing ability to reliably infer unobservable mental states in others (Strachan et al., 2024;Holterman and Deemter, 2023;Moghaddam and Honey, 2023).”
From Machine Psychology (2023) · cites this paper as Strachan et al., 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Specifically, these subjects are adult native English speakers (aged 18-70) with no history of psychiatric conditions or dyslexia, as specified in Strachan et al. (2024).”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Strachan et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“These models are increasingly achieving or surpassing human performance benchmarks in various domains, including medical diagnostic dialogue28, persuasive communication29, theory of mind30, making people feel heard31, responding to relationship issues32 and helping people reframe negative situations to reduce negative emotions33.”
From “It happened to be the perfect thing”: experiences of generative AI chatbots for mental health (2024) · cites this paper as [30]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Proceedings of the National Academy of Sciences
Richards, J. W. (1915) · Science · doi:10.1126/science.42.1074.161 · Cited by 13,983
Does the chimpanzee have a theory of mind?
Premack, D., & Woodruff, G. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00076512 · Cited by 8,743 · cited 1×
“3), dozens of tasks have been developed to study it, including indirect measures of belief attribution using reaction times 4–6 and looking or searching behaviour 7–9 , tasks examining the ability to infer mental states from photographs of eyes 10 , and language-based tasks assessing false belief…”
BackgroundWimmer, H. (1983) · Cognition · doi:10.1016/0010-0277(83)90004-5 · Cited by 6,305
Baron‐Cohen, S., Wheelwright, S., Hill, J., Raste, Y., & Plumb, I. (2001) · Journal of Child Psychology and Psychiatry · doi:10.1111/1469-7610.00715 · Cited by 4,555 · cited 1×
“…indirect measures of belief attribution using reaction times 4–6 and looking or searching behaviour 7–9 , tasks examining the ability to infer mental states from photographs of eyes 10 , and language-based tasks assessing false belief understanding 11,12 and pragmatic language comprehension 13–16 .”
The principles of psychology, Vol I.
James, W. (1890) · doi:10.1037/10538-000 · Cited by 4,514
Happé, F. G. E. (1994) · Journal of Autism and Developmental Disorders · doi:10.1007/bf02172093 · Cited by 2,352 · cited 2×
“We selected a set of well-established theory of mind tests spanning different abilities: the hinting task 14 , the false belief task 11,33 , the recognition of faux pas 13 , and the strange stories 15,16 .”
BackgroundMethodThe Neural Basis of Mentalizing
Frith, C. D., & Frith, U. (2006) · Neuron · doi:10.1016/j.neuron.2006.05.001 · Cited by 1,855
Three-year-olds’ difficulty with false belief: The case for a conceptual deficit
Perner, J., Leekam, S. R., & Wimmer, H. (1987) · British Journal of Developmental Psychology · doi:10.1111/j.2044-835x.1987.tb01048.x · Cited by 1,550 · cited 1×
“…indirect measures of belief attribution using reaction times 4–6 and looking or searching behaviour 7–9 , tasks examining the ability to infer mental states from photographs of eyes 10 , and language-based tasks assessing false belief understanding 11,12 and pragmatic language comprehension 13–16 .”
BackgroundSparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 3×
“One example is the false belief test, one of the most widely used tools so far for testing the performance of LLMs on social cognitive tasks 19,21–23, 25, 51 , 52 .”
BackgroundMethodToward a second-person neuroscience
Schilbach, L., Timmermans, B., Reddy, V., Costall, A., Bente, G., Schlicht, T., & Vogeley, K. (2013) · Behavioral and Brain Sciences · doi:10.1017/s0140525x12000660 · Cited by 1,510
Research works printed but not linked: 1
- Gandhi, K., Fränken, J.-P., Gerstenberg, T. & Goodman, N. D. Understanding social reasoning in language models with language models. In Advances in Neural Information Processing Systems Vol. 36 (MIT Press, 2023).
Cited by
Evaluating large language models in theory of mind tasks
Kosinski, M. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2405460121 · Cited by 339
“58 and 59 ), others found that LLMs’ performance was inconsistent and brittle ( 26 , 60 , 61 ).”
BackgroundTorous, J., Linardon, J., Goldberg, S. B., Sun, S., Bell, I., Nicholas, J., Hassan, L., Hua, Y., Milton, A., & Firth, J. (2025) · World Psychiatry · doi:10.1002/wps.21299 · Cited by 323
“It happened to be the perfect thing”: experiences of generative AI chatbots for mental health
Siddals, S., Torous, J., & Coxon, A. (2024) · npj Mental Health Research · doi:10.1038/s44184-024-00097-4 · Cited by 241
“…or surpassing humanperformancebenchmarksin variousdomains,including medical diagnostic dialogue 28 , persuasive communication 29 , theory of mind 30 , making people feel heard 31 , responding to relationship issues 32 and helping people reframe negative situations to reduce negative emotions 33…”
Why human–AI relationships need socioaffective alignment
Kirk, H. R., Gabriel, I., Summerfield, C., Vidgen, B., & Hale, S. A. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04532-5 · Cited by 131
Human-AI agency in the age of generative AI
Krakowski, S. (2025) · Information and Organization · doi:10.1016/j.infoandorg.2025.100560 · Cited by 162
Metadata
- Authors
- Strachan, J. W. A., Albergo, D., Borghini, G., Pansardi, O., Scaliti, E., Gupta, S., Saxena, K., Rufo, A., Panzeri, S., Manzi, G., Graziano, M. S. A., & Becchio, C.
- Affiliations
- University Medical Center Hamburg-Eppendorf (DE)
- University of Trento (IT)
- Italian Institute of Technology (IT)
- Universität Hamburg (DE)
- Princeton University (US)
- Corresponding author
- Strachan, James W. A.
- Becchio, Cristina
- ORCID
- Venue
- Nature Human Behaviour
- Published
- 2024-05-20
- Volume
- 8
- Issue
- 7
- Pages
- 1285-1295
- ISSN
- 2397-3374 (ISSN-L)
- PMC
- PMC11272575
- OpenAlex
- W4398143508
- Cited by
- 299
- References
- 58
- Citations by year
- 2026: 125 · 2025: 136 · 2024: 37
- FWCI
- 232.53
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Nature Human Behaviour (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Institutional Research Information System University of Turin (University of Turin) (submittedVersion, cc-by-nc)
- HAL (submittedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- OSF Preprints (OSF Preprints) (submittedVersion)
- Funders
- Alexander von Humboldt-Stiftung
- European Commission (101071191)
- Universitätsklinikum Hamburg-Eppendorf
- Topics
- Social Sciences › Psychology › Developmental and Educational Psychology › Child and Animal Learning Development
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Arts and Humanities › Philosophy › Philosophy and Theoretical Science
- Keywords
- theory of mind, large language models, false belief understanding, indirect requests, irony detection, faux pas, misdirection, GPT-4, Llama 2, human-AI comparison, belief attribution
- MeSH terms
- Adolescent, Adult, Female, Humans, Language, Male, Middle Aged, Models, Psychological, Young Adult, Theory of Mind
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Testing theory of mind in large language models and humans. https://app.cochat.ai/papers/10.1038/s41562-024-01882-z
- The paper
- Strachan, J. W. A., Albergo, D., Borghini, G., Pansardi, O., Scaliti, E., Gupta, S., Saxena, K., Rufo, A., Panzeri, S., Manzi, G., Graziano, M. S. A., & Becchio, C. (2024). Testing theory of mind in large language models and humans. Nature Human Behaviour. https://doi.org/10.1038/s41562-024-01882-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.