HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models
Open accessAbstract
Theory of Mind (ToM) is the ability to reason about one's own and others' mental states. ToM plays a critical role in the development of intelligence, language understanding, and cognitive processes. While previous work has primarily focused on first and second-order ToM, we explore higher-order ToM, which involves recursive reasoning on others' beliefs. We introduce HI-TOM, a Higher Order Theory of Mind benchmark. Our experimental evaluation using various Large Language Models (LLMs) indicates a decline in performance on higher-order ToM tasks, demonstrating the limitations of current LLMs. We conduct a thorough analysis of different failure cases of LLMs, and share our thoughts on the implications of our findings on the future of NLP.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hao, X.-B., & Zhang, G.-G. (2017) · Encycl. Semantic Comput. Robotic Intell. · doi:10.1142/s2425038416300184 · Cited by 86,177
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations
Rumelhart, D. E., McClelland, J. L., & AU (1986) · doi:10.7551/mitpress/5236.001.0001 · Cited by 15,686 · see citing works
Does the chimpanzee have a theory of mind?
Premack, D., & Woodruff, G. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00076512 · Cited by 8,743 · cited 3× · see citing works
“Theory of Mind (ToM) refers to the ability to understand and reason the mental states, such as intentions and beliefs, of others and distinguish them from one’s own (Premack & Woodruff, 1978).”
Does the autistic child have a “theory of mind” ?
Baron-Cohen, S., Leslie, A. M., & Frith, U. (1985) · Cognition · doi:10.1016/0010-0277(85)90022-8 · Cited by 8,599 · cited 2× · see citing works
“Researchers from cognitive science have investigated second-order and higher-order ToM among young children via complex forms of false-belief tests, such as the Sally-Anne false-belief experiment (Baron-Cohen et al., 1985) in Figure 1.”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1× · see citing works
“Furthermore, we test chain-of-thought prompting (Wei et al., 2022) and conduct a thorough analysis.”
Understanding the Representational Mind
Perner, J. (1993) · The MIT Press eBooks · doi:10.7551/mitpress/6988.001.0001 · Cited by 2,424 · cited 1× · see citing works
“Thus, the ToM has been widely used as a protocol to evaluate language understanding and reasoning ability of intelligence agents (Premack & Woodruff, 1978; Takano et al., 2006), such as young children (Osterhaus & Koerber, The milk is on the Where is the milk?”
Development and neurophysiology of mentalizing
Frith, U., & Frith, C. D. (2003) · Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rstb.2002.1218 · Cited by 2,306 · cited 1× · see citing works
“Such an ability has been considered as a crucial pointer in the development of intelligence functions (Premack & Woodruff, 1978; Bretherton & Beeghly, 1982; Frith & Frith, 2003), where researchers find that the ToM reasoning is highly related to linguistic and cognitive processes.”
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187 · see citing works
Large language models are zero-shot reasoners
Kojima, T., Gu, S. (., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1613 · Cited by 2,013 · cited 2× · see citing works
“Recent works have found that CoT prompting significantly improves LLMs’ performance on various reasoning tasks (Kojima et al., 2022; Zhang et al., 2023; Liu et al., 2023).”
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · cited 3× · see citing works
“Kosinski (2023); Bubeck et al. (2023) show the promising performance of recent LLMs such as GPT-3.”
Pragmatics, modularity and mindreading
Sperber, D., & Wilson, D. (2002) · Mind & Language · doi:10.1111/1468-0017.00186 · Cited by 1,027 · see citing works
Multimodal Chain-of-Thought Reasoning in Language ModelsPreprint
Zhang, Z., Zhang, A., Li, M., Zhao, H., Karypis, G., & Smola, A. J. (2023) · Trans. Mach. Learn. Res. · doi:10.48550/arxiv.2302.00923 · Cited by 994 · cited 1× · see citing works
“Recent works have found that CoT prompting significantly improves LLMs’ performance on various reasoning tasks (Kojima et al., 2022; Zhang et al., 2023; Liu et al., 2023).”
Talking About Internal States: The Acquisition of an Explicit Theory of Mind
Bretherton, I., & Beeghly, M. (1982) · Developmental Psychology · doi:10.1037/0012-1649.18.6.906 · Cited by 985 · cited 1× · see citing works
“Such an ability has been considered as a crucial pointer in the development of intelligence functions (Premack & Woodruff, 1978; Bretherton & Beeghly, 1982; Frith & Frith, 2003), where researchers find that the ToM reasoning is highly related to linguistic and cognitive processes.”
Human problem solving: The state of the theory in 1970.
Simon, H. A., & Newell, A. (1971) · American Psychologist · doi:10.1037/h0030806 · Cited by 635 · see citing works
QLoRA: Efficient Finetuning of Quantized LLMs
Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023) · arXiv · doi:10.48550/arxiv.2305.14314 · Cited by 507 · cited 1× · see citing works
“For demonstration purposes, we only show different orders for 3 agents as the proof-of-concept.”
Evaluating Large Language Models in Theory of Mind Tasks
Kosinski, M. (2023) · arXiv · doi:10.48550/arxiv.2302.02083 · Cited by 339 · cited 2× · see citing works
“Kosinski (2023); Bubeck et al. (2023) show the promising performance of recent LLMs such as GPT-3.”
Data Contamination: From Memorization to Exploitation
Magar, I., & Schwartz, R. (2022) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2203.08242 · Cited by 325 · cited 1×
“Also, it is crucial to pay attention to preventing LLMs from taking shortcuts to arrive at correct answers, as this would lead to overestimations of their true reasoning capabilities.”
Sources of Hallucination by Large Language Models on Inference TasksPreprint
McKenna, N., Li, T., Cheng, L., Hosseini, M., Johnson, M., & Steedman, M. (2023) · Conference on Empirical Methods in Natural Language Processing · doi:10.48550/arxiv.2305.14552 · Cited by 297 · see citing works
The overlapping relationship between emotion perception and theory of mind.
Mitchell, R. L., & Phillips, L. (2015) · Neuropsychologia · doi:10.1016/j.neuropsychologia.2015.02.018 · Cited by 237 · cited 1× · see citing works
“For demonstration purposes, we only show different orders for 3 agents as the proof-of-concept.”
Higher-order theory of mind and social competence in school-age children
Liddle, B., & Nettle, D. (2006) · Journal of Cultural and Evolutionary Psychology · doi:10.1556/jcep.4.2006.3-4.3 · Cited by 231 · cited 1× · see citing works
“…higher-order ToM (third-order and beyond), which requires recursively reasoning on others’ beliefs, is essential to communicate effectively in complicated scenarios, such as multi-party conversations (Liddle & Nettle, 2006; De Weerd et al., 2015; Ridinger & McBride, 2017; De Weerd et al., 2022).”
Revisiting the Evaluation of Theory of Mind through Question Answering
Le, M., Boureau, Y.-L., & Nickel, M. (2019) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d19-1598 · Cited by 220 · cited 4× · see citing works
“Most of the prior works focus on first or second-order ToM (Nematzadeh et al., 2018; Le et al., 2019; Sap et al., 2022), while higher-order ToM (third order and beyond 1 ) remains under-explored.”
InfluentialThinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
Miech, A., Alayrac, J.-B., Laptev, I., Sivic, J., & Zisserman, A. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00970 · Cited by 149 · see citing works
Neuro-Symbolic Artificial Intelligence: The State of the Art
Hitzler, P., & Sarker, M. K. (2021) · Neuro-Symbolic Artificial Intelligence · doi:10.3233/faia342 · Cited by 130 · see citing works
Zero-shot Temporal Relation Extraction with ChatGPTPreprint
Yuan, C.-H., Xie, Q., & Ananiadou, S. (2023) · Workshop on Biomedical Natural Language Processing · doi:10.48550/arxiv.2304.05454 · Cited by 123 · see citing works
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs
Sap, M., LeBras, R., Fried, D., & Choi, Y. (2022) · arXiv · doi:10.48550/arxiv.2210.13312 · Cited by 118 · cited 4× · see citing works
“Most of the prior works focus on first or second-order ToM (Nematzadeh et al., 2018; Le et al., 2019; Sap et al., 2022), while higher-order ToM (third order and beyond 1 ) remains under-explored.”
InfluentialData Contamination: From Memorization to Exploitation
Magar, I., & Schwartz, R. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/2022.acl-short.18 · Cited by 88 · see citing works
Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks
Ullman, T. (2023) · arXiv · doi:10.48550/arxiv.2302.08399 · Cited by 81 · cited 2× · see citing works
“Ullman (2023) find that for GPT-3.”
Osterhaus, C., & Koerber, S. (2021) · Child Development · doi:10.1111/cdev.13627 · Cited by 80 · cited 1× · see citing works
“Where does Anne think the milk is?”
Lee, L.-W. (2019) · Industrial Organization · doi:10.1007/978-3-030-26237-2_5 · Cited by 79
Can Large Language Models Infer Causation from Correlation?Preprint
Jin, Z., Liu, J., Lyu, Z., Poff, S., Sachan, M., Mihalcea, R. F., Diab, M., & Schölkopf, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.05836 · Cited by 67 · see citing works
Self-citationCombining Symbolic and Neural Learning
Shavlik, J. (1994) · Machine-mediated learning · doi:10.1023/a:1022665814563 · Cited by 60 · see citing works
Evaluating Theory of Mind in Question Answering
Nematzadeh, A., Burns, K., Grant, E., Gopnik, A., & Griffiths, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1261 · Cited by 53 · cited 3× · see citing works
“Most of the prior works focus on first or second-order ToM (Nematzadeh et al., 2018; Le et al., 2019; Sap et al., 2022), while higher-order ToM (third order and beyond 1 ) remains under-explored.”
Higher-order theory of mind in the Tacit Communication Game
Weerd, H. D., Verbrugge, R., & Verheij, B. (2015) · Biologically Inspired Cognitive Architectures · doi:10.1016/j.bica.2014.11.010 · Cited by 42 · cited 1× · see citing works
“…higher-order ToM (third-order and beyond), which requires recursively reasoning on others’ beliefs, is essential to communicate effectively in complicated scenarios, such as multi-party conversations (Liddle & Nettle, 2006; De Weerd et al., 2015; Ridinger & McBride, 2017; De Weerd et al., 2022).”
Boosting Theory-of-Mind Performance in Large Language Models via Prompting
Moghaddam, S. R., & Honey, C. J. (2023) · arXiv · doi:10.48550/arxiv.2304.11490 · Cited by 35 · cited 1× · see citing works
“Following Kojima et al. (2022); Moghaddam & Honey (2023), we adopt the zero-shot setting in our experiments, where we prompt the model with the story, question, and several possible answers without any prior demonstration.”
Grounded Language Learning Fast and Slow
Hill, F., Tieleman, O., von Glehn, T., Wong, N., Merzic, H., & Clark, S. (2020) · arXiv · doi:10.48550/arxiv.2009.01719 · Cited by 35 · see citing works
Large Language Models are not Fair Evaluators
Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., & Sui, Z. (2023) · arXiv · doi:10.48550/arxiv.2305.17926 · Cited by 34 · cited 1× · see citing works
“A correct third step is “Ava would believe that William also thinks that Isabella thinks the pineapple is still in the red basket, as she doesn’t know that William moved the pineapple and Isabella saw the movement” .”
Federated Prompting and Chain-of-Thought Reasoning for Improving LLMs Answering
Liu, X., Pang, T., & Fan, C. (2023) · Knowledge Science, Engineering and Management · doi:10.48550/arxiv.2304.13911 · Cited by 34 · cited 1×
“Recent works have found that CoT prompting significantly improves LLMs’ performance on various reasoning tasks (Kojima et al., 2022; Zhang et al., 2023; Liu et al., 2023).”
Higher-order theory of mind is especially useful in unpredictable negotiations
de Weerd, H., Verbrugge, R., & Verheij, B. (2022) · Autonomous Agents and Multi-Agent Systems · doi:10.1007/s10458-022-09558-6 · Cited by 30 · cited 2× · see citing works
“…higher-order ToM (third-order and beyond), which requires recursively reasoning on others’ beliefs, is essential to communicate effectively in complicated scenarios, such as multi-party conversations (Liddle & Nettle, 2006; De Weerd et al., 2015; Ridinger & McBride, 2017; De Weerd et al., 2022).”
Eliciting Latent Predictions from Transformers with the Tuned Lens
Belrose, N., Ostrovsky, I., McKinney, L., Furman, Z., Smith, L., Halawi, D., Biderman, S., & Steinhardt, J. (2023) · arXiv · doi:10.48550/arxiv.2303.08112 · Cited by 27 · see citing works
Emergence of Maps in the Memories of Blind Navigation Agents
Wijmans, E., Savva, M., Essa, I., Lee, S., Morcos, A. S., & Batra, D. (2023) · AI Matters · doi:10.1145/3609468.3609471 · Cited by 17 · see citing works
Flagged first, then by citations
11 of the 53 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, and Ben He. 2023. Chatgpt is a knowledgeable but inexperienced solver: An investigation of com- monsense problem in large language models. arXiv preprint arXiv:2303.16421. · arXiv
- Harmen de Weerd and Bart Verheij. 2011. The advan- tage of higher-order theory of mind in the game of limited bidding. In Proceedings Workshop 'Reason- ing about other Minds', CEUR Workshop Proceed- ings, volume 751, pages 149-164.
- Ziqiao Ma, Jacob Sansom, Run Peng, and Joyce Chai. 2023. Towards a holistic landscape of situated theory of mind in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational Linguistics.
- Dan Sperber and Deirdre Wilson. 2002. Pragmatics, modularity and mind-reading. Mind & language, 17(1-2):3-23.
- Masanori Takano, Takaya Arita, et al. 2006. Asymmetry between even and odd levels of recursion in a theory of mind. Proceedings of ALife X, pages 405-411.
Cited by
Sorted by citations
How developments in natural language processing help us in understanding human behaviour
Mihalcea, R. F., Biester, L., Boyd, R. L., Jin, Z., Pérez‐Rosas, V., Wilson, S. R., & Pennebaker, J. W. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01938-0 · Cited by 34 · see citing works
Self-citationRe-evaluating Theory of Mind evaluation in large language models
Hu, J., Sosa, F., & Ullman, T. (2025) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2023.0499 · Cited by 10 · see citing works
Large Language Model Reasoning Failures
Song, P., Han, P., & Goodman, N. (2026) · arXiv · doi:10.48550/arxiv.2602.06176 · Cited by 3 · see citing works
- Authors
- He, Y., Wu, Y., Jia, Y., Mihalcea, R., Chen, Y., & Deng, N.
- Affiliations
- University of Michigan (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2310.16755
- OpenAlex
- W4387964152
- Cited by
- 1
- References
- 53
- Citations by year
- 2024: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- Instituto de Ciencias del Mar y Limnología, Universidad Nacional Autónoma de México
- Topics
- Social Sciences › Psychology › Developmental and Educational Psychology › Child and Animal Learning Development
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
- Keywords
- recursive reasoning, mental state reasoning, large language models, benchmark, LLM limitations, failure case analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
