Inferring the intentional states of autonomous virtual agents
Abstract
Inferring the mental states of other agents, including their goals and intentions, is a central problem in cognition. A critical aspect of this problem is that one cannot observe mental states directly, but must infer them from observable actions. To study the computational mechanisms underlying this inference, we created a two-dimensional virtual environment populated by autonomous agents with independent cognitive architectures. These agents navigate the environment, collecting "food" and interacting with one another. The agents' behavior is modulated by a small number of distinct goal states: attacking, exploring, fleeing, and gathering food. We studied subjects' ability to detect and classify the agents' continually changing goal states on the basis of their motions and interactions. Although the programmed ground truth goal state is not directly observable, subjects' responses showed both high validity (correlation with this ground truth) and high reliability (correlation with one another). We present a Bayesian model of the inference of goal states, and find that it accounts for subjects' responses better than alternative models. Although the model is fit to the actual programmed states of the agents, and not to subjects' responses, its output actually conforms better to subjects' responses than to the ground truth goal state of the agents.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Bayesian models of cognition have primarily flourished as explanations of the complex dependency of behavior on the environment in domains including intuitive physics, causal learning, perception, motor control, and language.
“Bayesian models of cognition have primarily flourished as explanations of the complex dependency of behavior on the environment in domains including intuitive physics, causal learning, perception, motor control, and language [1-10]”
From Bayesian Brains without Probabilities (2016) · cites this paper as [3]
Cites
Model selection and multimodel inference: a practical information-theoretic approach.
Guthery, F. S., Burnham, K. P., & Anderson, D. R. (2003) · The Journal of Wildlife Management · doi:10.2307/3802723 · Cited by 41,495 · see citing works
Brainard, D. H. (1997) · Spatial Vision · doi:10.1163/156856897x00357 · Cited by 19,308 · see citing works
The VideoToolbox software for visual psychophysics: transforming numbers into movies
Pelli, D. G. (1997) · Spatial Vision · doi:10.1163/156856897x00366 · Cited by 11,110 · see citing works
(1987) · Medical Entomology and Zoology · Cited by 4,269 · see citing works
Planning and acting in partially observable stochastic domains
Kaelbling, L. P., Littman, M. L., & Cassandra, A. R. (1998) · Artificial Intelligence · doi:10.1016/s0004-3702(98)00023-x · Cited by 4,149 · see citing works
Kleiner, M., Brainard, D., & Pelli, D. G. (2007) · MPG.PuRe (Max Planck Society) · Cited by 3,751 · see citing works
Simple Heuristics That Make Us Smart
Gigerenzer, G., & Todd, P. M. (1999) · Cited by 3,562 · see citing works
An Experimental Study of Apparent Behavior
Heider, F., & Simmel, M. (1944) · The American Journal of Psychology · doi:10.2307/1416950 · Cited by 3,340 · see citing works
Perception as Bayesian Inference
(1996) · doi:10.1017/cbo9780511984037 · Cited by 1,425 · see citing works
Taking the intentional stance at 12 months of age
Gergely, G., Nádasdy, Z., Csibra, G., & Bı́ró, S. (1995) · Cognition · doi:10.1016/0010-0277(95)00661-h · Cited by 1,399 · see citing works
Simulating minds: the philosophy, psychology, and neuroscience of mindreading
(2006) · Choice Reviews Online · doi:10.5860/choice.44-2399 · Cited by 1,238 · see citing works
Event structure in perception and conception.
Zacks, J. M., & Tversky, B. (2001) · Psychological Bulletin · doi:10.1037/0033-2909.127.1.3 · Cited by 1,208 · see citing works
Action understanding as inverse planning
Baker, C. L., Saxe, R., & Tenenbaum, J. B. (2009) · Cognition · doi:10.1016/j.cognition.2009.07.005 · Cited by 920 · see citing works
Self-citationProceedings of the 33rd Annual Conference of the Cognitive Science Society
Dziemianko, M., Keller, F., & Coco, M. I. (2011) · Cited by 730 · see citing works
Carruthers, P., & Smith, P. K. (1996) · Cited by 718 · see citing works
(1996) · Cambridge University Press eBooks · doi:10.1017/cbo9780511597985 · Cited by 679 · see citing works
Attribution of Dispositional States by 12-Month-Olds
Kuhlmeier, V. A., Wynn, K., & Bloom, P. (2003) · Psychological Science · doi:10.1111/1467-9280.01454 · Cited by 506 · see citing works
Flatland: a Romance of Many Dimensions
TUCKER, R. (1884) · Nature · doi:10.1038/031076a0 · Cited by 490 · see citing works
Flatland: A Romance of Many Dimensions
Abbott, E. A. (2023) · Routledge eBooks · doi:10.4324/9781003056355-4 · Cited by 462 · see citing works
Perception of Animacy from the Motion of a Single Object
Tremoulet, P. D., & Feldman, J. (2000) · Perception · doi:10.1068/p3101 · Cited by 412 · see citing works
Self-citationPixel independence: measuring spatial interactions on a CRT display
Pelli, D. G. (1997) · Spatial Vision · doi:10.1163/156856897x00375 · Cited by 369 · see citing works
Distinguishing between animates and inanimates: not by motion alone
Gelman, R., Durgin, F. H., & Kaufman, L. (1996) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780198524021.003.0006 · Cited by 244 · see citing works
The psychophysics of chasing: A case study in the perception of animacy
Gao, T., Newman, G. E., & Scholl, B. J. (2009) · Cognitive Psychology · doi:10.1016/j.cogpsych.2009.03.001 · Cited by 233 · see citing works
Shao, W., & Terzopoulos, D. (2007) · Graphical Models · doi:10.1016/j.gmod.2007.09.001 · Cited by 228 · see citing works
Varieties of Memory and Consciousness
Roediger, H. L., III, & Craik, F. I. M. (2014) · Psychology Press eBooks · doi:10.4324/9781315801841 · Cited by 214 · see citing works
Inferring the goals of a nonhuman agent
Johnson, S. C., Booth, A. E., & O’Hearn, K. M. (2001) · Cognitive Development · doi:10.1016/s0885-2014(01)00043-0 · Cited by 190 · see citing works
Accurate judgments of intention from motion cues alone: A cross-cultural study
Barrett, H. C., Todd, P. M., Miller, G. F., & Blythe, P. W. (2005) · Evolution and Human Behavior · doi:10.1016/j.evolhumbehav.2004.08.015 · Cited by 190 · see citing works
Using movement and intentions to understand human activity
Zacks, J. M., Kumar, S., Abrams, R. A., & Mehta, R. (2009) · Cognition · doi:10.1016/j.cognition.2009.03.007 · Cited by 188 · see citing works
Visual Perception of Intentional Motion
Dittrich, W. H., & Lea, S. E. G. (1994) · Perception · doi:10.1068/p230253 · Cited by 179 · see citing works
Gao, T., McCarthy, G., & Scholl, B. J. (2010) · Psychological Science · doi:10.1177/0956797610388814 · Cited by 170 · see citing works
Help or Hinder: Bayesian Models of Social Goal Inference
Ullman, T. D., Baker, C. L., Macindoe, O., Evans, O., Goodman, N. D., & Tenenbaum, J. B. (2009) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 156 · see citing works
Self-citationSocial behavior recognition in continuous video
Burgos-Artizzu, X. P., Dollár, P., Lin, D., Anderson, D. J., & Perona, P. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247817 · Cited by 148 · see citing works
A Review of Theory in Physiological Psychology
Pribram, K. H. (1960) · Annual Review of Psychology · doi:10.1146/annurev.ps.11.020160.000245 · Cited by 136 · see citing works
Tremoulet, P. D., & Feldman, J. (2006) · Perception & Psychophysics · doi:10.3758/bf03193364 · Cited by 129 · see citing works
Self-citationDennett, D. C. (1978) · Behavioral and Brain Sciences · doi:10.1017/s0140525x00059859 · Cited by 127 · see citing works
Geisler, W. S. (2003) · The MIT Press eBooks · doi:10.7551/mitpress/7131.003.0061 · Cited by 123 · see citing works
Simulation, theory, and content
Heal, J. (1996) · Cambridge University Press eBooks · doi:10.1017/cbo9780511597985.006 · Cited by 105 · see citing works
Chasing vs. stalking: Interrupting the perception of animacy.
Gao, T., & Scholl, B. J. (2011) · Journal of Experimental Psychology Human Perception & Performance · doi:10.1037/a0020735 · Cited by 98 · see citing works
How motion reveals intention: Categorizing social interactions
Blythe, P. W., Todd, P. M., & Miller, G. F. (1999) · Max Planck Institute for Plasma Physics · Cited by 90 · see citing works
The Computational Perception of Scene Dynamics
Mann, R., Jepson, A. S., & Siskind, J. M. (1997) · Computer Vision and Image Understanding · doi:10.1006/cviu.1996.0576 · Cited by 83 · see citing works
The Blackwell Guide to Philosophy of Mind
(2002) · Blackwell Publishing Ltd eBooks · doi:10.1111/b.9780631217756.2002.00007.x · Cited by 74 · see citing works
Simulation, Theory and Content
Heal, J. (2003) · Cambridge University Press eBooks · doi:10.1017/cbo9780511615894.006 · Cited by 69 · see citing works
Artificial Perception of Actions
Thibadeau, R. H. (1986) · Cognitive Science · doi:10.1016/s0364-0213(86)80001-5 · Cited by 58 · see citing works
A rational analysis of human memory.
Anderson, J. R. (1989) · Cited by 42 · see citing works
Artificial Perception of Actions
Thibadeau, R. H. (1986) · Cognitive Science · doi:10.1207/s15516709cog1002_1 · Cited by 36 · see citing works
Understanding intention from minimal displays of human activity
McAleer, P., & Pollick, F. E. (2008) · Behavior Research Methods · doi:10.3758/brm.40.3.830 · Cited by 29 · see citing works
Stewart, J. A. (1982) · Scholarly Commons (University of Pennsylvania) · Cited by 24 · see citing works
Controlling a Robot with Intention Derived from Motion
Crick, C., & Scassellati, B. (2010) · Topics in Cognitive Science · doi:10.1111/j.1756-8765.2009.01073.x · Cited by 13 · see citing works
Chasing vs. Stalking: Interrupting the Perception of Animacy
Gao, T., & Scholl, B. J. (2010) · Journal of Vision · doi:10.1167/10.7.239 · Cited by 10 · see citing works
Exploring the mental space of autonomous intentional agents
Pantelis, P. C., & Feldman, J. (2011) · Attention Perception & Psychophysics · doi:10.3758/s13414-011-0215-6 · Cited by 10 · see citing works
Self-citationUsing modified incremental chart parsing to ascribe intentions to animated geometric figures
Pautler, D., Koenig, B. L., Quek, B., & Ortony, A. (2011) · Behavior Research Methods · doi:10.3758/s13428-011-0128-2 · Cited by 7 · see citing works
Recognizing behaviors and the internal state of the participants
Kerr, W., & Cohen, P. R. (2010) · IEEE International Conference on Development and Learning (ICDL) · doi:10.1109/devlrn.2010.5578868 · Cited by 7 · see citing works
Perception of intentions and mental states in autonomous virtual agents
Pantelis, P. C., Cholewiak, S. A., Ringstad, P., Sanik, K., Weinstein, A., Wu, C.-C., & Feldman, J. (2011) · Journal of Vision · doi:10.1167/11.11.733 · Cited by 5 · see citing works
Self-citationPerception of intentions and mental states in autonomous virtual agents
Pantelis, P. C., Cholewiak, S. A., Ringstad, P., Sanik, K., Weinstein, A., Wu, C.-C., & Feldman, J. (2011) · Cited by 2 · see citing works
Self-citation
Flagged first, then by citations
1 of the 56 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Cognitive computational neuroscience
Kriegeskorte, N., & Douglas, P. K. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0210-5 · Cited by 535 · see citing works
Rational quantitative attribution of beliefs, desires and percepts in human mentalizing
Baker, C. L., Jara-Ettinger, J., Saxe, R., & Tenenbaum, J. B. (2017) · Nature Human Behaviour · doi:10.1038/s41562-017-0064 · Cited by 515 · see citing works
Self-citationBayesian Brains without Probabilities
Sanborn, A. N., & Chater, N. (2016) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2016.10.003 · Cited by 307 · see citing works
Plan, Activity, and Intent Recognition
Sukthankar, G., Geib, C., Bui, H., Pynadath, D. V., & Goldman, R. P. (2014) · Elsevier eBooks · doi:10.1016/c2012-0-01169-6 · Cited by 157 · see citing works
Modeling Human Plan Recognition Using Bayesian Theory of Mind
Baker, C. L., & Tenenbaum, J. B. (2014) · Elsevier eBooks · doi:10.1016/b978-0-12-398532-3.00007-5 · Cited by 114 · see citing works
Self-citationAdjustable autonomy: a systematic literature review
Mostafa, S. A., Ahmad, M. S., & Mustapha, A. (2017) · Artificial Intelligence Review · doi:10.1007/s10462-017-9560-8 · Cited by 98 · see citing works
Control blindness: Why people can make incorrect inferences about the intentions of others
Willett, A. S., Marken, R. S., Parker, M. G., & Mansell, W. (2017) · Attention Perception & Psychophysics · doi:10.3758/s13414-016-1268-3 · Cited by 72 · see citing works
Lucky or clever? From expectations to responsibility judgments
Gerstenberg, T., Ullman, T. D., Nagel, J., Max Kleiman-weiner, A., Lagnado, D. A., & Tenenbaum, J. B. (2018) · Cognition · doi:10.1016/j.cognition.2018.03.019 · Cited by 64 · see citing works
Self-citationCan People Learn about ‘Black Swans’? Experimental Evidence
Payzan-LeNestour, É. (2018) · Review of Financial Studies · doi:10.1093/rfs/hhy040 · Cited by 40 · see citing works
Papagni, G., & Koeszegi, S. T. (2021) · Minds and Machines · doi:10.1007/s11023-021-09567-6 · Cited by 38 · see citing works
A decision network account of reasoning about other people’s choices
Jern, A., & Kemp, C. (2015) · Cognition · doi:10.1016/j.cognition.2015.05.006 · Cited by 29 · see citing works
The Cognitive Architecture of Perceived Animacy: Intention, Attention, and Memory
Gao, T., Baker, C. L., Tang, N., Xu, H., & Tenenbaum, J. B. (2019) · Cognitive Science · doi:10.1111/cogs.12775 · Cited by 25 · see citing works
Self-citationThe Multiple Realizability of Sentience in Living Systems and Beyond
Rouleau, N., & Levin, M. G. (2023) · eNeuro · doi:10.1523/eneuro.0375-23.2023 · Cited by 23 · see citing works
Reasoning about social sources to learn from actions and outcomes.
Hawthorne-Madell, D., & Goodman, N. D. (2017) · Decision · doi:10.1037/dec0000088 · Cited by 22 · see citing works
Lucky or clever? From expectations to responsibility judgmentsPreprint
Gerstenberg, T., Ullman, T. D., Nagel, J., Max Kleiman-weiner, A., Lagnado, D. A., & Tenenbaum, J. B. (2018) · doi:10.31234/osf.io/jc72g · Cited by 13 · see citing works
Self-citationInteractive cognitive maps support flexible behavior under threat
Wise, T., Charpentier, C. J., Dayan, P., & Mobbs, D. (2023) · Cell Reports · doi:10.1016/j.celrep.2023.113008 · Cited by 11 · see citing works
Lee, Y., & Malcein, L. A. (2020) · Human Behavior and Emerging Technologies · doi:10.1002/hbe2.212 · Cited by 9 · see citing works
Perception in dynamic scenes: What is your Heider capacity?
Wick, F., Soce, A. A., Garg, S., Grace, R. C., & Wolfe, J. M. (2019) · Journal of Experimental Psychology General · doi:10.1037/xge0000557 · Cited by 9 · see citing works
Toward a Perceptual Account of Mindreading
Varga, S. (2018) · Philosophy and Phenomenological Research · doi:10.1111/phpr.12556 · Cited by 8 · see citing works
A unified psychological space for human perception of physical and social events
Shu, T., Peng, Y., Zhu, S. C., & Lu, H. (2021) · Cognitive Psychology · doi:10.1016/j.cogpsych.2021.101398 · Cited by 8 · see citing works
Sampling as the Human Approximation to Probabilistic Inference
Sanborn, A. N., Zhu, J.-Q., Spicer, J., Sundh, J., León-Villagrá, P., & Nick Chater, A. (2021) · Oxford University Press eBooks · doi:10.1093/oso/9780198862536.003.0021 · Cited by 7 · see citing works
Bayesian Inference of Self-intention Attributed by Observer
Fukuchi, Y., Osawa, M., Yamakawa, H., Takahashi, T., & Imai, M. (2018) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3284432.3284438 · Cited by 6 · see citing works
Agency and rationality: Adopting the intentional stance toward evolved virtual agents.
Pantelis, P. C., Gerstner, T., Sanik, K., Weinstein, A., Cholewiak, S. A., Kharkwal, G., Wu, C.-C., & Feldman, J. (2015) · Decision · doi:10.1037/dec0000042 · Cited by 6 · see citing works
Self-citationFormal Approaches to a Definition of AgentsPreprint
Biehl, M. (2017) · arXiv (Cornell University) · doi:10.18745/th.17955 · Cited by 5 · see citing works
A Dynamical Generative Model of Social Interactions
Salatiello, A., Hovaidi-Ardestani, M., & Giese, M. A. (2021) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2021.648527 · Cited by 3 · see citing works
Yin, J., Chen, M., Wang, X., & Ding, X. (2018) · Attention Perception & Psychophysics · doi:10.3758/s13414-018-1508-9 · Cited by 3 · see citing works
Understanding Dialogue Acts by Bayesian Inference and Reinforcement Learning
Matsushima, A., Oka, N., Fukada, C., & Tanaka, K. (2019) · International Conference on Human-Agent Interaction (HAI) · doi:10.1145/3349537.3352786 · Cited by 2 · see citing works
Hellou, M., Vinanzi, S., & Cangelosi, A. (2025) · Springer proceedings in advanced robotics · doi:10.1007/978-3-031-81688-8_15 · Cited by 1 · see citing works
Cognitive computational neurosciencePreprint
Kriegeskorte, N., & Douglas, P. K. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1807.11819 · Cited by 1 · see citing works
Empirical investigation of memory-based strategy on anomalous behavior
Matsubayashi, S., Miwa, K., & Terai, H. (2019) · The Japanese journal of psychology · doi:10.4992/jjpsy.90.18018
A Bayesian Approach to Identifying Representational ErrorsPreprint
Ramakrishnan, R., Unhelkar, V. V., Kamar, E. S., & Shah, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.15171
Wise, T., Charpentier, C. J., Dayan, P., & Mobbs, D. (2022) · doi:10.31234/osf.io/6d4z7
Exploring users' mental models of conversational agents: a systematic review
Cena, F., & Grasso, F. (2025) · Behaviour and Information Technology · doi:10.1080/0144929x.2025.2573436
Hellou, M., Angelopoulos, G., Vinanzi, S., Rossi, A., Rossi, S., & Cangelosi, A. (2025) · IEEE RO-MAN · doi:10.1109/ro-man63969.2025.11217739
- Authors
- Pantelis, P. C., Baker, C. L., Cholewiak, S. A., Sanik, K., Weinstein, A., Wu, C.-C., Tenenbaum, J. B., & Feldman, J.
- Affiliations
- Rutgers, The State University of New Jersey (US)
- Massachusetts Institute of Technology (US)
- Corresponding author
- Pantelis, Peter C.
- Venue
- Cognition
- Publisher
- Elsevier BV
- Published
- 2014-03
- Volume
- 130
- Issue
- 3
- Pages
- 360-379
- ISSN
- 0010-0277 (ISSN-L)
- OpenAlex
- W2032255178
- Cited by
- 34
- References
- 56
- Citations by year
- 2025: 3 · 2023: 2 · 2022: 1 · 2021: 5 · 2020: 1 · 2019: 4 · 2018: 8 · 2017: 5 · 2016: 1 · 2015: 2 · 2014: 2
- FWCI
- 7.82
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Cognition (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Psychology › Developmental and Educational Psychology › Child and Animal Learning Development
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Psychology › Social Psychology › Action Observation and Synchronization
- Keywords
- goal inference, intention inference, mental state inference, Bayesian model, social cognition, agent navigation
- MeSH terms
- Cognition, Computer Simulation, Female, Humans, Male, Comprehension, Intention, Young Adult, Theory of Mind
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review