CoChat
Sign Up
Cited by 83Multimodal Machine Learning ApplicationsLogic, Reasoning, and KnowledgeCategorization, perception, and language

Visual cognition in multimodal large language models

Open access
Schulze Buschoff, Luca M., Akata, Elif, Bethge, Matthias, Schulz, Eric·2025·Nature Machine Intelligence
DOI 10.1038/s42256-024-00963-y
Open this paper in CoChat — free accountOpen PDF

Abstract

A chief goal of artificial intelligence is to build machines that think like people. Yet it has been argued that deep neural network architectures fail to accomplish this. Researchers have asserted these models’ limitations in the domains of causal reasoning, intuitive physics and intuitive psychology. Yet recent advancements, namely the rise of large language models, particularly those designed for visual processing, have rekindled interest in the potential to emulate human-like cognitive abilities. This paper evaluates the current state of vision-based large language models in the domains of intuitive physics, causal reasoning and intuitive psychology. Through a series of controlled experiments, we investigate the extent to which these modern models grasp complex physical interactions, causal relationships and intuitive understanding of others’ preferences. Our findings reveal that, while some of these models demonstrate a notable proficiency in processing and interpreting visual data, they still fall short of human capabilities in these areas. Our results emphasize the need for integrating more robust mechanisms for understanding causality, physical dynamics and social cognition into modern-day, vision-based language models, and point out the importance of cognitively inspired benchmarks.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s42256-024-00963-y

Cites

24 of 24 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Uncorrected perceptual errors can invalidate subsequent reasoning.

    “Since downstream reasoning cannot reliably recover visual content that was missed, misread, or hallucinated, uncorrected perceptual errors can invalidate subsequent reasoning (Buschoff et al. 2025; Gao, Zihao Huang, et al. 2025).”

    From PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models (2026) · cites this paper as Buschoff et al. 2025

    The Discussion (page unavailable) says that poor basic visual processing may partially explain the models’ weak performance in some domains, supporting the possibility that perceptual errors impair reasoning. It does not establish that downstream reasoning cannot reliably recover missed, misread, or hallucinated visual content, or that such errors invalidate the subsequent reasoning. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Downstream reasoning cannot reliably recover visual content that was missed, misread, or hallucinated.

    “Since downstream reasoning cannot reliably recover visual content that was missed, misread, or hallucinated, uncorrected perceptual errors can invalidate subsequent reasoning (Buschoff et al. 2025; Gao, Zihao Huang, et al. 2025).”

    From PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models (2026) · cites this paper as Buschoff et al. 2025

    The Discussion (page unavailable) says that weak performance in some reasoning domains can partly be explained by poor basic visual processing. That supports the concern that perceptual errors can affect later reasoning, but the paper does not establish that downstream reasoning cannot reliably recover missed, misread, or hallucinated visual content. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 66

  1. Mitchell, M. Artificial Intelligence: A Guide for Thinking Humans (Penguin, 2019).
  2. Devlin, J., Chang, M.-W., Lee, K. & Toutanova, K. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) (eds Burstein, J. et al.) 4171-4186 (Association for Computational Linguistics, 2019).
  3. Vaswani, A. et al. Attention is all you need. In Proc. Advances in Neural Information Processing Systems 30 (NIPS 2017) (eds Guyon, I. et al.) 5998-6008 (2017).
  4. Brown, T. et al. Language models are few-shot learners. In Proc. Advances in Neural Information Processing Systems 33 (NeurIPS 2020) (eds Larochelle, H. et al.) 1877-1901 (Curran Associates, 2020).
  5. Bubeck, S. et al. Sparks of artificial general intelligence: early experiments with GPT-4. Preprint at https://arxiv.org/ abs/2303.12712 (2023). · arXiv
  6. Wei, J. et al. Emergent abilities of large language models. Trans. Mach. Learn. Res. https://openreview.net/forum?id=yzkSU5zdwD (2022).
  7. Sawicki, P. et al. On the power of special-purpose gpt models to create and evaluate new poetry in old styles. In Proc. 14th International Conference on Computational Creativity (ICCC'23) (eds Pease, A. et al.) 10-19 (Association for Computational Creativity, 2023).
  8. Borsos, Z. et al. Audiolm: a language modeling approach to audio generation. IEEE/ACM Trans. Audio Speech Lang. Process 31, 2523-2533 (2023).
  9. Poldrack, R. A., Lu, T. & Beguš, G. Ai-assisted coding: experiments with GPT-4. Preprint at https://arxiv.org/abs/2304.13187 (2023). · arXiv
  10. Elkins, K. & Chun, J. Can GPT-3 pass a writer's Turing test? J. Cult. Anal. 5, 2 (2020).
  11. Akata, E. et al. Playing repeated games with large language models. Preprint at https://arxiv.org/abs/2305.16867 (2023). · arXiv
  12. Rumelhart, D. E. et al. Parallel Distributed Processing, Vol.1 (MIT Press, 1987).
  13. Bowers, J. S. et al. On the importance of severely testing deep learning models of cognition. Cogn. Syst. Res. 82, 101158 (2023).
  14. Marcus, G. Deep learning: a critical appraisal. Preprint at https://arxiv.org/abs/1801.00631 (2018). · arXiv
  15. Lake, B. M., Ullman, T. D., Tenenbaum, J. B. & Gershman, S. J. Building machines that learn and think like people. Behav. Brain Sci. 40, e253 (2017).
  16. Bates, C. J., Yildirim, I., Tenenbaum, J. B. & Battaglia, P. Modeling human intuitions about liquid flow with particle-based simulation. PLoS Comput. Biol. 15, e1007210 (2019).
  17. Battaglia, P. et al. Computational models of intuitive physics. Proc. Annu. Meet. Cogn. Sci. Soc. 34, 32-33 (2012).
  18. Hamrick, J., Battaglia, P. & Tenenbaum, J. B. Probabilistic internal physics models guide judgments about object dynamics. Proc. Annu. Meet. Cogn. Sci. Soc. 33, 1545-1550 (2011).
  19. Battaglia, P. W., Hamrick, J. B. & Tenenbaum, J. B. Simulation as an engine of physical scene understanding. Proc. Natl Acad. Sci. USA 110, 18327-18332 (2013).
  20. Bakhtin, A., van der Maaten, L., Johnson, J., Gustafson, L. & Girshick, R. PHYRE: a new benchmark for physical reasoning. In Proc. Advances in Neural Information Processing Systems 32 (NeurIPS 2019) (eds Wallach, H. et al.) 5082-5093 (Curran Associates, 2019).
  21. Riochet, R. et al. Intphys: a framework and benchmark for visual intuitive physics reasoning. Preprint at https://arxiv.org/ abs/1803.07616 (2018). · arXiv
  22. Schulze Buschoff, L. M., Schulz, E. & Binz, M. The acquisition of physical knowledge in generative neural networks. In Proc. 40th International Conference on Machine Learning (eds Krause, A. et al.) 202, 30321-30341 (JMLR, 2023).
  23. Waldmann, M. The Oxford Handbook of Causal Reasoning (Oxford Univ. Press, 2017).
  24. Cheng, P. W. From covariation to causation: a causal power theory. Psychol. Rev. 104, 367 (1997).
  25. Pearl, J. Causality (Cambridge Univ. Press, 2009).
  26. Griffiths, T. L. & Tenenbaum, J. B. Theory-based causal induction. Psychol. Rev. 116, 661 (2009).
  27. Lagnado, D. A., Waldmann, M. R., Hagmayer, Y. & Sloman, S. A. in Causal Learning: Psychology, Philosophy, and Computation (eds Gopnik, A. and Schulz, L.) 154-172 (Oxford Univ. Press, 2007).
  28. Carey, S. On the Origin of Causal Understanding (Clarendon Press/ Oxford Univ. Press, 1995).
  29. Gopnik, A. et al. A theory of causal learning in children: causal maps and Bayes nets. Psychol. Rev. 111, 3 (2004).
  30. Bramley, N. R., Gerstenberg, T., Mayrhofer, R. & Lagnado, D. A. Time in causal structure learning. J. Exp. Psychol. Learn. Mem. Cogn. 44, 1880 (2018).
  31. Schulz, L., Kushnir, T. & Gopnik, A. in Causal Learning: Psychology, Philosophy, and Computation (eds Gopnik, A. and Schulz, L.) 67-85 (Oxford Univ. Press, 2007).
  32. Gerstenberg, T. What would have happened? Counterfactuals, hypotheticals and causal judgements. Philos. Trans. R. Soc. B 377, 20210339 (2022).
  33. Gerstenberg, T., Goodman, N. D., Lagnado, D. A. & Tenenbaum, J. B. A counterfactual simulation model of causal judgments for physical events. Psychol. Rev. 128, 936 (2021).
  34. Jin, Z. et al. CLadder: Assessing causal reasoning in language models. In Proc. Advances in Neural Information Processing Systems 36 (NeurIPS 2023) (eds Oh, A. et al.) 31038-31065 (Curran Associates, 2023).
  35. Dasgupta, I. et al. Causal reasoning from meta-reinforcement learning. Preprint at https://arxiv.org/abs/1901.08162 (2019). · arXiv
  36. Baker, C. L. & Tenenbaum, J. B. in Plan, Activity, and Intent Recognition: Theory and Practice (eds Sukthankar, G. et al.) 177-204 (Morgan Kaufmann, 2014).
  37. Spelke, E. S., Bernier, E. P. & Skerry, A. Core Social Cognition (Oxford Univ. Press, 2013).
  38. Baker, C., Saxe, R. & Tenenbaum, J. Bayesian theory of mind: modeling joint belief-desire attribution. Proc. Annu. Meet. Cogn. Sci. Soc. 33, 2469-2474 (2011).
  39. Baker, C. L. et al. Intuitive theories of mind: a rational approach to false belief. Proc. Annu. Meet. Cogn. Sci. Soc. 28, 69k8c7v6 (2006).
  40. Shum, M., Kleiman-Weiner, M., Littman, M. L. & Tenenbaum, J. B. Theory of minds: understanding behavior in groups through inverse planning. In Proc. 33rd AAAI Conference on Artificial Intelligence 6163-6170 (Curran Associates, 2019).
  41. Baker, C. L., Jara-Ettinger, J., Saxe, R. & Tenenbaum, J. B. Rational quantitative attribution of beliefs, desires and percepts in human mentalizing. Nat. Hum. Behav. 1, 0064 (2017).
  42. Zhi-Xuan, T. et al. Solving the baby intuitions benchmark with a hierarchically Bayesian theory of mind. Preprint at https://arxiv. org/abs/2208.02914 (2022).
  43. Rabinowitz, N. et al. Machine theory of mind. In Proc. 35th International Conference on Machine Learning (eds Dy, J. & Krause, A.) 80, 4218-4227 (JMLR, 2018).
  44. Kosinski, M. Theory of mind may have spontaneously emerged in large language models. Preprint at https://arxiv.org/abs/ 2302.02083 (2023).
  45. Ullman, T. Large language models fail on trivial alterations to theory-of-mind tasks. Preprint at https://arxiv.org/abs/2302.08399 (2023). Article https://doi.org/10.1038/s42256-024-00963-y · arXiv
  46. Ullman, T. D. On the Nature and Origin of Intuitive Theories: Learning, Physics and Psychology. PhD thesis, Massachusetts Institute of Technology (2015).
  47. Sawada, T. et al. ARB: Advanced reasoning benchmark for large language models. Preprint at https://arxiv.org/abs/2307.13692 (2023). · arXiv
  48. Wei, J. et al. Chain-of-thought prompting elicits reasoning in large language models. In Proc. Advances in Neural Information Processing Systems 35 (NeurIPS 2022) (eds Koyejo, S. et al.) 24824-24837 (Curran Associates, 2022).
  49. Coda-Forno, J. et al. Inducing anxiety in large language models increases exploration and bias. Preprint at https://arxiv.org/ abs/2304.11111 (2023). · arXiv
  50. Eisape, T. et al. A systematic comparison of syllogistic reasoning in humans and language models. In Proc. 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (eds Duh, K. et al.) 8425-8444 (Association for Computational Linguistics, 2024).
  51. Ettinger, A. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Trans. Assoc. Comput. Linguist. 8, 34-48 (2020).
  52. Jones, C. R. et al. Distrubutional semantics still can't account for affordances. Proc. Annu. Meet. Cogn. Sci. Soc. 44, 482-489 (2022).
  53. Schulz, E. & Dayan, P. Computational psychiatry for computers. iScience 23, 12 (2020).
  54. Zhang, Y., Pan, J., Zhou, Y., Pan, R. & Chai, J. Grounding visual illusions in language: do vision-language models perceive illusions like humans? In Proc. 2023 Conference on Empirical Methods in Natural Language (eds Bouamor, H. et al.) 5718-5728 (Association for Computational Linguistics, 2023).
  55. Mitchell, M., Palmarini, A. B. & Moskvichev, A. Comparing humans, GPT-4, and GPT-4v on abstraction and reasoning tasks. Preprint at https://arxiv.org/abs/2311.09247 (2023). · arXiv
  56. Zečević, M., Willig, M., Dhami, D. S. & Kersting, K. Causal parrots: large language models may talk causality but are not causal. Trans. Mach. Learn. Res. https://openreview.net/ pdf?id=tv46tCzs83 (2023).
  57. Zhang, C., Wong, L., Grand, G. & Tenenbaum, J. Grounded physical language understanding with probabilistic programs and simulated worlds. Proc. Annu. Meet. Cogn. Sci. Soc. 45, 3476-3483 (2023).
  58. Jassim, S. et al. GRASP: a novel benchmark for evaluating language grounding and situated physics understanding in multimodal language models. Preprint at https://arxiv.org/ abs/2311.09048 (2023). · arXiv
  59. Kosoy, E. et al. Towards understanding how machines can learn causal overhypotheses. Proc. Annu. Meet. Cogn. Sci. Soc. 45, 363-374 (2023).
  60. Gandhi, K., Fränken, J.-P., Gerstenberg, T. & Goodman, N. D. Understanding social reasoning in language models with language models. In Proc. 37th International Conference on Neural Information Processing Systems (NIPS '23) (eds Oh, A. et al.) 13518-13529 (Curran Associates, 2024).
  61. Baltrušaitis, T., Ahuja, C. & Morency, L.-P. Multimodal machine learning: a survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41, 423-443 (2018).
  62. Reed, S. et al. Generative adversarial text to image synthesis. In Proc. 33rd International Conference on Machine Learning (eds
  63. Balcan, M. F. & Weinberger, K. Q.) 48, 1060-1069 (JMLR, 2016).
  64. Manmadhan, S. & Kovoor, B. C. Visual question answering: a state-of-the-art review. Artif. Intell. Rev. 53, 5705-5745 (2020).
  65. Lerer, A., Gross, S. & Fergus, R. Learning physical intuition of block towers by example. In Proc. 33rd International Conference on Machine Learning (eds Balcan, M. F. & Weinberger, K. Q.) 48, 430-438 (JMLR, 2016).
  66. Binz, M. & Schulz, E. Turning large language models into cognitive models. In Proc. 12th International Conference on Learning Representations (ICLR) https://openreview.net/ forum?id=eiC4BKypf1 (OpenReview, 2024). 108. Rahmanzadehgervi, P., Bolton, L., Taesiri, M. R. & Nguyen, A. T. Vision language models are blind. In Proc. Asian Conference on Computer Vision (ACCV) 18-34 (Computer Vision Foundation, 2024). 109. Ju, C., Han, T., Zheng, K., Zhang, Y. & Xie, W. Prompting visual-language models for efficient video understanding. In Proc. Computer Vision -ECCV 2022: 17th European Conference (eds Avidan, S. et al.) 105-124 (Springer, 2022). 110.

Cited by

Sorted by citations

Authors
Schulze Buschoff, L. M., Akata, E., Bethge, M., & Schulz, E.
Affiliations
  • Helmholtz Munich (DE)
  • University of Tübingen (DE)
ORCID
Venue
Nature Machine Intelligence
Publisher
Springer Science and Business Media LLC
Published
2025-01-15
Volume
7
Issue
1
Pages
96-106
ISSN
2522-5839 (ISSN-L)
DOI
10.1038/s42256-024-00963-y
OpenAlex
W4406400870
Cited by
83
References
24
Citations by year
2026: 48 · 2025: 31 · 2024: 4
FWCI
24.02
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Machine Intelligence (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • Max-Planck-Gesellschaft
  • Bundesministerium für Bildung und Forschung (01IS18039A)
  • Deutsche Forschungsgemeinschaft (EXC 2064/1 – 390727645)
  • Volkswagen Foundation
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Logic, Reasoning, and Knowledge
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Categorization, perception, and language
Keywords
visual cognition, multimodal large language models, intuitive physics, causal reasoning, intuitive psychology, social cognition, human-like cognition

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review