The DOI matches its Crossref record.
GRASP: A Novel Benchmark for Evaluating Language GRounding and Situated Physics Understanding in Multimodal Language Models
Open accessAbstract
This paper presents GRASP, a novel benchmark to evaluate the language grounding and physical understanding capabilities of video-based multimodal large language models (LLMs). This evaluation is accomplished via a two-tier approach leveraging Unity simulations. The first level tests for language grounding by assessing a model's ability to relate simple textual descriptions with visual information. The second level evaluates the model's understanding of "Intuitive Physics" principles, such as object permanence and continuity. In addition to releasing the benchmark, we use it to evaluate several state-of-the-art multimodal LLMs. Our evaluation reveals significant shortcomings in the language grounding and intuitive physics capabilities of these models. Although they exhibit at least some grounding capabilities, particularly for colors and shapes, these capabilities depend heavily on the prompting strategy. At the same time, all models perform below or at the chance level of 50% in the Intuitive Physics tests, while human subjects are on average 80% correct. These identified limitations underline the importance of using benchmarks like GRASP to monitor the progress of future models in developing these competencies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The low performance of pixel prediction and multimodal LLMs corroborates previous findings.
“The low performance of pixel prediction and multimodal LLMs corroborates previous findings (Riochet et al., 2022; Jassim et al., 2024).”
From Intuitive physics understanding emerges from self-supervised pretraining on natural videos (2025) · cites this paper as Jassim et al., 2024
The Results section reports that the tested multimodal LLMs performed at or below chance on the intuitive-physics tests, supporting that part of the citing sentence. The paper does not report results for pixel-prediction models, so it does not support that part of the claim. Page numbers are unavailable in our copy. (LLM, conf=0.86)
Experimental — accuracy not yet measuredGRASP uses simulated data, covers a wider range of properties (10 compared to 3) with more total videos than IntPhys.
“GRASP (Jassim et al., 2024) is similar to IntPhys in the sense that it also uses simulated data, but covers a wider range of properties (10 compared to 3) with more total videos.”
From Intuitive physics understanding emerges from self-supervised pretraining on natural videos (2025) · cites this paper as Jassim et al., 2024
The Multimodal LLMs section says that GRASP and IntPhys use simulated videos and that GRASP covers more concepts and scenes per concept. But §3.2 specifies eight GRASP concepts and events, not ten; our copy also does not establish that GRASP has more total videos than IntPhys. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThese results do not mean that LLMs or pixel prediction models cannot achieve intuitive physics understanding, but merely that this seemingly simple task remains difficult even for frontier models.
“These results, however, do not mean that LLMs or pixel prediction models cannot achieve intuitive physics understanding, but merely that this seemingly simple task remains difficult even for frontier models (Jassim et al., 2024; Kang et al., 2024; Bansal et al., 2024).”
From Intuitive physics understanding emerges from self-supervised pretraining on natural videos (2025) · cites this paper as Jassim et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
No references could be resolved yet.
Research works printed but not linked: 37
- Pulkit Agrawal, Ashvin V. Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Gar- nett, editors, Proceedings of the 30th International Confer- ence on Neural Information Processing Systems (NeurIPS), volume 29.
- Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Ja- cob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ri- cardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. ArXiv Preprint, arXiv:2204.14198, 2022. · arXiv
- Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision- and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceed- ings of the IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), 2018.
- Stanislaw Antol, Aishwarya Agrawal, Ji- asen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2015.
- Renée Baillargeon, Maayan Sta- vans, Di Wu, Yael Gertner, Peipei Setoh, Audrey K. Kit- tredge, and Amélie Bernard. Object individuation and physical reasoning in infancy: An integrative account. Lan- guage learning and development: The official journal of the Society for Language Development, 8(1):4-46, 2012.
- Baillargeon, 1987] Renée Baillargeon. Object permanence in 3½-and 4½-month-old infants. Developmental Psychol- ogy, 23(5):655-664, 1987.
- Renee Baillargeon. Physical reasoning in infancy. In Michael S. Gazzaniga, editor, The Cognitive Neurosciences, pages 181-204. MIT Press, 1995.
- Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray kavukcuoglu. Interaction networks for learning about ob- jects, relations and physics. In Proceedings of the 30th International Conference on Neural Information Process- ing Systems (NeurIPS), pages 4509-4517, Red Hook, NY, USA, 2016.
- Yonatan Bisk, Ari Holtzman, Jesse Thoma- son, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nis- nevich, Nicolas Pinto, and Joseph Turian. Experience grounds language. In Bonnie Webber, Trevor Cohn, Yu- lan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8718-8735, Online, 2020. As- sociation for Computational Linguistics.
- Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Em- bodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
- Dasgupta et al., 2021a] Arijit Dasgupta, Jiafei Duan, Marcelo H Ang Jr, Yi Lin, Su-hua Wang, Renée Bail- largeon, and Cheston Tan. A benchmark for modeling violation-of-expectation in physical reasoning across event categories. ArXiv Preprint, arXiv:2111.0882, 2021.
- Dasgupta et al., 2021b] Arijit Dasgupta, Jiafei Duan, Marcelo H Ang Jr, and Cheston Tan. Avoe: A synthetic 3d dataset on understanding violation of expectation for artificial cognition. ArXiv Preprint, arXiv:2110.05836, 2021. · arXiv
- Duan et al., 2022] Jiafei Duan, Arijit Dasgupta, Jason Fis- cher, and Cheston Tan. A survey on machine learning approaches for modelling intuitive physics. In Lud De Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 5444-5452. International Joint Conferences on Artificial Intelligence Organization, 2022. Survey Track.
- Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Ar- mand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. ArXiv Preprint, arXiv:2305.05665, 2023. · arXiv
- Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Eneko Agirre, Johan Bos, Mona Diab, Suresh Manandhar, Yuval Marton, and Deniz Yuret, edi- tors, *SEM 2012: The First Joint Conference on Lexical and Computational Semantics -Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Seman- tic Evaluation (SemEval 2012), pages 394-398, Montréal, Canada, 2012. Association for Computational Linguistics.
- Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask lan- guage understanding. In International Conference on Learning Representations (ICLR), 2021.
- Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for com- positional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
- Kim and Spelke, 1992] In K. Kim and Elizabeth S. Spelke. Infants' sensitivity to effects of gravity on visible object motion. Journal of Experimental Psychology: Human Perception and Performance, 18(2):385-393, 1992.
- Kim and Spelke, 1999] In K. Kim and Elizabeth S. Spelke. Perception and understanding of effects of gravity and iner- tia on object motion. Developmental Science, 2(3):339-362, 1999.
- Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Ma- chine Learning (ICML), volume 48, pages 430-438. PMLR, 2016.
- Li et al., 2023a] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. ArXiv Preprint, arXiv:2301.12597, 2023. · arXiv
- Li et al., 2023b] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. ArXiv Preprint, arXiv:2305.06355, 2023. · arXiv
- Liu et al., 2023] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv Preprint, arXiv:2304.08485, 2023. · arXiv
- Maaz et al., 2023] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-ChatGPT: Towards detailed video understanding via large vision and language models. ArXiv Preprint, arXiv:2306.05424, 2023. · arXiv
- Margaret Mitchell, Jesse Dodge, Amit Goyal, Kota Yamaguchi, Karl Stratos, Xufeng Han, Alyssa Mensch, Alex Berg, Tamara Berg, and Hal Daumé III. Midge: Generating image descriptions from computer vi- sion detections. In Walter Daelemans, editor, Proceedings of the 13th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 747-756, Avignon, France, 2012. Association for Computational Lin- guistics.
- Luis S. Piloto, Ari Weinstein, Peter Battaglia, and Matthew Botvinick. Intuitive physics learn- ing in a deep-learning model inspired by developmental psychology. Nature Human Behaviour, 6(9):1257-1267, Sep 2022.
- Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning trans- ferable visual models from natural language supervision. ArXiv Preprint, arXiv:2103.00020, 2021. · arXiv
- Ronan Riochet, Mario Ynocente Cas- tro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. Intphys 2019: A bench- mark for visual intuitive physics understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):5016-5025, 2022.
- Elizabeth S. Spelke, Karen Breinlinger, Janet Macomber, and Kristen Jacobson. Origins of knowl- edge. Psychological review, 99(4):605-632, 1992.
- Elizabeth S. Spelke, Gary Katz, Susan E. Purcell, Sheryl M. Ehrlich, and Karen Breinlinger. Early knowledge of object motion: continuity and inertia. Cogni- tion, 51(1):131-176, 1994.
- Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. PandaGPT: One model to instruction-follow them all. ArXiv Preprint, arXiv:2305.16355, 2023. · arXiv
- Nicholas Watters, Andrea Tacchetti, Théophane Weber, Razvan Pascanu, Peter Battaglia, and Daniel Zoran. Visual interaction networks: Learning a physics simulator from video. In Proceedings of the 31st International Conference on Neural Information Process- ing Systems (NeurIPS), pages 4542-4550, Red Hook, NY, USA, 2017.
- Luca Weihs, Amanda Yuile, Renée Bail- largeon, Cynthia Fisher, Gary Marcus, Roozbeh Mottaghi, and Aniruddha Kembhavi. Benchmarking progress to infant-level physical reasoning in AI. Transactions on Machine Learning Research, 2022.
- Kexin Yi, Chuang Gan, Yunzhu Li, Push- meet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. CLEVRER: Collision events for video repre- sentation and reasoning. In International Conference on Learning Representations (ICLR), 2020.
- Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
- Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMa: An instruction-tuned audio-vidual lan- guage model for video understanding. ArXiv Preprint, arXiv:2306.02858, 2023. · arXiv
- Zhu et al., 2023] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xi- ang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large lan- guage models. ArXiv Preprint, arXiv:2304.10592, 2023. · arXiv
Cited by
Sorted by citations
Intuitive physics understanding emerges from self-supervised pretraining on natural videos
Garrido, Q., Ballas, N., Assran, M., Bardes, A., Najman, L., Rabbat, M., Dupoux, E., & LeCun, Y. (2025) · arXiv · doi:10.48550/arxiv.2502.11831 · Cited by 3
Metadata
- Authors
- Jassim, S., Holubar, M., Richter, A., Wolff, C., Ohmer, X., & Bruni, E.
- Affiliations
- Osnabrück University (DE)
- University of Amsterdam (NL)
- ORCID
- Ohmer, Xenia
- Venue
- Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence
- Published
- 2024-08
- Pages
- 6297-6305
- OpenAlex
- W4401024343
- Cited by
- 2
- References
- 0
- Citations by year
- 2025: 2
- FWCI
- 0.41
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Joint Conference on Artificial Intelligence (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- language grounding, intuitive physics, multimodal large language models, benchmark, video-based assessment, Unity simulation, object permanence, continuity, prompting strategies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
