The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index; none of the 6 sources its claims cite is checked for support yet.
Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties
Open accessAbstract
General physical scene understanding requires more than simply localizing and recognizing objects -- it requires knowledge that objects can have different latent properties (e.g., mass or elasticity), and that those properties affect the outcome of physical events. While there has been great progress in physical and video prediction models in recent years, benchmarks to test their performance typically do not require an understanding that objects have individual physical properties, or at best test only those properties that are directly observable (e.g., size or color). This work proposes a novel dataset and benchmark, termed Physion++, that rigorously evaluates visual physical prediction in artificial systems under circumstances where those predictions rely on accurate estimates of the latent physical properties of objects in the scene. Specifically, we test scenarios where accurate prediction relies on estimates of properties such as mass, friction, elasticity, and deformability, and where the values of those properties can only be inferred by observing how objects move and interact with other objects or fluids. We evaluate the performance of a number of state-of-the-art prediction models that span a variety of levels of learning vs. built-in knowledge, and compare that performance to a set of human predictions. We find that models that have been trained using standard regimes and datasets do not spontaneously learn to make inferences about latent properties, but also that models that encode objectness and physical states tend to make better predictions. However, there is still a huge gap between all models and human performance, and all models' predictions correlate poorly with those made by humans, suggesting that no state-of-the-art model is learning to make physical predictions in a human-like way. Project page: https://dingmyu.github.io/physion_v2/
Overview
CoChat Summary
TL;DR
Physion++ is a new benchmark of 9 simulated physical scenarios across mass, friction, elasticity, and deformability, designed to test whether models can infer latent object properties online and use them for contact prediction. State-of-the-art video and dynamics models mostly perform near chance, fail to use property inference, and correlate poorly with human predictions.
Key findings
- Across the benchmark, most models perform only slightly above chance; unified training is typically worse than separate per-property training, and models with property inference perform similarly to those without it, indicating they are not using the inference phase.
- DPI-Net achieves the best overall performance (60.2% with property, 57.5% without, 92.2% fully observed) and is the only model showing clear mass-property inference (75.9% with property vs. 60.1% without).
- Humans perform reliably above chance but not especially well (60% accuracy); all video models perform slightly worse, and only DPI-Net is comparable. No model correlates well with humans: SlotFormer has the highest correlation (r=0.12 vs. split-half human correlation r=0.37).
- Models underperform humans on “easy” trials but outperform humans on “hard” trials; most models do better than humans on deformability but worse on mass, friction, and elasticity. Ablations show performance near random at 200 training trials and saturating by 1000–2000 trials; adding explicit property estimation as an auxiliary task barely changes performance.
Methods
- Stimuli were generated with the ThreeDWorld simulator across 9 scenarios covering mass, friction, elasticity, and deformability. Each property has a dynamics training set of 2,000 trials, a readout fitting set of 192 trials (96 pairs), and a testing set of 192 trials (96 pairs); paired test videos share the same first prediction-phase frame but differ in latent properties and outcomes.
- Models evaluated include MCVD, pRESNET-mlp, pVGG-mlp, pDEIT-mlp, SlotFormer, ALOE, and DPI-Net. Training protocols were “separate” per property or “unified” across properties; testing variants were “w/ property,” “w/o property,” and “fully observed.” Readout models map learned representations to YES/NO contact predictions.
- Human data came from 200 Prolific participants (50 per mechanical property), paid $15.50/hour. Each participant saw 32 stimuli from one property, balanced 16 YES and 16 NO; analyses compared model and human accuracy and Pearson correlation, including easy trials (>67% human accuracy) and hard trials (<33% human accuracy).
Limitations
- The paper states that property inference may be difficult, models were not designed for it, 2,000 scenarios may be insufficient for it to emerge, and models may struggle with multi-stage long-term dependencies; task videos last 13–20 seconds, longer than typical training videos. The checklist also reports no error bars across random seeds.
- The paper states the dataset can only be used to measure physical prediction based on latent property inference and that third-party contributions are not contemplated.
- Evident from the text: all stimuli are simulator-generated TDW scenes rather than real-world videos, human data are limited to 200 participants (50 per property), and the benchmark’s test set is 192 trials per property.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 3 Not checked · 83 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We generate 2000 trials for each mechanical property without YES/NO labels for dynamics pretraining.”
§ Download and unzip the three data files --Training_data ... --Readout_data ... --Testing_data --Folder organized by physical properties # folders named with copy0 and copy1 indicate two matched trials ( e .g ., the same video location from copy0 will have a different property / outcome in copy1 but the same initial conditions ) --Subfolder organized by scenarios and objects # e .g .bouncy_platformu s e _ b l o c k e r _ w i t h _ h o l e =0 -target = cube --[ id ]. json # object id and instance information over time --[ id ]. pkl # meta information of the physical event and the video --[ id ] _image .mp4 # raw input , RGB video --[ id ] _seg .mp4 # segmentation masks of the video --[ id ] _map .png # indicating the yellow and red objects # Below are lists of test filenames and ids (192 entries per physical property ) --Mass .csv --Friction .csv --Elasticity .csv --Deformability .csv > Training set.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“yet all video models are performing slightly worse than humans.”
§ Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties > 4.2 Human experiments
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Only DPI-Net, which operates on ground truth states from the physics engine performs comparably to humans.”
§ Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties > 4.2 Human experiments
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“DPI-Net achieves the best overall performance”
§ Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties > 4.1 Model Performance
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Most models perform only slightly above chance”
§ Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties > 4.1 Model Performance
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
35 more claims
Create a free account to see all 86 claims and every source verdict.
- The quote only partly states this claim. 3
- Probably not a substantial claim. 31
- It may state two separate claims as one. 22
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties > 4.1 Model Performance
“Based on these representations we train a readout model as logistic regression.”
§ Download and unzip the three data files --Training_data ... --Readout_data ... --Testing_data --Folder organized by physical properties # folders named with copy0 and copy1 indicate two matched trials ( e .g ., the same video location from copy0 will have a different property / outcome in copy1 but the same initial conditions ) --Subfolder organized by scenarios and objects # e .g .bouncy_platformu s e _ b l o c k e r _ w i t h _ h o l e =0 -target = cube --[ id ]. json # object id and instance information over time --[ id ]. pkl # meta information of the physical event and the video --[ id ] _image .mp4 # raw input , RGB video --[ id ] _seg .mp4 # segmentation masks of the video --[ id ] _map .png # indicating the yellow and red objects # Below are lists of test filenames and ids (192 entries per physical property ) --Mass .csv --Friction .csv --Elasticity .csv --Deformability .csv > D Pipelines and settings
“We implement a multilayer perceptron (MLP) with intermediate dimensions of 256 and 64 as our readout model, which is trained on rollout scene representations from the readout set to classify whether the two cued objects contact.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| | | fully observed | 59.9 | 63.9 | 61.2 | 56.0 | 55.5 | 65.1 | 92.2 |”
§ Download and unzip the three data files --Training_data ... --Readout_data ... --Testing_data --Folder organized by physical properties # folders named with copy0 and copy1 indicate two matched trials ( e .g ., the same video location from copy0 will have a different property / outcome in copy1 but the same initial conditions ) --Subfolder organized by scenarios and objects # e .g .bouncy_platformu s e _ b l o c k e r _ w i t h _ h o l e =0 -target = cube --[ id ]. json # object id and instance information over time --[ id ]. pkl # meta information of the physical event and the video --[ id ] _image .mp4 # raw input , RGB video --[ id ] _seg .mp4 # segmentation masks of the video --[ id ] _map .png # indicating the yellow and red objects # Below are lists of test filenames and ids (192 entries per physical property ) --Mass .csv --Friction .csv --Elasticity .csv --Deformability .csv > G Checklist
Reported 56 · recomputed 59.65
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“We selected 192 trials from each scenario, then shuffle and split the 192 trials into 8 disjoint sets, where all sets have half of the trials containing contact between the red and yellow objects, and half do not.Each participant was shown 32 stimuli drawn from one of these disjoint sets of trials drawn from a single mechanical property”
§ Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties > 4.2 Human experiments
Reported 32 · recomputed 24
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments
Ionescu, C., Papava, D., Olaru, V., & Sminchisescu, C. (2014) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2013.248 · Cited by 3,653
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Learning Latent Dynamics for Planning from Pixels
Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., & Davidson, J. (2018) · arXiv · doi:10.48550/arxiv.1811.04551 · Cited by 708
Deep visual foresight for planning robot motion
Finn, C., & Levine, S. (2017) · 2017 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2017.7989324 · Cited by 599
CLEVRER: CoLlision Events for Video REpresentation and ReasoningPreprint
Yi, K., Gan, C., Li, Y., Kohli, P., Wu, J., Torralba, A. B., & Tenenbaum, J. B. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.01442 · Cited by 578
Self-citationDream to Control: Learning Behaviors by Latent Imagination
Hafner, D., Lillicrap, T., Ba, J., & Norouzi, M. (2019) · arXiv · doi:10.48550/arxiv.1912.01603 · Cited by 558
Learning to Poke by Poking: Experiential Learning of Intuitive PhysicsPreprint
Agrawal, P., Nair, A., Abbeel, P., Malik, J., & Levine, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.07419 · Cited by 404
Research works printed but not linked: 5
- Yichen Li, YingQiao Wang, Tal Boger, Kevin Smith, Samuel J Gershman, and Tomer Ullman. An approximate representation of objects underlies physical reasoning. PsyArXiv, 2022.
- Jiajun Wu, Ilker Yildirim, Joseph J Lim, Bill Freeman, and Josh Tenenbaum. Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. Advances in neural information processing systems, 28, 2015.
- Yunzhu Li, Toru Lin, Kexin Yi, Daniel Bear, Daniel L.K. Yamins, Jiajun Wu, Joshua B. Tenenbaum, and Antonio Torralba. Visual grounding of learned physical models. In International Conference on Machine Learning, 2020.
- Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
- Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Yes] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes]
Cited by
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 154
Self-citationContPhy: Continuum Physical Concept Learning and Reasoning from VideosPreprint
Zheng, Z., Yan, X., Chen, Z., Wang, J., Lim, Q. Z. E., Tenenbaum, J. B., & Gan, C. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.06119 · Cited by 14
Self-citationDo Vision-Language Models Have Internal World Models? Towards an Atomic EvaluationPreprint
Gao, Q., Pi, X., Liu, K., Chen, J., Yang, R., Huang, X., Fang, X., Sun, L., Kishore, G., Ai, B., et al. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.21876 · Cited by 12
Self-citationBenchmarks for Physical Reasoning AIPreprint
Melnik, A., Schiewer, R., Lange, M., Muresanu, A. I., Saeidi, M., Garg, A., & Ritter, H. J. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.10728 · Cited by 10
Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluation
Gao, Q., Pi, X., Liu, K., Chen, J., Yang, R., Huang, X., Fang, X., Sun, L., Kishore, G., Ai, B., et al. (2025) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2025.findings-acl.1342 · Cited by 7
Self-citation
Metadata
- Authors
- Tung, H.-Y., Ding, M., Chen, Z., Bear, D., Gan, C., Tenenbaum, J., Yamins, D., Fan, J., & Smith, K.
- Venue
- Advances in Neural Information Processing Systems 36
- Published
- 2023
- Pages
- 67048-67068
- ISSN
- 1049-5258 (ISSN-L)
- OpenAlex
- W7133212649
- Cited by
- 8
- References
- 16
- Citations by year
- 2026: 4 · 2025: 1 · 2024: 1 · 2023: 2
- FWCI
- 0.86
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- neural information processing systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotics and Sensor-Based Localization
- Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
- Keywords
- online inference, physical properties
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties. https://app.cochat.ai/papers/10.52202/075280-2929
- The paper
- Tung, H.-Y., Ding, M., Chen, Z., Bear, D., Gan, C., Tenenbaum, J., Yamins, D., Fan, J., & Smith, K. (2023). Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties. Advances in Neural Information Processing Systems 36. https://doi.org/10.52202/075280-2929
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.