Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index; none of the 6 sources its claims cite is checked for support yet.

Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties

Open access
Tung, Hsiao-Yu, Ding, Mingyu, Chen, Zhenfang, Bear, Daniel, Gan, Chuang, Tenenbaum, Josh, Yamins, Dan, Fan, Judith, Smith, Kevin · 2023 · Advances in Neural Information Processing Systems 36

Abstract

General physical scene understanding requires more than simply localizing and recognizing objects -- it requires knowledge that objects can have different latent properties (e.g., mass or elasticity), and that those properties affect the outcome of physical events. While there has been great progress in physical and video prediction models in recent years, benchmarks to test their performance typically do not require an understanding that objects have individual physical properties, or at best test only those properties that are directly observable (e.g., size or color). This work proposes a novel dataset and benchmark, termed Physion++, that rigorously evaluates visual physical prediction in artificial systems under circumstances where those predictions rely on accurate estimates of the latent physical properties of objects in the scene. Specifically, we test scenarios where accurate prediction relies on estimates of properties such as mass, friction, elasticity, and deformability, and where the values of those properties can only be inferred by observing how objects move and interact with other objects or fluids. We evaluate the performance of a number of state-of-the-art prediction models that span a variety of levels of learning vs. built-in knowledge, and compare that performance to a set of human predictions. We find that models that have been trained using standard regimes and datasets do not spontaneously learn to make inferences about latent properties, but also that models that encode objectness and physical states tend to make better predictions. However, there is still a huge gap between all models and human performance, and all models' predictions correlate poorly with those made by humans, suggesting that no state-of-the-art model is learning to make physical predictions in a human-like way. Project page: https://dingmyu.github.io/physion_v2/

Overview

CoChat Summary

TL;DR

Physion++ is a new benchmark of 9 simulated physical scenarios across mass, friction, elasticity, and deformability, designed to test whether models can infer latent object properties online and use them for contact prediction. State-of-the-art video and dynamics models mostly perform near chance, fail to use property inference, and correlate poorly with human predictions.

Key findings

  • Across the benchmark, most models perform only slightly above chance; unified training is typically worse than separate per-property training, and models with property inference perform similarly to those without it, indicating they are not using the inference phase.
  • DPI-Net achieves the best overall performance (60.2% with property, 57.5% without, 92.2% fully observed) and is the only model showing clear mass-property inference (75.9% with property vs. 60.1% without).
  • Humans perform reliably above chance but not especially well (60% accuracy); all video models perform slightly worse, and only DPI-Net is comparable. No model correlates well with humans: SlotFormer has the highest correlation (r=0.12 vs. split-half human correlation r=0.37).
  • Models underperform humans on “easy” trials but outperform humans on “hard” trials; most models do better than humans on deformability but worse on mass, friction, and elasticity. Ablations show performance near random at 200 training trials and saturating by 1000–2000 trials; adding explicit property estimation as an auxiliary task barely changes performance.

Methods

  • Stimuli were generated with the ThreeDWorld simulator across 9 scenarios covering mass, friction, elasticity, and deformability. Each property has a dynamics training set of 2,000 trials, a readout fitting set of 192 trials (96 pairs), and a testing set of 192 trials (96 pairs); paired test videos share the same first prediction-phase frame but differ in latent properties and outcomes.
  • Models evaluated include MCVD, pRESNET-mlp, pVGG-mlp, pDEIT-mlp, SlotFormer, ALOE, and DPI-Net. Training protocols were “separate” per property or “unified” across properties; testing variants were “w/ property,” “w/o property,” and “fully observed.” Readout models map learned representations to YES/NO contact predictions.
  • Human data came from 200 Prolific participants (50 per mechanical property), paid $15.50/hour. Each participant saw 32 stimuli from one property, balanced 16 YES and 16 NO; analyses compared model and human accuracy and Pearson correlation, including easy trials (>67% human accuracy) and hard trials (<33% human accuracy).

Limitations

  • The paper states that property inference may be difficult, models were not designed for it, 2,000 scenarios may be insufficient for it to emerge, and models may struggle with multi-stage long-term dependencies; task videos last 13–20 seconds, longer than typical training videos. The checklist also reports no error bars across random seeds.
  • The paper states the dataset can only be used to measure physical prediction based on latent property inference and that third-party contributions are not contemplated.
  • Evident from the text: all stimuli are simulator-generated TDW scenes rather than real-world videos, human data are limited to 200 participants (50 per property), and the benchmark’s test set is 192 trials per property.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.52202/075280-2929

Cites

49 of 49 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 0 Not valid · 3 Not checked · 83 Own results and interpretations

35 more claims

Create a free account to see all 86 claims and every source verdict.

See all claims — free account
  • The quote only partly states this claim. 3
  • Probably not a substantial claim. 31
  • It may state two separate claims as one. 22

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

3 of 3

Citations

Cites

Show all 16 references

Research works printed but not linked: 5

  1. Yichen Li, YingQiao Wang, Tal Boger, Kevin Smith, Samuel J Gershman, and Tomer Ullman. An approximate representation of objects underlies physical reasoning. PsyArXiv, 2022.
  2. Jiajun Wu, Ilker Yildirim, Joseph J Lim, Bill Freeman, and Josh Tenenbaum. Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. Advances in neural information processing systems, 28, 2015.
  3. Yunzhu Li, Toru Lin, Kexin Yi, Daniel Bear, Daniel L.K. Yamins, Jiajun Wu, Joshua B. Tenenbaum, and Antonio Torralba. Visual grounding of learned physical models. In International Conference on Machine Learning, 2020.
  4. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
  5. Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Yes] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes]

Cited by

See all 23

Metadata

Authors
Tung, H.-Y., Ding, M., Chen, Z., Bear, D., Gan, C., Tenenbaum, J., Yamins, D., Fan, J., & Smith, K.
Venue
Advances in Neural Information Processing Systems 36
Published
2023
Pages
67048-67068
ISSN
1049-5258 (ISSN-L)
OpenAlex
W7133212649
Cited by
8
References
16
Citations by year
2026: 4 · 2025: 1 · 2024: 1 · 2023: 2
FWCI
0.86
Type
conference paper
Language
en
Open access
Locations
  • neural information processing systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Robotics and Sensor-Based Localization
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
Keywords
online inference, physical properties

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties. https://app.cochat.ai/papers/10.52202/075280-2929
The paper
Tung, H.-Y., Ding, M., Chen, Z., Bear, D., Gan, C., Tenenbaum, J., Yamins, D., Fan, J., & Smith, K. (2023). Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties. Advances in Neural Information Processing Systems 36. https://doi.org/10.52202/075280-2929

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.