CoChat
Sign Up
PreprintCited by 7Advanced Vision and ImagingRobot Manipulation and LearningAdvanced Neural Network Applications

The DOI matches its DataCite record; 1 of the 1 references it cites were found in a registry or an index.

Occlusion resistant learning of intuitive physics from videos

Open access
Riochet, Ronan, Sivic, Josef, Laptev, Ivan, Dupoux, Emmanuel·2020·arXiv
DOI 10.48550/arxiv.2005.00069arXiv 2005.00069
Open this paper in CoChat — free accountOpen PDF

Abstract

To reach human performance on complex tasks, a key ability for artificial systems is to understand physical interactions between objects, and predict future outcomes of a situation. This ability, often referred to as intuitive physics, has recently received attention and several methods were proposed to learn these physical rules from video sequences. Yet, most of these methods are restricted to the case where no, or only limited, occlusions occur. In this work we propose a probabilistic formulation of learning intuitive physics in 3D scenes with significant inter-object occlusions. In our formulation, object positions are modeled as latent variables enabling the reconstruction of the scene. We then propose a series of approximations that make this problem tractable. Object proposals are linked across frames using a combination of a recurrent interaction network, modeling the physics in object space, and a compositional renderer, modeling the way in which objects project onto pixel space. We demonstrate significant improvements over state-of-the-art in the intuitive physics benchmark of IntPhys. We apply our method to a second dataset with increasing levels of occlusions, showing it realistically predicts segmentation masks up to 30 frames in the future. Finally, we also show results on predicting motion of objects in real videos.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2005.00069

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Show all 1 references

Research works printed but not linked: 41

  1. Agrawal, P., Nair, A., Abbeel, P., Malik, J., and Levine, S. Learning to Poke by Poking: Experiential Learning of Intuitive Physics. CoRR , abs/1606.07419, 2016. URL http://arxiv.org/abs/1606.07419 . · arXiv
  2. Baillargeon, R. and Carey, S. Core Cognition and Beyond. In Pauen, S. (ed.), Early childhood development and later outcome , pp. 33-65. Cambridge University Press, New York, 2012.
  3. Battaglia, P., Pascanu, R., Lai, M., Rezende, D. J., and others. Interaction networks for learning about objects, relations and physics. In Advances in Neural Information Processing Systems , pp. 4502-4510, 2016.
  4. Battaglia, P. W., Hamrick, J. B., and Tenenbaum, J. B. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences of the United States of America , 110(45), 2013. URL http: //www.pnas.org/content/110/45/18327 .
  5. Burgess, C. P., Matthey, L., Watters, N., Kabra, R., Higgins, I., Botvinick, M., and Lerchner, A. Monet: Unsupervised scene decomposition and representation. CoRR , abs/1901.11390, 2019. URL http://arxiv.org/ abs/1901.11390 . · arXiv
  6. Carey, S. The origin of concepts . Oxford series in cognitive development. Oxford University Press, Oxford ; New York, 2009. ISBN 978-0-19-536763-8 0-19-536763-4.
  7. Chang, M. B., Ullman, T., Torralba, A., and Tenenbaum, J. B. A compositional object-based approach to learning physical dynamics. arXiv preprint arXiv:1612.00341 , 2016. · arXiv
  8. Ebert, F., Finn, C., Dasari, S., Xie, A., Lee, A., and Levine, S. Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control. arXiv:1812.00568 [cs] , December 2018. URL http://arxiv.org/abs/1812.00568 . arXiv: 1812.00568. · arXiv
  9. Ehrhardt, S., Monszpart, A., Mitra, N. J., and Vedaldi, A. Learning A Physical Long-term Predictor. CoRR , abs/1703.00247, 2017a. · arXiv
  10. Finn, C. and Levine, S. Deep visual foresight for planning robot motion. In Robotics and Automation (ICRA), 2017 IEEE International Conference on , pp. 2786-2793. IEEE, 2017.
  11. Finn, C., Goodfellow, I., and Levine, S. Unsupervised learning for physical interaction through video prediction. In Advances in neural information processing systems , pp. 64-72, 2016.
  12. Greff, K., Kaufman, R. L., Kabra, R., Watters, N., Burgess, C., Zoran, D., Matthey, L., Botvinick, M., and Lerchner, A. Multi-Object Representation Learning with Iterative Variational Inference. arXiv:1903.00450 [cs, stat] , March 2019. URL http://arxiv.org/abs/ 1903.00450 . arXiv: 1903.00450. · arXiv
  13. He, K., Gkioxari, G., Dollár, P., and Girshick, R. Mask R-CNN. arXiv:1703.06870 [cs] , January 2018. URL http://arxiv.org/abs/1703.06870 . arXiv: 1703.06870. · arXiv
  14. Janner, M., Levine, S., Freeman, W. T., Tenenbaum, J. B., Finn, C., and Wu, J. Reasoning About Physical Interactions with Object-Oriented Prediction and Planning. arXiv:1812.10972 [cs, stat] , December 2018. URL http://arxiv.org/abs/1812.10972 . arXiv: 1812.10972. · arXiv
  15. Jelinek, F. Statistical methods for speech recognition . MIT press, 1997.
  16. Kellman, P. J. and Spelke, E. S. Perception of partly occluded objects in infancy. Cognitive psychology , 15(4): 483-524, 1983.
  17. Kosiorek, A., Kim, H., Teh, Y. W., and Posner, I. Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31 , pp. 8606-8616. Curran Associates, Inc., 2018.
  18. Lan, T., Chen, T.-C., and Savarese, S. A hierarchical representation for future action prediction. In European Conference on Computer Vision , pp. 689-704. Springer, 2014.
  19. Lerer, A., Gross, S., and Fergus, R. Learning physical intuition of block towers by example. arXiv preprint arXiv:1603.01312 , 2016. · arXiv
  20. Li, W., Leonardis, A., and Fritz, M. To Fall Or Not To Fall: AVisual Approach to Physical Stability Prediction. arXiv preprint , 2016a. URL https://arxiv.org/abs/ 1604.00066 .
  21. Li, W., Leonardis, A., and Fritz, M. Visual stability prediction and its application to manipulation. arXiv preprint arXiv:1609.04861 , 2016b. · arXiv
  22. Li, Y., Wu, J., Tedrake, R., Tenenbaum, J. B., and Torralba, A. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. CoRR , abs/1810.01566, 2018. URL http://arxiv.org/ abs/1810.01566 . · arXiv
  23. Liu, R., Lehman, J., Molino, P., Such, F. P., Frank, E., Sergeev, A., and Yosinski, J. An intriguing failing of convolutional neural networks and the coordconv solution. CoRR , abs/1807.03247, 2018. URL http://arxiv. org/abs/1807.03247 . · arXiv
  24. Marco Fraccaro, Simon Kamronn, U. P. O. W. A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning. Advances in Neural Information Processing Systems 30, NIPS , 2017.
  25. Mathieu, M., Couprie, C., and LeCun, Y. Deep multiscale video prediction beyond mean square error. arXiv preprint arXiv:1511.05440 , 2015. · arXiv
  26. Minderer, M., Sun, C., Villegas, R., Cole, F., Murphy, K., and Lee, H. Unsupervised learning of object structure and dynamics from videos. CoRR , abs/1906.07889, 2019. URL http://arxiv.org/abs/1906.07889 . · arXiv
  27. Mirza, M., Courville, A., and Bengio, Y. Generalizable features from unsupervised learning. ICLR Workshop submission , 2017. URL https://openreview.net/ pdf?id=BynzZolYg .
  28. Mrowca, D., Zhuang, C., Wang, E., Haber, N., Fei-Fei, L. F., Tenenbaum, J., and Yamins, D. L. Flexible neural representation for physics prediction. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31 , pp. 8813-8824. Curran Associates, Inc., 2018.
  29. Riochet, R., Ynocente Castro, M., Bernard, M., Lerer, A., Fergus, R., Izard, V., and Dupoux, E. IntPhys: A Framework and Benchmark for Visual Intuitive Physics Reasoning. ArXiv e-prints , March 2018.
  30. Saxe, R. and Carey, S. The perception of causality in infancy. Acta psychologica , 123(1):144-165, 2006.
  31. Smith, K. A., Mei, L., Yao, S., jun Wu, J., Spelke, E. S., Tenenbaum, J., and Ullman, T. Modeling expectation violation in intuitive physics with coarse probabilistic object representations. In NeurIPS , 2019.
  32. Spelke, E. S., Kestenbaum, R., Simons, D. J., and Wein, D. Spatiotemporal continuity, smoothness of motion and object identity in infancy. British Journal of Developmental Psychology , 13(2):113-142, 1995.
  33. van Steenkiste, S., Chang, M., Greff, K., and Schmidhuber, J. Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions. arXiv:1802.10353 [cs] , February 2018. URL http://arxiv.org/abs/1802.10353 . arXiv: 1802.10353. · arXiv
  34. Watters, N., Tacchetti, A., Weber, T., Pascanu, R., Battaglia, P., and Zoran, D. Visual Interaction Networks. CoRR , abs/1706.01433, 2017. URL http://arxiv.org/ abs/1706.01433 . · arXiv
  35. Wichers, N., Villegas, R., Erhan, D., and Lee, H. Hierarchical long-term video prediction without supervision. CoRR , abs/1806.04768, 2018. URL http://arxiv. org/abs/1806.04768 . · arXiv
  36. Wu, J., Lim, J. J., Zhang, H., Tenenbaum, J. B., and Freeman, W. T. Physics 101: Learning Physical Object Properties from Unlabeled Videos. In BMVC , 2016.
  37. Wu, J., Lu, E., Kohli, P., Freeman, B., and Tenenbaum, J. Learning to see physics via visual de-animation. In Advances in Neural Information Processing Systems , pp. 152-163, 2017a.
  38. Wu, J., Tenenbaum, J. B., and Kohli, P. Neural scene derendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pp. 699-707, 2017b.
  39. Xu, Z., Wu, J., Zeng, A., Tenenbaum, J. B., and Song, S. Densephysnet: Learning dense physical object representations via multi-step dynamic interactions. CoRR , abs/1906.03853, 2019. URL http://arxiv.org/ abs/1906.03853 . · arXiv
  40. Xue, T., Wu, J., Bouman, K., and Freeman, B. Visual dynamics: Probabilistic future frame synthesis via cross convolutional networks. In Advances in Neural Information Processing Systems , pp. 91-99, 2016.
  41. Zhu, G., Huang, Z., and Zhang, C. Object-Oriented Dynamics Predictor. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31 , pp. 9826-9837. Curran Associates, Inc., 2018. URL http://papers.nips.cc/paper/ 8187-object-oriented-dynamics-predictor. pdf .

Cited by

Sorted by citations

See all 7

Metadata

Authors
Riochet, R., Sivic, J., Laptev, I., & Dupoux, E.
Affiliations
École Normale Supérieure - PSL (FR)
ORCID
Riochet, Ronan
Venue
arXiv
Publisher
arXiv
Published
2020
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2005.00069
arXiv
2005.00069
OpenAlex
W3005274572
Cited by
7
References
1
Citations by year
2023: 2 · 2021: 4 · 2020: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • HAL (submittedVersion, other-oa)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
Keywords
intuitive physics, occlusion, stochastic modeling, latent variables, object motion prediction, 3D scene reconstruction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review