CoChat
Sign Up
PreprintCited by 183D Shape Modeling and AnalysisRobot Manipulation and LearningAdvanced Vision and Imaging

3DP3: 3D Scene Perception via Probabilistic Programming

Open access
Gothoskar, Nishad, Cusumano-Towner, Marco, Zinberg, Ben, Ghavamizadeh, Matin, Pollok, Falk, Garrett, Austin, Tenenbaum, Joshua B., Gutfreund, Dan +1 more·2021·arXiv
DOI 10.48550/arxiv.2111.00312arXiv 2111.00312
Open this paper in CoChat — free accountOpen PDF

Abstract

We present 3DP3, a framework for inverse graphics that uses inference in a structured generative model of objects, scenes, and images. 3DP3 uses (i) voxel models to represent the 3D shape of objects, (ii) hierarchical scene graphs to decompose scenes into objects and the contacts between them, and (iii) depth image likelihoods based on real-time graphics. Given an observed RGB-D image, 3DP3's inference algorithm infers the underlying latent 3D scene, including the object poses and a parsimonious joint parametrization of these poses, using fast bottom-up pose proposals, novel involutive MCMC updates of the scene graph structure, and, optionally, neural object detectors and pose estimators. We show that 3DP3 enables scene understanding that is aware of 3D shape, occlusion, and contact structure. Our results demonstrate that 3DP3 is more accurate at 6DoF object pose estimation from real images than deep learning baselines and shows better generalization to challenging scenes with novel viewpoints, contact, and partial observability.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2111.00312

Cites

2 of 2 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • For AI systems, probabilistic programs have been shown to be a promising formalism for world modeling.

    “For AI systems, probabilistic programs have been shown to be a promising formalism for world modeling (e.g., Milch et al., 2007;Matheos et al., 2020;Gothoskar et al., 2021;Fremont et al., 2022).”

    From Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems (2024) · cites this paper as Gothoskar et al., 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Part of this paper’s reference list could not be split into single entries.

Flagged first, then by citations

Research works printed but not linked: 39

  1. [1][1] Sameer Agarwal, Noah Snavely, Steven M Seitz, and Richard Szeliski. Bundle Adjustment in the Large. In ECCV 2010: Proceedings of the European Conference on Computer Vision, pages 29–42. Springer, 2010.
  2. [2][2] Christophe Andrieu, Gareth O Roberts, et al. The pseudo-marginal approach for efficient Monte Carlo computations. The Annals of Statistics, 37(2):697–725, 2009.
  3. [3][3] Iro Armeni, Zhi-Yang He, JunYoung Gwak, Amir R Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera. In ICCV 2019: Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5664–5673, 2019.
  4. [5][5] Sid Yingze Bao and Silvio Savarese. Semantic structure from motion. In CVPR 2011: Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2025–2032. IEEE, 2011.
  5. [6][6] Paul J Besl and Neil D McKay. Method for registration of 3D shapes. In Sensor Fusion IV: Control Paradigms and Data Structures, volume 1611, pages 586–606. International Society for Optics and Photonics, 1992.
  6. [7][7] Berk Calli, Aaron Walsman, Arjun Singh, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M. Dollar. Benchmarking in Manipulation Research: Using the Yale-CMU-Berkeley Object and Model Set. IEEE Robotics Automation Magazine, 22(3):36–52, 2015.
  7. [8][8] Arthur Cayley. A Theorem on Trees, volume 13 of Cambridge Library Collection - Mathematics, page 26–28. Cambridge University Press, 2009.
  8. [9][9] Xiaotong Chen, Rui Chen, Zhiqiang Sui, Zhefan Ye, Yanqi Liu, R Iris Bahar, and Odest Chad- wicke Jenkins. GRIP: Generative Robust Inference and Perception for Semantic Robot Manip- ulation in Adversarial Environments. In IROS 2019: IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3988–3995. IEEE, 2019.
  9. [10][10] Yixin Chen, Siyuan Huang, Tao Yuan, Siyuan Qi, Yixin Zhu, and Song-Chun Zhu. Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense. In ICCV 2019: Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8648–8657, 2019.
  10. [11][11] James H. Clark. Hierarchical Geometric Models for Visible Surface Algorithms. Commun. ACM, 19(10):547–554, October 1976.
  11. [12][12] Marco Cusumano-Towner, Alexander K Lew, and Vikash K Mansinghka. Automating Involutive MCMC using Probabilistic and Differentiable Programming. arXiv preprint arXiv:2007.09871, 2020. · arXiv
  12. [13][13] Marco F Cusumano-Towner, Feras A Saad, Alexander K Lew, and Vikash K Mansinghka. Gen: A General-Purpose Probabilistic Programming System with Programmable Inference. In PLDI 2019: Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 221–236, 2019.
  13. [14][14] Fei Deng, Zhuo Zhi, and Sungjin Ahn. Generative Hierarchical Models for Parts, Objects, and Scenes. arXiv preprint arXiv:1910.09119, 2019. · arXiv
  14. [15][15] Fei Deng, Zhuo Zhi, Donghun Lee, and Sungjin Ahn. Generative Scene Graph Networks. In ICLR 2021: International Conference on Learning Representations, 2021.
  15. [16][16] Xinke Deng, Arsalan Mousavian, Yu Xiang, Fei Xia, Timothy Bretl, and Dieter Fox. PoseRBPF: A Rao-Blackwellized Particle Filter for 6-D Object Pose Tracking. IEEE Transactions on Robotics, 2021.
  16. [18][18] Yilun Du, Zhijian Liu, Hector Basevi, Ales Leonardis, Bill Freeman, Josh Tenenbaum, and Jiajun Wu. Learning to Exploit Stability for 3D Scene Parsing. In S. Bengio, H. Wallach, 11 https://vrs.amsi.org.au/wp-content/uploads/sites/78/2017/05/baldwin_jonathan_vrs-report.pdf https://vrs.amsi.org.au/wp-content/uploads/sites/78/2017/05/baldwin_jonathan_vrs-report.pdf H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, NIPS 2018: Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  17. [19][19] SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, Infer, Repeat: Fast Scene Understanding with Generative Models. NIPS 2016: Advances in Neural Information Processing Systems, 29:3225–3233, 2016.
  18. [20][20] Jean Gallier. The Quaternions and the Spaces S3, SU(2), SO(3), and RP3, pages 248–266. Springer New York, New York, NY, 2001.
  19. [23][23] Tomáš Hodaň, Frank Michel, Eric Brachmann, Wadim Kehl, Anders Glent Buch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, Caner Sahin, Fabian Manhardt, Federico Tombari, Tae-Kyun Kim, Jiří Matas, and Carsten Rother. BOP: Benchmark for 6D object pose estimation. ECCV 2018: Proceedings of the European Conference on Computer Vision, 2018.
  20. [24][24] Siyuan Huang, Siyuan Qi, Yixin Zhu, Yinxue Xiao, Yuanlu Xu, and Song-Chun Zhu. Holistic 3D scene parsing and reconstruction from a single RGB image. In ECCV 2018: Proceedings of the European Conference on Computer Vision, pages 187–203, 2018.
  21. [25][25] D Knill D Kersten and A Yuille. Introduction: A Bayesian Formulation of Visual Perception. Perception as Bayesian inference, pages 1–21, 1996.
  22. [27][27] Nilesh Kulkarni, Ishan Misra, Shubham Tulsiani, and Abhinav Gupta. 3D-RelNet: Joint object and relational network for 3D prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2212–2221, 2019.
  23. [28][28] Tejas D Kulkarni, Pushmeet Kohli, Joshua B Tenenbaum, and Vikash Mansinghka. Picture: A Probabilistic Programming Language for Scene Perception. In CVPR 2015: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4390–4399, 2015.
  24. [29][29] J.M. Lee. Introduction to Smooth Manifolds. Graduate Texts in Mathematics. Springer, 2003.
  25. [31][31] Vikash K Mansinghka, Tejas D Kulkarni, Yura N Perov, and Josh Tenenbaum. Approxi- mate Bayesian Image Interpretation using Generative Probabilistic Graphics Programs. NIPS 2013:Advances in Neural Information Processing Systems, 26:1520–1528, 2013.
  26. [32][32] Vikash K Mansinghka, Ulrich Schaechtle, Shivam Handa, Alexey Radul, Yutian Chen, and Martin Rinard. Probabilistic Programming with Programmable Inference. In PLDI 2018: Proceedings of the 39th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 603–616, 2018.
  27. [33][33] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV 2020: Proceedings of the European Conference on Computer Vision, pages 405–421. Springer, 2020.
  28. [34][34] Pol Moreno, Christopher KI Williams, Charlie Nash, and Pushmeet Kohli. Overcoming Occlusion with Inverse Graphics. In ECCV 2016: Proceedings of the European Conference on Computer Vision, pages 170–185. Springer, 2016.
  29. [35][35] Lukasz Romaszko, Christopher K. I. Williams, Pol Moreno, and Pushmeet Kohli. Vision-as- Inverse-Graphics: Obtaining a Rich 3D Explanation of a Scene from a Single Image. In ICCVW 2017: IEEE International Conference on Computer Vision Workshops, pages 940–948, 2017. 12
  30. [36][36] Antoni Rosinol, Arjun Gupta, Marcus Abate, Jingnan Shi, and Luca Carlone. 3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans. arXiv preprint arXiv:2002.06289, 2020. · arXiv
  31. [37][37] Ruwen Schnabel, Roland Wahl, and Reinhard Klein. Efficient RANSAC for point-cloud shape detection. In Computer graphics forum, volume 26, pages 214–226. Wiley Online Library, 2007.
  32. [38][38] Johannes L. Schönberger and Jan-Michael Frahm. Structure-from-Motion Revisited. In CVPR 2016: IEEE Conference on Computer Vision and Pattern Recognition, pages 4104–4113, 2016.
  33. [39][39] Meng Tian, Liang Pan, Marcelo H Ang Jr, and Gim Hee Lee. Robust 6D Object Pose Estimation by Learning RGB-D Features. In ICRA 2020: International Conference on Robotics and Automation, 2020.
  34. [40][40] Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, and Stan Birchfield. Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects. In CORL 2018: Conference on Robot Learning, 2018.
  35. [41][41] Zhuowen Tu and Song-Chun Zhu. Image segmentation by data-driven Markov chain Monte Carlo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(5):657–673, 2002.
  36. [42][42] Chen Wang, Danfei Xu, Yuke Zhu, Roberto Martín-Martín, Cewu Lu, Li Fei-Fei, and Silvio Savarese. DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion. In CVPR 2019: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3343–3352, 2019.
  37. [44][44] Anna Yershova, Swati Jain, Steven M. LaValle, and Julie C. Mitchell. Generating Uniform Incremental Grids on SO(3) Using the Hopf Fibration. The International Journal of Robotics Research, 29(7):801–812, 2010.
  38. [45][45] Alan Yuille and Daniel Kersten. Vision as Bayesian Inference: Analysis by Synthesis? Trends in Cognitive Sciences, 10(7):301–308, 2006.
  39. [46][46] Song Chun Zhu and Alan Yuille. Region competition: Unifying snakes, region growing, and bayes/mdl for multiband image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(9):884–900, 1996.

Cited by

Sorted by citations

Authors
Gothoskar, N., Cusumano-Towner, M., Zinberg, B., Ghavamizadeh, M., Pollok, F., Garrett, A., Tenenbaum, J. B., Gutfreund, D., & Mansinghka, V. K.
Affiliations
IBM (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2021
Volume
34
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2111.00312
arXiv
2111.00312
OpenAlex
W3209050319
Cited by
18
References
2
Citations by year
2026: 1 · 2025: 3 · 2024: 7 · 2023: 4 · 2022: 3
Type
posted content
Language
en
Open access
Licence
creativecommons.org/licenses/by/4.0/legalcode
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
  • neural information processing systems
Funders
  • Defense Advanced Research Projects Agency
  • Defence Science and Technology Agency - Singapore
Topics
  • Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
  • Physical Sciences › Engineering › Control and Systems Engineering › Robot Manipulation and Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Vision and Imaging
Keywords
inverse graphics, 3D scene perception, probabilistic programming, generative models, voxel model, RGB-D, Markov chain Monte Carlo, occlusion, contact structures, scene understanding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review