CoChat
Sign Up
Cited by 2383D Shape Modeling and AnalysisAdvanced Neural Network Applications3D Surveying and Cultural Heritage

Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes From a Single Image

Open access
Nie, Yinyu, Han, Xiaoguang, Guo, Shihui, Zheng, Yujian, Chang, Jian, Zhang, Jian Jun·2020·2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI 10.1109/cvpr42600.2020.00013
Open this paper in CoChat — free accountOpen PDF

Abstract

Semantic reconstruction of indoor scenes refers to both scene understanding and object reconstruction. Existing works either address one part of this problem or focus on independent objects. In this paper, we bridge the gap between understanding and reconstruction, and propose an end-to-end solution to jointly reconstruct room layout, object bounding boxes and meshes from a single image. Instead of separately resolving scene understanding and object reconstruction, our method builds upon a holistic scene context and proposes a coarse-to-fine hierarchy with three components: 1. room layout with camera pose; 2. 3D object bounding boxes; 3. object meshes. We argue that understanding the context of each component can assist the task of parsing the others, which enables joint understanding and reconstruction. The experiments on the SUN RGB-D and Pix3D datasets demonstrate that our method consistently outperforms existing methods in indoor layout estimation, 3D object detection and mesh reconstruction.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/cvpr42600.2020.00013

Cites

6 of 6 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Gemini 2.0's 3D detection performance is comparable to existing state-of-the-art expert models.

    “Gemini 2.0's 3D detection performance is comparable to existing state-of-the-art expert models”

    From Gemini Robotics: Bringing AI into the Physical World (2025) · cites this paper as Nie et al., 2020

    The 3D Object Detection discussion (Section 4.3, p. 6) compares the paper’s own method with earlier methods, not Gemini 2.0. Searches for Gemini 2.0 and a comparable 3D detection benchmark found no passage assessing its performance, so this copy cannot verify the claim. (LLM, conf=0.76)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 46

  1. [1][1] Yixin Chen, Siyuan Huang, Tao Yuan, Siyuan Qi, Yixin Zhu, and Song-Chun Zhu. Holistic++ scene understanding: Single-view 3d holistic scene parsing and human pose estimation with human-object interaction and physical commonsense. arXiv preprint arXiv:1909.01507 , 2019. - · arXiv
  2. [2][2] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5939-5948, 2019. -
  3. [3][3] Wongun Choi, Yu-Wei Chao, Caroline Pantofaru, and Silvio Savarese. Understanding indoor scenes using 3d geometric phrases. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 33-40, 2013. -
  4. [4][4] Christopher Bongsoo Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 628-644, 2016. -
  5. [5][5] Saumitro Dasgupta, Kuan Fang, Kevin Chen, and Silvio Savarese. Delay: Robust spatial layout estimation for cluttered indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 616-624, 2016. -
  6. [6][6] Theo Deprelle, Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. Learning elementary structures for 3d shape generation and matching. arXiv preprint arXiv:1908.04725 , 2019. - · arXiv
  7. [7][7] Yilun Du, Zhijian Liu, Hector Basevi, Ales Leonardis, Bill Freeman, Josh Tenenbaum, and Jiajun Wu. Learning to exploit stability for 3d scene parsing. In Advances in Neural Information Processing Systems , pages 1726-1736, 2018. -
  8. [8][8] Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 605-613, 2017. -
  9. [9][9] Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh r-cnn. arXiv preprint arXiv:1906.02739 , 2019. - · arXiv
  10. [10][10] Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan Russell, and Mathieu Aubry. AtlasNet: A Papier-Mˆ ach´ e Approach to Learning 3D Surface Generation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. -
  11. [12][12] Varsha Hedau, Derek Hoiem, and David Forsyth. Recovering the spatial layout of cluttered rooms. In 2009 IEEE 12th international conference on computer vision , pages 18491856. IEEE, 2009. -
  12. [13][13] Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3588-3597, 2018. -
  13. [14][14] Siyuan Huang, Siyuan Qi, Yinxue Xiao, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Cooperative holistic scene understanding: Unifying 3d object, layout, and camera pose estimation. In Advances in Neural Information Processing Systems , pages 207-218, 2018. -
  14. [15][15] Siyuan Huang, Siyuan Qi, Yixin Zhu, Yinxue Xiao, Yuanlu Xu, and Song-Chun Zhu. Holistic 3d scene parsing and reconstruction from a single rgb image. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 187-203, 2018. -
  15. [16][16] Moos Hueting, Pradyumna Reddy, Vladimir Kim, Ersin Yumer, Nathan Carr, and Niloy Mitra. Seethrough: finding chairs in heavily occluded indoor scene images. arXiv preprint arXiv:1710.10473 , 2017. - · arXiv
  16. [17][17] Hamid Izadinia, Qi Shan, and Steven M Seitz. Im2cad. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 5134-5143, 2017. -
  17. [18][18] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 39073916, 2018. -
  18. [19][19] Nilesh Kulkarni, Ishan Misra, Shubham Tulsiani, and Abhinav Gupta. 3d-relnet: Joint object and relational network for 3d prediction. International Conference on Computer Vision (ICCV) . -
  19. [20][20] Andrey Kurenkov, Jingwei Ji, Animesh Garg, Viraj Mehta, JunYoung Gwak, Christopher Choy, and Silvio Savarese. Deformnet: Free-form deformation network for 3d shape reconstruction from a single image. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 858-866. IEEE, 2018. -
  20. [21][21] David C Lee, Martial Hebert, and Takeo Kanade. Geometric reasoning for single image structure recovery. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 2136-2143. IEEE, 2009. -
  21. [22][22] Lin Li, Salman Khan, and Nick Barnes. Silhouette-assisted 3d object instance reconstruction from a cluttered scene. In Proceedings of the IEEE International Conference on Computer Vision Workshops , pages 0-0, 2019. -
  22. [23][23] Yiyi Liao, Simon Donne, and Andreas Geiger. Deep marching cubes: Learning explicit surface representations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2916-2925, 2018. -
  23. [24][24] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740-755. Springer, 2014. -
  24. [25][25] Arun Mallya and Svetlana Lazebnik. Learning informative edge maps for indoor scene layout prediction. In Proceedings of the IEEE international conference on computer vision , pages 936-944, 2015. -
  25. [26][26] Priyanka Mandikal, Navaneet KL, and R Venkatesh Babu. 3d-psrnet: Part segmented 3d point cloud reconstruction from a single image. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 0-0, 2018. -
  26. [27][27] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4460-4470, 2019. -
  27. [28][28] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson. Deep level sets: Implicit surface representations for 3d shape inference. arXiv preprint arXiv:1901.06802 , 2019. - · arXiv
  28. [29][29] KL Navaneet, Priyanka Mandikal, Mayank Agarwal, and R Venkatesh Babu. Capnet: Continuous approximation projection for 3d point cloud reconstruction using 2d supervision. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 8819-8826, 2019. -
  29. [30][30] Junyi Pan, Xiaoguang Han, Weikai Chen, Jiapeng Tang, and Kui Jia. Deep mesh reconstruction from single rgb images via topology modification networks. In Proceedings of the IEEE International Conference on Computer Vision , pages 9964-9973, 2019. -
  30. [31][31] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. arXiv preprint arXiv:1901.05103 , 2019. - · arXiv
  31. [32][32] Despoina Paschalidou, Ali Osman Ulusoy, and Andreas Geiger. Superquadrics revisited: Learning 3d shape parsing beyond cuboids. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1034410353, 2019. -
  32. [35][35] Yuzhuo Ren, Shangwen Li, Chen Chen, and C-C Jay Kuo. A coarse-to-fine indoor layout estimation (cfile) method. In Asian Conference on Computer Vision , pages 36-51. Springer, 2016. -
  33. [36][36] Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. Octnet: Learning deep 3d representations at high resolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3577-3586, 2017. -
  34. [37][37] Lawrence G Roberts. Machine perception of threedimensional solids . PhD thesis, Massachusetts Institute of Technology, 1963. -
  35. [38][38] Alexander G Schwing, Sanja Fidler, Marc Pollefeys, and Raquel Urtasun. Box in the box: Joint 3d layout and object reasoning from single images. In Proceedings of the IEEE International Conference on Computer Vision , pages 353-360, 2013. -
  36. [42][42] Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B Tenenbaum, and William T Freeman. Pix3d: Dataset and methods for single-image 3d shape modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2974-2983, 2018. -
  37. [43][43] Jiapeng Tang, Xiaoguang Han, Junyi Pan, Kui Jia, and Xin Tong. A skeleton-bridged deep learning approach for generating meshes of complex topologies from single rgb images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4541-4550, 2019. -
  38. [44][44] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In Proceedings of the IEEE International Conference on Computer Vision , pages 2088-2096, 2017. -
  39. [45][45] Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis, William T Freeman, Joshua B Tenenbaum, and Jiajun Wu. Learning to infer and execute 3d shape programs. arXiv preprint arXiv:1901.02875 , 2019. - · arXiv
  40. [46][46] Shubham Tulsiani, Saurabh Gupta, David F Fouhey, Alexei A Efros, and Jitendra Malik. Factoring shape, pose, and layout from the 2d image of a 3d scene. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 302-310, 2018. -
  41. [47][47] Shubham Tulsiani, Hao Su, Leonidas J Guibas, Alexei A Efros, and Jitendra Malik. Learning shape abstractions by assembling volumetric primitives. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2635-2643, 2017. -
  42. [48][48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pages 5998-6008, 2017. -
  43. [49][49] Bram Wallace and Bharath Hariharan. Few-shot generalization for single-image 3d reconstruction via priors. In Proceedings of the IEEE International Conference on Computer Vision , pages 3818-3827, 2019. -
  44. [50][50] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 52-67, 2018. -
  45. [51][51] Peng-Shuai Wang, Chun-Yu Sun, Yang Liu, and Xin Tong. Adaptive o-cnn: a patch-based deep representation of 3d shapes. In SIGGRAPH Asia 2018 Technical Papers , page 217. ACM, 2018. -
  46. [52][52] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. arXiv preprint arXiv:1905.10711 , 2019. · arXiv

Cited by

Sorted by citations

Authors
Nie, Y., Han, X., Guo, S., Zheng, Y., Chang, J., & Zhang, J. J.
Affiliations
  • Shenzhen Research Institute of Big Data (CN)
  • Xiamen University (CN)
  • Bournemouth University (GB)
ORCID
Venue
2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
IEEE
Published
2020-06
Pages
52-61
ISSN
1063-6919 (ISSN-L)
DOI
10.1109/cvpr42600.2020.00013
OpenAlex
W3035424742
Cited by
238
References
6
Citations by year
2026: 23 · 2025: 40 · 2024: 44 · 2023: 54 · 2022: 42 · 2021: 29 · 2020: 6
FWCI
28.90
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • Bournemouth University Research Online (Bournemouth University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Bournemouth University
  • National Natural Science Foundation of China (NSFC‐61629101, 61629101, 2018B030338001, ZDSYS201707251409055, 2018YFB1800800, 61702433, 61902334, 61661146002, 2017ZT07X152)
  • China Scholarship Council
  • Innovate UK
  • European Regional Development Fund
  • Fundamental Research Funds for the Central Universities (2018YFB1800800, 61702433, NSFC-61629101, 2018B030338001, NSFC-61902334, ZDSYS201707251409055, 2017ZT07X152)
  • Interreg
Topics
  • Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Earth and Planetary Sciences › Geology › 3D Surveying and Cultural Heritage
Keywords
indoor scene understanding, room layout estimation, 3D object pose estimation, 3D mesh reconstruction, single-view reconstruction, 3D object detection, camera pose estimation, semantic reconstruction

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review