CoChat
Sign Up
Cited by 3503D Surveying and Cultural Heritage3D Shape Modeling and AnalysisAdvanced Image and Video Retrieval Techniques

3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera

Open access
Armeni, Iro, He, Zhi-Yang, Zamir, Amir, Gwak, Junyoung, Malik, Jitendra, Fischer, Martin, Savarese, Silvio·2019·2019 IEEE/CVF International Conference on Computer Vision (ICCV)
DOI 10.1109/iccv.2019.00576
Open this paper in CoChat — free accountOpen PDF

Abstract

A comprehensive semantic understanding of a scene is important for many applications - but in what space should diverse semantic information (e.g., objects, scene categories, material types, 3D shapes, etc.) be grounded and what should be its structure? Aspiring to have one unified structure that hosts diverse types of semantics, we follow the Scene Graph paradigm in 3D, generating a 3D Scene Graph. Given a 3D mesh and registered panoramic images, we construct a graph that spans the entire building and includes semantics on objects (e.g., class, material, shape and other attributes), rooms (e.g., function, illumination type, etc.) and cameras (e.g., location, etc.), as well as the relationships among these entities. However, this process is prohibitively labor heavy if done manually. To alleviate this we devise a semi-automatic framework that employs existing detection methods and enhances them using two main constraints: I. framing of query images sampled on panoramas to maximize the performance of 2D detectors, and II. multi-view consistency enforcement across 2D detections that originate in different camera locations.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/iccv.2019.00576

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • 3D scene graphs allow efficient representation of larger scenes.

    “By representing objects or spatial concepts as nodes and their relations as edges, 3D scene graphs allow to efficiently represent larger scenes [36, 18, 19].”

    From Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation (2024) · cites this paper as [36]

    The Abstract and Section 3 describe a graph spanning an entire building, with nodes for rooms and objects and edges for their relationships. This supports representing larger scenes, but the paper does not establish that the graph represents them efficiently; its efficiency claim in Section 2 concerns automated generation of semantic information. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • The home evaluation environment is an existing 3D scene graph from the Stanford 3D Scene Graph dataset, comprising a family home (Klickitat) with 28 rooms over 3 floors and 112 assets and objects that the agent can interact with.

    “Home: An existing 3D scene graph from the Stanford 3D Scene Graph dataset [11] which consists of a family home environment (Klickitat) spanning 28 rooms across 3 floors and contains 112 assets and objects that the agent can interact with.”

    From SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning (2023) · cites this paper as [11]

    The Introduction says the authors made 3D scene graphs for the Gibson Environment publicly available, and Section 3 describes graphs containing buildings, rooms, and objects. Our copy does not identify Klickitat or substantiate the claimed 28 rooms, 3 floors, 112 assets and objects, or agent interactivity (searched for Klickitat and those counts). Page numbers were not available in our copy. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 56 · Cited by 350

Research works printed but not linked: 43

  1. David Acuna, Huan Ling, Amlan Kar, and Sanja Fidler. Ef- ficient interactive annotation of segmentation datasets with polygon-rnn++. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 859-868, 2018.
  2. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image cap- tion evaluation. In European Conference on Computer Vi- sion, pages 382-398. Springer, 2016.
  3. Mykhaylo Andriluka, Jasper RR Uijlings, and Vittorio Ferrari. Fluid annotation: a human-machine collabora- tion interface for full image annotation. arXiv preprint arXiv:1806.07527, 2018. · arXiv
  4. Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese. Joint 2d-3d-semantic data for indoor scene understanding. arXiv preprint arXiv:1702.01105, 2017. · arXiv
  5. Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 1534-1543, 2016.
  6. Sean Bell, Paul Upchurch, Noah Snavely, and Kavita Bala. Material recognition in the wild with the materials in con- text database. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3479-3487, 2015.
  7. Lluis Castrejon, Kaustav Kundu, Raquel Urtasun, and Sanja Fidler. Annotating object instances with a polygon-rnn. In CVPR, volume 1, page 2, 2017.
  8. Wongun Choi, Yu-Wei Chao, Caroline Pantofaru, and Silvio Savarese. Understanding indoor scenes using 3d geometric phrases. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pages 33-40, 2013.
  9. Angela Dai, Angel X Chang, Manolis Savva, Maciej Hal- ber, Thomas A Funkhouser, and Matthias Nießner. Scan- net: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, volume 2, page 10, 2017.
  10. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248-255. Ieee, 2009.
  11. Nikita Dvornik, Konstantin Shmelkov, Julien Mairal, and Cordelia Schmid. Blitznet: A real-time deep network for scene understanding. In ICCV 2017-International Confer- ence on Computer Vision, page 11, 2017.
  12. Kiana Ehsani, Roozbeh Mottaghi, and Ali Farhadi. Segan: Segmenting and generating the invisible. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6144-6153, 2018.
  13. Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal com- pact bilinear pooling for visual question answering and vi- sual grounding. arXiv preprint arXiv:1606.01847, 2016. · arXiv
  14. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask r-cnn. 2017 IEEE International Conference on Computer Vision (ICCV), pages 2980-2988, 2017.
  15. Siyuan Huang, Siyuan Qi, Yixin Zhu, Yinxue Xiao, Yuanlu Xu, and Song-Chun Zhu. Holistic 3d scene parsing and re- construction from a single rgb image. In Proceedings of the European Conference on Computer Vision (ECCV), pages 187-203, 2018.
  16. Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. Structural-rnn: Deep learning on spatio-temporal graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5308-5317, 2016.
  17. Chenfanfu Jiang, Siyuan Qi, Yixin Zhu, Siyuan Huang, Jenny Lin, Lap-Fai Yu, Demetri Terzopoulos, and Song- Chun Zhu. Configurable 3d scene synthesis and 2d im- age rendering with per-pixel ground truth using stochas- tic grammars. International Journal of Computer Vision, 126(9):920-941, 2018.
  18. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elemen- tary visual reasoning. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pages 2901-2910, 2017.
  19. Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional localization networks for dense caption- ing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4565-4574, 2016.
  20. Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 3668-3678, 2015.
  21. Philipp Krähenbühl and Vladlen Koltun. Efficient inference in fully connected crfs with gaussian edge potentials. In Ad- vances in neural information processing systems, pages 109- 117, 2011.
  22. Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. A hierarchical approach for generating descriptive image paragraphs. In Computer Vision and Pattern Recogni- tion (CVPR), 2017 IEEE Conference on, pages 3337-3345. IEEE, 2017.
  23. Ke Li and Jitendra Malik. Amodal instance segmentation. In European Conference on Computer Vision, pages 677-693. Springer, 2016.
  24. Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang, and Xi- aogang Wang. Scene graph generation from objects, phrases and region captions. In ICCV, 2017.
  25. Xiaodan Liang, Lisa Lee, and Eric P Xing. Deep variation- structured reinforcement learning for visual relationship and attribute detection. In Computer Vision and Pattern Recogni- tion (CVPR), 2017 IEEE Conference on, pages 4408-4417. IEEE, 2017.
  26. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
  27. Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei- Fei. Visual relationship detection with language priors. In European Conference on Computer Vision, pages 852-869. Springer, 2016.
  28. Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei- Fei. Visual relationship detection with language priors. In European Conference on Computer Vision, pages 852-869. Springer, 2016.
  29. Tomasz Malisiewicz and Alyosha Efros. Beyond categories: The visual memex model for reasoning about object relation- ships. In Advances in neural information processing systems, pages 1222-1230, 2009.
  30. Duc Thanh Nguyen, Binh-Son Hua, Lap-Fai Yu, and Sai-Kit Yeung. A robust 3d-2d interactive tool for scene segmenta- tion and annotation. IEEE transactions on visualization and computer graphics, 24(12):3005-3018, 2018.
  31. Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu. Learning human-object interac- tions by graph parsing neural networks. arXiv preprint arXiv:1808.07962, 2018. · arXiv
  32. Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U- net: Convolutional networks for biomedical image segmen- tation. In International Conference on Medical image com- puting and computer-assisted intervention, pages 234-241. Springer, 2015.
  33. Bryan C Russell and Antonio Torralba. Building a database of 3d scenes from user annotations. In 2009 IEEE Con- ference on Computer Vision and Pattern Recognition, pages 2711-2718. IEEE, 2009.
  34. Gabriel Sepulveda, Juan Carlos Niebles, and Alvaro Soto. A deep learning based behavioral approach to indoor au- tonomous navigation. arXiv preprint arXiv:1803.04119, 2018. · arXiv
  35. Lyne Tchapmi, Christopher Choy, Iro Armeni, JunYoung Gwak, and Silvio Savarese. Segcloud: Semantic segmen- tation of 3d point clouds. In 3D Vision (3DV), 2017 Interna- tional Conference on, pages 537-547. IEEE, 2017.
  36. Fei Xia, Amir R Zamir, Zhiyang He, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: Real-world percep- tion for embodied agents. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pages 9068-9079, 2018.
  37. Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
  38. Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 2, 2017.
  39. Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, and Tat- Seng Chua. Visual translation embedding network for visual relation detection. In CVPR, volume 1, page 5, 2017.
  40. Yibiao Zhao and Song-Chun Zhu. Scene parsing by integrat- ing function, geometry and appearance models. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3119-3126, 2013.
  41. Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Tor- ralba, and Aude Oliva. Learning deep features for scene recognition using places database. In Advances in neural information processing systems, pages 487-495, 2014.
  42. Yuke Zhu, Alireza Fathi, and Li Fei-Fei. Reasoning about object affordances in a knowledge base representation. In European conference on computer vision, pages 408-424. Springer, 2014.
  43. Yan Zhu, Yuandong Tian, Dimitris Metaxas, and Piotr Dollár. Semantic amodal segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1464-1472, 2017.

Authors
Armeni, I., He, Z.-Y., Zamir, A., Gwak, J., Malik, J., Fischer, M., & Savarese, S.
Affiliations
  • Stanford University (US)
  • University of California, Berkeley (US)
ORCID
Venue
2019 IEEE/CVF International Conference on Computer Vision (ICCV)
Publisher
IEEE
Published
2019-10
Pages
5663-5672
DOI
10.1109/iccv.2019.00576
OpenAlex
W2978091520
Cited by
350
References
56
Citations by year
2026: 58 · 2025: 81 · 2024: 58 · 2023: 48 · 2022: 48 · 2021: 33 · 2020: 24
FWCI
131.33
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • Nvidia
  • Multidisciplinary University Research Initiative
  • Office of Naval Research (N000141410671, N00014)
Topics
  • Physical Sciences › Earth and Planetary Sciences › Geology › 3D Surveying and Cultural Heritage
  • Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
Keywords
3D scene graphs, semantic scene understanding, 3D building reconstruction, panoramic images, 3D mesh, object detection, room segmentation, multi-view consistency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review