CoChat
Sign Up
Cited by 202Robotics and Sensor-Based Localization3D Surveying and Cultural Heritage3D Shape Modeling and Analysis

Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization

Open access
Hughes, Nathan, Chang, Yun, Carlone, Luca·2022·Robotics: Science and Systems XVIII
DOI 10.15607/rss.2022.xviii.050
Open this paper in CoChat — free accountOpen PDF

Abstract

3D scene graphs have recently emerged as a powerful high-level representation of 3D environments. A 3D scene graph describes the environment as a layered graph where nodes represent spatial concepts at multiple levels of abstraction and edges represent relations between concepts. While 3D scene graphs can serve as an advanced "mental model" for robots, how to build such a rich representation in real-time is still uncharted territory. This paper describes a real-time Spatial Perception System, a suite of algorithms to build a 3D scene graph from sensor data in real-time. Our first contribution is to develop real-time algorithms to incrementally construct the layers of a scene graph as the robot explores the environment; these algorithms build a local Euclidean Signed Distance Function (ESDF) around the current robot location, extract a topological map of places from the ESDF, and then segment the places into rooms using an approach inspired by community-detection techniques. Our second contribution is to investigate loop closure detection and optimization in 3D scene graphs. We show that 3D scene graphs allow defining hierarchical descriptors for loop closure detection; our descriptors capture statistics across layers in the scene graph, ranging from low-level visual appearance to summary statistics about objects and places. We then propose the first algorithm to optimize a 3D scene graph in response to loop closures; our approach relies on embedded deformation graphs to simultaneously correct all layers of the scene graph. We implement the proposed Spatial Perception System into a architecture named Hydra, that combines fast early and mid-level perception processes with slower high-level perception. We evaluate Hydra on simulated and real data and show it is able to reconstruct 3D scene graphs with an accuracy comparable with batch offline methods despite running online.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.15607/rss.2022.xviii.050

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • HOV-SG region segmentation has slightly lower precision but significantly greater recall of 83.59% compared to Hydra's 77.55% across eight scenes on HM3DSem.

    “We observe slightly lower precision but a significantly greater recall of 83.59% compared to 77.55%.”

    From Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation (2024) · cites this paper as [19]

    The Results and Ablation Study (pp. 7–8) evaluates Hydra’s room segmentation using precision and recall, so the paper supports identifying Hydra as a comparison method. It does not report the 77.55% recall, an eight-scene HM3DSem evaluation, or a comparison with HOV-SG; those figures appear to be the citing authors’ own observations. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • 3D scene graphs allow efficient representation of larger scenes.

    “By representing objects or spatial concepts as nodes and their relations as edges, 3D scene graphs allow to efficiently represent larger scenes [36, 18, 19].”

    From Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation (2024) · cites this paper as [19]

    The Introduction (p. 1) describes 3D scene graphs as nodes for spatial concepts and edges for their relations. The paper also describes a memory-bounded method for constructing its graph (Section III, p. 3), but it does not establish that the graph representation itself efficiently represents larger scenes. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • HOV-SG's naïve morphological heuristic for region segregation does not work well on more complex, semantically ambiguous room layouts such as combined kitchen and living rooms.

    “Similar to Hydra [19], our approach utilizes a naı̈ve morphological heuristic to segregate regions, which does not work well on more complex, seman- tically ambiguous room layouts such as combined kitchen and living rooms.”

    From Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation (2024) · cites this paper as [19]

  • 3D scene graphs allow a tight coupling between semantics and highly accurate mapping approaches utilizing meshes.

    “3D scene graphs also allow a tight coupling between semantics and highly accurate mapping approaches utilizing e.g. meshes to represent environments [19].”

    From Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigation (2024) · cites this paper as [19]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 71 · Cited by 202

Research works printed but not linked: 60

  1. [1][1] C. Agia, J. Krishna Murthyand M. Khodeir, O. Miksik, V. Vineet, M. Mukadam, L. Paull, and F. Shkurti. Taskography: Evalutation robot task planning over large 3D scene graphs. In Conference on Robot Learning (CoRL) , 2021. -
  2. [2][2] P. Antonante, V. Tzoumas, H. Yang, and L. Carlone. Outlier-robust estimation: Hardness, minimally tuned algorithms, and applications. IEEE Trans. Robotics , 38(1): 281-301, 2021. (pdf). -
  3. [3][3] R. Arandjelovic, P. Gronat, A. Torii, T. Pajdla, and J. Sivic. NetVLAD: CNN architecture for weakly supervised place recognition. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , pages 52975307, 2016. -
  4. [4][4] I. Armeni, Z. He, J. Gwak, A. Zamir, M. Fischer, J. Malik, and S. Savarese. 3D scene graph: A structure for unified semantics, 3D space, and camera. In Intl. Conf. on Computer Vision (ICCV) , pages 5664-5673, 2019. -
  5. [8][8] Vincent D Blondel, Jean-Loup Guillaume, Renaud Lambiotte, and Etienne Lefebvre. Fast unfolding of communities in large networks. Journal of statistical mechanics: theory and experiment , 2008(10):P10008, 2008. -
  6. [9][9] Richard Bormann, Florian Jordan, Wenzhe Li, Joshua Hampp, and Martin Hagele. Room segmentation: Survey, implementation, and analysis. In 2016 IEEE International Conference on Robotics and Automation (ICRA) , pages 1019-1026, 2016. doi: 10.1109/ICRA. 2016.7487234. - · DOI
  7. [10][10] S.L. Bowman, N. Atanasov, K. Daniilidis, and G.J. Pappas. Probabilistic data association for semantic SLAM. In IEEE Intl. Conf. on Robotics and Automation (ICRA) , pages 1722-1729, 2017. -
  8. [11][11] C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. Reid, and J.J. Leonard. Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Trans. Robotics , 32(6):1309-1332, 2016. ISSN 1552-3098. doi: 10.1109/ TRO.2016.2624754. arxiv preprint: 1606.05830, (pdf). - · arXiv
  9. [12][12] R. Chatila and J.-P. Laumond. Position referencing and consistent world modeling for mobile robots. In IEEE Intl. Conf. on Robotics and Automation (ICRA) , pages 138-145, 1985. -
  10. [13][13] Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, and Christian Theobalt. Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface reintegration. ACM Transactions on Graphics (ToG) , 36(4):1, 2017. -
  11. [14][14] J. Dong, X. Fei, and S. Soatto. Visual-Inertial-Semantic scene representation for 3D object detection. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2017. -
  12. [16][16] Stephen Friedman, Hanna Pasula, and Dieter Fox. Voronoi random fields: Extracting the topological structure of indoor environments via place labeling. In Intl. Joint Conf. on AI (IJCAI) , page 2109-2114, San Francisco, CA, USA, 2007. Morgan Kaufmann Publishers Inc. -
  13. [17][17] Y. Furukawa, B. Curless, S. M. Seitz, and R. Szeliski. Reconstructing building interiors from images. In Intl. Conf. on Computer Vision (ICCV) , 2009. -
  14. [18][18] C. Galindo, A. Saffiotti, S. Coradeschi, P. Buschka, J.A. Fernández-Madrigal, and J. González. Multi-hierarchical semantic maps for mobile robotics. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) , pages 3492-3497, 2005. -
  15. [19][19] Dorian Gálvez-López and J. D. Tardós. Bags of binary words for fast place recognition in image sequences. IEEE Transactions on Robotics , 28(5):1188-1197, October 2012. ISSN 1552-3098. doi: 10.1109/TRO.2012. 2197158. - · DOI
  16. [20][20] Sourav Garg and Michael Milford. Seqnet: Learning descriptors for sequence-based hierarchical place recognition. IEEE Robotics and Automation Letters , 6(3): 4305-4312, 2021. doi: 10.1109/LRA.2021.3067633. - · DOI
  17. [21][21] Abel Gawel, Carlo Del Don, Roland Siegwart, Juan Nieto, and Cesar Cadena. X-view: Graph-based semantic multi-view localization. IEEE Robotics and Automation Letters , 3(3):1687-1694, 2018. doi: 10.1109/LRA.2018. 2801879. - · DOI
  18. [22][22] N. Gothoskar, M. Cusumano-Towner, B. Zinberg, M. Ghavamizadeh, F. Pollok, A. Garrett, J.B. Tenenbaum, D. Gutfreund, and V.K. Mansinghka. 3DP3: 3D scene perception via probabilistic programming. In ArXiv preprint: 2111.00312 , 2021. - · arXiv
  19. [23][23] M. Grinvald, F. Furrer, T. Novkovic, J. J. Chung, C. Cadena, R. Siegwart, and J. Nieto. Volumetric InstanceAware Semantic Mapping and 3D Object Discovery. IEEE Robotics and Automation Letters , 4(3):3037-3044, 2019. -
  20. [24][24] Shihua Huang, Zhichao Lu, Ran Cheng, and Cheng He. FaPN: Feature-aligned pyramid network for dense image prediction. In International Conference on Computer Vision (ICCV) , 2021. -
  21. [25][25] Gregory Izatt and Russ Tedrake. Scene understanding and distribution modeling with mixed-integer scene parsing. In technical report (under review) , 2021. -
  22. [26][26] U. Kim, J. Park, T. Song, and J. Kim. 3-D scene graph: A sparse and semantic representation of physical environments for intelligent agents. IEEE Trans. Cybern. , PP:1-13, Aug. 2019. -
  23. [27][27] Alexander Kleiner, Rodrigo Baravalle, Andreas Kolling, Pablo Pilotti, and Mario Munich. A solution to roomby-room coverage for autonomous cleaning robots. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 5346-5352, 2017. doi: 10.1109/IROS.2017.8206429. - · DOI
  24. [30][30] Boris Lau, Christoph Sprunk, and Wolfram Burgard. Efficient grid-based spatial representations for robot navigation in dynamic environments. Robot. Auton. Syst. , 61(10):1116-1130, October 2013. ISSN 0921-8890. doi: 10.1016/j.robot.2012.08.010. URL https://doi.org/ 10.1016/j.robot.2012.08.010. - · DOI
  25. [31][31] C. Li, H. Xiao, K. Tateno, F. Tombari, N. Navab, and G. D. Hager. Incremental scene understanding on dense SLAM. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) , pages 574-581, 2016. -
  26. [32][32] K. Lianos, J. Schönberger, M. Pollefeys, and T. Sattler. Vso: Visual semantic odometry. In European Conf. on Computer Vision (ECCV) , pages 246-263, 2018. -
  27. [33][33] Shiqi Lin, Jikai Wang, Meng Xu, Hao Zhao, and Zonghai Chen. Topology aware object-level semantic mapping towards more robust loop closure. IEEE Robotics and 34. Automation Letters , 6(4):7041-7048, 2021. doi: 10.1109/ LRA.2021.3097242. -
  28. [34][34] Chen Liu, Jiaye Wu, and Yasutaka Furukawa. FloorNet: A unified framework for floorplan reconstruction from 3d scans. In Proceedings of the European Conference on Computer Vision (ECCV) , September 2018. -
  29. [35][35] Yu Liu, Yvan Petillot, David Lane, and Sen Wang. Global Localization with Object-Level Semantics and Topology. In 2019 International Conference on Robotics and Automation (ICRA) , pages 4909-4915, May 2019. doi: 10.1109/ICRA.2019.8794475. ISSN: 2577-087X. - · DOI
  30. [37][37] R. Lukierski, S. Leutenegger, and A. J. Davison. Room layout estimation from rapid omnidirectional exploration. In IEEE Intl. Conf. on Robotics and Automation (ICRA) , pages 6315-6322, 2017. -
  31. [38][38] J. McCormac, A. Handa, A. J. Davison, and S. Leutenegger. SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks. In IEEE Intl. Conf. on Robotics and Automation (ICRA) , 2017. -
  32. [39][39] J. McCormac, R. Clark, M. Bloesch, A.J. Davison, and S. Leutenegger. Fusion++: Volumetric object-level SLAM. In Intl. Conf. on 3D Vision (3DV) , pages 32-41, 2018. -
  33. [40][40] G. Narita, T. Seno, T. Ishikawa, and Y. Kaji. Panopticfusion: Online volumetric semantic mapping at the level of stuff and things. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) , 2019. -
  34. [41][41] L. Nicholson, M. Milford, and N. Sünderhauf. QuadricSLAM: Dual quadrics from object detections as landmarks in object-oriented SLAM. IEEE Robotics and Automation Letters , 4:1-8, 2018. -
  35. [42][42] Kyel Ok, Katherine Liu, and Nicholas Roy. Hierarchical object map estimation for efficient and robust navigation. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 1132-1139, 2021. doi: 10. 1109/ICRA48506.2021.9561225. -
  36. [43][43] Helen Oleynikova, Zachary Taylor, Marius Fehr, Roland Siegwart, and Juan Nieto. Voxblox: Incremental 3d euclidean signed distance fields for on-board mav planning. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) , pages 1366-1373. IEEE, 2017. -
  37. [44][44] Helen Oleynikova, Zachary Taylor, Roland Siegwart, and Juan Nieto. Sparse 3D topological graphs for microaerial vehicle planning. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) , 2018. -
  38. [45][45] Cao Qin, Yunzhou Zhang, Yingda Liu, and Guanghao Lv. Semantic loop closure detection based on graph matching in multi-objects scenes. Journal of Visual Communication and Image Representation , 76:103072, 2021. -
  39. [46][46] Z. Ravichandran, L. Peng, N. Hughes, J.D. Griffith, and L. Carlone. Hierarchical representations and explicit memory: Learning effective navigation policies on 3D 48. scene graphs using graph neural networks. In IEEE Intl. Conf. on Robotics and Automation (ICRA) , 2022. (pdf). -
  40. [47][47] V. Reijgwart, A. Millane, H. Oleynikova, R. Siegwart, C. Cadena, and J. Nieto. Voxgraph: Globally consistent, volumetric mapping using signed distance function submaps. IEEE Robotics and Automation Letters , 2020. -
  41. [48][48] A. Rosinol, M. Abate, Y. Chang, and L. Carlone. Kimera: an open-source library for real-time metricsemantic localization and mapping. In IEEE Intl. Conf. on Robotics and Automation (ICRA) , 2020. doi: 10. 1109/ICRA40945.2020.9196885. arXiv preprint arXiv: 1910.02490, (video), (code), (pdf). - · arXiv
  42. [49][49] A. Rosinol, A. Gupta, M. Abate, J. Shi, and L. Carlone. 3D dynamic scene graphs: Actionable spatial perception with places, objects, and humans. In Robotics: Science and Systems (RSS) , 2020. doi: 10.15607/RSS.2020.XVI.079. URL http://news.mit.edu/ 2020/robots-spatial-perception-0715. (pdf), (media), (video). - · DOI
  43. [50][50] A. Rosinol, A. Violette, M. Abate, N. Hughes, Y. Chang, J. Shi, A. Gupta, and L. Carlone. Kimera: from SLAM to spatial perception with 3D dynamic scene graphs. Intl. J. of Robotics Research , 40(12-14):1510-1546, 2021. arXiv preprint arXiv: 2101.06894, (pdf). - · arXiv
  44. [53][53] R. F. Salas-Moreno, R. A. Newcombe, H. Strasdat, P. H. J. Kelly, and A. J. Davison. SLAM++: Simultaneous localisation and mapping at the level of objects. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2013. -
  45. [54][54] Lukas Schmid, Jeffrey Delmerico, Johannes Schönberger, Juan Nieto, Marc Pollefeys, Roland Siegwart, and Cesar Cadena. Panoptic multi-tsdfs: a flexible representation for online multi-resolution volumetric mapping and long-term dynamic scene consistency. arXiv preprint arXiv:2109.10165 , 2021. - · arXiv
  46. [55][55] Stefan Schubert, Peer Neubert, and Peter Protzel. Fast and memory efficient graph optimization via ICM for visual place recognition. In Proc. of Robotics: Science and Systems (RSS) , 2021. -
  47. [56][56] M. Shan, Q. Feng, and N. Atanasov. Object residual constrained visual-inertial odometry. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) , pages 5104-5111, 2020. -
  48. [58][58] J. Stückler and S. Behnke. Multi-resolution surfel maps for efficient dense 3d modeling and tracking. J. Vis. Comun. Image Represent. , 25(1):137-147, 2014. -
  49. [59][59] R. Sumner, J. Schmid, and M. Pauly. Embedded deformation for shape manipulation. ACM SIGGRAPH 2007 papers on - SIGGRAPH '07 , 2007. doi: 10.1145/ 1275808.1276478. -
  50. [60][60] R. Talak, S. Hu, L. Peng, and L. Carlone. Neural trees for learning on graphs. In Conference on Neural Information Processing Systems (NeurIPS) , 2021. (pdf). -
  51. [61][61] K. Tateno, F. Tombari, and N. Navab. Real-time and scalable incremental segmentation on dense SLAM. In IEEE/RSJ Intl. Conf. on Intelligent Robots and Systems (IROS) , pages 4465-4472, 2015. -
  52. [62][62] S. Thrun. Robotic mapping: a survey. In Exploring artificial intelligence in the new millennium , pages 1-35. Morgan Kaufmann, Inc., 2003. -
  53. [63][63] Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3D semantic scene graphs from 3D indoor reconstructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3961-3970, 2020. -
  54. [65][65] T. Whelan, M. Kaess, H. Johannsson, M. Fallon, J. Leonard, and J. McDonald. Real-time large-scale dense RGB-D SLAM with volumetric fusion. Intl. J. of Robotics Research , 34(4-5):598-626, 2015. -
  55. [67][67] S. Wu, J. Wald, K. Tateno, N. Navab, and F. Tombari. SceneGraphFusion: Incremental 3D scene graph prediction from RGB-D sequences. In IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2021. -
  56. [68][68] B. Xu, W. Li, D. Tzoumanikas, M. Bloesch, A. Davison, and S. Leutenegger. MID-Fusion: Octree-based objectlevel multi-instance dynamic SLAM. pages 5231-5237, 2019. -
  57. [69][69] H. Yang, J. Shi, and L. Carlone. TEASER: Fast and Certifiable Point Cloud Registration. IEEE Trans. Robotics , 37(2):314-333, 2020. extended arXiv version 2001.07715 (pdf). -
  58. [70][70] H. Zender, O. Martínez Mozos, P. Jensfelt, G.-J.M. Kruijff, and W. Burgard. Conceptual spatial representations for indoor mobile robots. Robotics and Autonomous Systems , 56(6):493-502, 2008. From Sensors to Human Spatial Concepts. -
  59. [71][71] Tian Zheng, Guoqing Zhang, Lei Han, Lan Xu, and Lu Fang. Building fusion: Semantic-aware structural building-scale 3d reconstruction. IEEE Transactions on Pattern Analysis and Machine Intelligence , pages 1-1, 2020. doi: 10.1109/TPAMI.2020.3042881. - · DOI
  60. [72][72] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2017.

Authors
Hughes, N., Chang, Y., & Carlone, L.
Affiliations
Massachusetts Institute of Technology (US)
ORCID
Venue
Robotics: Science and Systems XVIII
Publisher
Robotics: Science and Systems Foundation
Published
2022-06-27
DOI
10.15607/rss.2022.xviii.050
OpenAlex
W4283332944
Cited by
202
References
71
Citations by year
2026: 46 · 2025: 69 · 2024: 55 · 2023: 27 · 2022: 5
FWCI
68.31
Type
conference paper
Language
en
Open access
Locations
  • Robotics: Science and Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
Office of Naval Research (N00014-18-1-2828, W911NF-17-2-0181, N00014, RAIDER N00014-18-1-2828)
Topics
  • Physical Sciences › Engineering › Aerospace Engineering › Robotics and Sensor-Based Localization
  • Physical Sciences › Earth and Planetary Sciences › Geology › 3D Surveying and Cultural Heritage
  • Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
Keywords
3D scene graphs, room segmentation, community detection, loop closure detection, hierarchical perception, robot exploration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review