The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
SemanticFusion: Dense 3D semantic mapping with convolutional neural networks
Open accessAbstract
Ever more robust, accurate and detailed mapping using visual sensing has proven to be an enabling factor for mobile robots across a wide variety of applications. For the next level of robot intelligence and intuitive user interaction, maps need to extend beyond geometry and appearance - they need to contain semantics. We address this challenge by combining Convolutional Neural Networks (CNNs) and a state-of-the-art dense Simultaneous Localization and Mapping (SLAM) system, ElasticFusion, which provides long-term dense correspondences between frames of indoor RGB-D video even during loopy scanning trajectories. These correspondences allow the CNN's semantic predictions from multiple view points to be probabilistically fused into a map. This not only produces a useful semantic 3D map, but we also show on the NYUv2 dataset that fusing multiple predictions leads to an improvement even in the 2D semantic labelling over baseline single frame predictions. We also show that for a smaller reconstruction dataset with larger variation in prediction viewpoint, the improvement over single frame segmentation increases. Our system is efficient enough to allow real-time interactive use at frame-rates of ≈25Hz.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Microsoft COCO: Common Objects in Context
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., & Zitnick, C. L. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10602-1_48 · Cited by 45,460
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
The pascal visual object classes (VOC) challenge
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2010) · International Journal of Computer Vision · doi:10.1007/s11263-009-0275-4 · Cited by 19,599 · cited 2×
“The majority of other approaches to indoor semantic labelling either focuses on offline batch mapping methods [24], [11] or on single-frame 2D segmentations which do not aim to produce a semantically annotated 3D map [3], [20], [15], [22].”
BackgroundMethodCaffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201 · cited 1×
“Future research could incorporate CNN compression [9], which would not only enable the incorporation of other modalities, but also offer exciting new directions in real-time semantic segmentation on low memory and power mobile devices.”
Indoor segmentation and support inference from rgbd images
Silberman, N., Hoiem, D., Kohli, P., & Fergus, R. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-33715-4_54 · Cited by 5,760
Learning Deconvolution Network for Semantic Segmentation
Noh, H., Hong, S., & Han, B. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.178 · Cited by 4,422
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Krähenbühl, P., & Koltun, V. (2012) · arXiv · doi:10.48550/arxiv.1210.5644 · Cited by 2,908
Cited by
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T., & Niessner, M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.261 · Cited by 4,368
(2020) · doi:10.1109/icra40945.2020.9196885 · Cited by 633
Pointwise Convolutional Neural Networks
Hua, B.-S., Tran, M.-K., & Yeung, S.-K. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00109 · Cited by 615
Tangent Convolutions for Dense Prediction in 3D
Tatarchenko, M., Park, J., Koltun, V., & Zhou, Q.-Y. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00409 · Cited by 593
SuMa++: Efficient LiDAR-based Semantic SLAM
Chen, X., Milioto, A., Palazzolo, E., Giguère, P., Behley, J., & Stachniss, C. (2019) · 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8967704 · Cited by 514
Matsuki, H., Murai, R., Kelly, P. H. J., & Davison, A. J. (2024) · 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52733.2024.01708 · Cited by 493
Self-citationMaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects
Runz, M., Buffier, M., & Agapito, L. (2018) · 2018 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) · doi:10.1109/ismar.2018.00024 · Cited by 415
ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
Dai, A., Chang, A. X., Savva, M., Halber, M., Funkhouser, T. A., & Nießner, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.04405 · Cited by 412
In-Place Scene Labelling and Understanding with Implicit Scene Representation
Zhi, S., Laidlow, T., Leutenegger, S., & Davison, A. J. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01554 · Cited by 396
Self-citationOpenScene: 3D Scene Understanding with Open Vocabularies
Peng, S., Genova, K., Jiang, C., Tagliasacchi, A., Pollefeys, M., & Funkhouser, T. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00085 · Cited by 395
Visual Language Maps for Robot Navigation
Huang, C., Mees, O., Zeng, A., & Burgard, W. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10160969 · Cited by 376
3DMV: Joint 3D-Multi-view Prediction for 3D Semantic Scene Segmentation
Dai, A., & Nießner, M. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01249-6_28 · Cited by 328
QuadricSLAM: Dual Quadrics From Object Detections as Landmarks in Object-Oriented SLAM
Nicholson, L., Milford, M., & Sunderhauf, N. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2866205 · Cited by 325
Fusion++: Volumetric Object-Level SLAM
Mccormac, J., Clark, R., Bloesch, M., Davison, A., & Leutenegger, S. (2018) · 2018 International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2018.00015 · Cited by 308
Self-citationDetect-SLAM: Making Object Detection and SLAM Mutually Beneficial
Zhong, F., Wang, S., Zhang, Z., Chen, C., & Wang, Y. (2018) · 2018 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv.2018.00115 · Cited by 307
ImVoteNet: Boosting 3D Object Detection in Point Clouds With Image Votes
Qi, C. R., Chen, X., Litany, O., & Guibas, L. J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00446 · Cited by 294
SceneNet RGB-D: Can 5M Synthetic Images Beat Generic ImageNet Pre-training on Indoor Segmentation?
McCormac, J., Handa, A., Leutenegger, S., & Davison, A. J. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.292 · Cited by 291
Self-citationKimera-Multi: Robust, Distributed, Dense Metric-Semantic SLAM for Multi-Robot Systems
Tian, Y., Chang, Y., Herrera Arias, F., Nieto-Granda, C., How, J. P., & Carlone, L. (2022) · IEEE Transactions on Robotics · doi:10.1109/tro.2021.3137751 · Cited by 288
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs
Rosinol, A., Violette, A., Abate, M., Hughes, N., Chang, Y., Shi, J., Gupta, A., & Carlone, L. (2021) · The International Journal of Robotics Research · doi:10.1177/02783649211056674 · Cited by 271
Atlas: End-to-End 3D Scene Reconstruction from Posed Images
Murez, Z., van As, T., Bartolozzi, J., Sinha, A., Badrinarayanan, V., & Rabinovich, A. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-58571-6_25 · Cited by 266
ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning
Gu, Q., Kuwajerwala, A., Morin, S., Jatavallabhula, K. M., Sen, B., Agarwal, A., Rivera, C., Paul, W., Ellis, K., Chellappa, R., et al. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10610243 · Cited by 406
“Aside from reconstructing 3D geometry, recent works also use deep learning-based object detection and segmentation models to reconstruct the 3D scene representations with dense semantic mapping [8]–[11] or object-level decomposition [12]–[15].”
MethodConceptFusion: Open-set Multimodal 3D Mapping
Jatavallabhula, K. M., Kuwajerwala, A., Gu, Q., Omama, M., Chen, T., Maalouf, A., Li, S., Iyer, G., Saryazdi, S., Keetha, N., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.07241 · Cited by 228
ConceptFusion: Open-set multimodal 3D mapping
Jatavallabhula, K., Kuwajerwala, A., Gu, Q., Omama, M., Iyer, G., Saryazdi, S., Chen, T., Maalouf, A., Li, S., Keetha, N., et al. (2023) · Robotics: Science and Systems XIX · doi:10.15607/rss.2023.xix.066 · Cited by 228
Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization
Hughes, N., Chang, Y., & Carlone, L. (2022) · Robotics: Science and Systems XVIII · doi:10.15607/rss.2022.xviii.050 · Cited by 202
“The literature has focused on both object-based maps [10, 14, 41, 42, 53, 56] and dense maps, including volumetric models [23, 38, 40], point clouds [6, 32, 61], and 3D meshes [48, 51].”
Deep semantic segmentation for automated driving: Taxonomy, roadmap and challenges
Siam, M., Elkerdawy, S., Jagersand, M., & Yogamani, S. (2017) · 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2017.8317714 · Cited by 178
Additive manufacturing using mobile robots: Opportunities and challenges for building construction
Dörfler, K., Dielemans, G., Lachmayer, L., Recker, T., Raatz, A., Lowke, D., & Gerke, M. (2022) · Cement and Concrete Research · doi:10.1016/j.cemconres.2022.106772 · Cited by 124
NodeSLAM: Neural object descriptors for multi-view shape reconstruction
Sucar, E., Wada, K., & Davison, A. (2020) · 2020 International Conference on 3D Vision (3DV) · doi:10.1109/3dv50981.2020.00105 · Cited by 108
Self-citationDeep Learning for Visual Localization and Mapping: A Survey
Chen, C., Wang, B., Lu, C. X., Trigoni, N., & Markham, A. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3309809 · Cited by 97
Global Pose Estimation with an Attention-Based Recurrent Network
Parisotto, E., Chaplot, D. S., Zhang, J., & Salakhutdinov, R. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2018.00061 · Cited by 88
A Cross-Season Correspondence Dataset for Robust Semantic Segmentation
Mans Larsson, M., Stenborg, E., Hammarstrand, L., Pollefeys, M., Sattler, T., & Kahl, F. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00976 · Cited by 71
Intelligent Robotic Perception Systems
Premebida, C., Ambrus, R., & Marton, Z.-C. (2019) · Applications of Mobile Robots · doi:10.5772/intechopen.79742 · Cited by 52
“[29] use the elastic fusion SLAM algorithm proposed by Whelan et al.”
SEAL: Self-supervised embodied active learning using exploration and 3D consistency
Chaplot, D. S., Dalal, M., Gupta, S., Malik, J., & Salakhutdinov, R. (2021) · arXiv · doi:10.48550/arxiv.2112.01001 · Cited by 22
“Researchers have also considered the task of associating semantics with 3D maps [32, 8].”
Background
Metadata
- Authors
- McCormac, J., Handa, A., Davison, A., & Leutenegger, S.
- Venue
- 2017 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2017-05
- Pages
- 4628-4635
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2523049145
- Cited by
- 682
- References
- 27
- FWCI
- 120.15
- Type
- conference paper
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.