SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud
Open accessAbstract
We address semantic segmentation of road-objects from 3D LiDAR point clouds. In particular, we wish to detect and categorize instances of interest, such as cars, pedestrians and cyclists. We formulate this problem as a point-wise classification problem, and propose an end-to-end pipeline called SqueezeSeg based on convolutional neural networks (CNN): the CNN takes a transformed LiDAR point cloud as input and directly outputs a point-wise label map, which is then refined by a conditional random field (CRF) implemented as a recurrent layer. Instance-level labels are then obtained by conventional clustering algorithms. Our CNN model is trained on LiDAR point clouds from the KITTI [1] dataset, and our point-wise segmentation labels are derived from 3D bounding boxes from KITTI. To obtain extra training data, we built a LiDAR simulator into Grand Theft Auto V (GTA-V), a popular video game, to synthesize large amounts of realistic training data. Our experiments show that SqueezeSeg achieves high accuracy with astonishingly fast and stable runtime (8.7±0.5 ms per frame), highly desirable for autonomous driving. Furthermore, additionally training on synthesized data boosts validation accuracy on real-world data. Our source code is open-source released1. The paper is accompanied by a video2containing a high level introduction and demonstrations of this work.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A spherical map represents each 3D point densely and compactly, making it suitable for point cloud segmentation, but its size can differ from camera images, making early fusion difficult.
“A spherical map is obtained by projecting each 3D point onto a sphere, characterized by azimuth and zenith angles. It has the advantage of representing each 3D point in a dense and compact way, making it a suitable representation for point cloud segmentation (e.g. [51]). However, the size of the representation can be different from camera images. Therefore, it is difficult to fuse them at an early”
From Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2021) · cites this paper as [51]
RaLF3D, using a SqueezeSeg backbone, achieves an mIoU of 37.8 on the KITTI dataset for 2D + 3D semantic segmentation.
“| RaLF3D [119] | SqueezeSeg [33] | 37.8 |”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [33]
Cites
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392 · see citing works
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897 · see citing works
Multi-view 3D Object Detection Network for Autonomous Driving
Chen, X., Ma, H., Wan, J., Li, B., & Xia, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.691 · Cited by 3,450 · see citing works
Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
Krähenbühl, P., & Koltun, V. (2012) · arXiv · doi:10.48550/arxiv.1210.5644 · Cited by 2,908 · see citing works
Conditional Random Fields as Recurrent Neural Networks
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., & Torr, P. H. S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.179 · Cited by 2,386 · cited 6× · see citing works
“In [11], mean-field iteration is reformulated as a recurrent neural network (RNN) module.”
BackgroundMethodInfluentialStanley: The robot that won the DARPA Grand Challenge
Thrun, S., Montemerlo, M., Dahlkamp, H., Stavens, D., Aron, A., Diebel, J., Fong, P., Gale, J., Halpenny, M., Hoffmann, G., et al. (2006) · Journal of Field Robotics · doi:10.1002/rob.20147 · Cited by 2,237 · see citing works
Playing for Data: Ground Truth from Computer Games
Richter, S. R., Vineet, V., Roth, S., & Koltun, V. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46475-6_7 · Cited by 1,928 · see citing works
Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?
Johnson‐Roberson, M., Barto, C., Mehta, R., Sridhar, S. N., Rosaen, K., & Vasudevan, R. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989092 · Cited by 649 · see citing works
Wu, B., Wan, A., Iandola, F., Jin, P. H., & Keutzer, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.60 · Cited by 614 · see citing works
Self-citationVehicle Detection from 3D Lidar Using Fully Convolutional Network
Li, B., Zhang, T., & Xia, T. (2016) · Robotics: Science and Systems XII · doi:10.15607/rss.2016.xii.042 · Cited by 577 · see citing works
On the segmentation of 3D LIDAR point clouds
Douillard, B., Underwood, J., Kuntz, N. R., Vlaskine, V., Quadros, A., Morton, P., & Frenkel, A. (2011) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2011.5979818 · Cited by 477 · see citing works
Fast segmentation of 3D point clouds: A paradigm on LiDAR data for autonomous vehicle applications
Zermas, D., Izzat, I. H., & Papanikolopoulos, N. P. (2017) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2017.7989591 · Cited by 371 · see citing works
Segmentation of 3D lidar data in non-flat urban environments using a local convexity criterion
Moosmann, F., Pink, O., & Stiller, C. (2009) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2009.5164280 · Cited by 302 · see citing works
Fast LIDAR-based road detection using fully convolutional neural networks
Caltagirone, L., Scheidegger, S., Svensson, L., & Wahde, M. (2017) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2017.7995848 · Cited by 246 · see citing works
3D Convolutional Neural Networks for landing zone detection from LiDAR
Maturana, D., & Scherer, S. (2015) · 2015 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2015.7139679 · Cited by 203 · see citing works
Fusing LIDAR and images for pedestrian detection using convolutional neural networks
Schlosser, J., Chow, C. K., & Kira, Z. (2016) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2016.7487370 · Cited by 138 · see citing works
LIDAR-based 3D Object Perception
Himmelsbach, M., Mueller, A., Luettel, T., et al. (2008) · Cited by 107 · see citing works
What could move? Finding cars, pedestrians and bicyclists in 3D laser data
Wang, D. Z., Posner, I., & Newman, P. M. (2012) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2012.6224734 · Cited by 103 · see citing works
Real-Time and Accurate Segmentation of 3-D Point Clouds Based on Gaussian Process Regression
Shin, M.-O., Oh, G.-M., Kim, S.-W., et al. (2017) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2017.2685523 · Cited by 50 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences
Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C., & Gall, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00939 · Cited by 2,294 · see citing works
RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds
Hu, Q., Yang, B., Xie, L., Rosa, S., Guo, Y., Wang, Z., Trigoni, N., & Markham, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01112 · Cited by 2,275 · see citing works
Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425 · see citing works
Robust Multi-Modality Multi-Object Tracking
Zhang, W., Zhou, H., Sun, S., Wang, Z., Shi, J., & Loy, C. C. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00245 · Cited by 250 · see citing works
xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation
Jaritz, M., Vu, T.-H., de Charette, R., Wirbel, E., & Perez, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01262 · Cited by 224 · see citing works
Perception-Aware Multi-Sensor Fusion for 3D LiDAR Semantic Segmentation
Zhuang, Z., Li, R., Jia, K., Wang, Q., Li, Y., & Tan, M. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.01597 · Cited by 216 · see citing works
RGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving
El Madawi, K., Rashed, H., El Sallab, A., Nasr, O., Kamel, H., & Yogamani, S. (2019) · 2019 IEEE Intelligent Transportation Systems Conference (ITSC) · doi:10.1109/itsc.2019.8917447 · Cited by 116 · see citing works
Precise Synthetic Image and LiDAR (PreSIL) Dataset for Autonomous Vehicle Perception
Hurl, B., Czarnecki, K., & Waslander, S. (2019) · 2019 IEEE Intelligent Vehicles Symposium (IV) · doi:10.1109/ivs.2019.8813809 · Cited by 122 · see citing works
Are We Hungry for 3D LiDAR Data for Semantic Segmentation? A Survey of Datasets and Methods
Gao, B., Pan, Y., Li, C., Geng, S., & Zhao, H. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3076844 · Cited by 105 · see citing works
LiDAR Panoptic Segmentation for Autonomous Driving
Milioto, A., Behley, J., McCool, C., & Stachniss, C. (2020) · 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9340837 · Cited by 98 · see citing works
FuseSeg: LiDAR Point Cloud Segmentation Fusing Multi-Modal Data
Krispel, G., Opitz, M., Waltner, G., Possegger, H., & Bischof, H. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093584 · Cited by 99 · see citing works
Emerging trends in autonomous vehicle perception: Multimodal fusion for 3D object detection
Alaba, S. Y., Gurbuz, A. C., & Ball, J. E. (2024) · World Electric Vehicle Journal · doi:10.3390/wevj15010020 · Cited by 82 · see citing works
Recent Advancements in Learning Algorithms for Point Clouds: An Updated Overview
Camuffo, E., Mari, D., & Milani, S. (2022) · Sensors · doi:10.3390/s22041357 · Cited by 73 · see citing works
Rizzoli, G., Barbato, F., & Zanuttigh, P. (2022) · Technologies · doi:10.3390/technologies10040090 · Cited by 58 · see citing works
Rock Segmentation in the Navigation Vision of the Planetary Rovers
Kuang, B., Wisniewski, M., Rana, Z. A., & Zhao, Y. (2021) · Mathematics · doi:10.3390/math9233048 · Cited by 31 · see citing works
- Authors
- Wu, B., Wan, A., Yue, X., & Keutzer, K.
- Affiliations
- University of California, Berkeley (US)
- Venue
- 2018 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2018-05
- Pages
- 1887-1893
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2962912109
- Cited by
- 967
- References
- 20
- Citations by year
- 2026: 53 · 2025: 110 · 2024: 124 · 2023: 162 · 2022: 140 · 2021: 155 · 2020: 119 · 2019: 88 · 2018: 14 · 2017: 1
- FWCI
- 28.33
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Nvidia
- SK Hynix
- Defense Advanced Research Projects Agency (HR0011-12-2-0016, HR0011)
- Samsung
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
- Keywords
- 3D LiDAR point clouds, semantic segmentation, conditional random fields, instance segmentation, autonomous driving, synthetic training data, lidar simulator, KITTI dataset, real-time segmentation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
