The DOI matches its Crossref record; 63 of the 65 references it cites were found in a registry or an index.
Perception-Aware Multi-Sensor Fusion for 3D LiDAR Semantic Segmentation
Abstract
3D LiDAR (light detection and ranging) semantic segmentation is important in scene understanding for many applications, such as auto-driving and robotics. For example, for autonomous cars equipped with RGB cameras and LiDAR, it is crucial to fuse complementary information from different sensors for robust and accurate segmentation. Existing fusion-based methods, however, may not achieve promising performance due to the vast difference between the two modalities. In this work, we investigate a collaborative fusion scheme called perception-aware multi-sensor fusion (PMF) to exploit perceptual information from two modalities, namely, appearance information from RGB images and spatio-depth information from point clouds. To this end, we first project point clouds to the camera coordinates to provide spatio-depth information for RGB images. Then, we propose a two-stream network to extract features from the two modalities, separately, and fuse the features by effective residual-based fusion modules. Moreover, we propose additional perception-aware losses to measure the perceptual difference between the two modalities. Extensive experiments on two benchmark data sets show the superiority of our method. For example, on nuScenes, our PMF outperforms the state-of-the-art method by 0.8% in mIoU.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“| PMF [107] | ResNet34 [16] | 63.9 |”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [107]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473
Focal Loss for Dense Object Detection
Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.324 · Cited by 30,581
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438
Are we ready for autonomous driving? The KITTI vision benchmark suite
Geiger, A., Lenz, P., & Urtasun, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248074 · Cited by 14,897
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433
YOLOv4: Optimal Speed and Accuracy of Object Detection
Bochkovskiy, A., Wang, C.-Y., & Liao, H.-Y. M. (2020) · arXiv · doi:10.48550/arxiv.2004.10934 · Cited by 10,307
PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric SpacePreprint
Qi, C. R., Yi, L., Su, H., & Guibas, L. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02413 · Cited by 10,099
PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
Charles, R., Su, H., Mo, K., & Guibas, L. J. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.16 · Cited by 10,029
Vision meets robotics: The KITTI dataset
Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013) · The International Journal of Robotics Research · doi:10.1177/0278364913491297 · Cited by 9,913
Rethinking Atrous Convolution for Semantic Image Segmentation
Chen, L.-C., Papandreou, G., Schroff, F., & Adam, H. (2017) · arXiv · doi:10.48550/arxiv.1706.05587 · Cited by 7,517
nuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152
Two-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756
SGDR: Stochastic Gradient Descent with Warm Restarts
Loshchilov, I., & Hutter, F. (2016) · arXiv · doi:10.48550/arxiv.1608.03983 · Cited by 5,309
Han, S., Mao, H., & Dally, W. J. (2015) · arXiv · doi:10.48550/arxiv.1510.00149 · Cited by 4,662
On Measures of Entropy and Information
Rényi, A. (1961) · Project Euclid (Cornell University) · Cited by 4,031
Pedestrian Detection: An Evaluation of the State of the Art
Dollár, P., Wojek, C., Schiele, B., & Perona, P. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.155 · Cited by 3,232
CCNet: Criss-Cross Attention for Semantic Segmentation
Huang, Z., Wang, X., Huang, L., Huang, C., Wei, Y., & Liu, W. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00069 · Cited by 3,045
Convolutional Two-Stream Network Fusion for Video Action Recognition
Feichtenhofer, C., Pinz, A. J., & Zisserman, A. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.213 · Cited by 2,842
Lego-loam: Lightweight and ground-optimized lidar odometry and mapping on variable terrain
Shan, T., & Englot, B. (2018) · 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2018.8594299 · Cited by 2,295
SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences
Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C., & Gall, J. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00939 · Cited by 2,294
RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds
Hu, Q., Yang, B., Xie, L., Rosa, S., Guo, Y., Wang, Z., Trigoni, N., & Markham, A. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01112 · Cited by 2,275
Understanding Convolution for Semantic Segmentation
Wang, P., Chen, P., Yuan, Y., Liu, D., Huang, Z., Hou, X., & Cottrell, G. W. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00163 · Cited by 2,184
4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks
Choy, C., Gwak, J., & Savarese, S. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00319 · Cited by 2,067
Zhang, Y., Xiang, T., Hospedales, T. M., & Lu, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00454 · Cited by 1,933
3D Semantic Segmentation with Submanifold Sparse Convolutional Networks
Graham, B., Engelcke, M., & Maaten, L. V. D. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00961 · Cited by 1,788
RangeNet ++: Fast and Accurate LiDAR Semantic Segmentation
Milioto, A., Vizzo, I., Behley, J., & Stachniss, C. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8967762 · Cited by 1,344
PointPainting: Sequential Fusion for 3D Object Detection
Vora, S., Lang, A. H., Helou, B., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00466 · Cited by 1,236
Berman, M., Triki, A. R., & Blaschko, M. B. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00464 · Cited by 1,163
Towards 3D Point cloud based object maps for household environments
Rusu, R. B., Márton, Z.-C., Blodow, N., Dolha, M. E., & Beetz, M. (2008) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2008.08.005 · Cited by 1,087
Segmentation and Recognition Using Structure from Motion Point Clouds
Brostow, G. J., Shotton, J., Fauqueur, J., & Cipolla, R. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_5 · Cited by 1,050
ESPNet: Efficient Spatial Pyramid of Dilated Convolutions for Semantic Segmentation
Mehta, S., Rastegari, M., Caspi, A., Shapiro, L., & Hajishirzi, H. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01249-6_34 · Cited by 1,029
Wu, B., Wan, A., Yue, X., & Keutzer, K. (2018) · 2018 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra.2018.8462926 · Cited by 967
Efficient Piecewise Training of Deep Structured Models for Semantic Segmentation
Lin, G., Shen, C., van den Hengel, A., & Reid, I. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.348 · Cited by 862
Once-for-All: Train One Network and Specialize it for Efficient Deployment
Cai, H., Gan, C., Wang, T., Zhang, Z., & Han, S. (2019) · arXiv · doi:10.48550/arxiv.1908.09791 · Cited by 845
Searching Efficient 3D Architectures with Sparse Point-Voxel Convolution
Tang, H., Liu, Z., Zhao, S., Lin, Y., Lin, J., Wang, H., & Han, S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58604-1_41 · Cited by 810
Wu, B., Zhou, X., Zhao, S., Yue, X., & Keutzer, K. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793495 · Cited by 785
Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation
Zhu, X., Zhou, H., Wang, T., Hong, F., Ma, Y., Li, W., Li, H., & Lin, D. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00981 · Cited by 680
PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation
Zhang, Y., Zhou, Z., David, P., Yue, X., Xi, Z., Gong, B., & Foroosh, H. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00962 · Cited by 621
OCNet: Object Context Network for Scene ParsingPreprint
Yuan, Y., Huang, L., Guo, J., Zhang, C., Chen, X., & Wang, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.00916 · Cited by 520
SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation
Xu, C., Wu, B., Wang, Z., Zhan, W., Vajda, P., Keutzer, K., & Tomizuka, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58604-1_1 · Cited by 448
LPD-Net: 3D Point Cloud Learning for Large-Scale Place Recognition and Environment Analysis
Liu, Z., Zhou, S., Suo, C., Yin, P., Chen, W., Wang, H., Li, H., & Liu, Y. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00292 · Cited by 336
DARTS: Deceiving Autonomous Cars with Toxic SignsPreprint
Sitawarin, C., Bhagoji, A. N., Mosenia, A., Chiang, M., & Mittal, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.06430 · Cited by 252
xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation
Jaritz, M., Vu, T.-H., de Charette, R., Wirbel, E., & Perez, P. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01262 · Cited by 224
SalsaNet: Fast Road and Vehicle Segmentation in LiDAR Point Clouds for Autonomous Driving
Aksoy, E. E., Baci, S., & Cavdar, S. (2020) · IEEE Intelligent Vehicles Symposium · doi:10.1109/iv47402.2020.9304694 · Cited by 194
Deep FusionNet for Point Cloud Semantic Segmentation
Zhang, F., Fang, J., Wah, B. W.-S., & Torr, P. H. S. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58586-0_38 · Cited by 185
Discrimination-aware Network Pruning for Deep Model Compression
Liu, J., Zhuang, B., Zhuang, Z., Guo, Y. F., Huang, J., Zhu, J., & Tan, M. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3066410 · Cited by 164
Self-citationSensor Fusion for Joint 3D Object Detection and Semantic Segmentation
Meyer, G. P., Charland, J., Hegde, D. K., Laddha, A., & Vallespi-Gonzalez, C. (2019) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2019.00162 · Cited by 162
Self-Supervised Moving Vehicle Tracking With Stereo Sound
Gan, C., Zhao, H., Chen, P., Cox, D., & Torralba, A. B. (2019) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00715 · Cited by 150
Generative Low-Bitwidth Data Free Quantization
Xu, S., Li, H., Zhuang, B., Liu, J., Cao, J., Liang, C., & Tan, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58610-2_1 · Cited by 141
Self-citationRGB and LiDAR fusion based 3D Semantic Segmentation for Autonomous Driving
El Madawi, K., Rashed, H., El Sallab, A., Nasr, O., Kamel, H., & Yogamani, S. (2019) · 2019 IEEE Intelligent Transportation Systems Conference (ITSC) · doi:10.1109/itsc.2019.8917447 · Cited by 116
Towards Accurate and Compact Architectures via Neural Architecture Transformer
Guo, Y. F., Zheng, Y., Tan, M., Chen, Q., Li, Z., Chen, J., Zhao, P., & Huang, J. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3086914 · Cited by 106
Self-citationFuseSeg: LiDAR Point Cloud Segmentation Fusing Multi-Modal Data
Krispel, G., Opitz, M., Waltner, G., Possegger, H., & Bischof, H. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093584 · Cited by 99
NAT: Neural Architecture Transformer for Accurate and Compact ArchitecturesPreprint
Guo, Y. F., Zheng, Y., Tan, M., Chen, Q., Chen, J., Zhao, P., & Huang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.14488 · Cited by 67
Self-citationSalsaNext: Fast Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving.
Cortinhal, T., Tzelepis, G., & Aksoy, E. E. (2020) · Cited by 37
Disturbance-immune weight sharing for neural architecture search
Niu, S., Wu, J., Zhang, Y., Guo, Y. F., Zhao, P., Huang, J., & Tan, M. (2021) · Neural Networks · doi:10.1016/j.neunet.2021.09.002 · Cited by 23
Self-citationMulti-Modality Cut and Paste for 3D Object Detection.Preprint
Zhang, W., Wang, Z., & Loy, C. C. (2020) · arXiv (Cornell University) · Cited by 18
Research works printed but not linked: 3
- xu, 2020, Generative lowbitwidth data free quantization, European Conference on Computer Vision, 1
- geiger, 2012, A toolbox for automatic calibration of range and camera sensors using a single shot, International Conference on Robotics and Automation
- nesterov, 1983, A method for solving the convex programming problem with convergence rate o (1/kˆ 2), Proceedings of the USSR Academy of Sciences, 269, 543
Cited by
Sorted by citations
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers
Zhang, J., Liu, H., Yang, K., Hu, X., Liu, R., & Stiefelhagen, R. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3300537 · Cited by 731
2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds
Yan, X., Gao, J., Zheng, C., Zheng, C., Zhang, R., Cui, S. R., & Li, Z. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19815-1_39 · Cited by 301
Spherical Transformer for LiDAR-Based 3D Recognition
Lai, X. Q., Chen, Y., Lu, F., Liu, J., & Jia, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01683 · Cited by 248
Muhammad, K., Hussain, T., Ullah, H., Ser, J. D., Rezaei, M., Kumar, N., Hijji, M., Bellavista, P., & de Albuquerque, V. H. C. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3207665 · Cited by 232
Delivering Arbitrary-Modal Semantic Segmentation
Zhang, J., Liu, R., Shi, H., Yang, K., Reiß, S., Peng, K., Fu, H., Wang, K., & Stiefelhagen, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00116 · Cited by 215
Metadata
- Authors
- Zhuang, Z., Li, R., Jia, K., Wang, Q., Li, Y., & Tan, M.
- Affiliations
- South China University of Technology (CN)
- Shenzhen Youjia Innov Tech Co., Ltd
- Venue
- 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
- Publisher
- IEEE
- Published
- 2021-10
- Pages
- 16260-16270
- OpenAlex
- W3173732446
- Cited by
- 216
- References
- 65
- Citations by year
- 2026: 26 · 2025: 58 · 2024: 68 · 2023: 47 · 2022: 16 · 2021: 1
- FWCI
- 10.79
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/CVF International Conference on Computer Vision (ICCV) (publishedVersion)
- Funders
- Research and Development
- Ministry of Science and Technology
- Fundamental Research Funds for the Central Universities
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › 3D Shape Modeling and Analysis
- Keywords
- multi-sensor fusion, RGB-LiDAR fusion, point cloud projection, two-stream network, autonomous driving, nuScenes
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
