CoChat
Sign Up
Cited by 17Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot Learning3D Shape Modeling and Analysis

Understanding the Challenges When 3D Semantic Segmentation Faces Class Imbalanced and OOD Data

Open access
Pan, Yancheng, Xie, Fan, Zhao, Huijing·2023·IEEE Transactions on Intelligent Transportation Systems
DOI 10.1109/tits.2023.3256442
Open this paper in CoChat — free accountOpen PDF

Abstract

3D semantic segmentation (3DSS) is an essential process in the creation of a safe autonomous driving system. However, deep learning models for 3D semantic segmentation often suffer from the class imbalance problem and out-of-distribution (OOD) data. In this study, we explore how the class imbalance problem affects 3DSS performance and whether the model can detect the category prediction correctness, or whether data is ID or OOD. For these purposes, we conduct two experiments using four representative 3DSS models and five trust scoring methods, and conduct both a confusion and feature analysis of each class. Furthermore, a data augmentation method for the 3D LiDAR dataset is proposed to create a new dataset based on SemanticKITTI and SemanticPOSS, called AugKITTI. We propose the wPre metric and TSD for a more in-depth analysis of the results, and follow are proposals with an insightful discussion. Based on the experimental results, we find that: 1) classes are not only imbalanced in their data size but also in the basic properties of each semantic category; 2) intraclass diversity and interclass ambiguity make class learning difficult and greatly limit the models’ performance, creating the challenges of semantic and data gaps; 3) trust scores are unreliable for classes whose features are confused with other classes. For 3DSS models, those misclassified ID classes and OODs may also be given high trust scores, making the 3DSS predictions unreliable, and leading to the challenges in judging 3DSS result trustworthiness. All of these outcomes point to several research directions for improving the performance and reliability of the 3DSS models used for real-world applications.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/tits.2023.3256442

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The highly-discussed dimension target class balance pays tribute to the technical requirements of ML.

    “Finally, the highly-discussed dimension target class balance pays tribute to the technical requirements of ML140,141,144,150,159.”

    From The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review (2024) · cites this paper as [144]

  • Most DL papers in the literature corpus investigate the robustness of a model, while only few investigate generalisability or distribution shift.

    “When it comes to the concrete behaviour change of the DL algorithm, most of the DL papers in the literature corpus investigate the robustness of a model, i.e. the stable behaviour of a model when facing erroneous or a limited amount of inputs. Only few records investigate generalisability119,144,145 or distribution shift139,192, a model’s capability of coping with new, unseen data.”

    From The METRIC-framework for assessing data quality for trustworthy AI in medicine: A systematic review (2024) · cites this paper as [144]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 55

  1. H. Yu, Z. Yang, L. Tan, Y. Wang, W. Sun, M. Sun, and Y. Tang, "Meth- ods and datasets on semantic segmentation: A review," Neurocomputing, vol. 304, pp. 82-103, 2018.
  2. F. Lateef and Y. Ruichek, "Survey on semantic segmentation using deep learning techniques," Neurocomputing, vol. 338, pp. 321-348, 2019.
  3. Y. Xie, J. Tian, and X. X. Zhu, "A review of point cloud semantic segmentation," arXiv preprint arXiv:1908.08854, 2019. · arXiv
  4. B. J. Patz, Y. Papelis, R. Pillat, G. Stein, and D. Harper, "A practical approach to robotic design for the DARPA urban challenge," Journal of Field Robotics, vol. 25, no. 8, pp. 528-566, 2008.
  5. X. Chen, H. Ma, J. Wan, B. Li, and T. Xia, "Multi-view 3D object detection network for autonomous driving," in IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 1907-1915.
  6. Y. Guo, H. Wang, Q. Hu, H. Liu, L. Liu, and M. Bennamoun, "Deep learning for 3D point clouds: A survey," arXiv preprint arXiv:1912.12033, 2019. · arXiv
  7. G. Haixiang, L. Yijing, J. Shang, G. Mingyun, H. Yuanyue, and G. Bing, "Learning from class-imbalanced data: Review of methods and applications," Expert Systems with Applications, vol. 73, pp. 220-239, 2017.
  8. D. Hendrycks and K. Gimpel, "A baseline for detecting misclassified and out-of-distribution examples in neural networks," arXiv preprint arXiv:1610.02136, 2016. · arXiv
  9. P. Koopman and M. Wagner, "Autonomous vehicle safety: An interdis- ciplinary challenge," IEEE Intelligent Transportation Systems Magazine, vol. 9, no. 1, pp. 90-96, 2017.
  10. M. Salehi, H. Mirzaei, D. Hendrycks, Y. Li, M. H. Rohban, and M. Sabokrou, "A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges," arXiv preprint arXiv:2110.14051, 2021. · arXiv
  11. A. Nguyen, J. Yosinski, and J. Clune, "Deep neural networks are easily fooled: High confidence predictions for unrecognizable images," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 427-436.
  12. C. R. Qi, L. Yi, H. Su, and L. J. Guibas, "PointNet++: Deep hierarchical feature learning on point sets in a metric space," in Advances in Neural Information Processing Systems, 2017, pp. 5099-5108.
  13. H. Zhou, X. Zhu, X. Song, Y. Ma, Z. Wang, H. Li, and D. Lin, "Cylinder3d: An effective 3d framework for driving-scene lidar semantic segmentation," arXiv preprint arXiv:2008.01550, 2020. · arXiv
  14. Q. Hu, B. Yang, L. Xie, S. Rosa, Y. Guo, Z. Wang, N. Trigoni, and A. Markham, "RandLA-Net: Efficient semantic segmentation of large- scale point clouds," arXiv preprint arXiv:1911.11236, 2019. · arXiv
  15. Y. Pan, B. Gao, J. Mei, S. Geng, C. Li, and H. Zhao, "SemanticPOSS: A point cloud dataset with large quantity of dynamic instances," arXiv preprint arXiv:2002.09147, 2020. · arXiv
  16. A. Kendall and Y. Gal, "What uncertainties do we need in bayesian deep learning for computer vision?" arXiv preprint arXiv:1703.04977, 2017. · arXiv
  17. A. Malinin, "Uncertainty estimation in deep learning with application to spoken language assessment," Ph.D. dissertation, University of Cam- bridge, 2019.
  18. S. Liang, Y. Li, and R. Srikant, "Enhancing the reliability of out- of-distribution image detection in neural networks," arXiv preprint arXiv:1706.02690, 2017. · arXiv
  19. K. Lee, K. Lee, H. Lee, and J. Shin, "A simple unified framework for detecting out-of-distribution samples and adversarial attacks," Advances in neural information processing systems, vol. 31, 2018.
  20. C. R. Qi, H. Su, K. Mo, and L. J. Guibas, "PointNet: Deep learning on point sets for 3D classification and segmentation," in IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 652-660.
  21. F. Engelmann, T. Kontogianni, A. Hermans, and B. Leibe, "Exploring spatial context for 3D semantic segmentation of point clouds," in IEEE International Conference on Computer Vision Workshops, 2017, pp. 716-724.
  22. M. Jiang, Y. Wu, T. Zhao, Z. Zhao, and C. Lu, "PointSIFT: A SIFT- like network module for 3D point cloud semantic segmentation," arXiv preprint arXiv:1807.00652, 2018. · arXiv
  23. F. Engelmann, T. Kontogianni, J. Schult, and B. Leibe, "Know what your neighbors do: 3D semantic segmentation of point clouds," in European Conference on Computer Vision, 2018.
  24. L.-Z. Chen, X.-Y. Li, D.-P. Fan, M.-M. Cheng, K. Wang, and S.-P. Lu, "LSANet: Feature learning on point sets by local spatial attention," arXiv preprint arXiv:1905.05442, 2019. · arXiv
  25. K. Zhiheng and L. Ning, "PyramNet: Point cloud pyramid attention net- work and graph embedding module for classification and segmentation," arXiv preprint arXiv:1906.03299, 2019. · arXiv
  26. H. Thomas, C. R. Qi, J.-E. Deschaud, B. Marcotegui, F. Goulette, and L. J. Guibas, "KPConv: Flexible and deformable convolution for point clouds," in IEEE International Conference on Computer Vision, 2019, pp. 6411-6420.
  27. W. Wu, Z. Qi, and L. Fuxin, "Pointconv: Deep convolutional networks on 3d point clouds," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 9621-9630.
  28. J. Huang and S. You, "Point cloud labeling using 3D convolutional neural network," in International Conference on Pattern Recognition. IEEE, 2016, pp. 2670-2675.
  29. L. Tchapmi, C. Choy, I. Armeni, J. Gwak, and S. Savarese, "Segcloud: Semantic segmentation of 3D point clouds," in 2017 international conference on 3D vision. IEEE, 2017, pp. 537-547.
  30. D. Rethage, J. Wald, J. Sturm, N. Navab, and F. Tombari, "Fully- convolutional point networks for large-scale point clouds," in European Conference on Computer Vision, 2018, pp. 596-611.
  31. B. Graham, M. Engelcke, and L. van der Maaten, "3D semantic segmentation with submanifold sparse convolutional networks," in IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 9224-9232.
  32. C. Zhang, W. Luo, and R. Urtasun, "Efficient convolutions for real-time semantic segmentation of 3D point clouds," in International Conference on 3D Vision. IEEE, 2018, pp. 399-408.
  33. B. Wu, A. Wan, X. Yue, and K. Keutzer, "SqueezeSeg: Convolutional neural nets with recurrent CRF for real-time road-object segmentation from 3D LiDAR point cloud," in IEEE International Conference on Robotics and Automation. IEEE, 2018, pp. 1887-1893.
  34. W. Zhang, C. Zhou, J. Yang, and K. Huang, "LiSeg: Lightweight road-object semantic segmentation in 3D LiDAR scans for autonomous driving," in IEEE Intelligent Vehicles Symposium. IEEE, 2018, pp. 1021-1026.
  35. Y. Wang, T. Shi, P. Yun, L. Tai, and M. Liu, "PointSeg: Real-time semantic segmentation based on 3D LiDAR point cloud," arXiv preprint arXiv:1807.06288, 2018. · arXiv
  36. A. Dewan and W. Burgard, "DeepTemporalSeg: Temporally consis- tent semantic segmentation of 3D LiDAR scans," arXiv preprint arXiv:1906.06962, 2019. · arXiv
  37. B. Wu, X. Zhou, S. Zhao, X. Yue, and K. Keutzer, "SqueezeSegV2: Improved model structure and unsupervised domain adaptation for road-object segmentation from a LiDAR point cloud," in International Conference on Robotics and Automation. IEEE, 2019, pp. 4376-4382.
  38. A. Milioto, I. Vizzo, J. Behley, and C. Stachniss, "RangeNet++: Fast and accurate LiDAR semantic segmentation," in IEEE/RSJ International Conference on Intelligent Robots and Systems, 2019.
  39. C. Xu, B. Wu, Z. Wang, W. Zhan, P. Vajda, K. Keutzer, and M. Tomizuka, "SqueezeSegV3: Spatially-adaptive convolution for effi- cient point-cloud segmentation," arXiv preprint arXiv:2004.01803, 2020. · arXiv
  40. J. M. Johnson and T. M. Khoshgoftaar, "Survey on deep learning with class imbalance," Journal of Big Data, vol. 6, no. 1, pp. 1-54, 2019.
  41. Y. Zhang, B. Kang, B. Hooi, S. Yan, and J. Feng, "Deep long-tailed learning: A survey," arXiv preprint arXiv:2110.04596, 2021. · arXiv
  42. K. Ghosh, C. Bellinger, R. Corizzo, B. Krawczyk, and N. Japkowicz, "On the combined effect of class imbalance and concept complexity in deep learning," arXiv preprint arXiv:2107.14194, 2021. · arXiv
  43. G. Menardi and N. Torelli, "Training and assessing classification rules with imbalanced data," Data mining and knowledge discovery, vol. 28, no. 1, pp. 92-122, 2014.
  44. S. Pouyanfar, Y. Tao, A. Mohan, H. Tian, A. S. Kaseb, K. Gauen, R. Dailey, S. Aghajanzadeh, Y.-H. Lu, S.-C. Chen et al., "Dynamic sampling in convolutional neural networks for imbalanced data classifi- cation," in 2018 IEEE conference on multimedia information processing and retrieval (MIPR). IEEE, 2018, pp. 112-117.
  45. T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, "Focal loss for dense object detection," in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2980-2988.
  46. Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie, "Class-balanced loss based on effective number of samples," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 9268- 9277.
  47. X. Chen, C. Zhang, G. Lin, and J. Han, "Compositional prototype network with multi-view comparision for few-shot point cloud semantic segmentation," arXiv preprint arXiv:2012.14255, 2020. · arXiv
  48. V. Sehwag, A. N. Bhagoji, L. Song, C. Sitawarin, D. Cullina, M. Chiang, and P. Mittal, "Analyzing the robustness of open-world machine learn- ing," in Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security, 2019, pp. 105-116.
  49. D. Hendrycks, M. Mazeika, and T. Dietterich, "Deep anomaly detection with outlier exposure," arXiv preprint arXiv:1812.04606, 2018. · arXiv
  50. A.-A. Papadopoulos, M. R. Rajati, N. Shaikh, and J. Wang, "Outlier exposure with confidence control for out-of-distribution detection," Neurocomputing, vol. 441, pp. 138-150, 2021.
  51. Y. Xia, X. Cao, F. Wen, G. Hua, and J. Sun, "Learning discriminative reconstructions for unsupervised outlier removal," in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 1511- 1519.
  52. L. Ruff, R. Vandermeulen, N. Goernitz, L. Deecke, S. A. Siddiqui, A. Binder, E. Müller, and M. Kloft, "Deep one-class classification," in International conference on machine learning. PMLR, 2018, pp. 4393-4402.
  53. B. Lakshminarayanan, A. Pritzel, and C. Blundell, "Simple and scalable predictive uncertainty estimation using deep ensembles," arXiv preprint arXiv:1612.01474, 2016. · arXiv
  54. H. Jiang, B. Kim, M. Y. Guan, and M. Gupta, "To trust or not to trust a classifier," arXiv preprint arXiv:1805.11783, 2018. · arXiv
  55. Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. V. Dillon, B. Lakshminarayanan, and J. Snoek, "Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift," arXiv preprint arXiv:1906.02530, 2019. · arXiv

Cited by

Sorted by citations

Authors
Pan, Y., Xie, F., & Zhao, H.
Affiliations
Peking University (CN)
ORCID
Venue
IEEE Transactions on Intelligent Transportation Systems
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2023-07
Volume
24
Issue
7
Pages
6955-6970
ISSN
  • 1524-9050 (ISSN-L)
  • 1558-0016
DOI
10.1109/tits.2023.3256442
OpenAlex
W4360584269
Cited by
17
References
8
Citations by year
2026: 4 · 2025: 7 · 2024: 4 · 2023: 2
FWCI
1.55
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Transactions on Intelligent Transportation Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • arXiv (Cornell University)
Funders
  • National Natural Science Foundation of China (61973004, U22A2061)
  • High-Performance Computing Platform of Peking University
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Engineering › Computational Mechanics › 3D Shape Modeling and Analysis
Keywords
3D semantic segmentation, class imbalance, out-of-distribution detection, trust evaluation, SemanticKITTI, autonomous driving, semantic gap, feature analysis, confidence calibration

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review