Robust Semantic Segmentation in Adverse Weather Conditions by means of Sensor Data Fusion
Open accessAbstract
A robust and reliable semantic segmentation in adverse weather conditions is very important for autonomous cars, but most state-of-the-art approaches only achieve high accuracy rates in optimal weather conditions. The reason is that they are only optimized for good weather conditions and given noise models. However, most of them fail, if data with unknown disturbances occur, and their performance decrease enormously. One possibility to still obtain reliable results is to observe the environment with different sensor types, such as camera and lidar, and to fuse the sensor data by means of neural networks, since different sensors behave differently in diverse weather conditions. Hence, the sensors can complement each other by means of an appropriate sensor data fusion. Nevertheless, the fusion-based approaches are still susceptible to disturbances and fail to classify disturbed image areas correctly. This problem can be solved by means of a special training method, the so called Robust Learning Method (RLM), a method by which the neural network learns to handle unknown noise. In this work, two different sensor fusion architectures for semantic segmentation are compared and evaluated on several datasets. Furthermore, it is shown that the RLM increases the robustness in adverse weather conditions enormously, and achieve good results although no disturbance model has been learned by the neural network.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
RSSAWC, using an ICNet backbone, achieves an mIoU of 65.09 on the Cityscapes dataset for 2.5D semantic segmentation.
“| RSSAWC [77] | ICNet [123] | 65.09 |”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [77]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Fully convolutional networks for semantic segmentationPreprint
Long, J., Shelhamer, E., & Darrell, T. (2014) · Computer Vision and Pattern Recognition · doi:10.48550/arxiv.1605.06211 · Cited by 43,601 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392 · see citing works
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392 · see citing works
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
Chen, L.-C., Zhu, Y., Papandreou, G., Schroff, F., & Adam, H. (2018) · arXiv · doi:10.48550/arxiv.1802.02611 · Cited by 20,063 · see citing works
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631 · see citing works
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · see citing works
ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
Paszke, A., Chaurasia, A., Kim, S., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1606.02147 · Cited by 1,826 · see citing works
ICNet for Real-Time Semantic Segmentation on High-Resolution Images
Zhao, H., Qi, X., Shen, X., Shi, J., & Jia, J. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01219-9_25 · Cited by 1,709 · see citing works
Semantic Foggy Scene Understanding with Synthetic Data
Sakaridis, C., Dai, D., & Gool, L. V. (2018) · International Journal of Computer Vision · doi:10.1007/s11263-018-1072-8 · Cited by 1,354 · see citing works
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · arXiv · doi:10.48550/arxiv.1604.01685 · Cited by 1,174 · see citing works
FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture
Hazirbas, C., Ma, L., Domokos, C., & Cremers, D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54181-5_14 · Cited by 902 · see citing works
Pyramid Scene Parsing NetworkPreprint
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.01105 · Cited by 321 · see citing works
RDFNet: RGB-D Multi-level Residual Feature Fusion for Indoor Semantic Segmentation
Lee, S., Park, S., & Hong, K.-S. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.533 · Cited by 294 · see citing works
Fast LIDAR-based road detection using fully convolutional neural networks
Caltagirone, L., Scheidegger, S., Svensson, L., & Wahde, M. (2017) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2017.7995848 · Cited by 246 · see citing works
Semantic Segmentation on Radar Point Clouds
Schumann, O., Hahn, M., Dickmann, J., & Wöhler, C. (2018) · International Conference on Information Fusion (FUSION) · doi:10.23919/icif.2018.8455344 · Cited by 185 · see citing works
Learning Common and Specific Features for RGB-D Semantic Segmentation with Deconvolutional Networks
Wang, J., Wang, Z., Tao, D., See, S. C. W., & Wang, G. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46454-1_40 · Cited by 169 · see citing works
Joint 3D Proposal Generation and Object Detection from View AggregationPreprint
Ku, J., Mozifian, M., Lee, J., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.02294 · Cited by 134 · see citing works
ICNet for Real-Time Semantic Segmentation on High-Resolution ImagesPreprint
Zhao, H., Qi, X., Shen, X., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1704.08545 · Cited by 129 · see citing works
Autonomous driving at Ulm University: A modular, robust, and sensor-independent fusion approach
Kunz, F., Nuß, D., Wiest, J., Deusch, H., Reuter, S., Gritschneder, F., Scheel, A., Stübler, M., Bach, M., Hatzelmann, P., et al. (2015) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2015.7225761 · Cited by 121 · see citing works
Self-citationI Can See Clearly Now: Image Restoration via De-Raining
Porav, H., Bruls, T., & Newman, P. M. (2019) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2019.8793486 · Cited by 103 · see citing works
Recurrent Fully Convolutional Networks for Video Segmentation
Valipour, S., Siam, M., Jägersand, M., & Ray, N. (2017) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2017.11 · Cited by 83 · see citing works
Boosting LiDAR-Based Semantic Labeling by Cross-modal Training Data Generation
Piewak, F., Pinggera, P., Schäfer, M., Peter, D., Schwarz, B., Schneider, N., Enzweiler, M., Pfeiffer, D., & Zöllner, M. J. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-11024-6_39 · Cited by 78 · see citing works
Multi-View 3D Object Detection Network for Autonomous DrivingPreprint
Chen, X., Ma, H., Wan, J., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.07759 · Cited by 65 · see citing works
Deep semantic classification for 3D LiDAR data
Dewan, A., Oliveira, G. L., & Burgard, W. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206198 · Cited by 60 · see citing works
Optimal Sensor Data Fusion Architecture for Object Detection in Adverse Weather Conditions
Pfeuffer, A., & Dietmayer, K. (2018) · International Conference on Information Fusion (FUSION) · doi:10.23919/icif.2018.8455757 · Cited by 54 · see citing works
Self-citationSemantic Segmentation of Video Sequences with Convolutional LSTMs
Pfeuffer, A., Schulz, K., & Dietmayer, K. (2019) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2019.8813852 · Cited by 45 · see citing works
Self-citationFast LIDAR-based Road Detection Using Fully Convolutional Neural NetworksPreprint
Caltagirone, L., Scheidegger, S., Svensson, L., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.03613 · Cited by 24 · see citing works
Multi-View Deep Learning for Consistent Semantic Mapping with RGB-D CamerasPreprint
Ma, L., Stückler, J., Kerl, C., et al. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.08866 · Cited by 19 · see citing works
Semantic Segmentation of RGBD Videos with Recurrent Fully Convolutional Neural Networks
Yurdakul, E. E., & Yemez, Y. (2017) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2017.51 · Cited by 16 · see citing works
Robustness Against Unknown Noise for Raw Data Fusing Neural Networks
Bijelic, M., Muench, C., Ritter, W., Kalnishkan, Y., & Dietmayer, K. (2018) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2018.8569911 · Cited by 14 · see citing works
Self-citationLearning Common and Specific Features for RGB-D Semantic Segmentation with Deconvolutional NetworksPreprint
Wang, J., Wang, Z., Tao, D., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.01082 · Cited by 12 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Rizzoli, G., Barbato, F., & Zanuttigh, P. (2022) · Technologies · doi:10.3390/technologies10040090 · Cited by 58 · see citing works
Missing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentation
Maheshwari, H., Liu, Y.-C., & Kira, Z. (2024) · 2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv57701.2024.00106 · Cited by 37 · see citing works
Transformer-based sensor fusion for autonomous vehicles: A comprehensive review
Abdulmaksoud, A., & Ahmed, R. (2025) · IEEE Access · doi:10.1109/access.2025.3545032 · Cited by 25 · see citing works
Das, A., Krizek, P., Sistu, G., Burger, F., Madasamy, S., Uricar, M., Kumar, V. R., & Yogamani, S. (2020) · 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc45102.2020.9294677 · Cited by 25 · see citing works
- Authors
- Pfeuffer, A., & Dietmayer, K.
- Affiliations
- Universität Ulm (DE)
- ORCID
- Dietmayer, Klaus
- Venue
- 2019 22th International Conference on Information Fusion (FUSION)
- Publisher
- IEEE
- Published
- 2019-07
- Pages
- 1-8
- OpenAlex
- W2946824359
- Cited by
- 51
- References
- 32
- Citations by year
- 2026: 1 · 2025: 10 · 2024: 6 · 2023: 13 · 2022: 6 · 2021: 7 · 2020: 3 · 2019: 5
- FWCI
- 5.23
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- International Conference on Information Fusion (FUSION) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- semantic segmentation, severe weather, sensor fusion, LiDAR-camera fusion, autonomous driving, unknown noise, robust neural networks, sensor fusion architecture
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
