AdapNet: Adaptive semantic segmentation in adverse environmental conditions
Abstract
Robust scene understanding of outdoor environments using passive optical sensors is a onerous and essential task for autonomous navigation. The problem is heavily characterized by changing environmental conditions throughout the day and across seasons. Robots should be equipped with models that are impervious to these factors in order to be operable and more importantly to ensure safety in the real-world. In this paper, we propose a novel semantic segmentation architecture and the convoluted mixture of deep experts (CMoDE) fusion technique that enables a multi-stream deep neural network to learn features from complementary modalities and spectra, each of which are specialized in a subset of the input space. Our model adaptively weighs class-specific features of expert networks based on the scene condition and further learns fused representations to yield robust segmentation. We present results from experimentation on three publicly available datasets that contain diverse conditions including rain, summer, winter, dusk, fall, night and sunset, and show that our approach exceeds the state-of-the-art. In addition, we evaluate the performance of autonomously traversing several kilometres of a forested environment using only the segmentation for perception.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
AdapNet is capable of robust segmentation in various environments by adaptively learning features of expert networks based on scene conditions.
“The robustness objective for semantic segmentation was tackled for optimization in AdapNet [58]. The model is capable of robust segmentation in various environments by adaptively learning features of expert networks based on scene conditions.”
From A survey of deep learning techniques for autonomous driving (2020) · cites this paper as [58]
AdapNet, using an AdapNet backbone, achieves an mIoU of 71.72 on the Cityscapes dataset for 2.5D semantic segmentation.
“| AdapNet [105] | AdapNet [105] | 71.72 |”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [105]
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2015) · arXiv · doi:10.48550/arxiv.1506.01497 · Cited by 70,812 · see citing works
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
Ren, S., He, K., Girshick, R., & Sun, J. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2577031 · Cited by 56,035 · see citing works
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392 · see citing works
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392 · see citing works
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631 · see citing works
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · see citing works
Caffe: Convolutional Architecture for Fast Feature Embedding
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., & Darrell, T. (2014) · Proceedings of the 22nd ACM international conference on Multimedia · doi:10.1145/2647868.2654889 · Cited by 11,077 · see citing works
Adaptive Mixtures of Local Experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384 · see citing works
End to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198 · see citing works
Ros, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. M. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.352 · Cited by 2,338 · see citing works
Deeper Depth Prediction with Fully Convolutional Residual Networks
Laina, I., Rupprecht, C., Belagiannis, V., Tombari, F., & Navab, N. (2016) · International Conference on 3D Vision (3DV) · doi:10.1109/3dv.2016.32 · Cited by 2,061 · see citing works
Deep learning for detecting robotic grasps
Lenz, I., Lee, H., & Saxena, A. (2013) · Int. J. Robotics Res. · doi:10.1177/0278364914549607 · Cited by 1,820 · see citing works
Efficient object localization using Convolutional Networks
Tompson, J., Goroshin, R., Jain, A., LeCun, Y., & Bregler, C. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298664 · Cited by 1,365 · see citing works
Multispectral pedestrian detection: Benchmark dataset and baseline
Hwang, S., Park, J., Kim, N., Choi, Y., & Kweon, I. S. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298706 · Cited by 1,249 · see citing works
ParseNet: Looking Wider to See BetterPreprint
Liu, W., Rabinovich, A., & Berg, A. C. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.04579 · Cited by 1,056 · see citing works
Multimodal deep learning for robust RGB-D object recognition
Eitel, A., Springenberg, J. T., Spinello, L., Riedmiller, M., & Burgard, W. (2015) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2015.7353446 · Cited by 647 · see citing works
Self-citationLaplacian Pyramid Reconstruction and Refinement for Semantic Segmentation
Ghiasi, G., & Fowlkes, C. C. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46487-9_32 · Cited by 502 · see citing works
Deep Multispectral Semantic Scene Understanding of Forested Environments Using Multimodal Fusion
Valada, A., Oliveira, G. L., Brox, T., & Burgard, W. (2017) · Springer proceedings in advanced robotics · doi:10.1007/978-3-319-50115-4_41 · Cited by 211 · see citing works
Self-citationLearning Factored Representations in a Deep Mixture of ExpertsPreprint
Eigen, D., Ranzato, M., & Sutskever, I. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1312.4314 · Cited by 134 · see citing works
Choosing smartly: Adaptive multimodal fusion for object detection in changing environments
Mees, O., Eitel, A., & Burgard, W. (2016) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2016.7759048 · Cited by 96 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
A survey of deep learning techniques for autonomous driving
Grigorescu, S., Trasnea, B., Cocias, T., & Macesanu, G. (2020) · Journal of Field Robotics · doi:10.1002/rob.21918 · Cited by 1,809 · see citing works
Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425 · see citing works
Deep multimodal fusion for semantic image segmentation: A survey
Zhang, Y., Sidibé, D., Morel, O., & Mériaudeau, F. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2020.104042 · Cited by 261 · see citing works
Self-supervised model adaptation for multimodal semantic segmentation
Valada, A., Mohan, R., & Burgard, W. (2020) · International Journal of Computer Vision · doi:10.1007/s11263-019-01188-y · Cited by 293 · see citing works
Self-citationSMSnet: Semantic motion segmentation using deep convolutional neural networks
Vertens, J., Valada, A., & Burgard, W. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8202211 · Cited by 69 · see citing works
Self-citationRizzoli, G., Barbato, F., & Zanuttigh, P. (2022) · Technologies · doi:10.3390/technologies10040090 · Cited by 58 · see citing works
A central multimodal fusion framework for outdoor scene image segmentation
Zhang, Y., Morel, O., Seulin, R., Mériaudeau, F., & Sidibé, D. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-020-10357-y · Cited by 5 · see citing works
- Authors
- Valada, A., Vertens, J., Dhall, A., & Burgard, W.
- Affiliations
- University of Freiburg (DE)
- Vellore Institute of Technology University (IN)
- Venue
- 2017 IEEE International Conference on Robotics and Automation (ICRA)
- Publisher
- IEEE
- Published
- 2017-05
- Pages
- 4644-4651
- ISSN
- 1050-4729 (ISSN-L)
- OpenAlex
- W2739253758
- Cited by
- 209
- References
- 24
- Citations by year
- 2026: 9 · 2025: 11 · 2024: 22 · 2023: 24 · 2022: 32 · 2021: 34 · 2020: 35 · 2019: 29 · 2018: 10 · 2017: 3
- FWCI
- 6.16
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- Proceedings - IEEE International Conference on Robotics and Automation/Proceedings (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Engineering › Media Technology › Remote-Sensing Image Classification
- Keywords
- semantic segmentation, adverse environmental conditions, feature fusion, passive optical sensors, outdoor scene understanding, autonomous navigation, forest environment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
