MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes
The DOI matches its Crossref record; 32 of the 33 references it cites were found in a registry or an index.
Abstract
This work addresses the semantic segmentation of images of street scenes for autonomous vehicles based on a new RGB-Thermal dataset, which is also introduced in this paper. An increasing interest in self-driving vehicles has brought the adaptation of semantic segmentation to self-driving systems. However, recent research relating to semantic segmentation is mainly based on RGB images acquired during times of poor visibility at night and under adverse weather conditions. Furthermore, most of these methods only focused on improving performance while ignoring time consumption. The aforementioned problems prompted us to propose a new convolutional neural network architecture for multi-spectral image segmentation that enables the segmentation accuracy to be retained during real-time operation. We benchmarked our method by creating an RGB-Thermal dataset in which thermal and RGB images are combined. We showed that the segmentation accuracy was significantly increased by adding thermal infrared information.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Compared to the object detection problem summarized in Sec. IV-A, there are fewer works on multi-modal semantic segmentation: [92], [119], [124] employ RGB and thermal images, [61] fuses RGB images and depth images from a stereo camera, [125]-[127] combine RGB, thermal, and depth images for semantic segmentation in diverse environments such as forests, [123] fuses RGB images and LiDAR point clouds”
From Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2021) · cites this paper as [92]
In the citing paper
“MSSSD/MF [75] is a real-world dataset and one of the few that provides multispectral (thermal + color) information.”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [75]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 4×
“, AlexNet [16], VGG16 [6], GoogLeNet [7], and ResNet [3].”
BackgroundMethodInfluentialVery Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148 · cited 12×
“Although it is usually necessary to process images with higher resolution (e.g., 480×640 or even higher) in image segmentation, SegNet or DeconvNet adopted VGG16 [6], which was designed to only process 224× 224 images.”
BackgroundInfluentialImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · cited 4×
“SqueezeNet [33] achieved a 50× reduction in the model size compared to AlexNet while maintaining the accuracy.”
BackgroundInfluentialGoing deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
ImageNet Large Scale Visual Recognition Challenge
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-015-0816-y · Cited by 39,380 · cited 1×
“Methods adopting CNNs have surpassed human performance for some computer vision tasks such as image classification [3], [4] or traffic sign”
MethodFully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Learning representations by back-propagating errors
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1986) · Nature · doi:10.1038/323533a0 · Cited by 31,892 · cited 1×
“As an optimizer, we adopted momentum stochastic gradient descent [29] with a learning rate of 0.”
MethodBatch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 1×
“The number of channels of the convolutional layers in MFNet is set to a relatively small number to increase the speed of the network, and each convolutional layer in MFNet is followed by batch normalization [27], following which the nonlinear activation function leaky-Rectified Linear Unit (leaky-ReLU) [28] is applied.”
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631
The Cityscapes Dataset for Semantic Urban Scene Understanding
Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., & Schiele, B. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.350 · Cited by 12,433 · cited 3×
“Public image segmentation datasets built for autonomous driving, such as CamVid [13], Cityscapes [14], and Daimler Urban Segmentation [15], are based on visible spectral (or”
BackgroundBackpropagation Applied to Handwritten Zip Code Recognition
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D. (1989) · Neural Computation · doi:10.1162/neco.1989.1.4.541 · Cited by 11,891 · cited 2×
“CNNs [2] have significantly improved the accuracy of image classification tasks [16] since 2012 and it has also been adopted in semantic segmentation in recent years.”
BackgroundMulti-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738 · cited 5×
“As with most recent image segmentation networks [10], [8], [26], [18], [11], MFNet adopted the Encoder-Decoder structure.”
BackgroundMethodInfluentialThe Pascal Visual Object Classes Challenge: A Retrospective
Everingham, M., Eslami, S. M. A., Van Gool, L., Williams, C. K. I., Winn, J., & Zisserman, A. (2015) · International Journal of Computer Vision · doi:10.1007/s11263-014-0733-5 · Cited by 7,139 · cited 1×
“Because only a few categories of each entire scene are labeled in our dataset, unlabeled pixels occupied the majority of all pixels (much sparser than VOC2012 [30]) and the number of pixels is extremely unbalanced between each class, similar to any other image segmentation dataset.”
Background(2011) · Cited by 6,993
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
Iandola, F. N., Han, S., Moskewicz, M. W., Ashraf, K., Dally, W. J., & Keutzer, K. (2016) · arXiv · doi:10.48550/arxiv.1602.07360 · Cited by 6,201
Learning Deconvolution Network for Semantic Segmentation
Noh, H., Hong, S., & Han, B. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.178 · Cited by 4,422 · cited 7×
“Although it is usually necessary to process images with higher resolution (e.g., 480×640 or even higher) in image segmentation, SegNet or DeconvNet adopted VGG16 [6], which was designed to only process 224× 224 images.”
BackgroundMethodInfluentialSemantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2014) · arXiv · doi:10.48550/arxiv.1412.7062 · Cited by 4,213 · cited 2×
“This can be used to simplify existing network architectures, as in [12], while only slightly reducing the accuracy.”
BackgroundMethodEigen, D., & Fergus, R. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.304 · Cited by 2,862 · cited 1×
“In subsequent years, considerable research went into the design of a larger decoder (than FCN) in their network architecture [10], [12], [11], [17], [18].”
BackgroundConditional Random Fields as Recurrent Neural Networks
Zheng, S., Jayasumana, S., Romera-Paredes, B., Vineet, V., Su, Z., Du, D., Huang, C., & Torr, P. H. S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.179 · Cited by 2,386 · cited 1×
“As with most recent image segmentation networks [10], [8], [26], [18], [11], MFNet adopted the Encoder-Decoder structure.”
MethodSUN RGB-D: A RGB-D scene understanding benchmark suite
Song, S., Lichtenberg, S. P., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298655 · Cited by 1,943 · cited 3×
“Some methods adopting CNN were designed for the RGB-Depth dataset [21], [20], which contains images that were acquired by using Kinect [22], [23].”
MethodENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
Paszke, A., Chaurasia, A., Kim, S., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1606.02147 · Cited by 1,826 · cited 4×
“We compared the performance of our method with that of SegNet [8] and ENet [19].”
BackgroundMethodResultInfluentialNickolls, J., Buck, I., Garland, M., & Skadron, K. (2008) · Queue · doi:10.1145/1365490.1365500 · Cited by 1,610
(2008) · doi:10.1145/1401132.1401152 · Cited by 1,575 · cited 1×
“5 [24], and the cuDNN 5.”
Learning Rich Features from RGB-D Images for Object Detection and Segmentation
Gupta, S., Girshick, R., Arbeláez, P., & Malik, J. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10584-0_23 · Cited by 1,528
Multi-column deep neural network for traffic sign classification
Cireşan, D., Meier, U., Masci, J., & Schmidhuber, J. (2012) · Neural Networks · doi:10.1016/j.neunet.2012.02.023 · Cited by 1,057
Segmentation and Recognition Using Structure from Motion Point Clouds
Brostow, G. J., Shotton, J., Fauqueur, J., & Cipolla, R. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-88682-2_5 · Cited by 1,050 · cited 3×
“Public image segmentation datasets built for autonomous driving, such as CamVid [13], Cityscapes [14], and Daimler Urban Segmentation [15], are based on visible spectral (or”
BackgroundcuDNN: Efficient Primitives for Deep Learning
Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J. J., Catanzaro, B., & Shelhamer, E. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.0759 · Cited by 943 · cited 2×
“0 backend [25] with a single NVIDIA Geforce Titan X GPU.”
BackgroundMethodFuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture
Hazirbas, C., Ma, L., Domokos, C., & Cremers, D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54181-5_14 · Cited by 902 · cited 4×
“[22] proposed a new CNN network named FuseNet, which contains two encoders that simultaneously extract features from RGB and depth images.”
BackgroundMethodInfluentialMaking Bertha Drive—An Autonomous Journey on a Historic Route
Ziegler, J., Bender, P., Schreiber, M., Lategahn, H., Strauss, T., Stiller, C., Dang, T., Franke, U., Appenrodt, N., Keller, C. G., et al. (2014) · IEEE Intelligent Transportation Systems Magazine · doi:10.1109/mits.2014.2306552 · Cited by 854
Learning Common and Specific Features for RGB-D Semantic Segmentation with Deconvolutional Networks
Wang, J., Wang, Z., Tao, D., See, S., & Wang, G. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46454-1_40 · Cited by 161
Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation
Hong, S., Noh, H., & Han, B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.04924 · Cited by 159 · cited 2×
“As with most recent image segmentation networks [10], [8], [26], [18], [11], MFNet adopted the Encoder-Decoder structure.”
BackgroundMethodEfficient Multi-cue Scene Segmentation
Scharwächter, T., Enzweiler, M., Franke, U., & Roth, S. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-40602-7_46 · Cited by 91 · cited 3×
“Public image segmentation datasets built for autonomous driving, such as CamVid [13], Cityscapes [14], and Daimler Urban Segmentation [15], are based on visible spectral (or”
Background
Research works printed but not linked: 3
- iandola, 2016, Squeezenet Alexnet-level accuracy with 50x fewer parameters and¡ 1mb model size
- tokui, 2015, Chainer: a next-generation open source framework for deep learning, NIPS
- maas, 2013, Rectifier nonlinearities improve neural network acoustic models, ICML
Cited by
nuScenes: A Multimodal Dataset for Autonomous Driving
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., & Beijbom, O. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01164 · Cited by 7,152
“cities Vistas [26] 2017 - - 25k 0 0 25k 0 Yes Yes Yes 6 continents BDD100k [60] 2017 100k 1k 100M 0 0 100k 0 Yes Yes Yes NY, SF M. obj. det. [51] 2017 - - 7.5k 0 0 7.5k 0 Yes No No Tokyo M. sem. seg. [29] 2017 - - 1.6k 0 0 1.6k 0 Yes No No - ApolloScape [33] 2018 - 100 144k 0 0 144k 70k Yes No No 4x China D2-City [6]…”
InfluentialA Survey of Autonomous Driving: Common Practices and Emerging Technologies
Yurtsever, E., Lambert, J., Carballo, A., & Takeda, K. (2020) · IEEE Access · doi:10.1109/access.2020.2983149 · Cited by 1,889
“Other camera types gaining interest for ADS include flash cameras [78], thermal cameras [80], [81], and”
BackgroundSwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer
Ma, J., Tang, L., Fan, F., Huang, J., Mei, X., & Ma, Y. (2022) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2022.105686 · Cited by 1,526
“The MSRS dataset [76], [77]1, containing 1,083 training image pairs and 361 testing image pairs, is selected for training and evaluating the visible and infrared image fusion task.”
MethodFeng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425
“IV-A, there are fewer works on multi-modal semantic segmentation: [91], [118], [123] employ RGB and thermal images, [60] fuses RGB images and depth images from a stereo camera, [124]–[126] combine RGB, thermal, and depth images for semantic segmentation in diverse environments such as forests, [122] fuses RGB images a…”
BackgroundTang, L., Yuan, J., & Ma, J. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.12.004 · Cited by 1,136
Metadata
- Authors
- Ha, Q., Watanabe, K., Karasawa, T., Ushiku, Y., & Harada, T.
- Affiliations
- The University of Tokyo (JP)
- Venue
- 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)
- Publisher
- IEEE
- Published
- 2017-09
- Pages
- 5108-5115
- ISSN
- 2153-0858 (ISSN-L)
- OpenAlex
- W2774839435
- Cited by
- 693
- References
- 33
- Citations by year
- 2026: 86 · 2025: 181 · 2024: 150 · 2023: 113 · 2022: 79 · 2021: 48 · 2020: 25 · 2019: 9 · 2018: 2
- FWCI
- 2.88
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- semantic segmentation, autonomous vehicles, multispectral imagery, thermal infrared imaging, real-time segmentation, street scenes, autonomous driving, low visibility, severe weather
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.