Rtfnet: Rgb-thermal fusion network for semantic segmentation of urban scenes
Abstract
Semantic segmentation is a fundamental capability for autonomous vehicles. With the advancements of deep learning technologies, many effective semantic segmentation networks have been proposed in recent years. However, most of them are designed using RGB images from visible cameras. The quality of RGB images is prone to be degraded under unsatisfied lighting conditions, such as darkness and glares of oncoming headlights, which imposes critical challenges for the networks that use only RGB images. Different from visible cameras, thermal imaging cameras generate images using thermal radiations. They are able to see under various lighting conditions. In order to enable robust and accurate semantic segmentation for autonomous vehicles, we take the advantage of thermal images and fuse both the RGB and thermal information in a novel deep neural network. The main innovation of this letter is the architecture of the proposed network. We adopt the encoder–decoder design concept. ResNet is employed for feature extraction and a new decoder is developed to restore the feature map resolution. The experimental results prove that our network outperforms the state of the arts.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
There are fewer works on multi-modal semantic segmentation compared to multi-modal object detection.
“Compared to the object detection problem summarized in Sec. IV-A, there are fewer works on multi-modal semantic segmentation: [92], [119], [124] employ RGB and thermal images, [61] fuses RGB images and depth images from a stereo camera, [125]-[127] combine RGB, thermal, and depth images for semantic segmentation in diverse environments such as forests, [123] fuses RGB images and LiDAR point clouds”
From Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2021) · cites this paper as [124]
RTFNet, using a ResNet152 backbone, achieves an mIoU of 53.2 on the MSSSD/MF dataset for RGB + Thermal semantic segmentation.
“| RTFNet [106] | ResNet152 [16] | 53.2 |”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [106]
RFNet, using a ResNet18 backbone, achieves an mIoU of 69.37 on the Cityscapes dataset for 2.5D semantic segmentation.
“| RFNet [106] | ResNet18 [16] | 69.37 |”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [106]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562 · see citing works
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834 · see citing works
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392 · see citing works
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631 · see citing works
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438 · see citing works
(2010) · Cited by 14,134 · see citing works
Fully convolutional networks for semantic segmentation
Shelhamer, E., Long, J., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2572683 · Cited by 11,898 · see citing works
Learning Deconvolution Network for Semantic Segmentation
Noh, H., Hong, S., & Han, B. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.178 · Cited by 4,422 · see citing works
Understanding Convolution for Semantic Segmentation
Wang, P., Chen, P., Yuan, Y., Liu, D., Huang, Z., Hou, X., & Cottrell, G. W. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00163 · Cited by 2,127 · see citing works
Neural Networks: Tricks of the Trade: Second Edition
(2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8 · Cited by 2,116 · see citing works
ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation
Romera, E., Álvarez, J. M., Bergasa, L. M., & Arroyo, R. (2017) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2017.2750080 · Cited by 1,491 · see citing works
FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture
Hazirbas, C., Ma, L., Domokos, C., & Cremers, D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54181-5_14 · Cited by 902 · see citing works
Thermal cameras and applications: A survey
Gade, R., & Moeslund, T. B. (2014) · Machine Vision and Applications · doi:10.1007/s00138-013-0570-5 · Cited by 780 · see citing works
MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes
Ha, Q., Watanabe, K., Karasawa, T., Ushiku, Y., & Harada, T. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206396 · Cited by 693 · see citing works
Survey on semantic segmentation using deep learning techniques
Lateef, F., & Ruichek, Y. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.02.003 · Cited by 561 · see citing works
Improving RGB-D SLAM in dynamic environments: A motion removal approach
Sun, Y., Liu, M., & Meng, M. Q. ‐. (2016) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2016.11.012 · Cited by 387 · see citing works
Self-citationVollmer, M., & Möllmann, K. (2017) · Wiley eBooks · doi:10.1002/9783527693306 · Cited by 320 · see citing works
Remote sensing of concrete bridge decks using unmanned aerial vehicle infrared thermography
Omar, T., & Nehdi, M. L. (2017) · Automation in Construction · doi:10.1016/j.autcon.2017.06.024 · Cited by 266 · see citing works
Motion removal for reliable RGB-D SLAM in dynamic environments
Sun, Y., Liu, M., & Meng, M. Q. ‐. (2018) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2018.07.002 · Cited by 200 · see citing works
Self-citationMonocular Semantic Occupancy Grid Mapping With Convolutional Variational Encoder–Decoder Networks
Lu, C., van de Molengraft, R., & Dubbelman, G. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2891028 · Cited by 169 · see citing works
Autonomous Robotic Exploration by Incremental Road Map Construction
Wang, C., Chi, W., Sun, Y., & Meng, M. Q. ‐. (2019) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2019.2894748 · Cited by 85 · see citing works
Self-citationHalmetschlager-Funek, G., Suchi, M., Kampel, M., & Vincze, M. (2018) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2018.2852795 · Cited by 82 · see citing works
Robust Intrinsic and Extrinsic Calibration of RGB-D Cameras
Basso, F., Menegatti, E., & Pretto, A. (2018) · IEEE Transactions on Robotics · doi:10.1109/tro.2018.2853742 · Cited by 76 · see citing works
Active Perception for Foreground Segmentation: An RGB-D Data-Based Background Modeling Method
Sun, Y., Liu, M., & Meng, M. Q. ‐. (2019) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2019.2893414 · Cited by 50 · see citing works
Self-citationObstacle Detection System for Small UAVs using ADS-B and Thermal Imaging
Carrio, A., Lin, Y., Saripalli, S., & Campoy, P. (2017) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-017-0529-2 · Cited by 31 · see citing works
A dense semantic mapping system based on CRF-RNN network
Cheng, J., Sun, Y., & Meng, M. Q. ‐. (2017) · International Conference on Advanced Robotics (ICAR) · doi:10.1109/icar.2017.8023671 · Cited by 24 · see citing works
Self-citationThermal-infrared based drivable region detection
Yoon, J. S., Park, K., Hwang, S., Kim, N. I., Choi, Y., Rameau, F., & Kweon, I. S. (2016) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2016.7535507 · Cited by 18 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425 · see citing works
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers
Zhang, J., Liu, H., Yang, K., Hu, X., Liu, R., & Stiefelhagen, R. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3300537 · Cited by 731 · see citing works
Review the state-of-the-art technologies of semantic segmentation based on deep learning
Mo, Y., Wu, Y. X., Yang, X., Liu, F., & Liao, Y. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.01.005 · Cited by 647 · see citing works
Zhao, F., Zhang, C., & Geng, B. (2024) · ACM Computing Surveys · doi:10.1145/3649447 · Cited by 480 · see citing works
Tang, L., Zhang, H., Xu, H., & Ma, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101870 · Cited by 396 · see citing works
Liu, J., Liu, Z., Wu, G., Ma, L., Liu, R., Zhong, W., Luo, Z., & Fan, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00745 · Cited by 346 · see citing works
GMNet: Graded-Feature Multilabel-Learning Network for RGB-Thermal Urban Scene Semantic Segmentation
Zhou, W., Liu, J., Lei, J., Yu, L., & Hwang, J.-N. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3109518 · Cited by 344 · see citing works
FuseSeg: Semantic segmentation of urban scenes based on RGB and thermal data fusion
Sun, Y., Zuo, W., Yun, P., Wang, H., & Liu, M. (2021) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2020.2993143 · Cited by 270 · see citing works
Self-citationPST900: RGB-Thermal Calibration, Dataset and Segmentation Network
Shivakumar, S. S., Rodrigues, N., Zhou, A., Miller, I. D., Kumar, V., & Taylor, C. J. (2020) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra40945.2020.9196831 · Cited by 269 · see citing works
Deep multimodal fusion for semantic image segmentation: A survey
Zhang, Y., Sidibé, D., Morel, O., & Mériaudeau, F. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2020.104042 · Cited by 261 · see citing works
Divide-and-Conquer: Confluent Triple-Flow Network for RGB-T Salient Object Detection
Tang, H., Li, Z., Zhang, D., He, S., & Tang, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3511621 · Cited by 259 · see citing works
MFFENet: Multiscale Feature Fusion and Enhancement Network For RGB–Thermal Urban Road Scene Parsing
Zhou, W., Lin, X., Lei, J., Yu, L., & Hwang, J. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3086618 · Cited by 251 · see citing works
Infrared and Visible Image Fusion: From Data Compatibility to Task Adaption
Liu, J., Wu, G., Liu, Z., Wang, D., Jiang, Z., Ma, L., Zhong, W., Fan, X., & Liu, R. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3521416 · Cited by 238 · see citing works
Zhang, Q., Zhao, S., Luo, Y., Zhang, D., Huang, N., & Han, J. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.00266 · Cited by 225 · see citing works
Blitz-SLAM: A semantic SLAM in dynamic environments
Fan, Y., Zhang, Q., Tang, Y., Liu, S., & Han, H. W. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.108225 · Cited by 213 · see citing works
Fan, R., Wang, H., Cai, P., & Liu, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58577-8_21 · Cited by 198 · see citing works
Self-citationRGB-T Semantic Segmentation With Location, Activation, and Sharpening
Li, G., Wang, Y., Liu, Z., Zhang, X., & Zeng, D. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3208833 · Cited by 196 · see citing works
Recent Advances in Thermal Imaging and its Applications Using Machine Learning: A Review
Wilson, A. N., Gupta, K. A., Koduru, B. H., Kumar, A., Jha, A., & Cenkeramaddi, L. R. (2023) · IEEE Sensors Journal · doi:10.1109/jsen.2023.3234335 · Cited by 183 · see citing works
Zhou, W., Dong, S., Lei, J., & Yu, L. (2022) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2022.3164899 · Cited by 179 · see citing works
Explicit Attention-Enhanced Fusion for RGB-Thermal Perception Tasks
Liang, M., Hu, J., Bao, C., Hua, F., Deng, F., & Lam, T. L. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3272269 · Cited by 179 · see citing works
FEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentation
Deng, F., Feng, H., Liang, M., Wang, H., Yang, Y., Gao, Y., Chen, J., Hu, J., Guo, X., & Lam, T. L. (2021) · 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636084 · Cited by 170 · see citing works
Zhou, W., Zhang, H., Yan, W., & Lin, W. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3275314 · Cited by 151 · see citing works
Edge-Aware Guidance Fusion Network for RGB–Thermal Scene Parsing
Zhou, W., Dong, S., Xu, C., & Qian, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20269 · Cited by 146 · see citing works
Zhang, S., Zhang, J., Tian, B., Lukasiewicz, T., & Xu, Z. (2022) · Medical Image Analysis · doi:10.1016/j.media.2022.102656 · Cited by 145 · see citing works
Context-Aware Interaction Network for RGB-T Semantic Segmentation
Lv, Y., Liu, Z., & Li, G. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3349072 · Cited by 138 · see citing works
FRNet: Feature Reconstruction Network for RGB-D Indoor Scene Parsing
Zhou, W., Yang, E., Lei, J., & Yu, L. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3174338 · Cited by 114 · see citing works
Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation
Wan, Z., Zhang, P., Wang, Y., Yong, S., Stepputtis, S., Sycara, K. P., & Xie, Y. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00176 · Cited by 114 · see citing works
EGFNet: Edge-Aware Guidance Fusion Network for RGB–Thermal Urban Scene Parsing
Dong, S., Zhou, W., Xu, C., & Yan, W. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3306368 · Cited by 107 · see citing works
Dynamic Fusion Module Evolves Drivable Area and Road Anomaly Detection: A Benchmark and Algorithms
Wang, H., Fan, R., Sun, Y., & Liu, M. (2021) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2021.3064089 · Cited by 104 · see citing works
Self-citationSGFNet: Semantic-Guided Fusion Network for RGB-Thermal Semantic Segmentation
Wang, Y., Li, G., & Liu, Z. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3281419 · Cited by 104 · see citing works
Mitigating Modality Discrepancies for RGB-T Semantic Segmentation
Zhao, S., Liu, Y., Jiao, Q., Zhang, Q., & Han, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3233089 · Cited by 102 · see citing works
Infrared and Visible Image Fusion Techniques Based on Deep Learning: A Review
Sun, C., Zhang, C., & Xiong, N. N. (2020) · Electronics · doi:10.3390/electronics9122162 · Cited by 101 · see citing works
Attention fusion network for multi-spectral semantic segmentation
Xu, J., Lu, K., & Wang, H. (2021) · Pattern Recognition Letters · doi:10.1016/j.patrec.2021.03.015 · Cited by 99 · see citing works
Attention-Based Multimodal Image Feature Fusion Module for Transmission Line Detection
Choi, H., Yun, J. P., Kim, B. J., Jang, H., & Kim, S. W. (2022) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2022.3147833 · Cited by 98 · see citing works
Zhou, W., Jianhui, J., Lei, J., & Yu, L. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3159032 · Cited by 94 · see citing works
RGB-T image analysis technology and application: A survey
Song, K., Zhao, Y., Huang, L., Yan, Y., & Meng, Q. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.105919 · Cited by 94 · see citing works
Multimodal Cross-Layer Bilinear Pooling for RGBT Tracking
Xu, Q., Mei, Y., Liu, J., & Li, C. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3055362 · Cited by 91 · see citing works
A Feature Divide-and-Conquer Network for RGB-T Semantic Segmentation
Zhao, S., & Zhang, Q. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3229359 · Cited by 91 · see citing works
HeatNet: Bridging the Day-Night Domain Gap in Semantic Segmentation with Thermal Images
Vertens, J., Zürn, J., & Burgard, W. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341192 · Cited by 89 · see citing works
Feng, Z., Guo, Y., & Sun, Y. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3247175 · Cited by 85 · see citing works
Self-citationFrom SLAM to Situational Awareness: Challenges and Survey
Bavle, H., Sánchez-López, J. L., Cimarelli, C., Tourani, A., & Voos, H. (2023) · Sensors · doi:10.3390/s23104849 · Cited by 76 · see citing works
Semantic Image Segmentation: Two Decades of Research
Csurka, G., Volpi, R., & Chidlovskii, B. (2022) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000095 · Cited by 76 · see citing works
MMNet: Multi-modal multi-stage network for RGB-T image semantic segmentation
Lan, X., Gu, X., & Gu, X. (2021) · Applied Intelligence · doi:10.1007/s10489-021-02687-7 · Cited by 74 · see citing works
Complementarity-aware cross-modal feature fusion network for RGB-T semantic segmentation
Wu, W., Chu, T., & Liu, Q. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108881 · Cited by 74 · see citing works
Deep Depth Estimation from Thermal Image
Shin, U., Park, J., & Kweon, I. S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00107 · Cited by 74 · see citing works
Wang, H., Fan, R., Sun, Y., & Liu, M. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341340 · Cited by 73 · see citing works
Self-citationGlass Segmentation With RGB-Thermal Image Pairs
Huo, D., Wang, J., Qian, Y., & Yang, Y. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3256762 · Cited by 73 · see citing works
Self-Supervised Drivable Area and Road Anomaly Segmentation Using RGB-D Data For Robotic Wheelchairs
Wang, H., Sun, Y., & Liu, M. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2932874 · Cited by 74 · see citing works
Self-citationWe Learn Better Road Pothole Detection: From Attention Aggregation to Adversarial Domain Adaptation
Fan, R., Wang, H., Bocus, M. J., & Liu, M. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-66823-5_17 · Cited by 71 · see citing works
Self-citationEmbedded Control Gate Fusion and Attention Residual Learning for RGB–Thermal Urban Scene Parsing
Zhou, W., Lv, Y. H., Lei, J., & Yu, L. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3242651 · Cited by 71 · see citing works
Object Detection from Thermal Infrared and Visible Light Cameras in Search and Rescue Scenes
Bañuls-Arias, A., Mandow, A., Vázquez-Martín, R., Morales, J., & Garcia-Cerezo, A. J. (2020) · IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR) · doi:10.1109/ssrr50563.2020.9292593 · Cited by 70 · see citing works
Self-Training for Class-Incremental Semantic Segmentation
Yu, L., Liu, X., & van de Weijer, J. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3155746 · Cited by 70 · see citing works
Li, J., Jiang, J., Liang, P., Ma, J., & Nie, L. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3541562 · Cited by 70 · see citing works
Zhou, W., Gong, T., Lei, J., & Yu, L. (2023) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2023.3298921 · Cited by 68 · see citing works
Hassan, T., Shafay, M., Akçay, S., Khan, S. H., Bennamoun, M., Damiani, E., & Werghi, N. (2020) · Sensors · doi:10.3390/s20226450 · Cited by 67 · see citing works
PSPNet-SLAM: A Semantic SLAM Detect Dynamic Object by Pyramid Scene Parsing Network
Long, X., Zhang, W., & Zhao, B. (2020) · IEEE Access · doi:10.1109/access.2020.3041038 · Cited by 67 · see citing works
ISSAFE: Improving Semantic Segmentation in Accidents by Fusing Event-based Data
Zhang, J., Yang, K., & Stiefelhagen, R. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636109 · Cited by 65 · see citing works
RoadFormer: Duplex Transformer for RGB-Normal Semantic Road Scene Parsing
Li, J., Zhang, Y., Peng, Y., Zhou, G., Chen, Q., & Fan, R. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3388726 · Cited by 65 · see citing works
Oral cancer detection using feature-level fusion and novel self-attention mechanisms
Khan, S. U. R., & Asif, S. (2024) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2024.106437 · Cited by 62 · see citing works
Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation
Reza, M. K., Prater-Bennette, A., & Asif, M. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3476487 · Cited by 61 · see citing works
Real-time infrared and visible image fusion network using adaptive pixel weighting strategy
Zhang, X., Zhai, H., Liu, J., Wang, Z., & Sun, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101863 · Cited by 61 · see citing works
MDNet: Mamba-Effective Diffusion-Distillation Network for RGB-Thermal Urban Dense Prediction
Zhou, W., Wu, H., & Jiang, Q. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3508058 · Cited by 61 · see citing works
Xiong, H., Cai, W., & Liu, Q. (2021) · Infrared Physics & Technology · doi:10.1016/j.infrared.2020.103628 · Cited by 60 · see citing works
MS-UDA: Multi-Spectral Unsupervised Domain Adaptation for Thermal Image Semantic Segmentation
Kim, Y.-H., Shin, U., Park, J., & Kweon, I. S. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3093652 · Cited by 60 · see citing works
He, X., Wang, M., Liu, T., Zhao, L., & Yue, Y. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3267529 · Cited by 60 · see citing works
SNE-RoadSeg+: Rethinking Depth-Normal Translation and Deep Supervision for Freespace Detection
Wang, H., Fan, R., Cai, P., & Liu, M. (2021) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636723 · Cited by 59 · see citing works
Self-citationRizzoli, G., Barbato, F., & Zanuttigh, P. (2022) · Technologies · doi:10.3390/technologies10040090 · Cited by 58 · see citing works
On the Role of Thermal Imaging in Automotive Applications: A Critical Review
Farooq, M. A., Shariff, W., O'Callaghan, D., Merla, A., & Corcoran, P. (2023) · IEEE Access · doi:10.1109/access.2023.3255110 · Cited by 58 · see citing works
CACFNet: Cross-Modal Attention Cascaded Fusion Network for RGB-T Urban Scene Parsing
Zhou, W., Dong, S., Fang, M., & Yu, L. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3314527 · Cited by 57 · see citing works
Xu, C. X., Li, Q., Jiang, X., Yu, D., & Zhou, Y. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3216313 · Cited by 56 · see citing works
PAIF: Perception-Aware Infrared-Visible Image Fusion for Attack-Tolerant Semantic Segmentation
Liu, Z., Liu, J., Zhang, B., Ma, L., Fan, X., & Liu, R. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611928 · Cited by 56 · see citing works
A Total Variation With Joint Norms For Infrared and Visible Image Fusion
Nie, R., Ma, C., Cao, J., Ding, H., & Zhou, D. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3065496 · Cited by 55 · see citing works
Semantic Segmentation for Thermal Images: A Comparative Survey
Kutuk, Z., & Algan, G. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00043 · Cited by 54 · see citing works
Multi-Modal Sensor Fusion-Based Semantic Segmentation for Snow Driving Scenarios
Vachmanus, S., Ravankar, A. A., Emaru, T., & Kobayashi, Y. (2021) · IEEE Sensors Journal · doi:10.1109/jsen.2021.3077029 · Cited by 53 · see citing works
Multispectral Fusion Transformer Network for RGB-Thermal Urban Scene Semantic Segmentation
Zhou, H., Tian, C., Zhang, Z., Huo, Q., Xie, Y., & Li, Z. (2022) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2022.3179721 · Cited by 53 · see citing works
GCNet: Grid-like context-aware network for RGB-thermal semantic segmentation
Liu, J., Zhou, W., Cui, Y., Yu, L., & Luo, T. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.07.041 · Cited by 53 · see citing works
DHFNet: dual-decoding hierarchical fusion network for RGB-thermal semantic segmentation
Cai, Y., Zhou, W., Zhang, L., Yu, L., & Luo, T. (2023) · The Visual Computer · doi:10.1007/s00371-023-02773-6 · Cited by 53 · see citing works
iCurb: Imitation Learning-Based Detection of Road Curbs Using Aerial Images for Autonomous Driving
Xu, Z., Sun, Y., & Liu, M. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3056344 · Cited by 53 · see citing works
Self-citationUTFNet: Uncertainty-Guided Trustworthy Fusion Network for RGB-Thermal Semantic Segmentation
Wang, Q., Yin, C., Song, H., Shen, T., & Gu, Y. (2023) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2023.3322452 · Cited by 52 · see citing works
Contrastive learning-based knowledge distillation for RGB-thermal urban scene semantic segmentation
Guo, X., Zhou, W., & Liu, T. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111588 · Cited by 52 · see citing works
Yi, S., Chen, M., Yuan, X., Si, G., & Wang, J. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.01.022 · Cited by 52 · see citing works
Vision-Based Trajectory Planning via Imitation Learning for Autonomous Vehicles
Cai, P., Sun, Y., Chen, Y., & Liu, M. (2019) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2019.8917149 · Cited by 52 · see citing works
Self-citationAlexander, Q. G., Hoskere, V., Narazaki, Y., Maxwell, A., & Spencer, B. F. (2022) · AI in Civil Engineering · doi:10.1007/s43503-022-00002-y · Cited by 50 · see citing works
IGT: Illumination-guided RGB-T object detection with transformers
Chen, K., Liu, J., & Zhang, H. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110423 · Cited by 50 · see citing works
Complementary Random Masking for RGB-Thermal Semantic Segmentation
Shin, U., Lee, K., Kweon, I. S., & Oh, J. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10611200 · Cited by 49 · see citing works
“Specifically, they proposed various types of feature fusion modules, such as the na ¨ ıve feature-level fusion [11], [36], [35], multi-scale feature fusion [36], [46], [37], [21], and attention-weighted fusion [44], [46], [8], [21].”
BackgroundMethodInfluentialYi (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.11.056 · Cited by 49 · see citing works
Residual spatial fusion network for RGB-thermal semantic segmentation
Li, P., Chen, J., Lin, B., & Xu, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127913 · Cited by 49 · see citing works
Liu, L., Feng, G., Beautemps, D., & Zhang, X.-P. (2020) · IEEE transactions on multimedia · doi:10.1109/tmm.2020.2976493 · Cited by 51 · see citing works
Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning
Dong, S., Feng, Y., Yang, Q., Huang, Y., Liu, D., & Fan, H. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801872 · Cited by 46 · see citing works
Mask-guided modality difference reduction network for RGB-T semantic segmentation
Liang, W., Yang, Y., Li, F., Long, X., & Shan, C. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.12.036 · Cited by 45 · see citing works
Semantic Segmentation to Develop an Indoor Navigation System for an Autonomous Mobile Robot
Teso-Fz-Betoño, D., Zulueta, E., Sánchez-Chica, A., Fernández‐Gámiz, U., & Saenz‐Aguirre, A. (2020) · Mathematics · doi:10.3390/math8050855 · Cited by 45 · see citing works
Robust semantic segmentation based on RGB-thermal in variable lighting scenes
Guo, Z., Xu, L., Xu, Q., & Sun, Z. (2021) · Measurement · doi:10.1016/j.measurement.2021.110176 · Cited by 44 · see citing works
Global contextually guided lightweight network for RGB-thermal urban scene understanding
Gong, T., Zhou, W., Qian, X., Lei, J., & Yu, L. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105510 · Cited by 43 · see citing works
MMSFormer: Multimodal Transformer for Material and Semantic Segmentation
Reza, M. K., Prater-Bennette, A., & Asif, M. S. (2024) · IEEE Open Journal of Signal Processing · doi:10.1109/ojsp.2024.3389812 · Cited by 42 · see citing works
Huang, J., Li, J., Ning, J., Sun, Y., Liu, C., Chen, Q., & Fan, R. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3448251 · Cited by 42 · see citing works
Self-citationImproving Visual Localization Accuracy in Dynamic Environments Based on Dynamic Region Removal
Cheng, J., Zhang, H., & Meng, M. Q. ‐. (2020) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2020.2964938 · Cited by 42 · see citing works
MSNet: multispectral semantic segmentation network for remote sensing images
Tao, C., Meng, Y., Li, J., Yang, B., Hu, F., Li, Y., Cui, C., & Zhang, W. (2022) · GIScience & Remote Sensing · doi:10.1080/15481603.2022.2101728 · Cited by 42 · see citing works
AGFNet: Adaptive Gated Fusion Network for RGB-T Semantic Segmentation
Zhou, X., Wu, X., Bao, L., Yin, H., Jiang, Q., & Zhang, J. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3528064 · Cited by 41 · see citing works
Gallagher, J., & Oughton, E. J. (2023) · Scientific Reports · doi:10.1038/s41598-023-34791-8 · Cited by 39 · see citing works
Liu, X., Wang, J., & Li, J. (2023) · Control Engineering Practice · doi:10.1016/j.conengprac.2023.105560 · Cited by 39 · see citing works
- Authors
- Sun, Y., Zuo, W., & Liu, M.
- Affiliations
- Hong Kong University of Science and Technology (HK)
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2019-07
- Volume
- 4
- Issue
- 3
- Pages
- 2576-2583
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W2921749009
- Cited by
- 546
- References
- 30
- Citations by year
- 2026: 87 · 2025: 127 · 2024: 102 · 2023: 94 · 2022: 51 · 2021: 45 · 2020: 31 · 2019: 9
- FWCI
- 15.22
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Funders
- Shenzhen Science, Technology, and Innovation Commission (JCYJ20160401100022706)
- National Natural Science Foundation of China (U1713211)
- Hong Kong University of Science and Technology Project (IGN16EG12)
- Hong Kong Research Grant Council (11210017, 21202816)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Keywords
- RGB-thermal fusion, semantic segmentation, urban scenes, autonomous vehicles, infrared thermography, encoder-decoder architecture, ResNet feature extraction, low-light conditions, headlight glare
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
