FuseSeg: Semantic segmentation of urban scenes based on RGB and thermal data fusion
Abstract
Semantic segmentation of urban scenes is an essential component in various applications of autonomous driving. It makes great progress with the rise of deep learning technologies. Most of the current semantic segmentation networks use single-modal sensory data, which are usually the RGB images produced by visible cameras. However, the segmentation performance of these networks is prone to be degraded when lighting conditions are not satisfied, such as dim light or darkness. We find that thermal images produced by thermal imaging cameras are robust to challenging lighting conditions. Therefore, in this article, we propose a novel RGB and thermal data fusion network named FuseSeg to achieve superior performance of semantic segmentation in urban scenes. The experimental results demonstrate that our network outperforms the state-of-the-art networks.Note to Practitioners—This article investigates the problem of semantic segmentation of urban scenes when lighting conditions are not satisfied. We provide a solution to this problem via information fusion with RGB and thermal data. We build an end-to-end deep neural network, which takes as input a pair of RGB and thermal images and outputs pixel-wise semantic labels. Our network could be used for urban scene understanding, which serves as a fundamental component of many autonomous driving tasks, such as environment modeling, obstacle avoidance, motion prediction, and planning. Moreover, the simple design of our network allows it to be easily implemented using various deep learning frameworks, which facilitates the applications on different hardware or software platforms.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
FuseSeg-Thermal, using a DenseNet161 backbone, achieves an mIoU of 54.5 on the MSSSD/MF dataset for RGB + Thermal semantic segmentation.
“| FuseSeg-Thermal [117] | DenseNet161 [127] | 54.5 |”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [117]
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566 · see citing works
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631 · see citing works
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438 · see citing works
(2010) · Cited by 14,134 · see citing works
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465 · see citing works
Fully convolutional networks for semantic segmentation
Shelhamer, E., Long, J., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2572683 · Cited by 11,898 · see citing works
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244 · see citing works
Deep High-Resolution Representation Learning for Human Pose Estimation
Sun, K., Xiao, B., Liu, D., & Wang, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00584 · Cited by 5,984 · see citing works
Stacked Hourglass Networks for Human Pose Estimation
Newell, A., Yang, K., & Deng, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_29 · Cited by 5,634 · see citing works
Learning Deconvolution Network for Semantic Segmentation
Noh, H., Hong, S., & Han, B. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.178 · Cited by 4,422 · see citing works
BiSeNet: Bilateral Segmentation Network for Real-Time Semantic Segmentation
Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., & Sang, N. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_20 · Cited by 2,968 · see citing works
Frustum PointNets for 3D Object Detection from RGB-D Data
Qi, C. R., Liu, W., Wu, C., Su, H., & Guibas, L. J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00102 · Cited by 2,655 · see citing works
Understanding Convolution for Semantic Segmentation
Wang, P., Chen, P., Yuan, Y., Liu, D., Huang, Z., Hou, X., & Cottrell, G. W. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00163 · Cited by 2,127 · see citing works
SUN RGB-D: A RGB-D scene understanding benchmark suite
Song, S., Lichtenberg, S. P., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298655 · Cited by 1,943 · see citing works
Wide Residual NetworksPreprint
Zagoruyko, S., & Komodakis, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07146 · Cited by 1,850 · see citing works
ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
Paszke, A., Chaurasia, A., Kim, S., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1606.02147 · Cited by 1,826 · see citing works
ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation
Romera, E., Álvarez, J. M., Bergasa, L. M., & Arroyo, R. (2017) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2017.2750080 · Cited by 1,491 · see citing works
Learning a Discriminative Feature Network for Semantic Segmentation
Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., & Sang, N. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00199 · Cited by 923 · see citing works
FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture
Hazirbas, C., Ma, L., Domokos, C., & Cremers, D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54181-5_14 · Cited by 902 · see citing works
High-Resolution Representations for Labeling Pixels and RegionsPreprint
Sun, K., Zhao, Y. S., Jiang, B., Cheng, T., Xiao, B., Liu, D., Mu, Y., Wang, X., Liu, W., & Wang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.04514 · Cited by 841 · see citing works
Kendall, A., Badrinarayanan, V., & Cipolla, R. (2017) · Procedings of the British Machine Vision Conference 2017 · doi:10.5244/c.31.57 · Cited by 784 · see citing works
MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes
Ha, Q., Watanabe, K., Karasawa, T., Ushiku, Y., & Harada, T. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206396 · Cited by 693 · see citing works
Information combination operators for data fusion: a comparative review with classification
Bloch, I. (1996) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/3468.477860 · Cited by 640 · see citing works
Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes
Pohlen, T., Hermans, A., Mathias, M., & Leibe, B. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.353 · Cited by 591 · see citing works
Rtfnet: Rgb-thermal fusion network for semantic segmentation of urban scenes
Sun, Y., Zuo, W., & Liu, M. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2904733 · Cited by 546 · see citing works
Self-citationObject Classification Using CNN-Based Fusion of Vision and LIDAR in Autonomous Vehicle Environment
Gao, H., Cheng, B., Wang, J., Li, K., Zhao, J., & Li, D. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2018.2822828 · Cited by 529 · see citing works
Improving RGB-D SLAM in dynamic environments: A motion removal approach
Sun, Y., Liu, M., & Meng, M. Q. ‐. (2016) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2016.11.012 · Cited by 387 · see citing works
Vollmer, M., & Möllmann, K. (2017) · Wiley eBooks · doi:10.1002/9783527693306 · Cited by 320 · see citing works
Depth-Aware CNN for RGB-D Segmentation
Wang, W., & Neumann, U. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_9 · Cited by 314 · see citing works
Kendall, A., Badrinarayanan, V., & Cipolla, R. (2015) · arXiv · doi:10.48550/arxiv.1511.02680 · Cited by 292 · see citing works
Motion removal for reliable RGB-D SLAM in dynamic environments
Sun, Y., Liu, M., & Meng, M. Q. ‐. (2018) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2018.07.002 · Cited by 200 · see citing works
Asymmetric Bimanual Control of Dual-Arm Exoskeletons for Human-Cooperative Manipulations
Li, Z., Huang, B., Ajoudani, A., Yang, C., Su, C., & Bicchi, A. (2017) · IEEE Transactions on Robotics · doi:10.1109/tro.2017.2765334 · Cited by 193 · see citing works
High-Speed Autonomous Drifting With Deep Reinforcement Learning
Cai, P., Mei, X., Tai, L., Sun, Y., & Liu, M. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2967299 · Cited by 163 · see citing works
Efficient Uncertainty Estimation for Semantic Segmentation in Videos
Huang, P., Hsu, W., Chiu, C.-Y., Wu, T., & Sun, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_32 · Cited by 123 · see citing works
Driver’s fatigue recognition based on yawn detection in thermal images
Knapik, M., & Cyganek, B. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.02.014 · Cited by 107 · see citing works
A Hardware Platform Framework for an Intelligent Vehicle Based on a Driving Brain
Li, D., & Gao, H. (2018) · Engineering · doi:10.1016/j.eng.2018.07.015 · Cited by 104 · see citing works
Improving monocular visual SLAM in dynamic environments: an optical-flow-based approach
Cheng, J., Sun, Y., & Meng, M. Q. ‐. (2019) · Advanced Robotics · doi:10.1080/01691864.2019.1610060 · Cited by 95 · see citing works
A Novel Point Cloud Compression Algorithm Based on Clustering
Sun, X., Ma, H., Sun, Y., & Liu, M. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2900747 · Cited by 91 · see citing works
Integrating Deep Semantic Segmentation Into 3-D Point Cloud Registration
Zaganidis, A., Sun, L., Duckett, T., & Cielniak, G. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2848308 · Cited by 89 · see citing works
Self-Supervised Drivable Area and Road Anomaly Segmentation Using RGB-D Data For Robotic Wheelchairs
Wang, H., Sun, Y., & Liu, M. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2932874 · Cited by 74 · see citing works
Wu, X., Li, Z., Kan, Z., & Gao, H. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2933019 · Cited by 72 · see citing works
Focal Loss in 3D Object Detection
Peng, Y., Tai, L., Wang, Y., Liu, C., & Liu, M. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2894858 · Cited by 68 · see citing works
PointTrackNet: An End-to-End Network For 3-D Object Detection and Tracking From Point Clouds
Wang, S., Sun, Y., Liu, C., & Liu, M. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2974392 · Cited by 61 · see citing works
Vision-Based Trajectory Planning via Imitation Learning for Autonomous Vehicles
Cai, P., Sun, Y., Chen, Y., & Liu, M. (2019) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2019.8917149 · Cited by 52 · see citing works
Active Perception for Foreground Segmentation: An RGB-D Data-Based Background Modeling Method
Sun, Y., Liu, M., & Meng, M. Q. ‐. (2019) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2019.2893414 · Cited by 50 · see citing works
Thermal Augmented Expression Recognition
Wang, S., Pan, B., Chen, H., & Ji, Q. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2786309 · Cited by 48 · see citing works
Efficient Velodyne SLAM with point and plane features
Grant, W. S., Voorhies, R. C., & Itti, L. (2018) · Autonomous Robots · doi:10.1007/s10514-018-9794-6 · Cited by 46 · see citing works
Facial Action Units-Based Image Retrieval for Facial Expression Recognition
Pham, T. T. D., Kim, S., Lu, Y., Jung, S., & Won, C. S. (2019) · IEEE Access · doi:10.1109/access.2018.2889852 · Cited by 44 · see citing works
Statistical Model of Total Target Registration Error in Image-Guided Surgery
Min, Z., Ren, H., & Meng, M. Q. ‐. (2019) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2019.2909646 · Cited by 37 · see citing works
Sun, Y., Zuo, W., & Liu, M. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2975414 · Cited by 37 · see citing works
Robust Semantic Mapping in Challenging Environments
Cheng, J., Sun, Y., & Meng, M. Q. ‐. (2019) · Robotica · doi:10.1017/s0263574719000584 · Cited by 32 · see citing works
Deep Neural Networks: A Comparison on Different Computing Platforms
Modasshir, M., Li, A. Q., & Rekleitis, I. M. (2018) · Conference on Computer and Robot Vision (CRV) · doi:10.1109/crv.2018.00060 · Cited by 25 · see citing works
Accurate Lane Detection with Atrous Convolution and Spatial Pyramid Pooling for Autonomous Driving
Sun, Y., Wang, L., Chen, Y., & Liu, M. (2019) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio49542.2019.8961705 · Cited by 21 · see citing works
Thermal-infrared based drivable region detection
Yoon, J. S., Park, K., Hwang, S., Kim, N. I., Choi, Y., Rameau, F., & Kweon, I. S. (2016) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2016.7535507 · Cited by 18 · see citing works
Metric Monocular Localization Using Signed Distance Fields
Huang, H., Sun, Y., Ye, H., & Liu, M. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8968033 · Cited by 13 · see citing works
Multiple-object Tracking based on Monocular Camera and 3-D Lidar Fusion for Autonomous Vehicles
Chen, H., Xue, C., Liu, S., Sun, Y., & Chen, Y. (2019) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio49542.2019.8961438 · Cited by 6 · see citing works
Vision-and-Lidar Based Real-time Outdoor Localization for Unmanned Ground Vehicles without GPS
Wu, F., He, B., Zhang, L., Chen, S., & Zhang, J. (2018) · IEEE International Conference on Information and Automation (ICIA) · doi:10.1109/icinfa.2018.8812420 · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers
Zhang, J., Liu, H., Yang, K., Hu, X., Liu, R., & Stiefelhagen, R. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3300537 · Cited by 731 · see citing works
Review the state-of-the-art technologies of semantic segmentation based on deep learning
Mo, Y., Wu, Y. X., Yang, X., Liu, F., & Liao, Y. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.01.005 · Cited by 647 · see citing works
GMNet: Graded-Feature Multilabel-Learning Network for RGB-Thermal Urban Scene Semantic Segmentation
Zhou, W., Liu, J., Lei, J., Yu, L., & Hwang, J.-N. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3109518 · Cited by 344 · see citing works
MFFENet: Multiscale Feature Fusion and Enhancement Network For RGB–Thermal Urban Road Scene Parsing
Zhou, W., Lin, X., Lei, J., Yu, L., & Hwang, J. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3086618 · Cited by 251 · see citing works
Delivering Arbitrary-Modal Semantic Segmentation
Zhang, J., Liu, R., Shi, H., Yang, K., Reiß, S., Peng, K., Fu, H., Wang, K., & Stiefelhagen, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00116 · Cited by 202 · see citing works
RGB-T Semantic Segmentation With Location, Activation, and Sharpening
Li, G., Wang, Y., Liu, Z., Zhang, X., & Zeng, D. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3208833 · Cited by 196 · see citing works
Multi-Modal 3D Object Detection in Autonomous Driving: A Survey
Wang, Y., Mao, Q., Zhu, H., Deng, J., Zhang, Y., Ji, J., Li, H., & Zhang, Y. (2023) · International Journal of Computer Vision · doi:10.1007/s11263-023-01784-z · Cited by 214 · see citing works
Explicit Attention-Enhanced Fusion for RGB-Thermal Perception Tasks
Liang, M., Hu, J., Bao, C., Hua, F., Deng, F., & Lam, T. L. (2023) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2023.3272269 · Cited by 179 · see citing works
Zhou, W., Dong, S., Lei, J., & Yu, L. (2022) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2022.3164899 · Cited by 179 · see citing works
Cross-Modal Fusion and Progressive Decoding Network for RGB-D Salient Object Detection
Hu, X., Sun, F., Sun, J., Wang, F., & Li, H. (2024) · International Journal of Computer Vision · doi:10.1007/s11263-024-02020-y · Cited by 150 · see citing works
Zhou, W., Zhang, H., Yan, W., & Lin, W. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3275314 · Cited by 151 · see citing works
FEANet: Feature-Enhanced Attention Network for RGB-Thermal Real-time Semantic Segmentation
Deng, F., Feng, H., Liang, M., Wang, H., Yang, Y., Gao, Y., Chen, J., Hu, J., Guo, X., & Lam, T. L. (2021) · 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros51168.2021.9636084 · Cited by 170 · see citing works
Self-citationEdge-Aware Guidance Fusion Network for RGB–Thermal Scene Parsing
Zhou, W., Dong, S., Xu, C., & Qian, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20269 · Cited by 146 · see citing works
Context-Aware Interaction Network for RGB-T Semantic Segmentation
Lv, Y., Liu, Z., & Li, G. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3349072 · Cited by 138 · see citing works
MODS—A USV-Oriented Object Detection and Obstacle Segmentation Benchmark
Bovcon, B., Muhovič, J., Vranac, D., Mozetič, D., Perš, J., & Kristan, M. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2021.3124192 · Cited by 121 · see citing works
Wu, H., Zhao, Z., & Wang, Z. (2023) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2023.3292373 · Cited by 112 · see citing works
SGFNet: Semantic-Guided Fusion Network for RGB-Thermal Semantic Segmentation
Wang, Y., Li, G., & Liu, Z. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3281419 · Cited by 104 · see citing works
Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation
Wan, Z., Zhang, P., Wang, Y., Yong, S., Stepputtis, S., Sycara, K. P., & Xie, Y. (2025) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61041.2025.00176 · Cited by 114 · see citing works
Mitigating Modality Discrepancies for RGB-T Semantic Segmentation
Zhao, S., Liu, Y., Jiao, Q., Zhang, Q., & Han, J. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2022.3233089 · Cited by 102 · see citing works
EGFNet: Edge-Aware Guidance Fusion Network for RGB–Thermal Urban Scene Parsing
Dong, S., Zhou, W., Xu, C., & Yan, W. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3306368 · Cited by 107 · see citing works
A Comprehensive Review on Deep Learning-Based Data Fusion
Hussain, M., O’Nils, M., Lundgren, J., & Mousavirad, S. J. (2024) · IEEE Access · doi:10.1109/access.2024.3508271 · Cited by 107 · see citing works
Zhou, W., Jianhui, J., Lei, J., & Yu, L. (2022) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2022.3159032 · Cited by 94 · see citing works
RGB-T image analysis technology and application: A survey
Song, K., Zhao, Y., Huang, L., Yan, Y., & Meng, Q. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.105919 · Cited by 94 · see citing works
A Feature Divide-and-Conquer Network for RGB-T Semantic Segmentation
Zhao, S., & Zhang, Q. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3229359 · Cited by 91 · see citing works
Deep Learning in Multimodal Fusion for Sustainable Plant Care: A Comprehensive Review
Yang, Z., Li, Y., Wang, R.-F., Hu, P., & Su, W. (2025) · Sustainability · doi:10.3390/su17125255 · Cited by 83 · see citing works
Glass Segmentation With RGB-Thermal Image Pairs
Huo, D., Wang, J., Qian, Y., & Yang, Y. (2023) · IEEE Transactions on Image Processing · doi:10.1109/tip.2023.3256762 · Cited by 73 · see citing works
From SLAM to Situational Awareness: Challenges and Survey
Bavle, H., Sánchez-López, J. L., Cimarelli, C., Tourani, A., & Voos, H. (2023) · Sensors · doi:10.3390/s23104849 · Cited by 76 · see citing works
Complementarity-aware cross-modal feature fusion network for RGB-T semantic segmentation
Wu, W., Chu, T., & Liu, Q. (2022) · Pattern Recognition · doi:10.1016/j.patcog.2022.108881 · Cited by 74 · see citing works
Deep Depth Estimation from Thermal Image
Shin, U., Park, J., & Kweon, I. S. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00107 · Cited by 74 · see citing works
Embedded Control Gate Fusion and Attention Residual Learning for RGB–Thermal Urban Scene Parsing
Zhou, W., Lv, Y. H., Lei, J., & Yu, L. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3242651 · Cited by 71 · see citing works
Zhou, W., Gong, T., Lei, J., & Yu, L. (2023) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2023.3298921 · Cited by 68 · see citing works
Multispectral Semantic Segmentation for Land Cover Classification: An Overview
Ramos, L., & Sappa, A. D. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3438620 · Cited by 65 · see citing works
MS-UDA: Multi-Spectral Unsupervised Domain Adaptation for Thermal Image Semantic Segmentation
Kim, Y.-H., Shin, U., Park, J., & Kweon, I. S. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3093652 · Cited by 60 · see citing works
MDNet: Mamba-Effective Diffusion-Distillation Network for RGB-Thermal Urban Dense Prediction
Zhou, W., Wu, H., & Jiang, Q. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3508058 · Cited by 61 · see citing works
Rizzoli, G., Barbato, F., & Zanuttigh, P. (2022) · Technologies · doi:10.3390/technologies10040090 · Cited by 58 · see citing works
A Total Variation With Joint Norms For Infrared and Visible Image Fusion
Nie, R., Ma, C., Cao, J., Ding, H., & Zhou, D. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3065496 · Cited by 55 · see citing works
Xu, C. X., Li, Q., Jiang, X., Yu, D., & Zhou, Y. (2022) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2022.3216313 · Cited by 56 · see citing works
On the Role of Thermal Imaging in Automotive Applications: A Critical Review
Farooq, M. A., Shariff, W., O'Callaghan, D., Merla, A., & Corcoran, P. (2023) · IEEE Access · doi:10.1109/access.2023.3255110 · Cited by 58 · see citing works
CACFNet: Cross-Modal Attention Cascaded Fusion Network for RGB-T Urban Scene Parsing
Zhou, W., Dong, S., Fang, M., & Yu, L. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3314527 · Cited by 57 · see citing works
Wu, H., Li, H., Chi, H.-L., Peng, Z., Chang, S., & Wu, Y. C. (2023) · Advanced Engineering Informatics · doi:10.1016/j.aei.2023.101939 · Cited by 55 · see citing works
Zhu, H., Huang, M., Ji, P.-Q., Xiao, F., & Zhang, Q. (2025) · Tunnelling and Underground Space Technology · doi:10.1016/j.tust.2025.106508 · Cited by 61 · see citing works
DHFNet: dual-decoding hierarchical fusion network for RGB-thermal semantic segmentation
Cai, Y., Zhou, W., Zhang, L., Yu, L., & Luo, T. (2023) · The Visual Computer · doi:10.1007/s00371-023-02773-6 · Cited by 53 · see citing works
Multispectral Fusion Transformer Network for RGB-Thermal Urban Scene Semantic Segmentation
Zhou, H., Tian, C., Zhang, Z., Huo, Q., Xie, Y., & Li, Z. (2022) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2022.3179721 · Cited by 53 · see citing works
GCNet: Grid-like context-aware network for RGB-thermal semantic segmentation
Liu, J., Zhou, W., Cui, Y., Yu, L., & Luo, T. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.07.041 · Cited by 53 · see citing works
Semantic Segmentation for Thermal Images: A Comparative Survey
Kutuk, Z., & Algan, G. (2022) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00043 · Cited by 54 · see citing works
Yi, S., Chen, M., Yuan, X., Si, G., & Wang, J. (2025) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2025.01.022 · Cited by 52 · see citing works
Yi (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.11.056 · Cited by 49 · see citing works
Contrastive learning-based knowledge distillation for RGB-thermal urban scene semantic segmentation
Guo, X., Zhou, W., & Liu, T. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111588 · Cited by 52 · see citing works
Residual spatial fusion network for RGB-thermal semantic segmentation
Li, P., Chen, J., Lin, B., & Xu, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127913 · Cited by 49 · see citing works
Mask-guided modality difference reduction network for RGB-T semantic segmentation
Liang, W., Yang, Y., Li, F., Long, X., & Shan, C. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.12.036 · Cited by 45 · see citing works
Real-time semantic segmentation for autonomous driving: A review of CNNs, Transformers, and Beyond
Elhassan, M. A. M., Zhou, C., Khan, A. H., Benabid, A., Adam, A. B. M., Mehmood, A., & Wambugu, N. M. (2024) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2024.102226 · Cited by 46 · see citing works
Efficient Multimodal Semantic Segmentation via Dual-Prompt Learning
Dong, S., Feng, Y., Yang, Q., Huang, Y., Liu, D., & Fan, H. (2024) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros58592.2024.10801872 · Cited by 46 · see citing works
Global contextually guided lightweight network for RGB-thermal urban scene understanding
Gong, T., Zhou, W., Qian, X., Lei, J., & Yu, L. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105510 · Cited by 43 · see citing works
AGFNet: Adaptive Gated Fusion Network for RGB-T Semantic Segmentation
Zhou, X., Wu, X., Bao, L., Yin, H., Jiang, Q., & Zhang, J. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3528064 · Cited by 41 · see citing works
Complementary Random Masking for RGB-Thermal Semantic Segmentation
Shin, U., Lee, K., Kweon, I. S., & Oh, J. (2024) · 2024 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra57147.2024.10611200 · Cited by 49 · see citing works
“Specifically, they proposed various types of feature fusion modules, such as the na ¨ ıve feature-level fusion [11], [36], [35], multi-scale feature fusion [36], [46], [37], [21], and attention-weighted fusion [44], [46], [8], [21].”
BackgroundMethodDuan, A., Victorova, M., Zhao, J., Sun, Y., Zheng, Y., & Navarro-Alarcón, D. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3186504 · Cited by 38 · see citing works
Shojaiee, F., & Baleghi, Y. (2022) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2022.105627 · Cited by 34 · see citing works
Min, Z., Wang, J., Pan, J., & Meng, M. Q. ‐. (2020) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2020.3014420 · Cited by 35 · see citing works
Zhou, W., Xiao, Y., Yan, W., & Yu, L. (2023) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2023.3313122 · Cited by 31 · see citing works
Feng, Z., Guo, Y., & Sun, Y. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3376534 · Cited by 34 · see citing works
Li, B., Wang, T., Zhai, Y., & Yuan, J. (2022) · Measurement · doi:10.1016/j.measurement.2022.112177 · Cited by 33 · see citing works
DooDLeNet: Double DeepLab Enhanced Feature Fusion for Thermal-color Semantic Segmentation
Frigo, O., Martin-Gaffe, L., & Wacongne, C. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw56347.2022.00341 · Cited by 32 · see citing works
Gallagher, J., & Oughton, E. J. (2023) · Scientific Reports · doi:10.1038/s41598-023-34791-8 · Cited by 39 · see citing works
Shin, U., Lee, K., Lee, S., & Kweon, I. S. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3137895 · Cited by 30 · see citing works
Multi-Branch Differential Bidirectional Fusion Network for RGB-T Semantic Segmentation
Liang, W., Shan, C., Yang, Y., & Han, J. (2024) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2024.3374793 · Cited by 30 · see citing works
Yan, R., Yang, K., & Wang, K. (2021) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio54168.2021.9739390 · Cited by 29 · see citing works
Light Transport Induced Domain Adaptation for Semantic Segmentation in Thermal Infrared Urban Scenes
Chen, J., Liu, Z., Jin, D., Wang, Y., Yang, F., & Bai, X. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3194931 · Cited by 28 · see citing works
Li, G., Lin, Y., Ouyang, D., Li, S., Luo, X., Qu, X., Pi, D., & Li, S. E. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3332350 · Cited by 27 · see citing works
Multispectral Video Semantic Segmentation: A Benchmark Dataset and Baseline
Ji, W., Li, J., Bian, C., Zhou, Z., Zhao, J., Yuille, A., & Cheng, L. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00112 · Cited by 26 · see citing works
DHFNet: Decoupled Hierarchical Fusion Network for RGB-T dense prediction tasks
Chen, H., Wang, Z., Qin, H., & Mu, X. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127594 · Cited by 24 · see citing works
ECFNet: Efficient cross-layer fusion network for real time RGB-Thermal urban scene parsing
Shen, Z., Wang, J., Weng, Y., Pan, Z., Li, Y., & Wang, J. (2024) · Digital Signal Processing · doi:10.1016/j.dsp.2024.104579 · Cited by 24 · see citing works
Multi-Modal Contrastive Learning for LiDAR Point Cloud Rail-Obstacle Detection in Complex Weather
Wen, L., Peng, Y., Lin, M., Gan, N., & Tan, R. (2024) · Electronics · doi:10.3390/electronics13010220 · Cited by 25 · see citing works
Resolving semantic conflicts in RGB-T semantic segmentation
Zhao, S., Jin, Z., Jiao, Q., Zhang, Q., & Han, J. (2025) · Pattern Recognition · doi:10.1016/j.patcog.2025.111398 · Cited by 22 · see citing works
Shin, U., Park, K., Lee, B.-U., Lee, K., & Kweon, I. S. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00575 · Cited by 24 · see citing works
MiLNet: Multiplex Interactive Learning Network for RGB-T Semantic Segmentation
Liu, J., Liu, H., Li, X., Ren, J., & Xu, X. (2025) · IEEE Transactions on Image Processing · doi:10.1109/tip.2025.3544484 · Cited by 23 · see citing works
Visible and thermal images fusion architecture for few-shot semantic segmentation
Bao, Y., Song, K., Wang, J., Huang, L., Dong, H., & Yan, Y. (2021) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2021.103306 · Cited by 23 · see citing works
Region-adaptive and context-complementary cross modulation for RGB-T semantic segmentation
Peng, F., Ding, Z., Chen, Z., Wang, G., Hui, T., Liu, S., & Shi, H. (2023) · Pattern Recognition · doi:10.1016/j.patcog.2023.110092 · Cited by 22 · see citing works
Ji, W., Li, J., Bian, C., Zhang, Z., & Cheng, L. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611738 · Cited by 22 · see citing works
StitchFusion: Weaving Any Visual Modalities to Enhance Multimodal Semantic Segmentation
Li, B., Zhang, D., Zhao, Z., Gao, J., & Li, X. (2025) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3746027.3755110 · Cited by 24 · see citing works
Shin, U., Lee, K., Lee, B.-U., & Kweon, I. S. (2022) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2022.3185382 · Cited by 21 · see citing works
MEFNET: Multi-expert fusion network for RGB-Thermal semantic segmentation
Lai, W., Zeng, F., Hu, X., Li, W., He, S., Liu, Z., & Jiang, Y. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.106638 · Cited by 22 · see citing works
Zhang, J., Zhang, R., Yuan, W., & Liu, Y. (2024) · Evolving Systems · doi:10.1007/s12530-024-09567-8 · Cited by 21 · see citing works
Caltech Aerial RGB-Thermal Dataset in the Wild
Lee, C., Anderson, M., Ranganathan, N., Zuo, X., Do, K., Gkioxari, G., & Chung, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73036-8_14 · Cited by 20 · see citing works
Learning Discriminative Feature Representation for Estimating Smoke Density of Smoky Vehicle Rear
Tao, H., & Duan, Q. (2022) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2022.3198047 · Cited by 20 · see citing works
Knowledge Distillation SegFormer-Based Network for RGB-T Semantic Segmentation
Zhou, W., Gong, T., & Yan, W. (2024) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2024.3517732 · Cited by 20 · see citing works
Towards efficient RGB-T semantic segmentation via feature generative distillation strategy
Zhao, S., Wang, J., Zhang, Q., & Han, J. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103282 · Cited by 18 · see citing works
Tang, G., Wu, Z., Li, J., Zhong, P., Ye, W., Chen, X., Lu, H., & Fan, R. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3586286 · Cited by 21 · see citing works
Deformation-Resilient Multigranularity Learning for Unaligned RGB–T Semantic Segmentation
Zhou, H., Zhang, Z., Li, C., Tian, C., Xie, Y., Li, Z., & Wu, X. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3585105 · Cited by 19 · see citing works
GEBNet: Graph-Enhancement Branch Network for RGB-T Scene Parsing
Dong, S., Zhou, W., Qian, X., & Yu, L. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3219350 · Cited by 20 · see citing works
MS-IRTNet: Multistage information interaction network for RGB-T semantic segmentation
Zhang, Z., Liu, Y., & Xue, W. (2023) · Information Sciences · doi:10.1016/j.ins.2023.119442 · Cited by 17 · see citing works
Multi-Modal 3D Object Detection in Autonomous Driving: a SurveyPreprint
Wang, Y., Mao, Q., Zhu, H., Deng, J., Zhang, Y., Ji, J., Li, H., & Zhang, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.12735 · Cited by 18 · see citing works
Chung, Y., Chou, C.-A., & Li, C.-Y. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18030961 · Cited by 17 · see citing works
Deo, A. J., Behera, S. K., & Das, D. P. (2023) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2023.3253574 · Cited by 17 · see citing works
IGFNet: Illumination-Guided Fusion Network for Semantic Scene Understanding using RGB-Thermal Images
Li, H., & Sun, Y. (2023) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio58561.2023.10354613 · Cited by 15 · see citing works
C⁴Net: Excavating Cross-Modal Context- and Content-Complementarity for RGB-T Semantic Segmentation
Zhao, S., Li, J., & Zhang, Q. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3485655 · Cited by 15 · see citing works
Graph Representation Learning for Infrared and Visible Image Fusion
Li, J., Bai, L., Yang, B., Li, C., & Ma, L. (2025) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2025.3557234 · Cited by 14 · see citing works
Shen, J., Zhan, H., Dong, S., Zuo, X., Yang, W., & Ling, H. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.103895 · Cited by 15 · see citing works
FSNet: Redesign Self-Supervised MonoDepth for Full-Scale Depth Prediction for Autonomous Driving
Liu, Y., Xu, Z., Huang, H., Wang, L., & Liu, M. (2023) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2023.3290348 · Cited by 13 · see citing works
On Exploring Shape and Semantic Enhancements for RGB-X Semantic Segmentation
Yang, Y., Shan, C., Zhao, F., Liang, W., & Han, J. (2023) · IEEE Transactions on Intelligent Vehicles · doi:10.1109/tiv.2023.3296219 · Cited by 13 · see citing works
A Correlation Analysis-Based Multivariate Alarm Method With Maximum Likelihood Evidential Reasoning
Weng, X., Xu, X., Feng, J., Shen, X., Meng, J., & Steyskal, F. (2023) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2023.3305524 · Cited by 13 · see citing works
- Authors
- Sun, Y., Zuo, W., Yun, P., Wang, H., & Liu, M.
- Affiliations
- Hong Kong University of Science and Technology (HK)
- Venue
- IEEE Transactions on Automation Science and Engineering
- Published
- 2021-07
- Volume
- 18
- Issue
- 3
- Pages
- 1000-1011
- ISSN
- 1545-5955 (ISSN-L)
- 1558-3783
- OpenAlex
- W3033020497
- Cited by
- 270
- References
- 60
- Citations by year
- 2026: 39 · 2025: 74 · 2024: 50 · 2023: 56 · 2022: 31 · 2021: 19 · 2020: 1
- FWCI
- 10.32
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Automation Science and Engineering (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Funders
- National Natural Science Foundation of China (U1713211)
- Research Grant Council of Hong Kong (11210017)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Keywords
- semantic segmentation, urban scenes, infrared thermography, autonomous driving, multimodal sensor fusion, low-light conditions, end-to-end neural network, obstacle detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
