The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
FuseSeg: Semantic Segmentation of Urban Scenes Based on RGB and Thermal Data Fusion
Abstract
Semantic segmentation of urban scenes is an essential component in various applications of autonomous driving. It makes great progress with the rise of deep learning technologies. Most of the current semantic segmentation networks use single-modal sensory data, which are usually the RGB images produced by visible cameras. However, the segmentation performance of these networks is prone to be degraded when lighting conditions are not satisfied, such as dim light or darkness. We find that thermal images produced by thermal imaging cameras are robust to challenging lighting conditions. Therefore, in this article, we propose a novel RGB and thermal data fusion network named FuseSeg to achieve superior performance of semantic segmentation in urban scenes. The experimental results demonstrate that our network outperforms the state-of-the-art networks.Note to Practitioners—This article investigates the problem of semantic segmentation of urban scenes when lighting conditions are not satisfied. We provide a solution to this problem via information fusion with RGB and thermal data. We build an end-to-end deep neural network, which takes as input a pair of RGB and thermal images and outputs pixel-wise semantic labels. Our network could be used for urban scene understanding, which serves as a fundamental component of many autonomous driving tasks, such as environment modeling, obstacle avoidance, motion prediction, and planning. Moreover, the simple design of our network allows it to be easily implemented using various deep learning frameworks, which facilitates the applications on different hardware or software platforms.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“| FuseSeg-Thermal [117] | DenseNet161 [127] | 54.5 |”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [117]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438
(2010) · Cited by 14,134
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465
Fully convolutional networks for semantic segmentation
Shelhamer, E., Long, J., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2572683 · Cited by 11,898
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
Cited by
Sorted by citations
Items 201–269 of 269
Evidence-Based Fusion for Low-Quality RGB-T Semantic Segmentation
Liang, W., Zhang, X., Yang, Y., & Shan, C. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2025.3593962 · Cited by 1
Hazra, S., & Bhattacharya, U. (2025) · Proceedings - International Conference on Image Processing · doi:10.1109/icip55913.2025.11084532 · Cited by 3
Ferrod, R., Dantas, C. F., Di, L., & Ienco, D. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-06078-5_28 · Cited by 1
CMPFNet: semantic segmentation network for cross-modal phased fusion in extreme light scenes
Han, J., Tian, Z., Gao, K., & Lyu, X. (2025) · Pattern Analysis and Applications · doi:10.1007/s10044-025-01569-9 · Cited by 1
A multi-scale U-shaped transformer neural network for low-light image enhancement
Shin, J. S., & Lee, H. S. (2025) · Image and Vision Computing · doi:10.1016/j.imavis.2025.105801 · Cited by 1
Multimodal Thermal and Depth Imaging for Back Acupoint Localization in Massage Robots
Zheng, X., Cui, Y., Chen, Y., Chen, C., Jiang, B., & Liu, L. (2025) · IEEE Sensors Journal · doi:10.1109/jsen.2025.3631312 · Cited by 1
Chen, J., Wang, F., Fang, J., Zhu, Y., Zhang, Y.-A., & Hu, Q. (2026) · iScience · doi:10.1016/j.isci.2026.114705 · Cited by 1
KACAR, A., & Türkoğlu, İ. (2026) · Türk doğa ve fen dergisi :/Türk doğa ve fen dergisi · doi:10.46810/tdfd.1783115 · Cited by 1
ASNet: An adaptive scene-aware network for RGB-thermal urban scene semantic segmentation
Guo, Y., Chen, Z., Rong, X., Liu, C., Song, L., & Li, Y. (2026) · Journal of Visual Communication and Image Representation · doi:10.1016/j.jvcir.2026.104822 · Cited by 1
IG-SAM: Implicitly guided SAM for RGB-thermal semantic segmentation
Zhang, Q., Xing, J., Zhang, H., Yan, R., Hao, M., Yan, L., & Li, Z. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.132804 · Cited by 1
The Effect Analysis of Thermal Infrared Colorization
Han, D., Shin, J., Kim, H., & Choi, Y. (2021) · International Conference on Ubiquitous Robots (UR) · doi:10.1109/ur52253.2021.9494691 · Cited by 1
Learning Multi-Modal Scale-Aware Attentions for Efficient and Robust Road Segmentation
Zhou, Y., Yang, J., Cao, H., Zeng, Z., Zou, H., & Xie, L. (2023) · Unmanned Systems · doi:10.1142/s2301385024410048 · Cited by 1
Cross-modal Augmentation: A Data Augmentation Scheme for RGB-Thermal Semantic Segmentation
Lai, W., Hu, X., Liu, Z., & Jiang, Y. (2023) · IEEE International Conference on Civil Aviation Safety and Information Technology (ICCASIT) · doi:10.1109/iccasit58768.2023.10351626 · Cited by 1
Jayasimhan, A., & Pabitha, P. (2024) · International Conference on Smart Electronics and Communication (ICOSEC) · doi:10.1109/icosec61587.2024.10722343 · Cited by 1
Cross-Modality Comprehensive Feature Aggregation Network for Road Scene Segmentation
Yang, Y., Zhang, Z., Hou, H., Zhang, W., & Zhuang, C. (2024) · International Symposium on Computer Science and Intelligent Control (ISCSIC) · doi:10.1109/iscsic64297.2024.00050 · Cited by 1
Advancing Textile Damage Segmentation: A Novel RGBT Dataset and Thermal Frequency Normalization
Rayhan, F., Joshi, J., Ren, G., Hernandez, L., Petreca, B. B., Baurley, S. L., Bianchi‐Berthouze, N., & Cho, Y. (2025) · Sensors · doi:10.3390/s25072306 · Cited by 1
CSANet: Context–Spatial Awareness Network for RGB-T Urban Scene Understanding
Li, R., Wang, Z., Guo, J., & Zhang, C. (2025) · Journal of Imaging · doi:10.3390/jimaging11060188 · Cited by 1
SEANet: RGB and Thermal Maritime Panoptic Dataset and Intermodal Alignment Procedure
Kafka, O., Kaufmann, J., & Rankl, C. (2025) · IEEE International Forum on Research and Technology for Society and Industry (RTSI) · doi:10.1109/rtsi64020.2025.11212215 · Cited by 1
Wang, C., & Wu, T. (2022) · Lecture notes in electrical engineering · doi:10.1007/978-981-16-9492-9_338 · Cited by 1
Optimal Input Selection for Single Object Tracking using RGB-Thermal Camera
Abdani, S. R., & Zulkifley, M. A. (2022) · IEEE Symposium on Computer Applications & Industrial Electronics (ISCAIE) · doi:10.1109/iscaie54458.2022.9794503 · Cited by 1
Combining Enhancement and Fusion for Low-Light Visible and Infrared Image
Li, S., Zhang, K., Liang, Y., Zhang, C., & Shangguan, X. (2023) · China Automation Congress (CAC) · doi:10.1109/cac59555.2023.10449977 · Cited by 1
An Incentive Stackelberg Strategy Towards Intelligent Connected Vehicles
Liu, F., Liang, W., & Meng, W. (2021) · IEEE International Conference on Unmanned Systems (ICUS) · doi:10.1109/icus52573.2021.9641464
2021 Index IEEE Transactions on Automation Science and Engineering Vol. 18
(2021) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2021.3120615
RoI Uncertainty for RGB-Thermal Image Segmentation
Hu, J., Gu, X., Wu, Y., & Gu, X. (2022) · International Conference on Machine Learning and Machine Intelligence (MLMI) · doi:10.1145/3568199.3568220
Improving information fusion in deep learning
Don, D. P. M. K. (2023) · Queensland University of Technology · doi:10.5204/thesis.eprints.241291
Chen, Y., Li, X., Zhang, J., Liu, C., Luan, C., Run, G., Zhu, Z., Liu, D., & Wu, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4752465
DCFNet: Dense Complementary Fusion for RGB-Thermal Urban Scene Perception
Zhang, Y.-W. M., Zhang, G., & Hu, X. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-4399-5_30
Enhanced Thermal-RGB Fusion for Anomaly Detection and Segmentation
Asadi, A., Salihoglu, S., & Moghaddass, R. (2025) · Annual Reliability and Maintainability Symposium (RAMS) · doi:10.1109/rams48127.2025.10935016
Edge-Supervised Attention-Aware Fusion Network for RGB-T Semantic Segmentation
Wang, M., Zhu, Z., Wang, Y., Tu, R., Weng, J., & Yu, X. (2025) · Electronics · doi:10.3390/electronics14081489
FuseForm: Multimodal Transformer for Semantic Segmentation
McMillen, J., & Yılmaz, Y. (2025) · IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) · doi:10.1109/wacvw65960.2025.00070
Semantic-Aware Autonomous Vehicle Navigation with Causal Transformers and Q-Learning
Pandya, M., Siddalingaswamy, P. C., & Singh, S. (2025) · ACM Journal on Autonomous Transportation Systems · doi:10.1145/3749992
Yang, Y., & Sun, W. (2025) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2025.3599549
MASTER: Multimodal Segmentation with Text Prompts
Liu, F., Lu, S., Mei, J., & Hu, Y. (2025) · Communications in computer and information science · doi:10.1007/978-981-95-0848-8_18
A Spatial-Frequency Domain Joint Mechanism Network for Cross-modal Semantic Segmentation
Qu, Y., Zhang, Z., & He, L. (2025) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme59968.2025.11210110
Meng, Y., Yue, Y., Yang, Y., & Fu, M. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.132060
Edge Aware Segmentation Network with Sparse Transformer Fusion for Multimodal Scene Parsing
Wang, C. (2025) · International Conference on Computers, Information Processing and Advanced Education (CIPAE) · doi:10.1109/cipae66821.2025.00063
Yuan, H., Zhang, L., Bao, R., Si, J., Wang, S., & Niu, T. (2025) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros60139.2025.11247613
Unsupervised multi-modal domain adaptation for RGB-T Semantic Segmentation
Chen, Z., Lin, C., Zhao, Y., & Tillo, T. (2025) · Computer Vision and Image Understanding · doi:10.1016/j.cviu.2025.104573
Modality Adaptive Representation Network for Efficient RGB-T Semantic Segmentation
Zhao, S., Jin, Z., & Zhang, Q. (2025) · IEEE Signal Processing Letters · doi:10.1109/lsp.2025.3619880
CFWA-Net: Cross-scale fusion and wavelet aggregation network for polyp segmentation
Xu, Z., Ji, J., Wang, F., Lu, K., Sun, T., & Li, J. (2025) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2025.109221
Evaluation of Combined RGB and Thermal Imaging Human Detection Model Performance
Subramaniam, S., Sarjerao, S. M., Natarajan, M., & Marimuthu, B. (2025) · Innovations in Power and Advanced Computing Technologies (i-PACT) · doi:10.1109/i-pact65952.2025.11308029
Complementary Information Mining for Redundancy and Weakly Aligned RGB-T Semantic Segmentation
Wang, Q., Ouyang, J., Shen, T., Gu, Y., Ullah, S., Al-Antary, M. T., Alasmary, H., Waqas, M. M., Tang, B., & Halim, Z. (2026) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2026.3654044
Leveraging Vision Foundation Models for RGB-Thermal Semantic Segmentation
Wang, C., & Gong, X. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5567-3_30
谈, 焜. (2026) · Artificial Intelligence and Robotics Research · doi:10.12677/airr.2026.152061
Wu, J., Du, J., Hao, F., & Hong, J. (2026) · Signal Processing Image Communication · doi:10.1016/j.image.2026.117551
Meng, Z., Fukao, T., Meng, L., & Tomiyama, H. (2026) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp55912.2026.11463169
Edge-Embedded Bidirectional Interactive Network for Lightweight Surface Defect Detection
Cui, X., Shao, F., Jin, D., Mu, B., & Chen, H. (2026) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2026.3690492
Boosting RGB-D Segmentation by Fusing Multi-Robot Perceptual Knowledge
Liu, J., Zhang, W., Li, W., & Liu, H. (2025) · Asian Conference on Artificial Intelligence Technology (ACAIT) · doi:10.1109/acait67930.2025.11522587
A Review of 3D Point Cloud Semantic Segmentation Methods for Complex Railway Scenes
石, 江., Wang, X., Hou, H., Yin, T., & Zeng, S. (2026) · Journal of Surveying Engineering · doi:10.1061/jsued2.sueng-1702
基于可形变VMamba与特征对齐的无人机多模态图像语义分割 (特邀)
Tao, 张. Z., Ruitao, 卢. L., Yaochen, 李. L., Xueli, 谢. X., Jiwei, 范. F., & Xiaogang, 杨. Y. (2026) · Infrared and Laser Engineering · doi:10.3788/irla20260160
Zhang, Z., Zhang, M., Hou, H., Shi, Q., Meng, F., Li, X., Yang, T., Ren, W., & Jing, P. (2026) · Digital Signal Processing · doi:10.1016/j.dsp.2026.106377
Multi-scale rectified mixture-of-experts fusion for RGB–Thermal semantic segmentation
Gozler, D., & Topal, C. (2026) · Image and Vision Computing · doi:10.1016/j.imavis.2026.106109
He, X., & Xia, Y. (2026) · International Conference on Artificial Intelligence, Big Data and Algorithms (CAIBDA) · doi:10.1109/caibda70336.2026.11621385
SAR: semantic ambiguity relief network for RGB-T semantic segmentation
Zhao, S., Li, J., Jiao, Q., Zhang, Q., & Han, J. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104671
Jin, Q., Zhang, Y., Peng, C., Hu, J., & Li, L. (2026) · Geo-spatial Information Science · doi:10.1080/10095020.2026.2707066
He, X., Guo, B., Zhao, S., & Han, Y. (2026) · Infrared Physics & Technology · doi:10.1016/j.infrared.2026.106793
HFFNet: Hierarchical feature fusion network for lightweight RGB-T semantic segmentation
Wu, X., Zhou, X., Bao, L., Shi, R., Liu, D., Liu, Z., & Zhang, J. (2026) · Neural Networks · doi:10.1016/j.neunet.2026.109510
Liu, Y., Zhang, W., Chen, X., Ma, Z., Ye, J., Zheng, G. Y., Wang, L., & Hou, W. (2026) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2026.3724831
Knowledge-guided and reliability-aware fusion network for robust RGB-thermal road scene segmentation
Guo, Y., Chen, Z., Liu, C., Zhao, Y., Jian, M., & Meng, Q. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.133975
Thermal-gradient-guided RGB–thermal fusion for intelligent facade hollow diagnosis
Huang, J., Diao, Y., Du, Y., & Pang, J. (2026) · Expert Systems with Applications · doi:10.1016/j.eswa.2026.134137
Zhen, F., & Sun, Y. (2026) · IEEE Canadian Atlantic Ocean Symposium (CAOS) · doi:10.1109/caos69069.2026.11666192
Cheng, P., Zhang, L., Wang, B., Liu, J., & Sui, H. (2026) · International Journal of Applied Earth Observation and Geoinformation · doi:10.1016/j.jag.2026.105572
DSDFNet: Dual-Stream Dense-Feedback Network for RGB-T Semantic Segmentation
Zhang, H., Dong, X., & Du, N. (2026) · International Conference on Advanced Computational Intelligence (ICACI) · doi:10.1109/icaci69731.2026.11677891
AV2T-Gen: Aerial Visible to Thermal Generation with Environment and Vehicle State Guidance
Yang, K., Liu, Y., Wang, Y., Yan, S., Zhang, M., Liu, Y., Wang, X., & Wang, Q. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37225-3_26
Depth Adaption SegNet for RGB-T Segmentation
Zhao, S., Zhang, C., Zhu, H., Liu, B., & Zhou, Y. (2026) · Applied Sciences · doi:10.3390/app16199702
Fan, Z., Li, Y., Shen, Z., Wang, J., Yang, X., & Wang, J. (2026) · Digital Signal Processing · doi:10.1016/j.dsp.2026.106525
A Survey on Degraded Image Segmentation
Chen, L., Fu, Y., Shangguan, J., Xu, J., Peng, Y., & Lyu, F. (2026) · Chinese Journal of Electronics · doi:10.23919/cje.2025.00.245 · Cited by 1
Exploring Dual-to-Single-Encoder Transitions for Real-Time RGB-T Semantic Segmentation
Kim, J., Jung, M., & Moon, B. (2026) · Sensors · doi:10.3390/s26206602
Sun, X. R., Li, J., Wang, Z., & You, Z. (2026) · Information Fusion · doi:10.1016/j.inffus.2026.104902
Metadata
- Authors
- Sun, Y., Zuo, W., Yun, P., Wang, H., & Liu, M.
- Affiliations
- Hong Kong University of Science and Technology (HK)
- Venue
- IEEE Transactions on Automation Science and Engineering
- Published
- 2021-07
- Volume
- 18
- Issue
- 3
- Pages
- 1000-1011
- ISSN
- 1545-5955 (ISSN-L)
- 1558-3783
- OpenAlex
- W3033020497
- Cited by
- 270
- References
- 60
- Citations by year
- 2026: 39 · 2025: 74 · 2024: 50 · 2023: 56 · 2022: 31 · 2021: 19
- FWCI
- 10.32
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Automation Science and Engineering (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Funders
- National Natural Science Foundation of China (U1713211)
- Research Grant Council of Hong Kong (11210017)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Keywords
- semantic segmentation, urban scenes, infrared thermography, autonomous driving, multimodal sensor fusion, low-light conditions, end-to-end neural network, obstacle detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
