The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
FuseSeg: Semantic Segmentation of Urban Scenes Based on RGB and Thermal Data Fusion
Abstract
Semantic segmentation of urban scenes is an essential component in various applications of autonomous driving. It makes great progress with the rise of deep learning technologies. Most of the current semantic segmentation networks use single-modal sensory data, which are usually the RGB images produced by visible cameras. However, the segmentation performance of these networks is prone to be degraded when lighting conditions are not satisfied, such as dim light or darkness. We find that thermal images produced by thermal imaging cameras are robust to challenging lighting conditions. Therefore, in this article, we propose a novel RGB and thermal data fusion network named FuseSeg to achieve superior performance of semantic segmentation in urban scenes. The experimental results demonstrate that our network outperforms the state-of-the-art networks.Note to Practitioners—This article investigates the problem of semantic segmentation of urban scenes when lighting conditions are not satisfied. We provide a solution to this problem via information fusion with RGB and thermal data. We build an end-to-end deep neural network, which takes as input a pair of RGB and thermal images and outputs pixel-wise semantic labels. Our network could be used for urban scene understanding, which serves as a fundamental component of many autonomous driving tasks, such as environment modeling, obstacle avoidance, motion prediction, and planning. Moreover, the simple design of our network allows it to be easily implemented using various deep learning frameworks, which facilitates the applications on different hardware or software platforms.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“| FuseSeg-Thermal [117] | DenseNet161 [127] | 54.5 |”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [117]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438
(2010) · Cited by 14,134
Aggregated Residual Transformations for Deep Neural Networks
Xie, S., Girshick, R., Dollar, P., Tu, Z., & He, K. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.634 · Cited by 12,465
Fully convolutional networks for semantic segmentation
Shelhamer, E., Long, J., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2572683 · Cited by 11,898
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
Deep High-Resolution Representation Learning for Human Pose Estimation
Sun, K., Xiao, B., Liu, D., & Wang, J. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00584 · Cited by 5,984
Stacked Hourglass Networks for Human Pose Estimation
Newell, A., Yang, K., & Deng, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_29 · Cited by 5,634
Learning Deconvolution Network for Semantic Segmentation
Noh, H., Hong, S., & Han, B. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.178 · Cited by 4,422
BiSeNet: Bilateral Segmentation Network for Real-Time Semantic Segmentation
Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., & Sang, N. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01261-8_20 · Cited by 2,968
Frustum PointNets for 3D Object Detection from RGB-D Data
Qi, C. R., Liu, W., Wu, C., Su, H., & Guibas, L. J. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00102 · Cited by 2,655
Understanding Convolution for Semantic Segmentation
Wang, P., Chen, P., Yuan, Y., Liu, D., Huang, Z., Hou, X., & Cottrell, G. W. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00163 · Cited by 2,184
SUN RGB-D: A RGB-D scene understanding benchmark suite
Song, S., Lichtenberg, S. P., & Xiao, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298655 · Cited by 1,943
Wide Residual NetworksPreprint
Zagoruyko, S., & Komodakis, N. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.07146 · Cited by 1,850
ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
Paszke, A., Chaurasia, A., Kim, S., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1606.02147 · Cited by 1,826
ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation
Romera, E., Álvarez, J. M., Bergasa, L. M., & Arroyo, R. (2017) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2017.2750080 · Cited by 1,514
High-Resolution Representations for Labeling Pixels and RegionsPreprint
Sun, K., Zhao, Y. S., Jiang, B., Cheng, T., Xiao, B., Liu, D., Mu, Y., Wang, X., Liu, W., & Wang, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.04514 · Cited by 941
Learning a Discriminative Feature Network for Semantic Segmentation
Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., & Sang, N. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00199 · Cited by 909
FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture
Hazirbas, C., Ma, L., Domokos, C., & Cremers, D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54181-5_14 · Cited by 902
Kendall, A., Badrinarayanan, V., & Cipolla, R. (2017) · Procedings of the British Machine Vision Conference 2017 · doi:10.5244/c.31.57 · Cited by 784
MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes
Ha, Q., Watanabe, K., Karasawa, T., Ushiku, Y., & Harada, T. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206396 · Cited by 693
Information combination operators for data fusion: a comparative review with classification
Bloch, I. (1996) · IEEE Transactions on Systems Man and Cybernetics - Part A Systems and Humans · doi:10.1109/3468.477860 · Cited by 640
Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes
Pohlen, T., Hermans, A., Mathias, M., & Leibe, B. (2017) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2017.353 · Cited by 600
Rtfnet: Rgb-thermal fusion network for semantic segmentation of urban scenes
Sun, Y., Zuo, W., & Liu, M. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2904733 · Cited by 546
Self-citationObject Classification Using CNN-Based Fusion of Vision and LIDAR in Autonomous Vehicle Environment
Gao, H., Cheng, B., Wang, J., Li, K., Zhao, J., & Li, D. (2018) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2018.2822828 · Cited by 529
Improving RGB-D SLAM in dynamic environments: A motion removal approach
Sun, Y., Liu, M., & Meng, M. Q. ‐. (2016) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2016.11.012 · Cited by 387
Vollmer, M., & Möllmann, K. (2017) · Wiley eBooks · doi:10.1002/9783527693306 · Cited by 320
Depth-Aware CNN for RGB-D Segmentation
Wang, W., & Neumann, U. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01252-6_9 · Cited by 314
Kendall, A., Badrinarayanan, V., & Cipolla, R. (2015) · arXiv · doi:10.48550/arxiv.1511.02680 · Cited by 292
Motion removal for reliable RGB-D SLAM in dynamic environments
Sun, Y., Liu, M., & Meng, M. Q. ‐. (2018) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2018.07.002 · Cited by 200
Asymmetric Bimanual Control of Dual-Arm Exoskeletons for Human-Cooperative Manipulations
Li, Z., Huang, B., Ajoudani, A., Yang, C., Su, C., & Bicchi, A. (2017) · IEEE Transactions on Robotics · doi:10.1109/tro.2017.2765334 · Cited by 193
High-Speed Autonomous Drifting With Deep Reinforcement Learning
Cai, P., Mei, X., Tai, L., Sun, Y., & Liu, M. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2967299 · Cited by 163
Efficient Uncertainty Estimation for Semantic Segmentation in Videos
Huang, P., Hsu, W., Chiu, C.-Y., Wu, T., & Sun, M. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_32 · Cited by 123
Driver’s fatigue recognition based on yawn detection in thermal images
Knapik, M., & Cyganek, B. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.02.014 · Cited by 107
A Hardware Platform Framework for an Intelligent Vehicle Based on a Driving Brain
Li, D., & Gao, H. (2018) · Engineering · doi:10.1016/j.eng.2018.07.015 · Cited by 104
Improving monocular visual SLAM in dynamic environments: an optical-flow-based approach
Cheng, J., Sun, Y., & Meng, M. Q. ‐. (2019) · Advanced Robotics · doi:10.1080/01691864.2019.1610060 · Cited by 95
A Novel Point Cloud Compression Algorithm Based on Clustering
Sun, X., Ma, H., Sun, Y., & Liu, M. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2900747 · Cited by 91
Integrating Deep Semantic Segmentation Into 3-D Point Cloud Registration
Zaganidis, A., Sun, L., Duckett, T., & Cielniak, G. (2018) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2018.2848308 · Cited by 89
Self-Supervised Drivable Area and Road Anomaly Segmentation Using RGB-D Data For Robotic Wheelchairs
Wang, H., Sun, Y., & Liu, M. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2932874 · Cited by 74
Wu, X., Li, Z., Kan, Z., & Gao, H. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2019.2933019 · Cited by 72
Focal Loss in 3D Object Detection
Peng, Y., Tai, L., Wang, Y., Liu, C., & Liu, M. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2894858 · Cited by 69
PointTrackNet: An End-to-End Network For 3-D Object Detection and Tracking From Point Clouds
Wang, S., Sun, Y., Liu, C., & Liu, M. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2974392 · Cited by 61
Vision-Based Trajectory Planning via Imitation Learning for Autonomous Vehicles
Cai, P., Sun, Y., Chen, Y., & Liu, M. (2019) · IEEE International Conference on Intelligent Transportation Systems (ITSC) · doi:10.1109/itsc.2019.8917149 · Cited by 52
Active Perception for Foreground Segmentation: An RGB-D Data-Based Background Modeling Method
Sun, Y., Liu, M., & Meng, M. Q. ‐. (2019) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2019.2893414 · Cited by 50
Thermal Augmented Expression Recognition
Wang, S., Pan, B., Chen, H., & Ji, Q. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2786309 · Cited by 48
Efficient Velodyne SLAM with point and plane features
Grant, W. S., Voorhies, R. C., & Itti, L. (2018) · Autonomous Robots · doi:10.1007/s10514-018-9794-6 · Cited by 46
Facial Action Units-Based Image Retrieval for Facial Expression Recognition
Pham, T. T. D., Kim, S., Lu, Y., Jung, S., & Won, C. S. (2019) · IEEE Access · doi:10.1109/access.2018.2889852 · Cited by 44
Statistical Model of Total Target Registration Error in Image-Guided Surgery
Min, Z., Ren, H., & Meng, M. Q. ‐. (2019) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2019.2909646 · Cited by 37
Sun, Y., Zuo, W., & Liu, M. (2020) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2020.2975414 · Cited by 37
Robust Semantic Mapping in Challenging Environments
Cheng, J., Sun, Y., & Meng, M. Q. ‐. (2019) · Robotica · doi:10.1017/s0263574719000584 · Cited by 32
Deep Neural Networks: A Comparison on Different Computing Platforms
Modasshir, M., Li, A. Q., & Rekleitis, I. M. (2018) · Canadian Conference on Computer and Robot Vision · doi:10.1109/crv.2018.00060 · Cited by 26
Accurate Lane Detection with Atrous Convolution and Spatial Pyramid Pooling for Autonomous Driving
Sun, Y., Wang, L., Chen, Y., & Liu, M. (2019) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio49542.2019.8961705 · Cited by 21
Thermal-infrared based drivable region detection
Yoon, J. S., Park, K., Hwang, S., Kim, N. I., Choi, Y., Rameau, F., & Kweon, I. S. (2016) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2016.7535507 · Cited by 18
Metric Monocular Localization Using Signed Distance Fields
Huang, H., Sun, Y., Ye, H., & Liu, M. (2019) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros40897.2019.8968033 · Cited by 13
Multiple-object Tracking based on Monocular Camera and 3-D Lidar Fusion for Autonomous Vehicles
Chen, H., Xue, C., Liu, S., Sun, Y., & Chen, Y. (2019) · IEEE International Conference on Robotics and Biomimetics (ROBIO) · doi:10.1109/robio49542.2019.8961438 · Cited by 6
Vision-and-Lidar Based Real-time Outdoor Localization for Unmanned Ground Vehicles without GPS
Wu, F., He, B., Zhang, L., Chen, S., & Zhang, J. (2018) · IEEE International Conference on Information and Automation (ICIA) · doi:10.1109/icinfa.2018.8812420 · Cited by 4
Cited by
Sorted by citations
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers
Zhang, J., Liu, H., Yang, K., Hu, X., Liu, R., & Stiefelhagen, R. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3300537 · Cited by 731
Review the state-of-the-art technologies of semantic segmentation based on deep learning
Mo, Y., Wu, Y. X., Yang, X., Liu, F., & Liao, Y. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.01.005 · Cited by 669
GMNet: Graded-Feature Multilabel-Learning Network for RGB-Thermal Urban Scene Semantic Segmentation
Zhou, W., Liu, J., Lei, J., Yu, L., & Hwang, J.-N. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3109518 · Cited by 344
MFFENet: Multiscale Feature Fusion and Enhancement Network For RGB–Thermal Urban Road Scene Parsing
Zhou, W., Lin, X., Lei, J., Yu, L., & Hwang, J. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3086618 · Cited by 253
Delivering Arbitrary-Modal Semantic Segmentation
Zhang, J., Liu, R., Shi, H., Yang, K., Reiß, S., Peng, K., Fu, H., Wang, K., & Stiefelhagen, R. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00116 · Cited by 215
Metadata
- Authors
- Sun, Y., Zuo, W., Yun, P., Wang, H., & Liu, M.
- Affiliations
- Hong Kong University of Science and Technology (HK)
- Venue
- IEEE Transactions on Automation Science and Engineering
- Published
- 2021-07
- Volume
- 18
- Issue
- 3
- Pages
- 1000-1011
- ISSN
- 1545-5955 (ISSN-L)
- 1558-3783
- OpenAlex
- W3033020497
- Cited by
- 270
- References
- 60
- Citations by year
- 2026: 39 · 2025: 74 · 2024: 50 · 2023: 56 · 2022: 31 · 2021: 19
- FWCI
- 10.32
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Automation Science and Engineering (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Funders
- National Natural Science Foundation of China (U1713211)
- Research Grant Council of Hong Kong (11210017)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Keywords
- semantic segmentation, urban scenes, infrared thermography, autonomous driving, multimodal sensor fusion, low-light conditions, end-to-end neural network, obstacle detection
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
