The DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
RTFNet: RGB-Thermal Fusion Network for Semantic Segmentation of Urban Scenes
Abstract
Semantic segmentation is a fundamental capability for autonomous vehicles. With the advancements of deep learning technologies, many effective semantic segmentation networks have been proposed in recent years. However, most of them are designed using RGB images from visible cameras. The quality of RGB images is prone to be degraded under unsatisfied lighting conditions, such as darkness and glares of oncoming headlights, which imposes critical challenges for the networks that use only RGB images. Different from visible cameras, thermal imaging cameras generate images using thermal radiations. They are able to see under various lighting conditions. In order to enable robust and accurate semantic segmentation for autonomous vehicles, we take the advantage of thermal images and fuse both the RGB and thermal information in a novel deep neural network. The main innovation of this letter is the architecture of the proposed network. We adopt the encoder–decoder design concept. ResNet is employed for feature extraction and a new decoder is developed to restore the feature map resolution. The experimental results prove that our network outperforms the state of the arts.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Compared to the object detection problem summarized in Sec. IV-A, there are fewer works on multi-modal semantic segmentation: [92], [119], [124] employ RGB and thermal images, [61] fuses RGB images and depth images from a stereo camera, [125]-[127] combine RGB, thermal, and depth images for semantic segmentation in diverse environments such as forests, [123] fuses RGB images and LiDAR point clouds”
From Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2021) · cites this paper as [124]
In the citing paper
“| RTFNet [106] | ResNet152 [16] | 53.2 |”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [106]
In the citing paper
“| RFNet [106] | ResNet18 [16] | 69.37 |”
From Multimodal semantic segmentation in autonomous driving: A review of current approaches and future perspectives (2022) · cites this paper as [106]
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 114,148
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
Fully convolutional networks for semantic segmentation
Long, J., Shelhamer, E., & Darrell, T. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298965 · Cited by 38,834
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2699184 · Cited by 22,392
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2016) · arXiv · doi:10.48550/arxiv.1606.00915 · Cited by 22,392
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631
Zhao, H., Shi, J., Qi, X., Wang, X., & Jia, J. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.660 · Cited by 16,438
(2010) · Cited by 14,134
Fully convolutional networks for semantic segmentation
Shelhamer, E., Long, J., & Darrell, T. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2572683 · Cited by 11,898
Learning Deconvolution Network for Semantic Segmentation
Noh, H., Hong, S., & Han, B. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.178 · Cited by 4,422
Understanding Convolution for Semantic Segmentation
Wang, P., Chen, P., Yuan, Y., Liu, D., Huang, Z., Hou, X., & Cottrell, G. W. (2018) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2018.00163 · Cited by 2,184
Neural Networks: Tricks of the Trade: Second Edition
(2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8 · Cited by 2,116
ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation
Romera, E., Álvarez, J. M., Bergasa, L. M., & Arroyo, R. (2017) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2017.2750080 · Cited by 1,514
FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture
Hazirbas, C., Ma, L., Domokos, C., & Cremers, D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54181-5_14 · Cited by 902
Thermal cameras and applications: A survey
Gade, R., & Moeslund, T. B. (2014) · Machine Vision and Applications · doi:10.1007/s00138-013-0570-5 · Cited by 780
MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes
Ha, Q., Watanabe, K., Karasawa, T., Ushiku, Y., & Harada, T. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206396 · Cited by 693
Survey on semantic segmentation using deep learning techniques
Lateef, F., & Ruichek, Y. (2019) · Neurocomputing · doi:10.1016/j.neucom.2019.02.003 · Cited by 577
Improving RGB-D SLAM in dynamic environments: A motion removal approach
Sun, Y., Liu, M., & Meng, M. Q. ‐. (2016) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2016.11.012 · Cited by 387
Self-citationVollmer, M., & Möllmann, K. (2017) · Wiley eBooks · doi:10.1002/9783527693306 · Cited by 320
Remote sensing of concrete bridge decks using unmanned aerial vehicle infrared thermography
Omar, T., & Nehdi, M. L. (2017) · Automation in Construction · doi:10.1016/j.autcon.2017.06.024 · Cited by 266
Motion removal for reliable RGB-D SLAM in dynamic environments
Sun, Y., Liu, M., & Meng, M. Q. ‐. (2018) · Robotics and Autonomous Systems · doi:10.1016/j.robot.2018.07.002 · Cited by 200
Self-citationMonocular Semantic Occupancy Grid Mapping With Convolutional Variational Encoder–Decoder Networks
Lu, C., van de Molengraft, R., & Dubbelman, G. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2891028 · Cited by 169
Autonomous Robotic Exploration by Incremental Road Map Construction
Wang, C., Chi, W., Sun, Y., & Meng, M. Q. ‐. (2019) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2019.2894748 · Cited by 85
Self-citationHalmetschlager-Funek, G., Suchi, M., Kampel, M., & Vincze, M. (2018) · IEEE Robotics & Automation Magazine · doi:10.1109/mra.2018.2852795 · Cited by 82
Robust Intrinsic and Extrinsic Calibration of RGB-D Cameras
Basso, F., Menegatti, E., & Pretto, A. (2018) · IEEE Transactions on Robotics · doi:10.1109/tro.2018.2853742 · Cited by 76
Active Perception for Foreground Segmentation: An RGB-D Data-Based Background Modeling Method
Sun, Y., Liu, M., & Meng, M. Q. ‐. (2019) · IEEE Transactions on Automation Science and Engineering · doi:10.1109/tase.2019.2893414 · Cited by 50
Self-citationObstacle Detection System for Small UAVs using ADS-B and Thermal Imaging
Carrio, A., Lin, Y., Saripalli, S., & Campoy, P. (2017) · Journal of Intelligent & Robotic Systems · doi:10.1007/s10846-017-0529-2 · Cited by 31
A dense semantic mapping system based on CRF-RNN network
Cheng, J., Sun, Y., & Meng, M. Q. ‐. (2017) · International Conference on Advanced Robotics (ICAR) · doi:10.1109/icar.2017.8023671 · Cited by 24
Self-citationThermal-infrared based drivable region detection
Yoon, J. S., Park, K., Hwang, S., Kim, N. I., Choi, Y., Rameau, F., & Kweon, I. S. (2016) · IEEE Intelligent Vehicles Symposium · doi:10.1109/ivs.2016.7535507 · Cited by 18
Flagged first, then by citations
Research works printed but not linked: 1
- vollmer, 2017, et�al
Cited by
Sorted by citations
Feng, D., Haase-Schutz, C., Rosenbaum, L., Hertlein, H., Glaser, C., Timm, F., Wiesbeck, W., & Dietmayer, K. (2021) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2020.2972974 · Cited by 1,425
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers
Zhang, J., Liu, H., Yang, K., Hu, X., Liu, R., & Stiefelhagen, R. (2023) · IEEE Transactions on Intelligent Transportation Systems · doi:10.1109/tits.2023.3300537 · Cited by 731
Review the state-of-the-art technologies of semantic segmentation based on deep learning
Mo, Y., Wu, Y. X., Yang, X., Liu, F., & Liao, Y. (2022) · Neurocomputing · doi:10.1016/j.neucom.2022.01.005 · Cited by 669
Zhao, F., Zhang, C., & Geng, B. (2024) · ACM Computing Surveys · doi:10.1145/3649447 · Cited by 480
Tang, L., Zhang, H., Xu, H., & Ma, J. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101870 · Cited by 418
Metadata
- Authors
- Sun, Y., Zuo, W., & Liu, M.
- Affiliations
- Hong Kong University of Science and Technology (HK)
- Venue
- IEEE Robotics and Automation Letters
- Published
- 2019-07
- Volume
- 4
- Issue
- 3
- Pages
- 2576-2583
- ISSN
- 2377-3766 (ISSN-L)
- 2377-3774
- OpenAlex
- W2921749009
- Cited by
- 546
- References
- 30
- Citations by year
- 2026: 87 · 2025: 127 · 2024: 102 · 2023: 94 · 2022: 51 · 2021: 45 · 2020: 31 · 2019: 9
- FWCI
- 15.22
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Robotics and Automation Letters (publishedVersion)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- Funders
- Shenzhen Science, Technology, and Innovation Commission (JCYJ20160401100022706)
- National Natural Science Foundation of China (U1713211)
- Hong Kong University of Science and Technology Project (IGN16EG12)
- Hong Kong Research Grant Council (11210017, 21202816)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Engineering › Automotive Engineering › Autonomous Vehicle Technology and Safety
- Keywords
- RGB-thermal fusion, semantic segmentation, urban scenes, autonomous vehicles, infrared thermography, encoder-decoder architecture, ResNet feature extraction, low-light conditions, headlight glare
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
