CoChat
Sign Up
PreprintCited by 3Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningAdvanced Image Fusion Techniques

The DOI matches its DataCite record.

Variational Probabilistic Fusion Network for RGB-T Semantic Segmentation

Open access
Lin, Baihong, Lin, Zengrong, Guo, Yulan, Zhang, Yulan, Zou, Jianxiao, Fan, Shicai·2023·arXiv
DOI 10.48550/arxiv.2307.08536arXiv 2307.08536
Open this paper in CoChat — free accountOpen PDF

Abstract

RGB-T semantic segmentation has been widely adopted to handle hard scenes with poor lighting conditions by fusing different modality features of RGB and thermal images. Existing methods try to find an optimal fusion feature for segmentation, resulting in sensitivity to modality noise, class-imbalance, and modality bias. To overcome the problems, this paper proposes a novel Variational Probabilistic Fusion Network (VPFNet), which regards fusion features as random variables and obtains robust segmentation by averaging segmentation results under multiple samples of fusion features. The random samples generation of fusion features in VPFNet is realized by a novel Variational Feature Fusion Module (VFFM) designed based on variation attention. To further avoid class-imbalance and modality bias, we employ the weighted cross-entropy loss and introduce prior information of illumination and category to control the proposed VFFM. Experimental results on MFNet and PST900 datasets demonstrate that the proposed VPFNet can achieve state-of-the-art segmentation performance.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2307.08536

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 41

  1. F. Nesti, G. Rossolini, S. Nair, A. Biondi, and G. Buttazzo, "Evaluating the robustness of semantic segmentation for autonomous driving against real-world adversarial patch attacks," in proc. IEEE Winter Conf. on Applications of Comput. Vis., 2022, pp. 2280-2289.
  2. H. Wang, Y. Chen, Y. Cai, L. Chen, Y. Li, M. A. Sotelo, and Z. Li, "Sfnet-n: An improved sfnet algorithm for semantic segmentation of low-light autonomous driving road scenes," IEEE Trans. Intell. Transp. Syst., vol. 23, no. 11, pp. 21 405-21 417, 2022.
  3. I. Alonso, L. Riazuelo, and A. C. Murillo, "Mininet: An efficient semantic segmentation convnet for real-time robotic applications," IEEE Trans. Robotics, vol. 36, no. 4, pp. 1340-1347, 2020.
  4. S. Ainetter and F. Fraundorfer, "End-to-end trainable deep neural net- work for robotic grasp detection and semantic segmentation from rgb," in IEEE Int. Conf. on Robotics and Automation (ICRA). IEEE, 2021, pp. 13 452-13 458.
  5. Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku, and T. Harada, "Mfnet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes," in IEEE In. Conf. on Intell. Robots and Systems (IROS). IEEE, 2017, pp. 5108-5115.
  6. W. Zhou, S. Dong, C. Xu, and Y. Qian, "Edge-aware guidance fusion network for rgb-thermal scene parsing," in proc. AAAI Conf. Artif. Intell., 2022, pp. 3571-3579.
  7. Q. Zhang, S. Zhao, Y. Luo, D. Zhang, N. Huang, and J. Han, "Abmdrnet: Adaptive-weighted bi-directional modality difference reduction network for rgb-t semantic segmentation," in proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2021, pp. 2633-2642.
  8. F. Deng, H. Feng, M. Liang, H. Wang, Y. Yang, Y. Gao, J. Chen, J. Hu, X. Guo, and T. L. Lam, "Feanet: Feature-enhanced attention network for rgb-thermal real-time semantic segmentation," in IEEE In. Conf. on Intell. Robots and Systems (IROS). IEEE, 2021, pp. 4467-4473.
  9. O. Frigo, L. Martin-Gaffe, and C. Wacongne, "Doodlenet: Double deeplab enhanced feature fusion for thermal-color semantic segmenta- tion," in proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022, pp. 3021-3029.
  10. W. Zhou, J. Liu, J. Lei, L. Yu, and J.-N. Hwang, "Gmnet: graded-feature multilabel-learning network for rgb-thermal urban scene semantic seg- mentation," IEEE Trans. Image Processing, vol. 30, pp. 7790-7802, 2021.
  11. W. Zhou, X. Lin, J. Lei, L. Yu, and J.-N. Hwang, "Mffenet: Multiscale feature fusion and enhancement network for rgb-thermal urban road scene parsing," IEEE Trans. Multimedia, vol. 24, pp. 2526-2538, 2021.
  12. T. Zhou, S. Ruan, and S. Canu, "A review: Deep learning for medical image segmentation using multi-modality fusion," Array, vol. 3, p. 100004, 2019.
  13. X. Peng, Y. Wei, A. Deng, D. Wang, and D. Hu, "Balanced multimodal learning via on-the-fly gradient modulation," in proc. IEEE Conf. Com- put. Vis. Pattern Recognit. (CVPR), 2022, pp. 8238-8247.
  14. X. Chen, K.-Y. Lin, J. Wang, W. Wu, C. Qian, H. Li, and G. Zeng, "Bi-directional cross-modality feature propagation with separation-and- aggregation gate for rgb-d semantic segmentation," in proc. Eur. Conf. Comput. Vis. (ECCV). Springer, 2020, pp. 561-577.
  15. D. P. Kingma and M. Welling, "Auto-encoding variational bayes," arXiv preprint arXiv:1312.6114, 2013. · arXiv
  16. B. Chen, Z. Yan, K. Li, P. Li, B. Wang, W. Zuo, and L. Zhang, "Variational attention: Propagating domain-specific knowledge for multi- domain learning in crowd counting," in proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2021, pp. 16 065-16 075.
  17. L. Breiman, "Random forests," Machine learning, vol. 45, pp. 5-32, 2001.
  18. J. Lafferty, A. McCallum, and F. C. Pereira, "Conditional random fields: Probabilistic models for segmenting and labeling sequence data," in proc. IEEE int. conf. on Machine Learning (ICML), 2001, pp. 282-289.
  19. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection," in proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), vol. 1. Ieee, 2005, pp. 886-893.
  20. W. Burger and M. J. Burge, Scale-Invariant Feature Transform (SIFT). London: Springer London, 2016, pp. 609-664. [Online]. Available: https://doi.org/10.1007/978-1-4471-6684-9 25 · DOI
  21. J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," in proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2015, pp. 3431-3440.
  22. O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in proc. Int. Conf. Medical Image Computing and Computer-Assisted Intervention (MICCAI). Springer, 2015, pp. 234-241.
  23. H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, "Pyramid scene parsing network," in proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2017, pp. 2881-2890.
  24. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, "Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs," IEEE Trans. Pattern Anal. Machine Intell., vol. 40, no. 4, pp. 834-848, 2017.
  25. J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu, "Dual attention network for scene segmentation," in proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2019, pp. 3146-3154.
  26. Z. Huang, X. Wang, L. Huang, C. Huang, Y. Wei, and W. Liu, "Ccnet: Criss-cross attention for semantic segmentation," in proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2019, pp. 603-612.
  27. Y. Sun, W. Zuo, and M. Liu, "Rtfnet: Rgb-thermal fusion network for semantic segmentation of urban scenes," IEEE Robotics and Automation Letters, vol. 4, no. 3, pp. 2576-2583, 2019.
  28. Y. Sun, W. Zuo, P. Yun, H. Wang, and M. Liu, "Fuseseg: Semantic segmentation of urban scenes based on rgb and thermal data fusion," IEEE Trans. Automation Science and Engineering, vol. 18, no. 3, pp. 1000-1011, 2020.
  29. S. S. Shivakumar, N. Rodrigues, A. Zhou, I. D. Miller, V. Kumar, and C. J. Taylor, "Pst900: Rgb-thermal calibration, dataset and segmentation network," in proc. IEEE int. conf. on robotics and automation (ICRA). IEEE, 2020, pp. 9441-9447.
  30. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2016, pp. 770-778.
  31. G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks," in proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2017, pp. 4700-4708.
  32. G. Li, Y. Wang, Z. Liu, X. Zhang, and D. Zeng, "Rgb-t semantic segmentation with location, activation, and sharpening," IEEE Trans. Circuits and Syst. Video Technol., 2022.
  33. Z. Guo, X. Li, Q. Xu, and Z. Sun, "Robust semantic segmentation based on rgb-thermal in variable lighting scenes," Measurement, vol. 186, p. 110176, 2021.
  34. J. Liu, W. Zhou, Y. Cui, L. Yu, and T. Luo, "Gcnet: Grid-like context- aware network for rgb-thermal semantic segmentation," Neurocomput- ing, vol. 506, pp. 60-67, 2022.
  35. A. Paszke, A. Chaurasia, S. Kim, and E. Culurciello, "Enet: A deep neural network architecture for real-time semantic segmentation," arXiv preprint arXiv:1606.02147, 2016. · arXiv
  36. I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner, "beta-vae: Learning basic visual concepts with a constrained variational framework," in proc. Int. conf. on learning representations (ICLR), 2017.
  37. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," in proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR). Ieee, 2009, pp. 248-255.
  38. L. Wright and N. Demeure, "Ranger21: a synergistic deep learning optimizer," arXiv preprint arXiv:2106.13731, 2021. · arXiv
  39. C. Hazirbas, L. Ma, C. Domokos, and D. Cremers, "Fusenet: Incorporat- ing depth into semantic segmentation via fusion-based cnn architecture," in proc. Asian Conf. on Comput. Vis. (ACCV). Springer, 2017, pp. 213- 228.
  40. W. Wang and U. Neumann, "Depth-aware cnn for rgb-d segmentation," in proc. Eur. Conf. Comput. Vis. (ECCV), 2018, pp. 135-150.
  41. S. Zhao, Y. Liu, Q. Jiao, Q. Zhang, and J. Han, "Mitigating modality discrepancies for rgb-t semantic segmentation," IEEE Trans. Neural Networks Learn. Syst., 2023.

Cited by

Sorted by citations

Metadata

Authors
Lin, B., Lin, Z., Guo, Y., Zhang, Y., Zou, J., & Fan, S.
Affiliations
University of Electronic Science and Technology of China (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2307.08536
arXiv
2307.08536
OpenAlex
W4384648234
Cited by
3
References
0
Citations by year
2025: 1 · 2024: 2
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Basic and Applied Basic Research Foundation of Guangdong Province
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
Keywords
thermal images, multimodal fusion, class imbalance, modality bias, weighted cross-entropy loss

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review