CoChat
Sign Up
PreprintCited by 8Advanced Neural Network ApplicationsDomain Adaptation and Few-Shot LearningRemote-Sensing Image Classification

The DOI matches its DataCite record.

SpiderMesh: Spatial-aware Demand-guided Recursive Meshing for RGB-T Semantic Segmentation

Open access
Fan, Siqi, Wang, Zhe, Wang, Yan, Liu, Jingjing·2023·arXiv
DOI 10.48550/arxiv.2303.08692arXiv 2303.08692
Open this paper in CoChat — free accountOpen PDF

Abstract

For semantic segmentation in urban scene understanding, RGB cameras alone often fail to capture a clear holistic topology in challenging lighting conditions. Thermal signal is an informative additional channel that can bring to light the contour and fine-grained texture of blurred regions in low-quality RGB image. Aiming at practical RGB-T (thermal) segmentation, we systematically propose a Spatial-aware Demand-guided Recursive Meshing (SpiderMesh) framework that: 1) proactively compensates inadequate contextual semantics in optically-impaired regions via a demand-guided target masking algorithm; 2) refines multimodal semantic features with recursive meshing to improve pixel-level semantic analysis performance. We further introduce an asymmetric data augmentation technique M-CutOut, and enable semi-supervised learning to fully utilize RGB-T labels only sparsely available in practical use. Extensive experiments on MFNet and PST900 datasets demonstrate that SpiderMesh achieves state-of-the-art performance on standard RGB-T segmentation benchmarks.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2303.08692

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Existing works on missing modality robustness vary widely in model architectures, fusion methods, training procedures, and missing feature generation methods, making a fair comparison based solely on model size or number of parameters infeasible.

    “However, existing works on missing modality robustness vary widely in terms of model architectures [32, 4], fusion methods [35, 63], training procedures [20, 32], and missing feature generation methods [23]. Due to this heterogeneity, a fair comparison based solely on model size/number of parameters is infeasible.”

    From Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation (2025) · cites this paper as [32]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • Our approach performs significantly better compared to SpiderMesh, VPFNet, and MDRNet models.

    “Our approach performs significantly better compared to partial masking and recursive meshing based SpiderMesh [32], variational pobabilistic fusion based VPFNet [35] and modality discrepancy reduction based MDRNet [67] models.”

    From Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation (2025) · cites this paper as [32]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Still being checked.

Research works printed but not linked: 37

  1. Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku, and T. Harada, "MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes," in IROS, 2017, pp. 5108-5115. 1, 2, 4
  2. Y. Sun, W. Zuo, and M. Liu, "RTFNet: Rgb-thermal fusion network for semantic segmentation of urban scenes," IEEE RAL, vol. 4, no. 3, pp. 2576-2583, 2019. 1, 2, 5, 6
  3. Y. Sun, W. Zuo, P. Yun, H. Wang, and M. Liu, "FuseSeg: Semantic segmentation of urban scenes based on rgb and thermal data fusion," IEEE TASE, vol. 18, no. 3, pp. 1000-1011, 2021. 1, 2, 5, 6
  4. F. Deng, H. Feng, M. Liang, H. Wang, Y. Yang, Y. Gao, J. Chen, J. Hu, X. Guo, and T. L. Lam, "FEANet: Feature-enhanced attention network for rgb-thermal real-time semantic segmentation," in IROS, 2021, pp. 4467-4473. 1, 2, 5
  5. W. Zhou, J. Liu, J. Lei, L. Yu, and J.-N. Hwang, "GMNet: Graded- feature multilabel-learning network for rgb-thermal urban scene se- mantic segmentation," IEEE TIP, vol. 30, pp. 7790-7802, 2021. 1, 2, 5
  6. W. Zhou, X. Lin, J. Lei, L. Yu, and J.-N. Hwang, "MFFENet: Multiscale feature fusion and enhancement network for rgb-thermal urban road scene parsing," IEEE TMM, vol. 24, pp. 2526-2538, 2022. 1, 2, 5
  7. Q. Zhang, S. Zhao, Y. Luo, D. Zhang, N. Huang, and J. Han, "ABMDRNet: Adaptive-weighted bi-directional modality difference reduction network for rgb-t semantic segmentation," in CVPR, 2021, pp. 2633-2642. 1, 2, 5
  8. S. Zhao, Y. Liu, Q. Jiao, Q. Zhang, and J. Han, "Mitigating modality discrepancies for rgb-t semantic segmentation," IEEE TNNLS, 2023. 1, 2
  9. X. He, M. Wang, T. Liu, L. Zhao, and Y. Yue, "Sfaf-ma: Spatial feature aggregation and fusion with modality adaptation for rgb- thermal semantic segmentation," IEEE TIM, vol. 72, pp. 1-10, 2023. 1, 2
  10. G. Li, Y. Wang, Z. Liu, X. Zhang, and D. Zeng, "Rgb-t semantic segmentation with location, activation, and sharpening," IEEE TCSVT, vol. 33, no. 3, pp. 1223-1235, 2022. 1, 2
  11. S. Zhao and Q. Zhang, "A feature divide-and-conquer network for rgb- t semantic segmentation," IEEE TCSVT, vol. 33, no. 6, pp. 2892-2905, 2023. 1, 2
  12. W. Zhou, S. Dong, J. Lei, and L. Yu, "Mtanet: Multitask-aware network with hierarchical multimodal fusion for rgb-t urban scene understanding," IEEE TIV, vol. 8, no. 1, pp. 48-58, 2022. 1, 2
  13. J. Zhang, R. Liu, H. Shi, K. Yang, S. Reiß, K. Peng, H. Fu, K. Wang, and R. Stiefelhagen, "Delivering arbitrary-modal semantic segmentation," in CVPR, June 2023, pp. 1136-1147. 1, 2
  14. M. Vollmer and K.-P. Möllmann, Infrared Thermal Imaging: Funda- mentals, Research and Applications. Wiley-VCH, 2010. 1
  15. J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," in CVPR, 2015, pp. 3431-3440. 2
  16. V. Badrinarayanan, A. Kendall, and R. Cipolla, "SegNet: A deep convolutional encoder-decoder architecture for image segmentation," IEEE TPAMI, vol. 39, pp. 2481-2495, 2017. 2
  17. O. Ronneberger, P. Fischer, and T. Brox, "U-Net: Convolutional networks for biomedical image segmentation," in MICCAI, 2015, pp. 234-241. 2, 5
  18. L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, "Re- thinking atrous convolution for semantic image segmentation," arXiv:1706.05587, 2017. 2, 3 · arXiv
  19. E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, "SegFormer: Simple and efficient design for semantic segmentation with transformers," NIPS, vol. 34, pp. 12 077-12 090, 2021. 2
  20. C. Hazirbas, L. Ma, C. Domokos, and D. Cremers, "Fusenet: In- corporating depth into semantic segmentation via fusion-based cnn architecture," in ACCV, 2017, pp. 213-228. 2
  21. W. Wang and U. Neumann, "Depth-aware cnn for rgb-d segmentation," in ECCV, 2018, pp. 135-150. 2
  22. Y. Cheng, R. Cai, Z. Li, X. Zhao, and K. Huang, "Locality-sensitive deconvolution networks with gated fusion for rgb-d indoor semantic segmentation," in CVPR, 2017, pp. 1475-1483. 2
  23. J. Jiao, Y. Wei, Z. Jie, H. Shi, R. W. H. Lau, and T. S. Huang, "Geometry-aware distillation for indoor semantic segmentation," in CVPR, 2019, pp. 2869-2878. 2
  24. Z. Xiong, Y. Yuan, N. Guo, and Q. Wang, "Variational context- deformable convnets for indoor scene parsing," in CVPR, 2020, pp. 3992-4002. 2
  25. H. Zhou, C. Tian, Z. Zhang, Q. Huo, Y. Xie, and Z. Li, "Multispectral fusion transformer network for rgb-thermal urban scene semantic segmentation," IEEE Geosci. Remote Sens. Lett., vol. 19, pp. 1-5, 2022. 2, 5, 6
  26. Y.-H. Kim, U. Shin, J. Park, and I. S. Kweon, "MS-UDA: Multi- spectral unsupervised domain adaptation for thermal image semantic segmentation," IEEE Robot. Autom., vol. 6, no. 4, pp. 6497-6504, 2021. 2
  27. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in CVPR, 2016, pp. 770-778. 2
  28. S. Woo, J. Park, J.-Y. Lee, and I. S. Kweon, "CBAM: Convolutional block attention module," in ECCV, 2018, pp. 3-19. 2
  29. T. DeVries and G. W. Taylor, "Improved regularization of convolu- tional neural networks with cutout," arXiv:1708.04552, 2017. 4 · arXiv
  30. X. Chen, Y. Yuan, G. Zeng, and J. Wang, "Semi-supervised semantic segmentation with cross pseudo supervision," in CVPR, 2021, pp. 2613-2622. 4
  31. S. Fan, F. Zhu, Z. Feng, Y. Lv, M. Song, and F.-Y. Wang, "Conservative-progressive collaborative learning for semi-supervised semantic segmentation," arXiv preprint arXiv:2211.16701, 2022. 4 · arXiv
  32. Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, "Swin transformer: Hierarchical vision transformer using shifted windows," in CVPR, 2021, pp. 10 012-10 022. 5
  33. C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, and N. Sang, "BiSeNet: Bilateral segmentation network for real-time semantic segmentation," in ECCV, 2018, pp. 325-341. 5
  34. X. Chen, K.-Y. Lin, J. Wang, W. Wu, C. Qian, H. Li, and G. Zeng, "Bi- directional cross-modality feature propagation with separation-and- aggregation gate for rgb-d semantic segmentation," in ECCV, 2020, pp. 561-577. 5
  35. X. Hu, K. Yang, L. Fei, and K. Wang, "ACNet: Attention based network to exploit complementary features for rgbd semantic segmen- tation," in ICIP, 2019, pp. 1440-1444. 5
  36. O. Frigo, L. Martin-Gaffe, and C. Wacongne, "DooDLeNet: Double deeplab enhanced feature fusion for thermal-color semantic segmen- tation," in CVPRW, 2022, pp. 3021-3029. 5
  37. S. S. Shivakumar, N. Rodrigues, A. Zhou, I. D. Miller, V. Kumar, and C. J. Taylor, "PST900: Rgb-thermal calibration, dataset and segmentation network," in ICRA, 2020, pp. 9441-9447. 4, 5

Cited by

Sorted by citations

Related

Metadata

Authors
Fan, S., Wang, Z., Wang, Y., & Liu, J.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2303.08692
arXiv
2303.08692
OpenAlex
W4327673055
Cited by
8
References
0
Citations by year
2025: 4 · 2024: 4
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Remote-Sensing Image Classification
Keywords
infrared thermography, multimodal fusion, semi-supervised learning, urban scene understanding

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review