Investigating Vulnerability to Adversarial Examples on Multimodal Data Fusion in Deep Learning
Open accessAbstract
The success of multimodal data fusion in deep learning appears to be attributed to the use of complementary in-formation between multiple input data. Compared to their predictive performance, relatively less attention has been devoted to the robustness of multimodal fusion models. In this paper, we investigated whether the current multimodal fusion model utilizes the complementary intelligence to defend against adversarial attacks. We applied gradient based white-box attacks such as FGSM and PGD on MFNet, which is a major multispectral (RGB, Thermal) fusion deep learning model for semantic segmentation. We verified that the multimodal fusion model optimized for better prediction is still vulnerable to adversarial attack, even if only one of the sensors is attacked. Thus, it is hard to say that existing multimodal data fusion models are fully utilizing complementary relationships between multiple modalities in terms of adversarial robustness. We believe that our observations open a new horizon for adversarial attack research on multimodal data fusion.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multimodal models may be equally or even more vulnerable to adversarial attacks than models using a single modality, due to negative impact of increasing input dimensions on adversarial robustness.
“The robustness of multimodal models is a topic of ongoing discussion because multimodal models may be equally or even more vulnerable to adversarial attacks than models using a single modality. This susceptibility to adversarial attacks results from the negative impact of increasing input dimensions on adversarial robustness49–51.”
From Orchestrating explainable artificial intelligence for multimodal and longitudinal data in medical imaging (2024) · cites this paper as [49]
The Introduction (p. 1) says the multimodal model in the authors’ experiment was as vulnerable to adversarial attacks as a single-modal model. It also reports that prior studies found increasing input dimensions harms adversarial robustness, but it does not establish that increased dimensions caused the vulnerability observed in its multimodal model. (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2644615 · Cited by 19,631 · see citing works
Explaining and Harnessing Adversarial Examples
Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014) · arXiv · doi:10.48550/arxiv.1412.6572 · Cited by 10,381 · see citing works
Towards Deep Learning Models Resistant to Adversarial Attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017) · arXiv · doi:10.48550/arxiv.1706.06083 · Cited by 6,602 · see citing works
Deep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139 · cited 1× · see citing works
“Many studies in computer vision took account of multimodal data fusion when researchers want to enhance the performance such as for medical image segmentation [4, 9], semantic segmentation [8, 15], and person re-identification [18, 14].”
FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-Based CNN Architecture
Hazirbas, C., Ma, L., Domokos, C., & Cremers, D. (2017) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-54181-5_14 · Cited by 902 · see citing works
A survey on deep learning for multimodal data fusion
Gao, J., Li, P., Chen, Z., & Zhang, J. (2020) · Neural Computation · doi:10.1162/neco_a_01273 · Cited by 821 · see citing works
MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes
Ha, Q., Watanabe, K., Karasawa, T., Ushiku, Y., & Harada, T. (2017) · 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2017.8206396 · Cited by 693 · see citing works
Rtfnet: Rgb-thermal fusion network for semantic segmentation of urban scenes
Sun, Y., Zuo, W., & Liu, M. (2019) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2019.2904733 · Cited by 546 · see citing works
Deep Learning-Based Image Segmentation on Multimodal Medical Imaging
Guo, Z., Li, X., Huang, H., Guo, N., & Li, Q. (2019) · IEEE Transactions on Radiation and Plasma Medical Sciences · doi:10.1109/trpms.2018.2890359 · Cited by 445 · see citing works
An overview of multi-modal medical image fusion
Du, J., Li, W., Lü, K., & Xiao, B. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.07.160 · Cited by 366 · see citing works
On the Robustness of Semantic Segmentation Models to Adversarial Attacks
Arnab, A., Mikšík, O., & Torr, P. H. S. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00099 · Cited by 323 · see citing works
Adversarial Examples Are a Natural Consequence of Test Error in NoisePreprint
Ford, N., Gilmer, J., Carlini, N., & Cubuk, E. D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.10513 · Cited by 197 · see citing works
Deep Multi-View Feature Learning for Person Re-Identification
Tao, D., Guo, Y., Yu, B., Pang, J., & Yu, Z. (2017) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2726580 · Cited by 109 · see citing works
Multimodal Person Reidentification Using RGB-D Cameras
Pala, F., Satta, R., Fumera, G., et al. (2015) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2015.2424056 · Cited by 92 · see citing works
First-order Adversarial Vulnerability of Neural Networks and Input DimensionPreprint
Simon-Gabriel, C.-J., Ollivier, Y., Bottou, L., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.01421 · Cited by 60 · see citing works
Semantic segmentation of RGBD images based on deep depth regression
Guo, Y., & Chen, T. (2017) · Pattern Recognition Letters · doi:10.1016/j.patrec.2017.08.026 · Cited by 35 · see citing works
Adversarial Vulnerability of Neural Networks Increases with Input DimensionPreprint
Simon-Gabriel, C.-J., Ollivier, Y., Schölkopf, B., et al. (2018) · arXiv (Cornell University) · Cited by 30 · see citing works
Deep Learning for Multimodal Data Fusion
Kanezaki, A., Kuga, R., Sugano, Y., & Matsushita, Y. (2019) · Elsevier eBooks · doi:10.1016/b978-0-12-817358-9.00008-1 · Cited by 15 · see citing works
DAPAS : Denoising Autoencoder to Prevent Adversarial attack in Semantic SegmentationPreprint
Cho, S. J., Jun, T. J., Oh, B., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.05195 · Cited by 5 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Pahud de Mortanges, A., Luo, H., Shu, S. Z., Kamath, A., Suter, Y., Shelan, M., Pöllinger, A., & Reyes, M. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01190-w · Cited by 93 · see citing works
“This susceptibility to adversarial 49 – 51 .”
- Authors
- Yu, Y., Lee, H. J., Kim, B. C., Kim, J. U., & Ro, Y. M.
- Affiliations
- Korea Advanced Institute of Science and Technology (KR)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2005.10987
- OpenAlex
- W3027535223
- Cited by
- 15
- References
- 19
- Citations by year
- 2025: 1 · 2024: 4 · 2023: 3 · 2022: 3 · 2021: 3 · 2020: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Digital Media Forensic Detection
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Keywords
- adversarial robustness, multimodal fusion, adversarial examples, FGSM, preimplantation genetic diagnosis, white-box attacks, semantic segmentation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
