Multimodal Fusion with Co-attention Mechanism
Abstract
Because the information from different modalities will complement each other when describing the same contents, multimodal information can be used to obtain better feature representations. Thus, how to represent and fuse the relevant information has become a current research topic. At present, most of the existing feature fusion methods consider the different levels of features representations, but they ignore the significant relevance between the local regions, especially in the high-level semantic representation. In this paper, a general multimodal fusion method based on the co-attention mechanism is proposed, which is similar to the transformer structure. We discuss two main issues: (1) Improving the applicability and generality of the transformer to different modal data; (2) By capturing and transmitting the relevant information between local features before fusion, the proposed method can allow for more robustness. We evaluate our model on the multimodal classification task, and the experiments demonstrate that our model can learn fused featnre representation effectively.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Post fusion combines decision-level independent processing of modalities.
“Post fusion is a key method in multimodal analysis, which combines the results of decision level indepen-dent processing of modalities [45].”
From Multimodal fusion and vision–language models: A survey for robot vision (2026) · cites this paper as [45]
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648 · see citing works
Very Deep Convolutional Networks for Large-Scale Image Recognition
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1409.1556 · Cited by 74,248 · see citing works
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · see citing works
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018 · see citing works
Residual Attention Network for Image Classification
Wang, F., Jiang, M., Qian, C., Yang, S., Li, C., Zhang, H., Wang, X., & Tang, X. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.683 · Cited by 4,073 · see citing works
Attention-Based Models for Speech Recognition
Chorowski, J., Bahdanau, D., Serdyuk, D., Cho, K., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1506.07503 · Cited by 2,021 · see citing works
The MIR flickr retrieval evaluation
Huiskes, M. J., & Lew, M. S. (2008) · doi:10.1145/1460096.1460104 · Cited by 1,866 · see citing works
Hierarchical Question-Image Co-Attention for Visual Question AnsweringPreprint
Lu, J., Yang, J., Batra, D., & Parikh, D. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.00061 · Cited by 1,303 · see citing works
Multimodal fusion for multimedia analysis: a survey
Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010) · Multimedia Systems · doi:10.1007/s00530-010-0182-0 · Cited by 1,279 · see citing works
Multimodal Learning with Deep Boltzmann Machines
Srivastava, N., & Salakhutdinov, R. (2012) · doi:10.5555/2627435.2697059 · Cited by 1,070 · see citing works
Hierarchical Question-Image Co-Attention for Visual Question Answering
Lu, J. (2024) · TIB Data Manager · doi:10.57702/uhdmfu1r · Cited by 357 · see citing works
Improved Multimodal Deep Learning with Variation of Information
Sohn, K., Shang, W., & Lee, H. (2014) · neural information processing systems · Cited by 142 · see citing works
A Deep Multi-Modal CNN for Multi-Instance Multi-Label Image Classification
Song, L., Liu, J., Qian, B., Sun, M., Yang, K., Sun, M., & Abbas, S. (2018) · IEEE Transactions on Image Processing · doi:10.1109/tip.2018.2864920 · Cited by 116 · see citing works
Dense Multimodal Fusion for Hierarchically Joint Representation
Hu, D., Wang, C., Nie, F., & Li, X. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683898 · Cited by 48 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
Rahate, A., Walambe, R., Ramanna, S., & Kotecha, K. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.12.003 · Cited by 214 · see citing works
A survey on machine learning based analysis of heterogeneous data in industrial automation
Kamm, S., Veekati, S. S., Müller, T., Jazdi, N., & Weyrich, M. (2023) · Computers in Industry · doi:10.1016/j.compind.2023.103930 · Cited by 78 · see citing works
Multimodal fusion and vision–language models: A survey for robot vision
Han, X., Chen, S., Fu, Z., Feng, Z., Fan, L., An, D., Wang, C., Guo, L., Meng, W., Zhang, X., et al. (2026) · Information Fusion · doi:10.1016/j.inffus.2025.103652 · Cited by 41 · see citing works
“A common attention mechanism similar to Figure 3: The above diagram intuitively illustrates the basic processes of three main strategies for multimodal fusion: early fusion, mid fusion, and late fusion.”
DENet: Dual-Path Edge Network With Global-Local Attention for Infrared Small Target Detection
Zuo, J., Pei, S., Li, Q., Huang, Y., & Wang, S. (2026) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2026.3658092 · Cited by 9 · see citing works
Multimodal Fusion for Talking Face Generation Utilizing Speech-Related Facial Action Units
Liu, Z., Liu, X., Chen, S., Liu, J., Wang, L., & Bi, C. (2024) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3672565 · Cited by 7 · see citing works
Zhang, L., Liu, C., & Jia, N. (2023) · Applied Sciences · doi:10.3390/app13179910 · Cited by 6 · see citing works
Azadmaleki, H., Haghbin, Y., Rashidi, S., Nezhad, M. J. M., Zolnour, A., & Zolnoori, M. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-02026-x · Cited by 5 · see citing works
An Architecture for Adaptive Machine Learning Models using Adversarial and Transfer Learning
Kamm, S., Kumar, P., V, Jazdi, N., & Weyrich, M. (2023) · Procedia CIRP · doi:10.1016/j.procir.2023.09.192 · Cited by 2 · see citing works
Zhang, X., Shang, J., Zhao, Y., Qin, B., Ren, Q., Feng, L., & Liu, J. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130362 · Cited by 1 · see citing works
Talking Head Generation for Media Interaction System with Feature Disentanglement
Zhang, L., Chen, Q., & Liu, Z. (2023) · IEEE International Conference on Parallel and Distributed Systems (ICPADS) · doi:10.1109/icpads56603.2022.00059
Analysis of Airframe Parameters of OAT-bicopter to Optimize Pitch Stability
Xu, J., Yu, C., Li, H., Zhao, C., Li, Q., & Li, X. (2023) · International Conference on Robotics and Control Engineering (RobCE) · doi:10.1145/3598151.3598434
Self-citationKamm, S., Suthandhira, P. R., Jazdi, N., & Weyrich, M. (2023) · IEEE International Conference on Emerging Technologies and Factory Automation (ETFA) · doi:10.1109/etfa54631.2023.10275533
Co-Attention Based Multi-Channel TF-GridNet for Speech Separation with Ad-Hoc Microphone Arrays
Guo, H., Feng, L., Chen, Y., Li, X., Zhu, B., Wang, H., Zhang, X.-L., & Li, X. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888193
Self-citationEAM-DG: emotion-aware memory network with dual-gating mechanism for dialogue emotion recognition
Wu, T., Li, Y., Zheng, Y., Wang, K., Ma, Y., Li, D., & Li, S. (2026) · Multimedia Systems · doi:10.1007/s00530-026-02688-x
- Authors
- Li, P., & Li, X.
- Affiliations
- Southeast University (BD)
- Venue
- 2020 IEEE 23rd International Conference on Information Fusion (FUSION)
- Publisher
- IEEE
- Published
- 2020-07
- Pages
- 1-8
- OpenAlex
- W3086578443
- Cited by
- 14
- References
- 16
- Citations by year
- 2026: 2 · 2025: 4 · 2024: 1 · 2023: 6 · 2021: 1
- FWCI
- 0.54
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- International Conference on Information Fusion (FUSION) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Image and Video Retrieval Techniques
- Keywords
- multimodal fusion, co-attention, transformer models, feature fusion, multimodal classification, cross-modal representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
