MFAS: Multimodal Fusion Architecture Search
Open accessAbstract
We tackle the problem of finding good architectures for multimodal classification problems. We propose a novel and generic search space that spans a large number of possible fusion architectures. In order to find an optimal architecture for a given dataset in the proposed search space, we leverage an efficient sequential model-based exploration approach that is tailored for the problem. We demonstrate the value of posing multimodal fusion as a neural architecture search problem by extensive experimentation on a toy dataset and two other real multimodal datasets. We discover fusion architectures that exhibit state-of-the-art performance for problems with different domain and dataset size, including the \ntu~dataset, the largest multimodal action recognition dataset available.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Several approaches define a set of atomic neural operations and use architecture search to automatically learn the best order of these operations for a given multimodal task.
“Finally, instead of hand-designing multimodal architectures, several approaches define a set of atomic neural operations (e.g., linear transformation, activation, attention, etc.) and use architecture search to automatically learn the best order of these operations for a given multimodal task [122, 173].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [122]
Section 3.1 (pp. 3–4) describes an architecture search for multimodal tasks that selects features, nonlinearities, and the number of fusion layers. It does not search the order of a general set of atomic operations: feature concatenation is fixed, and attention is not among the operations described. (LLM, conf=0.87)
Experimental — accuracy not yet measuredThe current SOTA on AV-MNIST is based on architecture search: MFAS.
“The current SOTA on AV-MNIST is based on architecture search: MFAS [122].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [122]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566 · see citing works
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260 · cited 1× · see citing works
“Deep neural networks have demonstrated to be effective models for solving a large variety of problems in several domains, including image [22] and video [5] classification, speech recognition [15], and machine translation [44], to name a few.”
BackgroundTwo-Stream Convolutional Networks for Action Recognition in Videos
Simonyan, K., & Zisserman, A. (2014) · arXiv · doi:10.48550/arxiv.1406.2199 · Cited by 6,756 · see citing works
Learning Transferable Architectures for Scalable Image Recognition
Zoph, B., Vasudevan, V., Shlens, J., & Le, Q. V. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00907 · Cited by 6,631 · see citing works
Stacked Hourglass Networks for Human Pose Estimation
Newell, A., Yang, K., & Deng, J. (2016) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-46484-8_29 · Cited by 5,634 · see citing works
Adaptive Mixtures of Local Experts
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., & Hinton, G. E. (1991) · Neural Computation · doi:10.1162/neco.1991.3.1.79 · Cited by 5,384 · see citing works
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · see citing works
Neural Architecture Search with Reinforcement LearningPreprint
Zoph, B., & Le, Q. V. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01578 · Cited by 5,091 · see citing works
(1989) · neural information processing systems · Cited by 3,736 · see citing works
NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis
Shahroudy, A., Liu, J., Ng, T.-T., & Wang, G. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.115 · Cited by 3,148 · see citing works
(2011) · Cited by 2,752 · see citing works
Sequential Model-Based Optimization for General Algorithm Configuration
Hutter, F., Hoos, H. H., & Leyton-Brown, K. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25566-3_40 · Cited by 2,469 · see citing works
ESC: Dataset for Environmental Sound Classification
Piczak, K. J. (2015) · ACM Multimedia · doi:10.1145/2733373.2806390 · Cited by 2,195 · see citing works
Progressive Neural Architecture Search
Liu, C., Zoph, B., Neumann, M., Shlens, J., Wei, H., Li, L.-J., Fei-Fei, L., Yuille, A., Huang, J., & Murphy, K. P. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01246-5_2 · Cited by 1,946 · see citing works
Deep Canonical Correlation Analysis
Andrew, G., Arora, R., Bilmes, J., & Livescu, K. (2013) · Cited by 1,603 · see citing works
(2018) · International Conference on Machine Learning · Cited by 1,465 · see citing works
Multiple kernel learning, conic duality, and the SMO algorithm
Bach, F. R., Lanckriet, G. R. G., & Jordan, M. I. (2004) · Twenty-first international conference on Machine learning - ICML '04 · doi:10.1145/1015330.1015424 · Cited by 1,453 · see citing works
Multimodal fusion for multimedia analysis: a survey
Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010) · Multimedia Systems · doi:10.1007/s00530-010-0182-0 · Cited by 1,279 · see citing works
Efficient Neural Architecture Search via Parameter SharingPreprint
Pham, H., Guan, M. Y., Zoph, B., Le, Q. V., & Dean, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.03268 · Cited by 1,082 · see citing works
An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data
Song, S., Lan, C., Xing, J., Zeng, W., & Liu, J. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11212 · Cited by 1,011 · see citing works
Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., & Bengio, Y. (2013) · arXiv · doi:10.48550/arxiv.1302.4389 · Cited by 935 · see citing works
Early versus late fusion in semantic video analysis
Snoek, C. G. M., Worring, M., & Smeulders, A. W. M. (2005) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/1101149.1101236 · Cited by 899 · see citing works
Sequential Deep Learning for Human Action Recognition
Baccouche, M., Mamalet, F., Wolf, C., García, C., & Baskurt, A. M. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-25446-8_4 · Cited by 793 · see citing works
Self-citationMUTAN: Multimodal Tucker Fusion for Visual Question Answering
Ben-younes, H., Cadène, R., Cord, M., & Thome, N. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.285 · Cited by 733 · see citing works
Li, C., Zhong, Q., Xie, D., & Pu, S. (2018) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/109 · Cited by 598 · see citing works
2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning
Luvizon, D. C., Picard, D., & Tabia, H. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00539 · Cited by 560 · see citing works
SMASH: One-Shot Model Architecture Search through HyperNetworksPreprint
Brock, A., Lim, T., Ritchie, J. M., & Weston, N. J. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.05344 · Cited by 500 · see citing works
ModDrop: Adaptive Multi-Modal Gesture Recognition
Neverova, N., Wolf, C., Taylor, G., & Nebout, F. (2016) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2015.2461544 · Cited by 420 · see citing works
Attention-Based Multimodal Fusion for Video Description
Hori, C., Hori, T., Lee, T.-Y., Zhang, Z., Harsham, B., Hershey, J. R., Marks, T. K., & Sumi, K. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.450 · Cited by 401 · see citing works
Jiang, Y., Wu, Z., Wang, J., Xue, X., & Chang, S. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2670560 · Cited by 401 · see citing works
Gated Multimodal Units for Information FusionPreprint
Arévalo, J., Solorio, T., Montes-y-Gómez, M., & Gonzalez, F. A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1702.01992 · Cited by 302 · see citing works
Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos
Shahroudy, A., Ng, T.-T., Gong, Y., & Wang, G. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2691321 · Cited by 266 · see citing works
Multi-scale Deep Learning for Gesture Detection and Localization
Neverova, N., Wolf, C., Taylor, G. W., & Nebout, F. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-16178-5_33 · Cited by 239 · see citing works
Optimal multimodal fusion for multimedia data analysis
Wu, Y., Chang, E. Y., Chang, K. C.-C., & Smith, J. R. (2004) · Proceedings of the 12th annual ACM international conference on Multimedia · doi:10.1145/1027527.1027665 · Cited by 229 · see citing works
Robust late fusion with rank minimization
Ye, G., Liu, D., Jhuo, I., & Chang, S.-F. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248032 · Cited by 193 · see citing works
Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points
Baradel, F., Wolf, C., Mille, J., & Taylor, G. W. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00056 · Cited by 184 · see citing works
Multimodal feature fusion for robust event detection in web videos
Natarajan, P., Wu, S., Vitaladevuni, S. N. P., Zhuang, X., Tsakalidis, S., Park, U., Prasad, R., & Natarajan, P. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6247814 · Cited by 180 · see citing works
CentralNet: A Multilayer Approach for Multimodal Fusion
Vielzeuf, V., Lechervy, A., Pateux, S., & Jurie, F. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-11024-6_44 · Cited by 166 · see citing works
Self-citationMultimodal Keyless Attention Fusion for Video Classification
Long, X., Gan, C., de Melo, G., Liu, X., Li, Y., Li, F., & Wen, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12319 · Cited by 125 · see citing works
Multilayer and Multimodal Fusion of Deep Neural Networks for Video Classification
Yang, X., Molchanov, P. A., & Kautz, J. (2016) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2964284.2964297 · Cited by 115 · see citing works
Finding Beans in Burgers: Deep Semantic-Visual Embedding with Localization
Engilberge, M., Chevallier, L., Pérez, P., & Cord, M. (2018) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2018.00419 · Cited by 108 · see citing works
Deep Bilinear Learning for RGB-D Action Recognition
Hu, J., Zheng, W., Pan, J., Lai, J., & Zhang, J. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-01234-2_21 · Cited by 104 · see citing works
Feature fusion of side face and gait for video-based human identification
Zhou, X., & Bhanu, B. (2007) · Pattern Recognition · doi:10.1016/j.patcog.2007.06.019 · Cited by 94 · see citing works
On the relationship between visual attributes and convolutional networks
Escorcia, V. A., Niebles, J. C., & Ghanem, B. (2015) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2015.7298730 · Cited by 82 · see citing works
Deep Multimodal Fusion: A Hybrid Approach
Amer, M. R., Shields, T. J., Siddiquie, B., Tamrakar, A., Divakaran, A., & Chai, S. M. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-017-0997-7 · Cited by 42 · see citing works
Learning Joint Multimodal Representation Based on Multi-fusion Deep Neural Networks
Gu, Z., Lang, B., Yue, T., & Huang, L. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-70096-0_29 · Cited by 19 · see citing works
Efficient Progressive Neural Architecture SearchPreprint
Pérez-Rúa, J.-M., Baccouche, M., & Pateux, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.00391 · Cited by 13 · see citing works
Self-citationModout: Learning to Fuse Modalities via Stochastic Regularization
Li, F., Neverova, N., Wolf, C., & Taylor, G. W. (2016) · Journal of Computational Vision and Imaging Systems · doi:10.15353/vsnl.v2i1.103 · Cited by 11 · see citing works
CentralNet: a Multilayer Approach for Multimodal Fusion
Vielzeuf, V., Lechervy, A., Pateux, S., & Jurie, F. (2018) · HAL · doi:10.48550/arxiv.1808.07275 · Cited by 3 · see citing works
Self-citation
Flagged first, then by citations
1 of the 52 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- [31][31] N. Neverova, C. Wolf, G. W. Taylor, and F. Nebout. Multiscale deep learning for gesture detection and localization. In ECCV Workshop , pages 474-490. Springer, 2014. -
Cited by
Sorted by citations
Trusted Multi-View Classification With Dynamic Evidential Fusion
Han, Z., Zhang, C., Fu, H., & Zhou, J. T. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3171983 · Cited by 661 · see citing works
Multimodal intelligence: Representation learning, information fusion, and applications
Zhang, C., Yang, Z., He, X., & Deng, L. (2020) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2020.2987728 · Cited by 511 · see citing works
MMTM: Multimodal Transfer Module for CNN Fusion
Joze, H. R. V., Shaban, A., Iuzzolino, M. L., & Koishida, K. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01330 · Cited by 323 · see citing works
Li, Y., El Habib Daho, M., Conze, P.-H., Zeghlache, R., Le Boité, H., Tadayoni, R., Cochener, B., Lamard, M., & Quellec, G. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108635 · Cited by 274 · see citing works
Ouhami, M., Hafiane, A., Es-Saady, Y., Hajji, M. E., & Canals, R. (2021) · Remote Sensing · doi:10.3390/rs13132486 · Cited by 267 · see citing works
Neural Feature Search for RGB-Infrared Person Re-Identification
Chen, Y., Wan, L., Li, Z., Jing, Q., & Sun, Z. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00065 · Cited by 267 · see citing works
Reconsidering Representation Alignment for Multi-view Clustering
Trosten, D. J., Løkse, S., Jenssen, R., & Kampffmeyer, M. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00131 · Cited by 252 · see citing works
Boulahia, S. Y., Amamra, A., Madi, M. R., & Daikh, S. (2021) · Machine Vision and Applications · doi:10.1007/s00138-021-01249-8 · Cited by 236 · see citing works
Rahate, A., Walambe, R., Ramanna, S., & Kotecha, K. (2022) · Information Fusion · doi:10.1016/j.inffus.2021.12.003 · Cited by 214 · see citing works
Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion
Sun, P., Zhang, W., Wang, H., Li, S., & Li, X. (2021) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr46437.2021.00146 · Cited by 208 · see citing works
A Review on Explainability in Multimodal Deep Neural Nets
Joshi, G., Walambe, R., & Kotecha, K. (2021) · IEEE Access · doi:10.1109/access.2021.3070212 · Cited by 204 · see citing works
Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions
Liang, P. P., Zadeh, A., & Morency, L. (2024) · ACM Computing Surveys · doi:10.1145/3656580 · Cited by 306 · see citing works
Are Multimodal Transformers Robust to Missing Modality?
Ma, M., Ren, J., Zhao, L., Testuggine, D., & Peng, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01764 · Cited by 198 · see citing works
RFNet: Region-aware Fusion Network for Incomplete Multi-modal Brain Tumor Segmentation
Ding, Y., Yu, X., & Yang, Y. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00394 · Cited by 190 · see citing works
Shaban, A., & Yousefi, S. (2024) · Springer optimization and its applications · doi:10.1007/978-3-031-53092-0_10 · Cited by 206 · see citing works
Supervised Multimodal Bitransformers for Classifying Images and TextPreprint
Kiela, D., Bhooshan, S., Firooz, H., Perez, E., & Testuggine, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.02950 · Cited by 271 · see citing works
Searching Multi-Rate and Multi-Modal Temporal Enhanced Networks for Gesture Recognition
Yu, Z., Zhou, B., Wan, J., Wang, P., Chen, H., Liu, X., Li, S. Z., & Zhao, G. (2021) · IEEE Transactions on Image Processing · doi:10.1109/tip.2021.3087348 · Cited by 135 · see citing works
Xue, Z., & Marculescu, R. (2023) · IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw59228.2023.00256 · Cited by 134 · see citing works
Graph-Based Multimodal Sequential Embedding for Sign Language Translation
Tang, S., Guo, D., Hong, R., & Wang, M. (2021) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2021.3117124 · Cited by 110 · see citing works
Multimodal research in vision and language: A review of current and emerging trends
Uppal, S., Bhagat, S., Hazarika, D., Majumder, N., Poria, S., Zimmermann, R., & Zadeh, A. (2021) · Information Fusion · doi:10.1016/j.inffus.2021.07.009 · Cited by 112 · see citing works
Deep Multimodal Neural Architecture Search
Zhou, Y., Cui, Y., Yu, J., Wang, M., Tao, D., & Tian, Q. C. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413977 · Cited by 103 · see citing works
Islam, M. M., & Iqbal, T. (2021) · IEEE Robotics and Automation Letters · doi:10.1109/lra.2021.3059624 · Cited by 93 · see citing works
VPN++: Rethinking Video-Pose Embeddings for Understanding Activities of Daily Living
Das, S., Dai, R., Yang, D., & Brémond, F. (2021) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3127885 · Cited by 92 · see citing works
HAMLET: A Hierarchical Multimodal Attention-based Human Activity Recognition Algorithm
Islam, M., & Iqbal, T. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9340987 · Cited by 92 · see citing works
Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
Xie, L., Chen, X., Bi, K., Wei, L., Xu, Y., Wang, L., Chen, Z., Xiao, A., Chang, J., Zhang, X., et al. (2021) · ACM Computing Surveys · doi:10.1145/3473330 · Cited by 108 · see citing works
Toward a Collective Agenda on AI for Earth Science Data Analysis
Tuia, D., Roscher, R., Wegner, J. D., Jacobs, N., Zhu, X. X., & Camps‐Valls, G. (2021) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2020.3043504 · Cited by 89 · see citing works
Deep Learning on Multimodal Sensor Data at the Wireless Edge for Vehicular Network
Salehi, B., Reus-Muns, G., Roy, D., Wang, Z., Jian, T., Dy, J., Ioannidis, S., & Chowdhury, K. R. (2022) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2022.3170733 · Cited by 83 · see citing works
MTL-NAS: Task-Agnostic Neural Architecture Search Towards General-Purpose Multi-Task Learning
Gao, Y., Bai, H., Jie, Z., Ma, J., Jia, K., & Liu, W. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.01156 · Cited by 83 · see citing works
Multi-Modal Machine Learning in Engineering Design: A Review and Future Directions
Song, B., Zhou, R., & Ahmed, F. (2023) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4063954 · Cited by 83 · see citing works
Multi-Modality Sensing and Data Fusion for Multi-Vehicle Detection
Roy, D., Li, Y., Jian, T., Tian, P., Chowdhury, K. R., & Ioannidis, S. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3145663 · Cited by 78 · see citing works
An effective multimodal representation and fusion method for multimodal intent recognition
Huang, X., Ma, T., Jia, L., Zhang, Y., Rong, H., & Al-Nabhan, N. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126373 · Cited by 72 · see citing works
Gimme Signals: Discriminative signal encoding for multimodal activity recognition
Memmesheimer, R., Theisen, N., & Paulus, D. W. R. (2020) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341699 · Cited by 69 · see citing works
Fusion-Based Correlation Learning Model for Cross-Modal Remote Sensing Image Retrieval
Lv, Y., Xiong, W., Zhang, X., & Cui, Y. (2021) · IEEE Geoscience and Remote Sensing Letters · doi:10.1109/lgrs.2021.3131592 · Cited by 62 · see citing works
AdaMML: Adaptive Multi-Modal Learning for Efficient Video Recognition
Panda, R., Chen, C.-F. R., Fan, Q., Sun, X., Saenko, K., Oliva, A., & Feris, R. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00748 · Cited by 59 · see citing works
Joint Discriminative Sparse Coding for Robust Hand-Based Multimodal Recognition
Li, S., & Zhang, B. (2021) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2021.3074315 · Cited by 60 · see citing works
Excavating multimodal correlation for representation learning
Mai, S., Ya, S., Zeng, Y., & Hu, H. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.003 · Cited by 57 · see citing works
MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records
Xu, Z., So, D. R., & Dai, A. M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17260 · Cited by 52 · see citing works
Graph Fusion Network-Based Multimodal Learning for Freezing of Gait Detection
Hu, K., Wang, Z., Martens, K. A. E., Hagenbuchner, M., Bennamoun, M., Tsoi, A. C., & Lewis, S. J. G. (2021) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3105602 · Cited by 56 · see citing works
Fusion-GCN: Multimodal Action Recognition Using Graph Convolutional Networks
Duhme, M., Memmesheimer, R., & Paulus, D. W. R. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-92659-5_17 · Cited by 46 · see citing works
Skeleton Sequence and RGB Frame Based Multi-Modality Feature Fusion Network for Action Recognition
Zhu, X., Zhu, Y., Wang, H., Wen, H.-L., Yan, Y., & Liu, P.-L. (2022) · ACM Trans. Multim. Comput. Commun. Appl. · doi:10.1145/3491228 · Cited by 49 · see citing works
MSAF: Multimodal Split Attention FusionPreprint
Su, L., Hu, C., Li, G., & Cao, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.07175 · Cited by 65 · see citing works
MuMu: Cooperative Multitask Learning-Based Guided Multimodal Fusion
Islam, M. M., & Iqbal, T. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i1.19988 · Cited by 42 · see citing works
Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend
Chai, W., & Wang, G. (2022) · Applied Sciences · doi:10.3390/app12136588 · Cited by 43 · see citing works
Incorporating stock prices and text for stock movement prediction based on information fusion
Zhang, Q., Zhang, Y., Bao, F., Liu, Y., Zhang, C., & Лю, П. (2023) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2023.107377 · Cited by 43 · see citing works
Multimodal Attention Network for Continuous-Time Emotion Recognition Using Video and EEG Signals
Choi, D. Y., Kim, D.-H., & Song, B. C. (2020) · IEEE Access · doi:10.1109/access.2020.3036877 · Cited by 39 · see citing works
Dense Modality Interaction Network for Audio-Visual Event Localization
Liu, S., Quan, W., Wang, C., Liu, Y., Liu, B., & Dong-ming Yan, A. (2022) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2022.3150469 · Cited by 37 · see citing works
“Subverting the Jewtocracy”: Online Antisemitism Detection Using Multimodal Deep Learning
Chandra, M., Pailla, D. R., Bhatia, H., Sanchawala, A., Gupta, M., Shrivastava, M., & Kumaraguru, P. (2021) · ACM Web Science Conference (WebSci) · doi:10.1145/3447535.3462502 · Cited by 37 · see citing works
Attention-Enhanced Multimodal Learning for Conceptual Design Evaluations
Song, B., Miller, S. R., & Ahmed, F. (2023) · Journal of Mechanical Design · doi:10.1115/1.4056669 · Cited by 37 · see citing works
Common Practices and Taxonomy in Deep Multiview Fusion for Remote Sensing Applications
Mena, F., Arenas, D., Nuske, M., & Dengel, A. R. (2024) · IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing · doi:10.1109/jstars.2024.3361556 · Cited by 35 · see citing works
Regional Attention with Architecture-Rebuilt 3D Network for RGB-D Gesture Recognition
Zhou, B., Li, Y., & Wan, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i4.16471 · Cited by 34 · see citing works
BM-NAS: Bilevel Multimodal Neural Architecture Search
Yin, Y., Huang, S., & Zhang, X. D. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i8.20872 · Cited by 33 · see citing works
Poster-Based Multiple Movie Genre Classification Using Inter-Channel Features
Wi, J., Jang, S., & Kim, Y. (2020) · IEEE Access · doi:10.1109/access.2020.2986055 · Cited by 32 · see citing works
2D and 3D Palmprint and Palm Vein Recognition Based on Neural Architecture Search
Jia, W., Xia, W., Zhao, Y., Min, H., & Chen, Y. (2021) · Machine Intelligence Research · doi:10.1007/s11633-021-1292-1 · Cited by 31 · see citing works
Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition
Liu, J., Chen, C., & Liu, M. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681015 · Cited by 30 · see citing works
Multimodal Boosting: Addressing Noisy Modalities and Identifying Modality Contribution
Mai, S., Ya, S., Xiong, A., Zeng, Y., & Hu, H. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3306489 · Cited by 29 · see citing works
Learn to Search a Lightweight Architecture for Target-Aware Infrared and Visible Image Fusion
Liu, J., Wu, Y., Wu, G., Liu, R., & Fan, X. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3180672 · Cited by 28 · see citing works
Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems
Hao, C., & Chen, D. (2021) · IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS) · doi:10.1109/aicas51828.2021.9458577 · Cited by 27 · see citing works
Hierarchical Action Classification with Network Pruning
Davoodikakhki, M., & Yin, K. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-64556-4_23 · Cited by 26 · see citing works
Deep multimodal learning for municipal solid waste sorting
Lu, G., Wang, Y., Xu, H., Yang, H., & Zou, J. T. (2021) · Science China Technological Sciences · doi:10.1007/s11431-021-1927-9 · Cited by 25 · see citing works
Multi-View And Multi-Modal Event Detection Utilizing Transformer-Based Multi-Sensor Fusion
Yasuda, M., Ohishi, Y., Saito, S., & Harado, N. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9746006 · Cited by 25 · see citing works
Deep learning on multi-view sequential data: a survey
Xie, Z., Yang, Y. Z., Zhang, Y., Wang, J., & Du, S. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10332-z · Cited by 28 · see citing works
K-armed Bandit based Multi-Modal Network Architecture Search for Visual Question Answering
Zhou, Y., Ji, R., Sun, X., Luo, G., Hong, X., Su, J., Ding, X., & Shao, L. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413998 · Cited by 24 · see citing works
Multi-modality fusion learning for the automatic diagnosis of optic neuropathy
Cao, Z., Sun, C., Wang, W., Zheng, X., Wu, J., & Gao, H. (2020) · Pattern Recognition Letters · doi:10.1016/j.patrec.2020.12.009 · Cited by 25 · see citing works
Zhang, Q., Zhang, Y., Yao, X., Li, S., Zhang, C., & Лю, П. (2023) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3611311 · Cited by 26 · see citing works
Heidarivincheh, F., McConville, R., Morgan, C., McNaney, R., Masullo, A., Mirmehdi, M., Whone, A. L., & Craddock, I. J. (2021) · Sensors · doi:10.3390/s21124133 · Cited by 23 · see citing works
MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
Liang, P. P., Lyu, Y., Fan, X., Wu, Z., Cheng, Y., Wu, J., Chen, L., Wu, P., Lee, M. A., Zhu, Y., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.07502 · Cited by 85 · see citing works
“…Architecture search: Instead of hand-designing architectures, several approaches define a set of atomic operations (e.g., linear transformation, activation, attention, etc.) and use architecture search to learn the best order of these operations for a given task [122, 173], which we call MFAS.”
BackgroundA technical view on neural architecture search
Hu, Y.-Q., & Yang, Y. (2020) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-020-01062-1 · Cited by 22 · see citing works
Yang, C., Li, Z., Xu, P., Huang, H., Huo, Y., & Wei, Y. (2023) · Advanced Engineering Informatics · doi:10.1016/j.aei.2023.102238 · Cited by 22 · see citing works
Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization GapPreprint
Xie, L., Chen, X., Bi, K., Wei, L., Xu, Y., Chen, Z., Wang, L., Xiao, A., Chang, J., Zhang, X., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.01475 · Cited by 29 · see citing works
Appearance-posture fusion network for distracted driving behavior recognition
Yang, X., Qiao, Y., Han, S., Feng, Z., & Chen, Y. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124883 · Cited by 21 · see citing works
Visual language integration: A survey and open challenges
Park, S., & Kim, Y. (2023) · Computer Science Review · doi:10.1016/j.cosrev.2023.100548 · Cited by 21 · see citing works
DC-NAS: Divide-and-Conquer Neural Architecture Search for Multi-Modal Classification
Liang, X., Fu, P., Guo, Q. J., Zheng, K., & Qian, Y. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i12.29281 · Cited by 21 · see citing works
Li, J., Zhao, Y., Zhang, H., Li, W. J., Fu, C., Lian, C., & Shan, P. (2024) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2024.3418415 · Cited by 20 · see citing works
Zaher, M., Ghoneim, A. S., Abdelhamid, L., & Atia, A. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109399 · Cited by 21 · see citing works
He, Y., Peng, S., Chen, M., Yang, Z., & Chen, Y. (2023) · IEEE Transactions on Neural Systems and Rehabilitation Engineering · doi:10.1109/tnsre.2023.3305363 · Cited by 21 · see citing works
Neural architecture search for multi-sensor information fusion-based intelligent fault diagnosis
Lin, T., Ren, Z., Zhu, L., Huang, K., Zhu, Y., Zeng, L., & Wan, J. (2024) · Advanced Engineering Informatics · doi:10.1016/j.aei.2024.102776 · Cited by 19 · see citing works
Gao, B. (2022) · Scientific Programming · doi:10.1155/2022/4667677 · Cited by 18 · see citing works
Meng, Z., Zhu, Y., Pang, W., Tian, J., Nie, F., & Wang, K. (2022) · IEEE Transactions on Medical Imaging · doi:10.1109/tmi.2022.3222541 · Cited by 18 · see citing works
Revisiting spatio-temporal layouts for compositional action recognition
Radevski, G., Moens, M., & Tuytelaars, T. (2021) · British Machine Vision Conference (BMVC) · doi:10.5244/c.35.278 · Cited by 17 · see citing works
Meng, R., Chen, W., Yang, S., Song, J., Lin, L., Xie, D., Pu, S., Wang, X., Song, M., & Zhuang, Y. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00700 · Cited by 17 · see citing works
Automatic fused multimodal deep learning for plant identification
Lapkovskis, A., Nefedova, N., & Beikmohammadi, A. (2025) · Frontiers in Plant Science · doi:10.3389/fpls.2025.1616020 · Cited by 16 · see citing works
Cui, Z., Zhang, J., Noh, G., & Park, H. J. (2022) · Applied Sciences · doi:10.3390/app12052688 · Cited by 16 · see citing works
ICARUS: Learning on IQ and Cycle Frequencies for Detecting Anomalous RF Underlay Signals
Roy, D., Chaudhury, V., Tassie, C., Spooner, C. M., & Chowdhury, K. R. (2023) · IEEE Conference on Computer Communications (INFOCOM) · doi:10.1109/infocom53939.2023.10228929 · Cited by 16 · see citing works
SepFusion: Finding Optimal Fusion Structures for Visual Sound Separation
Zhou, D., Zhou, X., Hu, D., Zhou, H., Bai, L., Liu, Z., & Ouyang, W. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20266 · Cited by 15 · see citing works
Relation-aware Hierarchical Attention Framework for Video Question Answering
Li, F., Bai, T., Cao, C., Liu, Z., Yan, C., & Wu, B. (2021) · International Conference on Multimedia Retrieval · doi:10.1145/3460426.3463635 · Cited by 14 · see citing works
Ding, X., Han, T., Fang, Y., & Larson, E. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04095-x · Cited by 13 · see citing works
Multi-modal cognitive computing
李, 学. (2022) · Scientia Sinica Informationis · doi:10.1360/ssi-2022-0226 · Cited by 13 · see citing works
Graph-based temporal action co-localization from an untrimmed video
Wang, L., Zhai, C.-B., Zhang, Q., Tang, W., Zheng, N., & Hua, G. (2021) · Neurocomputing · doi:10.1016/j.neucom.2020.12.126 · Cited by 12 · see citing works
Multimodal Semantic Consistency-Based Fusion Architecture Search for Land Cover Classification
Li, X., Lei, L., Zhang, C., & Kuang, G. (2022) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2022.3193273 · Cited by 12 · see citing works
NAPReg: Nouns As Proxies Regularization for Semantically Aware Cross-Modal Embeddings
Jawade, B., Mohan, D. D., Ali, N. M., Setlur, S., & Govindaraju, V. (2023) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv56688.2023.00119 · Cited by 13 · see citing works
CURRENT CHALLENGES IN OPERATIONAL VERY HIGH RESOLUTION LAND-COVER MAPPING
Mallet, C., & Bris, A. L. (2020) · The international archives of the photogrammetry, remote sensing and spatial information sciences/International archives of the photogrammetry, remote sensing and spatial information sciences · doi:10.5194/isprs-archives-xliii-b2-2020-703-2020 · Cited by 12 · see citing works
Tian, Y. (2023) · Process Safety and Environmental Protection · doi:10.1016/j.cherd.2023.09.007 · Cited by 11 · see citing works
A multiscale neural architecture search framework for multimodal fusion
Lv, J., Sun, Y., Ye, Q., Feng, W., & Lv, J. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121005 · Cited by 12 · see citing works
Graph-based stock prediction with multisource information and relational data fusion
Zhang, Q., Zhang, Y., Bao, F., Yang, N., Zhang, C., & Лю, П. (2024) · Information Sciences · doi:10.1016/j.ins.2024.121561 · Cited by 11 · see citing works
Multimodal transformer for early alarm prediction
Strem, N., Dhami, D. S., Schmidt, B., & Kersting, K. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.109643 · Cited by 11 · see citing works
Chameleon: A Multimodal Learning Framework Robust to Missing Modalities
Liaqat, M. I., Nawaz, S., Zaheer, M. Z., Saeed, M. S., Sajjad, H., Schepper, T. D., Nandakumar, K., Khan, M. H., Gallo, I., & Schedl, M. (2025) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-025-00370-y · Cited by 11 · see citing works
Camera-LiDAR Fusion Method with Feature Switch Layer for Object Detection Networks
Kim, T.-L., & Park, T.-H. (2022) · Sensors · doi:10.3390/s22197163 · Cited by 11 · see citing works
Li, X., Lei, L., & Kuang, G. (2021) · IEEE International Geoscience and Remote Sensing Symposium (IGARSS) · doi:10.1109/igarss47720.2021.9555029 · Cited by 11 · see citing works
Deep Multi-modality Soft-decoding of Very Low Bit-rate Face Videos
Guo, Y., Zhang, X., & Wu, X. (2020) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3394171.3413709 · Cited by 10 · see citing works
Multimodal Fusion Refiner Networks
Sankaran, S., Yang, D., & Lim, S.-N. (2021) · arXiv · doi:10.48550/arxiv.2104.03435 · Cited by 10 · see citing works
- Authors
- Perez-Rua, J.-M., Vielzeuf, V., Pateux, S., Baccouche, M., & Jurie, F.
- Affiliations
- Samsung
- Université de Caen Normandie (FR)
- Orange Labs
- Orange (France) (FR)
- Univ. of Caen
- ORCID
- Vielzeuf, Valentin
- Venue
- 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
- Publisher
- IEEE
- Published
- 2019-06
- Pages
- 6959-6968
- ISSN
- 1063-6919 (ISSN-L)
- OpenAlex
- W2922066459
- Cited by
- 193
- References
- 52
- Citations by year
- 2026: 10 · 2025: 22 · 2024: 33 · 2023: 25 · 2022: 33 · 2021: 41 · 2020: 25 · 2019: 4
- FWCI
- 8.39
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- HAL (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- Agence Nationale de la Recherche (ANR-16-CE23-0006)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Human Pose and Action Recognition
- Keywords
- multimodal fusion, neural architecture search, multimodal classification, search space design, sequential model-based optimization, multimodal action recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
