Multimodal Classification: Current Landscape, Taxonomy and Future Directions
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
Multimodal classification research has been gaining popularity with new datasets in domains such as satellite imagery, biometrics, and medicine. Prior research has shown the benefits of combining data from multiple sources compared to traditional unimodal data that has led to the development of many novel multimodal architectures. However, the lack of consistent terminologies and architectural descriptions makes it difficult to compare different solutions. We address these challenges by proposing a new taxonomy for describing multimodal classification models based on trends found in recent publications. Examples of how this taxonomy could be applied to existing models are presented as well as a checklist to aid in the clear and complete presentation of future models. Many of the most difficult aspects of unimodal classification have not yet been fully addressed for multimodal datasets, including big data, class imbalance, and instance-level difficulty. We also provide a discussion of these challenges and future directions of research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203 · cited 1×
“GANs have become a popular method for data augmentation [85] and have already been used with multimodal datasets.”
MethodHe, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
Borderline-SMOTE : A New Over-Sampling Method in Imbalanced Data Sets Learning
Han, H., Wang, W.-Y., & Mao, B.-H. (2005) · Lecture notes in computer science · doi:10.1007/11538059_91 · Cited by 4,331
Learning from imbalanced data: open challenges and future directions
Krawczyk, B. (2016) · Progress in Artificial Intelligence · doi:10.1007/s13748-016-0094-0 · Cited by 2,459 · cited 1×
“Class imbalanced data can be addressed by modifying the data itself, using algorithms inherently designed for addressing this issue, or a combination of both [53].”
MethodDeep multimodal learning: a survey on recent advances and trends
Ramachandram, D., & Taylor, G. W. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2738401 · Cited by 1,139 · cited 7×
“For the cross-modality style architectures, Ramachandram and Taylor [77] and Syed et al. [94] used the term intermediate, while Gao et al. [25] used deep shared modality.”
BackgroundMethodInfluentialMulti-view learning overview: Recent progress and new challenges
Zhao, J., Xie, X., Xu, X., & Sun, S. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.007 · Cited by 1,002
Huang, S.-C., Pareek, A., Seyyedi, S., Banerjee, I., & Lungren, M. P. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-00341-z · Cited by 934 · cited 2×
“While depictions of early and late fusion styles have been relatively consistent across multiple papers [9, 17, 37, 41, 75, 84, 119], there are still cases where other terms have been used.”
BackgroundA survey of multi-view machine learning
Sun, S. (2013) · Neural Computing and Applications · doi:10.1007/s00521-013-1362-6 · Cited by 878
A survey on deep learning for multimodal data fusion
Gao, J., Li, P., Chen, Z., & Zhang, J. (2020) · Neural Computation · doi:10.1162/neco_a_01273 · Cited by 821 · cited 4×
“This cross-modality fusion architecture is often also used with deep belief network (DBN) or autoencoder-style networks [25, 32, 59].”
BackgroundInfluentialMultimodal deep learning models for early detection of Alzheimer’s disease stage
Venugopalan, J., Tong, L., Hassanzadeh, H. R., & Wang, M. D. (2021) · Scientific Reports · doi:10.1038/s41598-020-74399-w · Cited by 732
Deep learning in medical image registration: a survey
Haskins, G., Kruger, U., & Yan, P. (2020) · Machine Vision and Applications · doi:10.1007/s00138-020-01060-x · Cited by 704
Rathore, S., Habes, M., Iftikhar, M. A., Shacklett, A., & Davatzikos, C. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.03.057 · Cited by 622 · cited 1×
“The ield of neuroimaging has also utilized multimodal imagery to improve scientiic understanding and diagnostic performance [18], including two surveys [68, 80] that focused on Alzheimer’s disease.”
BackgroundDeep Multimodal Representation Learning: A Survey
Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535
Mental health monitoring with multimodal sensing and machine learning: A survey
Garcia-Ceja, E., Riegler, M., Nordgreen, T., Jakobsen, P., Oedegaard, K. J., & Tørresen, J. (2018) · Pervasive and Mobile Computing · doi:10.1016/j.pmcj.2018.09.003 · Cited by 356 · cited 4×
“[27] also reviewed sensor-based systems addressing mental health issues.”
BackgroundMethodInfluentialMultimodal skin lesion classification using deep learning
Yap, J., Yolland, W., & Tschandl, P. (2018) · Experimental Dermatology · doi:10.1111/exd.13777 · Cited by 326 · cited 1×
“[117] combined CNN generated features with tabular clinical data and performed classiication with a three layer FCNN.”
MethodMultimodal Deep Learning for Activity and Context Recognition
Radu, V., Tong, C., Bhattacharya, S., Lane, N. D., Mascolo, C., Marina, M. K., & Kawsar, F. (2018) · Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies · doi:10.1145/3161174 · Cited by 281 · cited 1×
“In one work [76], deep learning techniques for activity and context recognition were investigated.”
BackgroundDeep multimodal fusion for semantic image segmentation: A survey
Zhang, Y., Sidibé, D., Morel, O., & Mériaudeau, F. (2021) · Image and Vision Computing · doi:10.1016/j.imavis.2020.104042 · Cited by 261
A review of multimodal medical image fusion techniques
Huang, B., Yang, F., Yin, M., Mo, X., & Zhong, C. (2020) · Computational and Mathematical Methods in Medicine · doi:10.1155/2020/8279342 · Cited by 220 · cited 2×
“Multiple data modalities are naturally present in many problem domains such as medicine [26, 34, 39, 48], hyperspacial imagery [7, 29], sentiment analysis [14, 81, 87], and many others.”
BackgroundLiu, T., Huang, J., Liao, T., Pu, R., Liu, S., & Peng, Y. (2022) · IRBM · doi:10.1016/j.irbm.2020.12.002 · Cited by 137 · cited 2×
“(2021) [65] CNN/DNN* late score concatenation CNN/DNN* score merge image/tabular Song et al.”
BackgroundMethodMultimodal Deep Learning Approach for Joint EEG-EMG Data Compression and Classification
Ben Said, A., Mohamed, A., Elfouly, T., Harras, K., & Wang, Z. J. (2017) · 2017 IEEE Wireless Communications and Networking Conference (WCNC) · doi:10.1109/wcnc.2017.7925709 · Cited by 86 · cited 2×
“In [80], an AE network was built for classifying EEG and electromyography (EMG) data.”
BackgroundMethodMultimodal Material Classification for Robots using Spectroscopy and High Resolution Texture Imaging
Erickson, Z., Xing, E., Srirangam, B., Chernova, S., & Kemp, C. C. (2020) · 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros45743.2020.9341165 · Cited by 50
A Joint Model for Multimodal Document Quality Assessment
Shen, A., Salehi, B., Baldwin, T., & Qi, J. (2019) · 2019 ACM/IEEE Joint Conference on Digital Libraries (JCDL) · doi:10.1109/jcdl.2019.00024 · Cited by 24 · cited 2×
“(2019) [82] LSTM/CNN* late deep concatenation LSTM/CNN* DNN image/text Tian et al.”
BackgroundMethod
Cited by
Artificial intelligence for multimodal data integration in oncology
Lipkova, J., Chen, R. J., Chen, B., Lu, M. Y., Barbieri, M., Shao, D., Vaidya, A. J., Chen, C., Zhuang, L., Williamson, D. F. K., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.09.012 · Cited by 777
A survey on imbalanced learning: latest research, applications and future directions
Chen, W., Yang, K., Yu, Z., Shi, Y., & Chen, C. L. P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10759-6 · Cited by 506
Li, Y., El Habib Daho, M., Conze, P.-H., Zeghlache, R., Le Boité, H., Tadayoni, R., Cochener, B., Lamard, M., & Quellec, G. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.108635 · Cited by 274
Effective Techniques for Multimodal Data Fusion: A Comparative Analysis
Pawłowski, M., Wróblewska, A., & Sysko-Romańczuk, S. (2023) · Sensors · doi:10.3390/s23052381 · Cited by 180
“Multimodal interpretability is the process of understanding multimodal deep neural nets [33,34].”
BackgroundMultimodal data integration for oncology in the era of deep neural networks: a review
Waqas, A., Tripathi, A., Ramachandran, R. P., Stewart, P. A., & Rasool, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1408843 · Cited by 165
Metadata
- Authors
- Sleeman, W. C., Kapoor, R., & Ghosh, P.
- Venue
- ACM Computing Surveys
- Published
- 2022-12-15
- Volume
- 55
- Issue
- 7
- Pages
- 1-31
- ISSN
- 0360-0300 (ISSN-L)
- 1557-7341
- OpenAlex
- W3199015608
- Cited by
- 124
- References
- 24
- FWCI
- 8.58
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.