Multimodal Machine Learning: A Survey and Taxonomy
Open accessThe DOI matches its Crossref record; 268 of the 281 references it cites were found in a registry or an index.
Abstract
Our experience of the world is multimodal - we see objects, hear sounds, feel texture, smell odors, and taste flavors. Modality refers to the way in which something happens or is experienced and a research problem is characterized as multimodal when it includes multiple such modalities. In order for Artificial Intelligence to make progress in understanding the world around us, it needs to be able to interpret such multimodal signals together. Multimodal machine learning aims to build models that can process and relate information from multiple modalities. It is a vibrant multi-disciplinary field of increasing importance and with extraordinary potential. Instead of focusing on specific multimodal applications, this paper surveys the recent advances in multimodal machine learning itself and presents them in a common taxonomy. We go beyond the typical early and late fusion categorization and identify broader challenges that are faced by multimodal machine learning, namely: representation, translation, alignment, fusion, and co-learning. This new taxonomy will enable researchers to better understand the state of the field and identify directions for future research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Early and late fusion have been the de-facto first-approach when tackling new multimodal problems.”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the effective fusion of complementary modalities has been shown to improve model accuracy, robustness, and generalization, whereas suboptimal fusion strategies can lead to degraded performance, instability, and overfitting [6].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Recent studies have proposed increasingly sophisticated fusion architectures, including attention-based models, variational autoencoders, and graph neural networks, often reporting improved predictive performance in complex biomedical and neuroimaging tasks [6,9,45,46,47].”
From Benchmarking Multimodal Deep Fusion Strategies for Heterogeneous Neuroimaging and Cognitive Data Using a Controlled Sex Classification Task (2026) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“multimodal data fusion—essential for integrating ultrasound, elastography, Doppler imaging, and metabolomics—is often superficial.”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Convolutional neural networks (CNNs) enable basic feature concatenation but lack mechanisms for semantic alignment across heterogeneous data streams, limiting their ability to model spatiotemporal dynamics (11).”
From Placenta-pulmonary coupling-guided multimodal AI for fetal lung maturity staging and individualized glucocorticoid therapy (2026) · cites this paper as [11]
Experimental — accuracy not yet measured
8 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Items 201–268 of 268
Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text
Venugopalan, S., Hendricks, L. A., Mooney, R., & Saenko, K. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1204 · Cited by 131
Multiple Kernel Learning in the Primal for Multimodal Alzheimer’s Disease Classification
Liu, F., Zhou, L., Shen, C., & Yin, J. (2014) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2013.2285378 · Cited by 129
The Long-Short Story of Movie Description
Rohrbach, A., Rohrbach, M., & Schiele, B. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24947-6_17 · Cited by 126
Visual Information in Semantic Representation
Feng, Y., & Lapata, M. (2010) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 124
Multi-View Learning in the Presence of View DisagreementPreprint
Christoudias, C. M., Urtasun, R., & Darrell, T. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.3242 · Cited by 122
Global Ranking Using Continuous Conditional Random Fields
Qin, T., Liu, T., Zhang, X., Wang, D., & Li, H. (2008) · Cited by 122
Choosing Linguistics over Vision to Describe Images
Gupta, A., Verma, Y., & Jawahar, C. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v26i1.8205 · Cited by 120
Multiple kernel learning for emotion recognition in the wild
Sikka, K., Dykstra, K., Sathyanarayana, S., Littlewort, G., & Bartlett, M. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531741 · Cited by 119
Trainable Methods for Surface Natural Language Generation
Ratnaparkhi, A. (2000) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0006028 · Cited by 116
Dynamic Units of Visual Speech
Taylor, S. L., Mahler, M., Theobald, B.-J., & Matthews, I. (2012) · doi:10.2312/sca/sca12/275-284 · Cited by 114
Weakly-Supervised Alignment of Video with Text
Bojanowski, P., Lajugie, R., Grave, E., Bach, F., Laptev, I., Ponce, J., & Schmid, C. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.507 · Cited by 111
Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion
Gebru, I. D., Ba, S., Li, X., & Horaud, R. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2017.2648793 · Cited by 111
Modeling Latent Discriminative Dynamic of Multi-dimensional Affective Signals
Ramírez, G. A., Baltrušaitis, T., & Morency, L. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-24571-8_51 · Cited by 110
Self-citationDimensional affect recognition using Continuous Conditional Random Fields
Baltrusaitis, T., Banda, N., & Robinson, P. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553785 · Cited by 108
Multi-modal Dimensional Emotion Recognition using Recurrent Neural Networks
Chen, S., & Jin, Q. (2015) · Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge · doi:10.1145/2808196.2811638 · Cited by 108
Listen, Attend, and Walk: Neural Mapping of Navigational Instructions to Action SequencesPreprint
Mei, H., Bansal, M., & Walter, M. R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.04089 · Cited by 107
Nonparametric Method for Data-driven Image Captioning
Mason, R., & Charniak, E. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.3115/v1/p14-2097 · Cited by 106
Emotion Recognition in the Wild with Feature Fusion and Multiple Kernel Learning
Chen, J., Chen, Z., Chi, Z., & Fu, H. (2014) · Proceedings of the 16th International Conference on Multimodal Interaction · doi:10.1145/2663204.2666277 · Cited by 105
Grounding Distributional Semantics in the Visual World
Baroni, M. (2016) · Language and Linguistics Compass · doi:10.1111/lnc3.12170 · Cited by 99
Grounded Models of Semantic Representation
Silberer, C., & Lapata, M. (2012) · RECERCAT (Consorci de Serveis Universitaris de Catalunya) · Cited by 98
An HMM-based system for automatic segmentation and alignment of speech
Sjölander, K. (2003) · Cited by 97
Is this a wampimuk? Cross-modal mapping between distributional semantics and the visual world
Lazaridou, A., Bruni, E., & Baroni, M. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1132 · Cited by 94
Modality-Dependent Cross-Media Retrieval
Wei, Y., Zhao, Y., Zhu, Z., Wei, S., Xiao, Y., Feng, J., & Yan, S. (2016) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/2775109 · Cited by 90
Statistical Parametric Speech Synthesis Based on Speaker and Language Factorization
Zen, H., Braunschweiler, N., Buchholz, S., Gales, M., Knill, K. M., Krstulović, S., & Latorre, J. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2012.2187195 · Cited by 89
Multi- and Cross-Modal Semantics Beyond Vision: Grounding in Auditory Perception
Kiela, D., & Clark, S. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1293 · Cited by 89
Book2Movie: Aligning video scenes with book chapters
Tapaswi, M., Bauml, M., & Stiefelhagen, R. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298792 · Cited by 87
A Novel Multiple Kernel Learning Framework for Heterogeneous Feature Fusion and Variable Selection
Yeh, Y.-R., Lin, T.-C., Chung, Y.-Y., & Wang, Y.-C. F. (2012) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2012.2188783 · Cited by 87
On the integration of grounding language and learning objects
Yu, C., & Ballard, D. H. (2004) · National Conference on Artificial Intelligence · Cited by 85
Expressive Visual Text-to-Speech Using Active Appearance Models
Anderson, R., Stenger, B., Wan, V., & Cipolla, R. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.434 · Cited by 85
Text to 3D Scene Generation with Rich Lexical Grounding
Chang, A., Monroe, W., Savva, M., Potts, C., & Manning, C. D. (2015) · doi:10.3115/v1/p15-1006 · Cited by 81
Multi-level Fusion of Audio and Visual Features for Speaker Identification
Wu, Z., Cai, L., & Meng, H. M. L. (2005) · Lecture notes in computer science · doi:10.1007/11608288_66 · Cited by 78
Multi-view CCA-based acoustic features for phonetic recognition across speakers and domains
Arora, R., & Livescu, K. (2013) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2013.6639047 · Cited by 78
Multi-view latent variable discriminative models for action recognition
Song, Y., Morency, L., & Davis, R. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6247918 · Cited by 77
Self-citationA coupled HMM for audio-visual speech recognition
Nefian, Liang, L., Pi, X., Xiao-xiang, L., Mao, M., & Murphy, M. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2002.1006167 · Cited by 77
Learning Multi-modal Similarity
McFeeBrian, & LanckrietGert (2011) · Journal of Machine Learning Research · doi:10.5555/1953048.1953063 · Cited by 75
Text-to-visual speech synthesis based on parameter generation from HMM
Masuko, T., Kobayashi, T., Tamura, M., Masubuchi, J., & Tokuda, K. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1998.679698 · Cited by 74
Multimodal Speaker Diarization
Noulas, A. K., Englebienne, G., & Kröse, B. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2011.47 · Cited by 71
Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation
Klein, B., Lev, G., Sadeh, G., & Wolf, L. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.7399 · Cited by 69
Multi-Relational Learning, Text Mining, and Semi-Supervised Learning for Functional Genomics
Krogel, M.-A., & Scheffer, T. (2004) · Machine Learning · doi:10.1023/b:mach.0000035472.73496.0c · Cited by 64
Multimodal human behavior analysis: learning correlation and interaction across modalities
Song, Y., Morency, L., & Davis, R. (2012) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2388676.2388684 · Cited by 61
Self-citationGenerating Images from Captions with Attention
Mansimov, E., Parisotto, E., Ba, J. L., & Salakhutdinov, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.02793 · Cited by 59
Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learning
Rajendran, J., Khapra, M. M., Chandar, S., & Ravindran, B. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1021 · Cited by 57
Deep correspondence restricted Boltzmann machine for cross-modal retrieval
Feng, F., Li, R., & Wang, X. (2014) · Neurocomputing · doi:10.1016/j.neucom.2014.12.020 · Cited by 56
Trigeorgis, G., Nicolaou, M. A., Zafeiriou, S. P., & Schuller, B. W. (2016) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2016.552 · Cited by 56
Nakov, P., & Ng, H. T. (2012) · Journal of Artificial Intelligence Research · doi:10.1613/jair.3540 · Cited by 55
Alignment of Speech to Highly Imperfect Text Transcriptions
Haubold, A., & Kender, J. R. (2007) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2007.4284627 · Cited by 52
Boosted learning in dynamic bayesian networks for multimodal speaker detection
Garg, A., Pavlovic, V., & Rehg, J. M. (2003) · Proceedings of the IEEE · doi:10.1109/jproc.2003.817119 · Cited by 50
Unsupervised Alignment of Natural Language Instructions with Video Segments
Naim, I., Song, Y., Liu, Q., Kautz, H., Luo, J., & Gildea, D. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8939 · Cited by 48
Dynamic modality weighting for multi-stream hmms inaudio-visual speech recognition
Gurban, M., Thiran, J.-P., Drugman, T., & Dutoit, T. (2008) · Proceedings of the 10th international conference on Multimodal interfaces · doi:10.1145/1452392.1452442 · Cited by 48
Grounding Semantics in Olfactory Perception
Kiela, D., Bulat, L., & Clark, S. (2015) · Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · doi:10.3115/v1/p15-2038 · Cited by 47
Lebret, R., Pinheiro, P. O., & Collobert, R. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1502.03671 · Cited by 45
Hidden Conditional Random Fields for Meeting Segmentation
Reiter, S. A., Schuller, B. W., & Rigoll, G. (2007) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2007.4284731 · Cited by 38
Video Description Generation using Audio and Visual Cues
Jin, Q., & Liang, J. (2016) · International Conference on Multimedia Retrieval · doi:10.1145/2911996.2912043 · Cited by 38
Aligning plot synopses to videos for story-based retrieval
Tapaswi, M., Bäuml, M., & Stiefelhagen, R. (2014) · International Journal of Multimedia Information Retrieval · doi:10.1007/s13735-014-0065-9 · Cited by 33
A Distributed Representation Based Query Expansion Approach for Image Captioning
Yagcioglu, S., Erdem, E., Erdem, A., & Çakıcı, R. (2015) · doi:10.3115/v1/p15-2018 · Cited by 33
Naim, I., Song, Y. C., Liu, Q., Huang, L., Kautz, H., Luo, J., & Gildea, D. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1017 · Cited by 33
Audio-to-text alignment for speech recognition with very limited resources
Anguera, X., Luque, J., & Gràcia, C. (2014) · Interspeech (USB) · doi:10.21437/interspeech.2014-345 · Cited by 32
Co-Adaptation of audio-visual speech and gesture classifiers
Christoudias, C. M., Saenko, K., Morency, L., & Darrell, T. (2006) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/1180995.1181013 · Cited by 31
Self-citationUnsupervised alignment of actions in video with text descriptions
Song, Y. C., Naim, I., Mamun, A. A., Kulkarni, K., Singla, P., Luo, J., Gildea, D., & Kautz, H. (2016) · International Joint Conference on Artificial Intelligence · Cited by 30
Isotonic CCA for sequence alignment and activity recognition
Shariat, S., & Pavlović, V. (2011) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2011.6126545 · Cited by 28
Learning Concept Taxonomies from Multi-modal Data
Zhang, H., Hu, Z., Deng, Y., Sachan, M., Yan, Z., & Xing, E. P. (2016) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p16-1169 · Cited by 26
Hierarchical Co-Attention for Visual Question AnsweringPreprint
Lu, J., Yang, J., Batra, D., & Parikh, D. (2016) · arXiv (Cornell University) · Cited by 24
Multimodal Dynamic Networks for Gesture Recognition
Wu, D., & Shao, L. (2014) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/2647868.2654969 · Cited by 23
The classification of multi-modal data with hidden conditional random field
Jiang, X., Wu, F. Q., Zhang, Y., Tang, S., Lü, W., & Zhuang, Y. (2014) · Pattern Recognition Letters · doi:10.1016/j.patrec.2014.08.005 · Cited by 23
Everybody loves a rich cousin: An empirical study of transliteration through bridge languages
Khapra, M. M., Kumaran, A., & Bhattacharyya, P. (2010) · North American Chapter of the Association for Computational Linguistics · Cited by 19
Speech-Driven Facial Animation Using a Shared Gaussian Process Latent Variable Model
Deena, S., & Galata, A. (2009) · International Symposium on Visual Computing · doi:10.1007/978-3-642-10331-5_9 · Cited by 19
Bimodal recognition experiments with recurrent neural networks
Cosi, P., Caldognetto, E. M., Vagges, K., Mian, G. A., & Contolini, M. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1994.389596 · Cited by 15
Multimodal Transfer Deep Learning for Audio Visual Recognition.
Moon, S., Kim, S., & Wang, H. (2014) · Cited by 10
Research works printed but not linked: 16
- [163] G. Potamianos, C. Neti, G. Gravier, A. Garg, and A. W. Senior, 'Recent advances in the automatic recognition of audio-visual speech,' Proceedings of the IEEE , 2003. -
- [175] A. Rohrbach, A. Torabi, M. Rohrbach, N. Tandon, C. Pal, H. Larochelle, A. Courville, and B. Schiele, 'Movie description,' International Journal of Computer Vision , 2017. -
- [60] S. Fidler, A. Sharma, and R. Urtasun, 'A Sentence is Worth a Thousand Pixels Holistic CRF model,' in CVPR , 2013. -
- [61] A. Frome, G. Corrado, and J. Shlens, 'DeViSE: A deep visualsemantic embedding model,' NIPS , 2013. -
- [90] N. Jaques, S. Taylor, A. Sano, and R. Picard, 'Multi-task , MultiKernel Learning for Estimating Individual Wellbeing,' in Multimodal Machine Learning Workshop in conjunction with NIPS , 2015. -
- [80] G. Hinton and R. S. Zemel, 'Autoencoders, minimum description length and Helmoltz free energy,' in NIPS , 1993. -
- [78] L. A. Hendricks, S. Venugopalan, M. Rohrbach, R. Mooney, K. Saenko, and T. Darrell, in CVPR , 2016. -
- [87] T.-H. K. Huang, F. Ferraro, N. Mostafazadeh, I. Misra, A. Agrawal, J. Devlin, R. Girshick, X. He, P. Kohli, D. Batra et al. , 'Visual storytelling.' NAACL, 2016. -
- [212] I. Vendrov, R. Kiros, S. Fidler, and R. Urtasun, 'OrderEmbeddings of Images and Language,' in ICLR , 2016.
- [220] W. Wang, R. Arora, K. Livescu, and J. Bilmes, 'On deep multiview representation learning,' in ICML , 2015.
- [215] --, 'Show and tell: A neural image caption generator,' in CVPR , 2015.
- [135] R. Mason and E. Charniak, 'Nonparametric Method for Datadriven Image Captioning,' in ACL , 2014. -
- [242] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, 'Modeling Context in Referring Expressions,' in ECCV , 2016.
- [105] R. Kiros, R. Salakhutdinov, and R. S. Zemel, 'Unifying VisualSemantic Embeddings with Multimodal Neural Language Models,' 2014. -
- [13] T. Baltruˇ saitis, N. Banda, and P. Robinson, 'Dimensional Affect Recognition using Continuous Conditional Random Fields,' in IEEE FG , 2013. -
- [17] L. W. Barsalou, 'Grounded cognition,' Annual review of psychology , 2008. -
Cited by
Recent Trends in Deep Learning Based Natural Language Processing [Review Article]
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2018.2840738 · Cited by 2,921
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1208 · Cited by 1,433
Self-citationAcosta, J. N., Falcone, G. J., Rajpurkar, P., & Topol, E. J. (2022) · Nature Medicine · doi:10.1038/s41591-022-01981-2 · Cited by 1,365
Recent Trends in Deep Learning Based Natural Language Processing
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.02709 · Cited by 1,192
Multimodal learning with transformers: A survey
Xu, P., Zhu, X., & Clifton, D. A. (2023) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2023.3275156 · Cited by 1,072
Metadata
- Authors
- Baltrusaitis, T., Ahuja, C., & Morency, L.-P.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- IEEE Transactions on Pattern Analysis and Machine Intelligence
- Published
- 2019-02-01
- Volume
- 41
- Issue
- 2
- Pages
- 423-443
- ISSN
- 0162-8828 (ISSN-L)
- 2160-9292
- 1939-3539
- OpenAlex
- W2619383789
- Cited by
- 5,018
- References
- 281
- Citations by year
- 2026: 1284 · 2025: 994 · 2024: 705 · 2023: 586 · 2022: 481 · 2021: 380 · 2020: 324 · 2019: 194
- FWCI
- 120.66
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Pattern Analysis and Machine Intelligence (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Multisensory perception and integration
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- multimodal learning, multimodal fusion, feature-level fusion, decision fusion, representation learning, cross-modal translation, cross-modal alignment, co-learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.