EmbraceNet: A robust deep learning architecture for multimodal classification
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Classification using multimodal data arises in many machine learning applications. It is crucial not only to model cross-modal relationship effectively but also to ensure robustness against loss of part of data or modalities. In this paper, we propose a novel deep learning-based multimodal fusion architecture for classification tasks, which guarantees compatibility with any kind of learning models, deals with cross-modal information carefully, and prevents performance degradation due to partial absence of data. We employ two datasets for multimodal classification tasks, build models based on our architecture and other state-of-the-art models, and analyze their performance on various situations. The results show that our architecture outperforms the other multimodal fusion architectures when some parts of data are not available.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,235
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Xiao, H., Rasul, K., & Vollgraf, R. (2017) · arXiv · doi:10.48550/arxiv.1708.07747 · Cited by 6,108
Flownet: Learning optical flow with convolutional networks
Dosovitskiy, A., Fischer, P., Ilg, E., Hausser, P., Hazirbas, C., Golkov, V., Smagt, P. V. D., Cremers, D., & Brox, T. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.316 · Cited by 4,313
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Ordóñez, F., & Roggen, D. (2016) · Sensors · doi:10.3390/s16010115 · Cited by 2,761 · cited 8×
“For the OPPORTUNITY dataset, we borrow an early integration-based model, “DeepConvLSTM”, which was used for classifying activities in the OPPORTUNITY dataset and showed better performance than other methods [3].”
BackgroundMethodInfluentialA review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582
Multimodal fusion for multimedia analysis: a survey
Atrey, P. K., Hossain, M. A., El Saddik, A., & Kankanhalli, M. S. (2010) · Multimedia Systems · doi:10.1007/s00530-010-0182-0 · Cited by 1,279
Big data deep learning: Challenges and perspectives
Chen, X.-W., & Lin, X. (2014) · IEEE Access · doi:10.1109/access.2014.2325029 · Cited by 1,255
Research works printed but not linked: 17
- Schmidhuber, 2015, Deep learning in neural networks: An overview, Neural Networks, 61, 85, 10.1016/j.neunet.2014.09.003 · DOI · Paper page
- Kwon, 2014, Unsupervised learning for human activity recognition using smartphone sensors, Expert Systems with Applications, 41, 6067, 10.1016/j.eswa.2014.04.037 · DOI · Paper page
- Vergara, 2013, On the performance of gas sensor arrays in open sampling systems using Inhibitory Support Vector Machines, Sensors and Actuators B: Chemical, 185, 462, 10.1016/j.snb.2013.05.027 · DOI · Paper page
- Matthews, 2002, Extraction of visual features for lipreading, IEEE Transactions on Pattern Analysis and Machine Intelligence, 24, 198, 10.1109/34.982900 · DOI · Paper page
- Higuera, 2015, Self-organizing feature maps identify proteins critical to learning in a mouse model of Down syndrome, PLOS ONE, 10, 1, 10.1371/journal.pone.0129126 · DOI
- Singh, 2014, Fusion of multimodal medical images using Daubechies complex wavelet transform – A multiresolution approach, Information Fusion, 19, 49, 10.1016/j.inffus.2012.09.005 · DOI · Paper page
- Lee, 2008, Robust audio-visual speech recognition based on late integration, IEEE Transactions on Multimedia, 10, 767, 10.1109/TMM.2008.922789 · DOI
- Moon, 2015, Perceptual experience analysis for tone-mapped HDR videos based on EEG and peripheral physiological signals, IEEE Transactions on Autonomous Mental Development, 7, 236, 10.1109/TAMD.2015.2449553 · doi:10.1109/tamd.2015.2449553
- Kusuma, 2011, PCA-based image recombination for multimodal 2D + 3D face recognition, Image and Vision Computing, 29, 306, 10.1016/j.imavis.2010.12.003 · DOI
- Bengio, 2004, Multimodal speech processing using asynchronous Hidden Markov Models, Information Fusion, 5, 81, 10.1016/j.inffus.2003.04.001 · DOI
- Kittler, 1998, On combining classifiers, IEEE Transactions on Pattern Analysis and Machine Intelligence, 20, 226, 10.1109/34.667881 · DOI · Paper page
- Bharadwaj, 2015, QFuse: Online learning framework for adaptive biometric system, Pattern Recognition, 48, 3428, 10.1016/j.patcog.2015.05.002 · DOI
- Costa, 2017, An evaluation of convolutional neural networks for music classification using spectrograms, Applied Soft Computing, 52, 28, 10.1016/j.asoc.2016.12.024 · DOI · Paper page
- Amer, 2018, Deep multimodal fusion: A hybrid approach, International Journal of Computer Vision, 126, 440, 10.1007/s11263-017-0997-7 · DOI
- Srivastava, 2014, Dropout: A simple way to prevent neural networks from overfitting, Journal of Machine Learning Research, 15, 1929
- Livnat, 2010, Sex, mixability, and modularity, Proceedings of the National Academy of Sciences, 107, 1452, 10.1073/pnas.0910734106 · DOI
- Algashaam, 2017, Multispectral periocular classification with multimodal compact multi-linear pooling, IEEE Access, 5, 14572, 10.1109/ACCESS.2017.2731118 · DOI
Cited by
Artificial intelligence for multimodal data integration in oncology
Lipkova, J., Chen, R. J., Chen, B., Lu, M. Y., Barbieri, M., Shao, D., Vaidya, A. J., Chen, C., Zhuang, L., Williamson, D. F. K., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.09.012 · Cited by 777
Gupta, A., Anpalagan, A., Guan, L., & Khwaja, A. S. (2021) · Array · doi:10.1016/j.array.2021.100057 · Cited by 565
Robust Multimodal Emotion Recognition from Conversation with Transformer-Based Crossmodality Fusion
Xie, B., Sidulova, M., & Park, C. H. (2021) · Sensors · doi:10.3390/s21144913 · Cited by 108
Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptation
Reza, M. K., Prater-Bennette, A., & Asif, M. S. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3476487 · Cited by 78
“[7] proposed a robust fusion strategy for multimodal classification.”
MethodDeep learning-based classification of microalgae using light and scanning electron microscopy images
Sonmez, M. E., Altinsoy, B., Ozturk, B. Y., Gumus, N. E., & Eczacioglu, N. (2023) · Micron · doi:10.1016/j.micron.2023.103506 · Cited by 38
Metadata
- Authors
- Choi, J.-H., & Lee, J.-S.
- Venue
- Information Fusion
- Publisher
- Elsevier BV
- Published
- 2019-11
- Volume
- 51
- Pages
- 259-270
- ISSN
- 1566-2535 (ISSN-L)
- OpenAlex
- W2936150952
- Cited by
- 156
- References
- 14
- FWCI
- 6.59
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- Funders
- MSIT (Ministry of Science and ICT), Korea (IITP-2018-2017-0-01015)
- IITP (R7124-16-0004)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: EmbraceNet: A robust deep learning architecture for multimodal classification. https://app.cochat.ai/papers/10.1016/j.inffus.2019.02.010
- The paper
- Choi, J.-H., & Lee, J.-S. (2019). EmbraceNet: A robust deep learning architecture for multimodal classification. Information Fusion. https://doi.org/10.1016/j.inffus.2019.02.010
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.