A Multi-Modal Fusion Method Based on Higher-Order Orthogonal Iteration Decomposition
Open accessThe DOI matches its Crossref record; 32 of the 33 references it cites were found in a registry or an index.
Abstract
Multi-modal fusion can achieve better predictions through the amalgamation of information from different modalities. To improve the performance of accuracy, a method based on Higher-order Orthogonal Iteration Decomposition and Projection (HOIDP) is proposed, in the fusion process, higher-order orthogonal iteration decomposition algorithm and factor matrix projection are used to remove redundant information duplicated inter-modal and produce fewer parameters with minimal information loss. The performance of the proposed method is verified by three different multi-modal datasets. The numerical results validate the accuracy of the performance of the proposed method having 0.4% to 4% improvement in sentiment analysis, 0.3% to 8% improvement in personality trait recognition, and 0.2% to 25% improvement in emotion recognition at three different multi-modal datasets compared with other 5 methods.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Moreover, MER can leverage the maximal mutual information to quantify the dependency between different modality features, extracting the most informative and discriminative features from each modality.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As such, MER has captured the attention of many researchers, aiming to combine information from various modalities, which can corroborate and complement each other, thus providing more comprehensive and accurate information for emotional judgment and significantly improving the performance of emotional judgments [12,13,14,15].”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [15]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Tensor Decompositions and Applications
Kolda, T. G., & Bader, B. W. (2009) · SIAM Review · doi:10.1137/07070111x · Cited by 10,301
DEAP: A Database for Emotion Analysis ;Using Physiological Signals
Koelstra, S., Muhl, C., Soleymani, M., Lee, J.-S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., & Patras, I. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.15 · Cited by 5,093
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
LXMERT: Learning Cross-Modality Encoder Representations from Transformers
Tan, H., & Bansal, M. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1514 · Cited by 2,319
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 4×
“Abbreviations The following abbreviations are used in this manuscript: TFN Tensor Fusion Network HOIDP Higher-order Orthogonal Iteration Decomposition and Projection Fusion DF Deep Fusion MARN Multi-attention Recurrent Network MFN Memory Fusion Network LMF Low-rank Multi-modal Fusion ACC Accuracy MAE Mean Absolute Err…”
BackgroundMethodInfluentialFast Human Detection Using a Cascade of Histograms of Oriented Gradients
Zhu, Q., Yeh, M.-C., Cheng, K.-T., & Avidan, S. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.119 · Cited by 1,399
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors
Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1209 · Cited by 1,170 · cited 8×
“To verify the improvement of the method, we compare our method with DF [22], MARN [23], MFN [24], TEN[16], and LMF [17] in sentiment analysis, personality trait recognition, and emotion recognition at three different multi-modal datasets.”
MethodInfluentialMemory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12021 · Cited by 855
COVAREP — A collaborative voice analysis repository for speech technologies
Degottex, G., Kane, J., Drugman, T., Raitio, T., & Scherer, S. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6853739 · Cited by 695
A Review and Meta-Analysis of Multimodal Affect Detection Systems
D'mello, S. K., & Kory, J. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 628
Speaker identification on the SCOTUS corpus
Yuan, J., & Liberman, M. (2008) · The Journal of the Acoustical Society of America · doi:10.1121/1.2935783 · Cited by 594
Deep Multimodal Representation Learning: A Survey
Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535
Multi-attention Recurrent Network for Human Communication Comprehension
Zadeh, A., Liang, P. P., Poria, S., Vij, P., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12024 · Cited by 505
Multimodal sentiment analysis with word-level fusion and reinforcement learning
Chen, M., Wang, S., Liang, P. P., Baltrušaitis, T., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3136801 · Cited by 334 · cited 1×
“The audio and video features can be obtained by calculating the average of feature values in the word time interval [29].”
BackgroundDeep multimodal fusion for persuasiveness prediction
Nojavanasghari, B., Gopinath, D., Koushik, J., Baltrušaitis, T., & Morency, L.-P. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2993176 · Cited by 237
Mai, S., Hu, H., & Xing, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5347 · Cited by 237
Multimedia classification and event detection using double fusion
Lan, Z., Bao, L., Yu, S.-I., Liu, W., & Hauptmann, A. G. (2013) · Multimedia Tools and Applications · doi:10.1007/s11042-013-1391-2 · Cited by 136
Park, S., Shim, H. S., Chatterjee, M., Sagae, K., & Morency, L.-P. (2014) · Proceedings of the 16th International Conference on Multimodal Interaction · doi:10.1145/2663204.2663260 · Cited by 126
Zero-Shot Event Detection Using Multi-modal Fusion of Weakly Supervised Concepts
Wu, S., Bondugula, S., Luisier, F., Zhuang, X., & Natarajan, P. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.341 · Cited by 111
BERT Representations for Video Question Answering
Yang, Z., Garcia, N., Chu, C., Otani, M., Nakashima, Y., & Takemura, H. (2020) · 2020 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv45572.2020.9093596 · Cited by 107
(2019) · neural information processing systems · Cited by 83
Audio-visual emotion recognition using an emotion space concept
Kanluan, I., Grimm, M., & Kroschel, K. (2008) · European Signal Processing Conference (EUSIPCO) · Cited by 75
Video2vec Embeddings Recognize Events When Examples Are Scarce
Habibian, A., Mensink, T., & Snoek, C. G. M. (2017) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2016.2627563 · Cited by 73
KnowIT VQA: Answering Knowledge-Based Questions about Videos
García, N., Otani, M., Chu, C., & Nakashima, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i07.6713 · Cited by 49
Cross-modal Bidirectional Translation via Reinforcement Learning
Qi, J., & Peng, Y. (2018) · Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2018/365 · Cited by 40
Xie, Z., & Guan, L. (2013) · International Journal of Semantic Computing · doi:10.1142/s1793351x13400023 · Cited by 30
Bhagat, S., Uppal, S., Yin, Z., & Lim, N. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58592-1_7 · Cited by 30
A Multilevel Fusion Approach for Audiovisual Emotion Recognition
Chetty, G., Wagner, M., & Goecke, R. (2015) · Wiley eBooks · doi:10.1002/9781118910566.ch17 · Cited by 21
Modality-based Factorization for Multimodal Fusion
Fung, P., Jebalbarezi sarbijan, E., & Momeni, P. (2019) · Rare & Special e-Zone (The Hong Kong University of Science and Technology) · Cited by 2
Research works printed but not linked: 1
- Zadeh, A., Zellers, R., Pincus, E., and Morency, L.P. (2016). MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos. arXiv.
Cited by
A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
Multimodal Hierarchical CNN Feature Fusion for Stress Detection
Kuttala, R., Subramanian, R., & Murthy, O. V. R. (2023) · IEEE Access · doi:10.1109/access.2023.3237545 · Cited by 104
Multimodal emotion recognition: A comprehensive review, trends, and challenges
Ramaswamy, M. P. A., & Palaniswamy, S. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1563 · Cited by 70
Fen, L., Chen, J., Li, K., Tan, W., Cai, C., & Ayub, M. S. (2022) · Entropy · doi:10.3390/e24121836 · Cited by 18
Self-citationAffective Interaction: Attentive Representation Learning for Multi-Modal Sentiment Classification
Zhang, Y., Tiwari, P., Rong, L., Chen, R., Alnajem, N. A., & Hossain, M. S. (2022) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3527175 · Cited by 21
Metadata
- Authors
- Liu, F., Chen, J., Tan, W., & Cai, C.
- Affiliations
- School of Marine Science and Technology, Northwestern Polytechnical University, Xi’an 710072, China
- Northwestern Polytechnical University (CN)
- Guizhou University (CN)
- Corresponding author
- Chen, Jianfeng
- Venue
- Entropy
- Publisher
- MDPI AG
- Published
- 2021-10-15
- Volume
- 23
- Issue
- 10
- Pages
- 1349
- ISSN
- 1099-4300 (ISSN-L)
- PMC
- PMC8534596
- OpenAlex
- W3207610499
- Cited by
- 13
- References
- 33
- Citations by year
- 2026: 3 · 2025: 1 · 2024: 3 · 2023: 3 · 2022: 3
- FWCI
- 1.02
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Entropy (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Natural Science Foundation of Shaanxi Province (2020JM-554)
- Yan'an University Scientific Research Project (YDY2019-18)
- Key Research and Development Projects of Shaanxi Province (2021NY-036)
- Topics
- Physical Sciences › Mathematics › Computational Mathematics › Tensor decomposition and applications
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal fusion, redundancy elimination, sentiment analysis, emotion recognition
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.