Modality-based Factorization for Multimodal Fusion
The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
We propose a novel method, Modality-based Redundancy Reduction Fusion (MRRF), for understanding and modulating the relative contribution of each modality in multimodal inference tasks. This is achieved by obtaining an (M +1)-way tensor to consider the high-order relationships between M modalities and the output layer of a neural network model. Applying a modality-based tensor factorization method, which adopts different factors for different modalities, results in removing information present in a modality that can be compensated by other modalities, with respect to model outputs. This helps to understand the relative utility of information in each modality. In addition it leads to a less complicated model with less parameters and therefore could be applied as a regularizer avoiding overfitting. We have applied this method to three different multimodal datasets in sentiment analysis, personality trait recognition, and emotion recognition. We are able to recognize relationships and relative importance of different modalities in these tasks and achieves a 1% to 4% improvement on several evaluation measures compared to the state-of-the-art for all three tasks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 12×
“The original tensor fusion approach as proposed in (Zadeh et al., 2017) flattened the tensor D which results in loss of the information included in the tensor structure, which is avoided in our approach.”
BackgroundMethodInfluentialFast Human Detection Using a Cascade of Histograms of Oriented Gradients
Zhu, Q., Yeh, M.-C., Cheng, K.-T., & Avidan, S. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.119 · Cited by 1,399
The Expression of a Tensor or a Polyadic as a Sum of Products
Hitchcock, F. L. (1927) · Journal of Mathematics and Physics · doi:10.1002/sapm192761164 · Cited by 1,345
Efficient Low-rank Multimodal Fusion With Modality-Specific Factors
Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1209 · Cited by 1,170 · cited 1×
“Late fusion combines the decision for each modality (either classification, or regression), by voting (Morvant et al., 2014), averaging (Shutova et al., 2016) or weighted sum of the outputs of the learned models (Glodek et al., 2011; Shutova et al., 2016).”
BackgroundContext-Dependent Sentiment Analysis in User-Generated Videos
Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1081 · Cited by 939
Memory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12021 · Cited by 855
MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · arXiv · doi:10.48550/arxiv.1606.06259 · Cited by 350
Multimodal sentiment analysis with word-level fusion and reinforcement learning
Chen, M., Wang, S., Liang, P. P., Baltrušaitis, T., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3136801 · Cited by 334 · cited 2×
“Pre-trained 300-dimensional Glove word embed-dings (Chen et al., 2017) were used to extract the language feature representations, which encodes a sequence of the transcribed words into a sequence of vectors.”
MethodFacial emotion recognition using multi-modal information
De Silva, L. C., Miyasato, T., & Nakatsu, R. (2002) · Proceedings of ICICS, 1997 International Conference on Information, Communications and Signal Processing. Theme: Trends in Information Systems Engineering and Wireless Multimedia Communications (Cat. No.97TH8237) · doi:10.1109/icics.1997.647126 · Cited by 319 · cited 1×
“Previous studies have found that some modalities are more informative than others (De Silva et al., 1997; Kampman et al., 2018), suggesting that allowing different compression rates for different modalities should improve performance.”
Background
Cited by
Gandhi, A., Adhvaryu, K., Poria, S., Cambria, E., & Hussain, A. (2023) · Information Fusion · doi:10.1016/j.inffus.2022.09.025 · Cited by 783
“[33] MRRF 2019 Tensor Fusion MRFF CMU-MOSI Modality-based Redundancy Reduction Fusion (MRRF) modality-based tensor factorization Modality-based Redundancy Reduction Fusion in MRRF is a tensor fusion and factorization method that allows for modalityspecific compression rates while also reducing parameter complexity.”
MethodA Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
“These include Temporal Tensor Fusion Network (T2FN) [101], Hierarchical Feature Fusion Network (HFFN) [102], Modality-based Redundancy Reduction Fusion (MRRF) [103], and Semi-Tensor Product (STP) [104].”
MethodUR-FUNNY: A Multimodal Language Dataset for Understanding Humor
Hasan, M. K., Rahman, W., Bagher Zadeh, A., Zhong, J., Tanveer, M. I., Morency, L.-P., & Hoque, M. (. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1211 · Cited by 174
“…Poria et al., 2017; Zadeh et al., 2017), multimodal fusion approaches (Liang et al., 2018; Tsai et al., 2018; Liu et al., 2018; Zadeh et al., 2018a; Barezi et al., 2018) as well as resources (Poria et al., 2018a; Zadeh et al., 2018c, 2016; Park et al., 2014; Rosas et al., 2013; W¨ollmer et al.,…”
BackgroundMai, S., Hu, H., & Xing, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1046 · Cited by 153
“For this purpose, a large body of prior work mostly treats the feature vectors of the modalities as the smallest units and fuse them at holistic level (Barezi et al., 2018; Poria et al., 2016a, 2017b; Liu et al., 2018).”
BackgroundInfluentialQuantum-inspired multimodal fusion for video sentiment analysis
Li, Q., Gkoumas, D., Lioma, C., & Melucci, M. (2021) · Information Fusion · doi:10.1016/j.inffus.2020.08.006 · Cited by 143
Metadata
- Authors
- Barezi, E. J., & Fung, P.
- Venue
- Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)
- Published
- 2019
- OpenAlex
- W2971004099
- Cited by
- 32
- References
- 13
- FWCI
- 1.52
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.