Efficient Low-rank Multimodal Fusion With Modality-Specific Factors
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, Louis-Philippe Morency. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“The experiment confirmed that the LMF effectively improves the training and testing efficiency compared to TFN.”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [100]
The cited source says what the claim attributes to it.
In the citing paper
“Several efficient variants have been proposed to approximate expensive full tensor products with cheaper variants while maintaining performance [71, 101, 106].”
From MultiBench: Multiscale Benchmarks for Multimodal Representation Learning (2021) · cites this paper as [106]
In the citing paper
“we compared vector concatenation and a low-rank implementation of Kronecker Product Fusion, which similarly outperform unimodal approaches (Table S3) (Liu et al., 2018).”
From Pan-cancer integrative histology-genomic analysis via multimodal deep learning (2022) · cites this paper as Liu et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 4×
“Multi-View LSTM Multi-View LSTM (MV-LSTM) (Rajagopalan et al., 2016) aims to capture both modality-specific and cross-modality interactions from multiple modalities by partitioning the memory cell and the gates corresponding to multiple modalities.”
MethodInfluentialCortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Language We use pre-trained 300-dimensional Glove word embeddings (Pennington et al., 2014) to encode a sequence of transcribed words into a sequence of word vectors.”
MethodIEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869 · cited 4×
“We perform our experiments on the following multi-modal datasets, CMU-MOSI (Zadeh et al., 2016a POM (Park et al., 2014b), and IEMOCAP (Busso et al., 2008) for sentiment analysis, speaker traits recognition, and emotion recognition task, where the goal is to identify speakers emotions based on the speakers’ verbal and…”
BackgroundMethodInfluentialTensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 12×
“In addition, in order to be able to model the interactions between any subset of modalities using one tensor, Zadeh et al. (2017) proposed a simple extension to append 1 s to the unimodal representations before taking the outer product.”
BackgroundMethodInfluentialSelf-citationMultimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · cited 6×
“, 2017), (Fukui et al., 2016) performs contextdependent fusion of multimodal data.”
BackgroundMethodResultInfluentialFast Human Detection Using a Cascade of Histograms of Oriented Gradients
Zhu, Q., Yeh, M.-C., Cheng, K.-T., & Avidan, S. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.119 · Cited by 1,399 · cited 2×
“Visual The library Facet1 is used to extract a set of visual features for each frame (sampled at 30Hz) including 20 facial action units, 68 facial landmarks, head pose, gaze tracking and HOG features (Zhu et al., 2006).”
MethodMemory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12021 · Cited by 855
Self-citationMultimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.94 · Cited by 852 · cited 6×
“We perform our experiments on the following multi-modal datasets, CMU-MOSI (Zadeh et al., 2016a POM (Park et al., 2014b), and IEMOCAP (Busso et al., 2008) for sentiment analysis, speaker traits recognition, and emotion recognition task, where the goal is to identify speakers emotions based on the…”
MethodInfluentialSelf-citationCOVAREP — A collaborative voice analysis repository for speech technologies
Degottex, G., Kane, J., Drugman, T., Raitio, T., & Scherer, S. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6853739 · Cited by 695 · cited 1×
“Acoustic We use COVAREP acoustic analysis framework (Degottex et al., 2014) to extract a set of low-level acoustic features, including 12 Mel frequency cepstral coefficients (MFCCs), pitch, voiced/unvoiced segmentation, glottal source, peak slope, and maxima dispersion quotient features.”
MethodConvolutional MKL Based Multimodal Emotion Recognition and Sentiment Analysis
Poria, S., Chaturvedi, I., Cambria, E., & Hussain, A. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0055 · Cited by 595 · cited 2×
“Several works that use this method of fusion (Poria et al., 2016) , (Wang et al.”
MethodSpeaker identification on the SCOTUS corpus
Yuan, J., & Liberman, M. (2008) · The Journal of the Acoustical Society of America · doi:10.1121/1.2935783 · Cited by 594 · cited 1×
“To reach the same time alignment across modalities, we perform word alignment using P2FA (Yuan and Liberman, 2008) which allows us to align the three modalities at the word granularity.”
MethodMulti-attention Recurrent Network for Human Communication Comprehension
Zadeh, A., Liang, P. P., Poria, S., Vij, P., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12024 · Cited by 505
Self-citationTowards multimodal sentiment analysis: harvesting opinions from the web
Morency, L.-P., Mihalcea, R., & Doshi, P. (2011) · Proceedings of the 13th international conference on multimodal interfaces · doi:10.1145/2070481.2070509 · Cited by 470 · cited 3×
“…recognition (Yuhas et al., 1989), emotion recognition, (De Silva et al., 1997), (Chen et al., 1998), (W¨ollmer et al., 2013), sentiment analysis, (Morency et al., 2011) ∗ equal contributions as well as speaker trait analysis and media description (Park et al., 2014a) have seen a great boost in…”
BackgroundMethodSelf-citationYoutube movie reviews: Sentiment analysis in an audio-visual context
Wollmer, M., Weninger, F., Knaup, T., Schuller, B., Sun, C., Sagae, K., & Morency, L.-P. (2013) · IEEE Intelligent Systems · doi:10.1109/mis.2013.34 · Cited by 403 · cited 2×
“, 1998), (Wöllmer et al., 2013), sentiment analysis, (Morency et al.”
BackgroundSelf-citationMultimodal Compact Bilinear Pooling for Visual Question Answering and Visual GroundingPreprint
Fukui, A., Park, D. H., Yang, D., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.01847 · Cited by 388 · cited 6×
“, 2017), (Fukui et al., 2016) performs contextdependent fusion of multimodal data.”
BackgroundMethodResultInfluentialMOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · arXiv · doi:10.48550/arxiv.1606.06259 · Cited by 350 · cited 7×
“We perform our experiments on the following multi-modal datasets, CMU-MOSI (Zadeh et al., 2016a POM (Park et al., 2014b), and IEMOCAP (Busso et al., 2008) for sentiment analysis, speaker traits recognition, and emotion recognition task, where the goal is to identify speakers emotions based on the…”
MethodInfluentialSelf-citationMultimodal sentiment analysis with word-level fusion and reinforcement learning
Chen, M., Wang, S., Liang, P. P., Baltrušaitis, T., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3136801 · Cited by 334 · cited 1×
“We calculate the visual and acoustic features by taking the average of their feature values over the word time interval (Chen et al., 2017).”
MethodSelf-citationFacial emotion recognition using multi-modal information
De Silva, L. C., Miyasato, T., & Nakatsu, R. (2002) · Proceedings of ICICS, 1997 International Conference on Information, Communications and Signal Processing. Theme: Trends in Information Systems Engineering and Wireless Multimedia Communications (Cat. No.97TH8237) · doi:10.1109/icics.1997.647126 · Cited by 319 · cited 1×
“Tasks such as speech recognition (Yuhas et al., 1989), emotion recognition, (De Silva et al., 1997), (Chen et al., 1998), (W¨ollmer et al., 2013), sentiment analysis, (Morency et al., 2011) ∗ equal contributions as well as speaker trait analysis and media description (Park et al., 2014a) have seen…”
BackgroundDeep multimodal fusion for persuasiveness prediction
Nojavanasghari, B., Gopinath, D., Koushik, J., Baltrušaitis, T., & Morency, L.-P. (2016) · Proceedings of the 18th ACM International Conference on Multimodal Interaction · doi:10.1145/2993148.2993176 · Cited by 237 · cited 2×
“Deep Fusion The Deep Fusion model (DF) (Nojavanasghari et al., 2016) trains one deep neural model for each modality and then combine the output of each modality network with a joint neural network.”
MethodSelf-citation(2013) · Cited by 236
Integration of acoustic and visual speech signals using neural networks
Yuhas, B. P., Goldstein, M. H., & Sejnowski, T. J. (1989) · IEEE Communications Magazine · doi:10.1109/35.41402 · Cited by 209 · cited 2×
“Tasks such as speech recognition (Yuhas et al., 1989), emotion recognition, (De Silva et al.”
BackgroundMultimodal human emotion/expression recognition
Chen, L. S., Huang, T. S., Miyasato, T., & Nakatsu, R. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670976 · Cited by 183 · cited 1×
“Tasks such as speech recognition (Yuhas et al., 1989), emotion recognition, (De Silva et al., 1997), (Chen et al., 1998), (W¨ollmer et al., 2013), sentiment analysis, (Morency et al., 2011) ∗ equal contributions as well as speaker trait analysis and media description (Park et al., 2014a) have seen…”
BackgroundExtending Long Short-Term Memory for Multi-View Structured Learning
Rajagopalan, S. S., Morency, L., Baltrušaitis, T., & Goecke, R. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46478-7_21 · Cited by 156 · cited 2×
“Multi-View LSTM Multi-View LSTM (MV-LSTM) (Rajagopalan et al., 2016) aims to capture both modality-specific and cross-modality interactions from multiple modalities by partitioning the memory cell and the gates corresponding to multiple modalities.”
BackgroundMethodSelf-citationDynamical Tensor Approximation
Koch, O., & Lubich, C. (2010) · SIAM Journal on Matrix Analysis and Applications · doi:10.1137/09076578x · Cited by 131 · cited 1×
“Razenshteyn et al. (2016) proposes a mod-ified weighted version of low-rank approximation, and Koch and Lubich (2010) applies the method towards temporally dependent data to obtain low-rank approximations.”
MethodPark, S., Shim, H. S., Chatterjee, M., Sagae, K., & Morency, L.-P. (2014) · Proceedings of the 16th International Conference on Multimodal Interaction · doi:10.1145/2663204.2663260 · Cited by 126
Self-citationMemory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.00927 · Cited by 111
Self-citationLow-Rank Tensors for Scoring Dependency Structures
Lei, T., Xin, Y., Zhang, Y., Barzilay, R., & Jaakkola, T. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-1130 · Cited by 103 · cited 1×
“As for applications, Lei et al. (2014) proposes a low-rank tensor technique for dependency parsing while Wang and Ahuja (2008) uses the method of low-rank approximation applied directly on multidimensional image data (Datum-as-is representation) to enhance computer vision applications.”
MethodAttribute-Enhanced Face Recognition with Neural Tensor Fusion Networks
Hu, G., Hua, Y., Yuan, Y., Zhang, Z., Lu, Z., Mukherjee, S. S., Hospedales, T. M., Robertson, N. M., & Yang, Y. (2017) · 2017 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.404 · Cited by 86 · cited 1×
“Hu et al. (2017) proposes a low-rank tensor-based fusion framework to improve the face recognition performance using the fusion of facial attribute information.”
BackgroundA Tensor Approximation Approach to Dimensionality Reduction
Wang, H., & Ahuja, N. (2007) · International Journal of Computer Vision · doi:10.1007/s11263-007-0053-0 · Cited by 83 · cited 1×
“As for applications, Lei et al. (2014) proposes a low-rank tensor technique for dependency parsing while Wang and Ahuja (2008) uses the method of low-rank approximation applied directly on multidimensional image data (Datum-as-is representation) to enhance computer vision applications.”
MethodWeighted low rank approximations with provable guarantees
Razenshteyn, I., Song, Z., & Woodruff, D. P. (2016) · ACM Symposium on Theory of Computing (STOC) · doi:10.1145/2897518.2897639 · Cited by 51 · cited 1×
“Razenshteyn et al. (2016) proposes a mod-ified weighted version of low-rank approximation, and Koch and Lubich (2010) applies the method towards temporally dependent data to obtain low-rank approximations.”
MethodWang, H., Meghawat, A., Morency, L., & Xing, E. P. (2016) · arXiv (Cornell University) · Cited by 32
Self-citationSelect-Additive Learning: Improving Generalization in Multimodal Sentiment AnalysisPreprint
Wang, H., Meghawat, A., Morency, L., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.05244 · Cited by 17 · cited 1×
“Several works that use this method of fusion (Poria et al., 2016) , (Wang et al., 2016) use input-level feature concatenation and use the concatenated features as input, sometimes even removing the temporal dependency present in the modalities (Morency et al., 2011).”
MethodSelf-citation
Research works printed but not linked: 5
- [6]6. Gilles Degottex, John Kane, Thomas Drugman, Tuomo Raitio, and Stefan Scherer. 2014. Covarepa collabo- rative voice analysis repository for speech technolo- gies. In Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, pages 960-964.
- [10]10. Othmar Koch and Christian Lubich. 2010. Dynami- cal tensor approximation. SIAM Journal on Matrix Analysis and Applications 31(5):2360-2375.
- [14]14. Sunghyun Park, Han Suk Shim, Moitreya Chatterjee, Kenji Sagae, and Louis-Philippe Morency. 2014a. Computational analysis of persuasiveness in social multimedia: A novel dataset and multimodal predic- tion approach. In Proceedings of the 16th Interna- tional Conference on Multimodal Interaction. ACM, pages 50-57.
- [22]22. Hongcheng Wang and Narendra Ahuja. 2008. A ten- sor approximation approach to dimensionality re- duction. International Journal of Computer Vision 76(3):217-229.
- [24]24. Torsten Wortwein and Stefan Scherer. 2017. What really mattersan information gain analysis of ques- tions and reactions in automated ptsd screenings. In 2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, pages 15-20.
Cited by
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Hazarika, D., Zimmermann, R., & Poria, S. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413678 · Cited by 1,073
Yu, W., Xu, H., Yuan, Z., & Wu, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17289 · Cited by 816
Harnessing multimodal data integration to advance precision oncology
Boehm, K. M., Khosravi, P., Vanguri, R., Gao, J., & Shah, S. P. (2022) · Nature Reviews Cancer · doi:10.1038/s41568-021-00408-3 · Cited by 662
“Advances in multimodal ML methods have been impressive in other fields, such as sentiment analysis 86,149–154 , with large benchmark datasets, but the largest multimodal oncological dataset, the TCGA dataset, contains limited data modalities and only a few hundred patients per cancer type.”
BackgroundPan-cancer integrative histology-genomic analysis via multimodal deep learning
Chen, R. J., Lu, M. Y., Williamson, D. F. K., Chen, T. Y., Lipkova, J., Noor, Z., Shaban, M., Shady, M., Williams, M., Joo, B., et al. (2022) · Cancer Cell · doi:10.1016/j.ccell.2022.07.004 · Cited by 660
What Makes Training Multi-modal Classification Networks Hard?
Wang, W., Tran, D., & Feiszli, M. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01271 · Cited by 588
“On a different fusion architecture with Low-Rank Multi-Modal Fusion (LMF) [35], GradientBlending gives 4.”
Method
Metadata
- Authors
- Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Bagher Zadeh, A., & Morency, L.-P.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2018
- Pages
- 2247-2256
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2964346351
- Cited by
- 1,170
- References
- 31
- Citations by year
- 2026: 213 · 2025: 262 · 2024: 255 · 2023: 176 · 2022: 126 · 2021: 85 · 2020: 39 · 2019: 12 · 2018: 2
- FWCI
- 10.40
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- National Science Foundation
- Topics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Engineering › Media Technology › Advanced Image Fusion Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- multimodal fusion, cross-modal representation learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.