The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
Open accessAbstract
Multimodal Sentiment Analysis is an active area of research that leverages multimodal signals for affective understanding of user-generated videos. The predominant approach, addressing this task, has been to develop sophisticated fusion techniques. However, the heterogeneous nature of the signals creates distributional modality gaps that pose significant challenges. In this paper, we aim to learn effective modality representations to aid the process of fusion. We propose a novel framework, MISA, which projects each modality to two distinct subspaces. The first subspace is modality-invariant, where the representations across modalities learn their commonalities and reduce the modality gap. The second subspace is modality-specific, which is private to each modality and captures their characteristic features. These representations provide a holistic view of the multimodal data, which is used for fusion that leads to task predictions. Our experiments on popular sentiment analysis benchmarks, MOSI and MOSEI, demonstrate significant gains over state-of-the-art models. We also consider the task of Multimodal Humor Detection and experiment on the recently proposed UR_FUNNY dataset. Here too, our model fares better than strong baselines, establishing MISA as a useful multimodal framework.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“On CMU-MOSEI dataset, our proposed C-MIB yields 0.7% improvement on Acc2 and 1.2% improvement on F1 score compared to the current state-of-the-art MISA [53]. The performance on Acc7 and Corr is also improved compared to the state-of-the-art methods.”
From Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations (2023) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 8×
“Inter-utterance context. et al. proposed one of the first models, bc-LSTM , which utilized”
MethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 1×
“However, following recent works [7], including the stateof-the-art ICCN [50], we utilize the pre-trained BERT [11] as the feature extractor for textual utterances.”
MethodGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“ndard low-level features that are provided by the respective benchmarks and utilized by the state-of-the-art methods. 4.3.1 Language Features. Traditionally, language modality features has been GloVe [38] embeddings for each token in the utterance. However, following recent works [7], including the stateof-the-art ICC…”
MethodInfluential(2008) · Journal of Machine Learning Research · Cited by 33,051
Multimodal Transformer for Unaligned Multimodal Language Sequences
Tsai, Y.-H. H., Bai, S., Liang, P. P., Kolter, J. Z., Morency, L.-P., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1656 · Cited by 2,259
Tensor Fusion Network for Multimodal Sentiment Analysis
Zadeh, A., Chen, M., Poria, S., Cambria, E., & Morency, L.-P. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1115 · Cited by 2,102 · cited 5×
“licly available models of [26, 51, 58]).”
BackgroundInfluentialSelf-citationEkman, P., & Rosenberg, E. L. (2005) · Oxford University Press eBooks · doi:10.1093/acprof:oso/9780195179644.001.0001 · Cited by 1,971 · cited 1×
“Both MOSI and MOSEI use Facet 3 to extract facial expression features, which include facial action units and face pose based on the Facial Action Coding System (FACS) [14].”
MethodOpenFace: An open source facial behavior analysis toolkit
Baltrušaitis, T., Robinson, P., & Morency, L. (2016) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv.2016.7477553 · Cited by 1,794 · cited 1×
“For UR_FUNNY, OpenFace [4], a facial behavioral analysis tool, is used to extract features related to the facial expressions of the speaker.”
MethodA review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Self-citationMultimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1044 · Cited by 1,582 · cited 1×
“g cross-modal dynamics using sophisticated fusion mechanisms. These works include variety of methods, such as, multiple kernel learning [42], and tensor-based fusion (including its low-rank variants) [14, 21, 26, 29, 31, 58]. While these works perform fusion over representations of utterances, another line of work tak…”
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1208 · Cited by 1,433 · cited 5×
“com/platform/ 4Please refer to [19, 60] and their respective SDKs (https://github.”
BackgroundMethodInfluentialSelf-citationSimultaneous Deep Transfer Across Domains and Tasks
Tzeng, E., Hoffman, J., Darrell, T., & Saenko, K. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.463 · Cited by 1,220
Bousmalis, K., Trigeorgis, G., Silberman, N., Krishnan, D., & Erhan, D. (2016) · arXiv · doi:10.48550/arxiv.1608.06019 · Cited by 1,208 · cited 4×
“Our framework is closely related to the Domain Separation Network (DSN) [5], which proposed the shared-private model for domain adaptation.”
BackgroundMethodInfluentialVisualBERT: A Simple and Performant Baseline for Vision and Language
Li, L. H., Yatskar, M., Yin, D., Hsieh, C.-J., & Chang, K.-W. (2019) · arXiv · doi:10.48550/arxiv.1908.03557 · Cited by 1,198 · cited 1×
“rmation from fellow representations that are synergistic towards to the overall affective orientation. Such cross-modality matching has been highly prominent in recent cross-modal learning approaches [22, 23, 27, 50, 57]. For self-attention, we set Q = K = V = M ∈R6×d h . The Transformer generates a new matrix M ¯= [h…”
BackgroundEfficient Low-rank Multimodal Fusion With Modality-Specific Factors
Liu, Z., Shen, Y., Lakshminarasimhan, V. B., Liang, P. P., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1209 · Cited by 1,170
From Utterance to Text: The Bias of Language in Speech and Writing
Olson, D. (1977) · Harvard Educational Review · doi:10.17763/haer.47.3.8840364413869005 · Cited by 1,065 · cited 1×
“1An utterance is a unit of speech bounded by breaths or pauses [34].”
BackgroundVL-BERT: Pre-training of Generic Visual-Linguistic Representations
Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., & Dai, J. (2019) · arXiv · doi:10.48550/arxiv.1908.08530 · Cited by 950 · cited 1×
“Such cross-modality matching has been highly prominent in recent cross-modal learning approaches [22, 23, 27, 49, 57].”
BackgroundContext-Dependent Sentiment Analysis in User-Generated Videos
Poria, S., Cambria, E., Hazarika, D., Majumder, N., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1081 · Cited by 939
Self-citationMemory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12021 · Cited by 855
Self-citationMultimodal Sentiment Intensity Analysis in Videos: Facial Gestures and Verbal Messages
Zadeh, A., Zellers, R., Pincus, E., & Morency, L.-P. (2016) · IEEE Intelligent Systems · doi:10.1109/mis.2016.94 · Cited by 852
COVAREP — A collaborative voice analysis repository for speech technologies
Degottex, G., Kane, J., Drugman, T., Raitio, T., & Scherer, S. (2014) · 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2014.6853739 · Cited by 695 · cited 1×
“The acoustic features contain various low-level statistical audio functions extracted from COVAREP [10] – an acoustic analysis framework.”
MethodAdversarial Multi-task Learning for Text Classification
Liu, P., Qiu, X., & Huang, X. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-1001 · Cited by 674 · cited 2×
“The non-redundancy is achieved by enforcing a soft orthogonality constraint between the two representations [5, 25, 47].”
BackgroundMethodPoria, S., Cambria, E., & Gelbukh, A. F. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1303 · Cited by 559 · cited 1×
“roposed works in this category have primarily focused on learning cross-modal dynamics using sophisticated fusion mechanisms. These works include variety of methods, such as, multiple kernel learning [42], and tensor-based fusion (including its low-rank variants) [14, 21, 26, 29, 31, 58]. While these works perform fus…”
BackgroundSelf-citationDeep Multimodal Representation Learning: A Survey
Guo, W., Wang, J., & Wang, S. (2019) · IEEE Access · doi:10.1109/access.2019.2916887 · Cited by 535 · cited 1×
“shared representation in a common subspace with distributional similarity constraints [18].”
BackgroundPham, H., Liang, P. P., Manzini, T., Morency, L.-P., & Póczos, B. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016892 · Cited by 512
Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
Wang, Y., Shen, Y., Liu, Z., Liang, P. P., Zadeh, A., & Morency, L. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33017216 · Cited by 501
Sun, Z., Sarma, P., Sethares, W., & Liang, Y. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6431 · Cited by 410
Multimodal sentiment analysis using hierarchical fusion with context modeling
Majumder, N., Hazarika, D., Gelbukh, A. F., Cambria, E., & Poria, S. (2018) · Knowledge-Based Systems · doi:10.1016/j.knosys.2018.07.041 · Cited by 392
Self-citationBeneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research
Poria, S., Hazarika, D., Majumder, N., & Mihalcea, R. F. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3038167 · Cited by 362
Self-citationMultimodal sentiment analysis with word-level fusion and reinforcement learning
Chen, M., Wang, S., Liang, P. P., Baltrušaitis, T., Zadeh, A., & Morency, L.-P. (2017) · Proceedings of the 19th ACM International Conference on Multimodal Interaction · doi:10.1145/3136755.3136801 · Cited by 334 · cited 1×
“Approaches includemultimodal-awareword embeddings [56], recurrent multi-stage fusion [24], graph-based fusion [30, 60], recurrent networks (RNNs), attention-models, memory mechanisms, and transformer-based models [8, 46, 46, 52, 56, 59–61].”
Peng, Y., & Qi, J. (2019) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3284750 · Cited by 292 · cited 1×
“cross-modal correlations using Canonical Correlation Analysis [3]; 3) Learning a new shared subspace where all the modalities are simultaneously mapped, using techniques such as adversarial learning [35, 37]. Similar to the third category, we also learn common modality-invariant subspaces. However, we do not use adver…”
MethodMulti-task Learning for Multi-modal Emotion Recognition and Sentiment Analysis
Akhtar, M. S., Chauhan, D. S., Ghosal, D., Poria, S., Ekbal, A., & Bhattacharyya, P. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1034 · Cited by 238
Self-citationMai, S., Hu, H., & Xing, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i01.5347 · Cited by 237
Detection of Glottal Closure Instants From Speech Signals: A Quantitative Review
Thomas Drugman, A., Thomas, M. R., Guðnason, J., Naylor, P. A., & Dutoit, T. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2170835 · Cited by 236 · cited 1×
“from COVAREP [9] – an acoustic analysis framework. Some of the features include 12 Mel-frequency ceptral coefficients, pitch, Voiced/Unvoiced segmenting features (VUV) [11], glottal source parameters [12], and other features related to emotions and tone of speech 4. The feature dimensions, da, are 74 for MOSI/MOSEI an…”
MethodInfluentialMultimodal Language Analysis with Recurrent Multistage Fusion
Liang, P. P., Liu, Z., Bagher Zadeh, A., & Morency, L.-P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1014 · Cited by 230 · cited 2×
“ver representations of utterances, another line of work takes a fine-grained view to perform fusion at the word level.Approachesincludemultimodal-awarewordembeddings[56], recurrent multi-stage fusion [24], graph-based fusion [30, 60], recurrent networks (RNNs), attention-models, memory mechanisms, and transformer-base…”
Method(1998) · doi:10.1093/oso/9780195104462.001.0001 · Cited by 223
Multi-level Multiple Attentions for Contextual Multimodal Sentiment Analysis
Poria, S., Cambria, E., Hazarika, D., Mazumder, N., Zadeh, A., & Morency, L. (2017) · Proceedings · doi:10.1109/icdm.2017.134 · Cited by 215 · cited 1×
“STMs for the inter-utterance representation learning, framing the overall problem as a structured prediction (sequence tagging) task [43]. Later works involved either improving fusion using attention [7, 16, 44], hierarchical fusion [32], or developing better contextual modeling [2, 6, 7, 15]. In-depth discussion on t…”
MethodSelf-citationContextual Inter-modal Attention for Multi-modal Sentiment Analysis
Ghosal, D., Akhtar, M. S., Chauhan, D. S., Poria, S., Ekbal, A., & Bhattacharyya, P. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1382 · Cited by 196 · cited 1×
“m as a structured prediction (sequence tagging) task [43]. Later works involved either improving fusion using attention [7, 16, 44], hierarchical fusion [32], or developing better contextual modeling [2, 6, 7, 15]. In-depth discussion on the models are provided in Appendix A. Our work is fundamentally different from t…”
BackgroundSelf-citationUnsupervised Domain Clusters in Pretrained Language Models
Aharoni, R., & Goldberg, Y. (2020) · arXiv · doi:10.48550/arxiv.2004.02105 · Cited by 187 · cited 1×
“ile GloVe features are 300 dimensional token embeddings, for BERT, we utilize the BERT-base-uncased pre-trained model. This model comprises of 12 stacked Transformer layers. Aligned with recent works [1], we choose the utterance vector ul to be the average representation of the tokens from the final768 dimensional hid…”
MethodStrong Baselines for Neural Semi-Supervised Learning under Domain Shift
Ruder, S., & Plank, B. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1096 · Cited by 183 · cited 1×
“The non-redundancy is achieved by enforcing a soft orthogonality constraint between the two representations [5, 25, 47].”
BackgroundUnsupervised Domain Clusters in Pretrained Language Models
Aharoni, R., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.692 · Cited by 181
UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
Hasan, M. K., Rahman, W., Bagher Zadeh, A., Zhong, J., Tanveer, M. I., Morency, L.-P., & Hoque, M. (. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1211 · Cited by 174 · cited 5×
“For MHD, we consider the recently proposed UR_FUNNY dataset [19].”
BackgroundMethodInfluentialMai, S., Hu, H., & Xing, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1046 · Cited by 153 · cited 2×
“g cross-modal dynamics using sophisticated fusion mechanisms. These works include variety of methods, such as, multiple kernel learning [42], and tensor-based fusion (including its low-rank variants) [14, 21, 26, 29, 31, 58]. While these works perform fusion over representations of utterances, another line of work tak…”
MethodExtending Long Short-Term Memory for Multi-View Structured Learning
Rajagopalan, S. S., Morency, L., Baltrušaitis, T., & Goecke, R. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-46478-7_21 · Cited by 139 · cited 1×
“egorized into utterance-level and interutterance contextual models. Utterance-level baselines include: •Networks which perform temporal modeling and fusion of utterances: MFN [59], MARN [61], MV-LSTM [46], RMFN [24]. •Models which utilize attention and transformer modules to improvetokenrepresentationsusingnon-verbals…”
MethodContext-aware Interactive Attention for Multi-modal Sentiment and Emotion Analysis
Chauhan, D. S., Akhtar, M. S., Ekbal, A., & Bhattacharyya, P. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1566 · Cited by 97 · cited 2×
“ntations (MFM [53]). Inter-utterance contextual baselines include: •RNN-based models: BC-LSTM [43], with hierarchical fusion – CH-Fusion [32]. •Inter-utterance attention and multi-tasking models: CIA [6], CIM-MTL [2], DFF-ATMF [7]. For detailed descriptions of the baseline models, please refer to Appendix A. 4.4.2 Sta…”
BackgroundMethodInfluentialDeep Variational Canonical Correlation Analysis
Wang, W., Yan, X., Lee, H., & Livescu, K. (2016) · arXiv · doi:10.48550/arxiv.1610.03454 · Cited by 89 · cited 1×
“[55] revisited this framework by proposing a probabilistic CCA – deep variational CCA.”
MethodMai, S., Xing, S., & Hu, H. (2020) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2019.2925966 · Cited by 89 · cited 2×
“ingnon-verbalsignals:RAVEN[56], MulT [52]. •Graph-based fusion models: Graph-MFN [60]. •Utterance-vector fusion approaches that use tensor-based fusion and low-rank variants: TFN [58], LMF [26], LMFN [31], HFFN [29]. •Common subspace learning models that use cyclic translations (MCTN[39]),adversarialauto-encoders(ARGF…”
MethodCross-Domain Sentiment Classification with Target Domain Specific Information
Peng, M., Zhang, Q., Jiang, Y., & Huang, X. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1233 · Cited by 88 · cited 1×
“First, we choose CMD over KL-divergence or MMD, since CMD is a popular metric [36], which efficiently performs explicit matching of higherorder moments without expensive distance and kernel matrix computations.”
MethodAttribute-Enhanced Face Recognition with Neural Tensor Fusion Networks
Hu, G., Yang, H., Yuan, Y., Zhang, Z., Lu, Z., Mukherjee, S. S., Hospedales, T. M., Robertson, N. M., & Yang, Y. (2017) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2017.404 · Cited by 86 · cited 1×
“These works include variety of methods, such as, multiple kernel learning [42], and tensor-based fusion (including its low-rank variants) [15, 21, 26, 29, 31, 58].”
BackgroundSeq2Seq2Sentiment: Multimodal Sequence to Sequence Models for Sentiment Analysis
Pham, H., Manzini, T., Liang, P. P., & Póczos, B. (2018) · doi:10.18653/v1/w18-3308 · Cited by 82 · cited 1×
“Works that attempt to learn cross-modal common subspaces can be broadly categorized into: 1) Translation-based models which translates one modality to another using methods such as sequence-to-sequence [40], cyclic translations [39], and adversarial auto-encoders [30]; 2) Correlation-based models [51] that cross-modal…”
MethodMultimodal sentiment analysis based on multi-head attention mechanism
Chen, X., Lu, G., & Yan, J. (2020) · International Conference on Machine Learning and Soft Computing (ICMLSC) · doi:10.1145/3380688.3380693 · Cited by 80 · cited 1×
“Such cross-modality matching has been highly prominent in recent cross-modal learning approaches [22, 23, 27, 49, 57].”
BackgroundMulti-attention Recurrent Network for Human Communication Comprehension
Zadeh, A., Liang, P. P., Poria, S., Vij, P., Cambria, E., & Morency, L. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12024 · Cited by 49
Self-citationHuman Conversation Analysis Using Attentive Multimodal Networks with Hierarchical Encoder-Decoder
Gu, Y., Li, X., Huang, K., Fu, S., Yang, K., Chen, S., Zhou, M., & Marsic, I. (2018) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3240508.3240714 · Cited by 38
Sun, Z., Sarma, P. K., Sethares, W. A., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.05544 · Cited by 26 · cited 7×
“ICCN C-MFN”
BackgroundMethodInfluentialImage-Text Multi-Modal Representation Learning by Adversarial BackpropagationPreprint
Park, G., & Im, W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1612.08354 · Cited by 23 · cited 1×
“cross-modal correlations using Canonical Correlation Analysis [3]; 3) Learning a new shared subspace where all the modalities are simultaneously mapped, using techniques such as adversarial learning [35, 37]. Similar to the third category, we also learn common modality-invariant subspaces. However, we do not use adver…”
MethodBeneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research
Poria, S., Hazarika, D., Majumder, N., & Mihalcea, R. (2020) · arXiv · doi:10.48550/arxiv.2005.00357
Self-citation
Research works printed but not linked: 15
- Andrew Galen, 2013, Proceedings of the 30th International Conference on Machine Learning, ICML 2013 (JMLR Workshop and Conference Proceedings, 1255
- Bousmalis Konstantinos, Advances in Neural Information Processing Systems 29. Curran Associates
- 10.5555/2354409.2355082 · DOI
- Devlin Jacob, 2019, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1, 19
- Drugman Thomas, 2011, Joint Robust Voicing Detection and Pitch Estimation Based on Residual Harmonics. In INTERSPEECH 2011, 12th Annual Conference of the International Speech Communication Association. ISCA
- Kiela Douwe, 2019, NeurIPS 2019 Workshop. Curran Associates, Inc.
- Lu Jiasen, Advances in Neural Information Processing Systems 32. Curran Associates
- 10.5555/2392963.2392965 · DOI
- Peng Minlong, 2018, Long Papers
- Rajagopalan Shyam Sundar, 2016, Proceedings, Part VII (Lecture Notes in Computer Science, 353
- Salzmann Mathieu, 2010, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010 (JMLR Proceedings, 708
- Hubert Tsai Yao-Hung, 2019, Learning Factorized Multimodal Representations. In 7th International Conference on Learning Representations, ICLR
- Vaswani Ashish, Advances in Neural Information Processing Systems 30. Curran Associates
- Zadeh Amir, 2018, Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, 2246
- Zellinger Werner, 2017, Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning. In 5th International Conference on Learning Representations, ICLR
Cited by
Sorted by citations
Yu, W., Xu, H., Yuan, Z., & Wu, J. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i12.17289 · Cited by 816
Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research
Poria, S., Hazarika, D., Majumder, N., & Mihalcea, R. F. (2020) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2020.3038167 · Cited by 362
Self-citationHybrid Contrastive Learning of Tri-Modal Representation for Multimodal Sentiment Analysis
Mai, S., Zeng, Y., Zheng, S., & Hu, H. (2022) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2022.3172360 · Cited by 307
UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition
Hu, G., Lin, T.-E., Zhao, Y., Lu, G., Wu, Y., & Li, Y. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.534 · Cited by 304
Sun, L., Lian, Z., Liu, B., & Tao, J. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3274829 · Cited by 288
“Following previous works [16], [29], [32], we use the powerful BERT [52] model to encode raw text tokens into contextual word embeddings.”
MethodInfluential
Metadata
- Authors
- Hazarika, D., Zimmermann, R., & Poria, S.
- Affiliations
- National University of Singapore (SG)
- Singapore University of Technology and Design (SG)
- Venue
- Proceedings of the 28th ACM International Conference on Multimedia
- Publisher
- ACM
- Published
- 2020-10-12
- Pages
- 1122-1131
- OpenAlex
- W3093051361
- Cited by
- 1,073
- References
- 45
- Citations by year
- 2026: 280 · 2025: 315 · 2024: 238 · 2023: 136 · 2022: 78 · 2021: 23 · 2020: 3
- FWCI
- 22.92
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM International Conference on Multimedia (ACM MM) (publishedVersion, cc-by)
- Funders
- Ministry of Education - Singapore (MOE2018-T2-1-103)
- DSO National Laboratories - Singapore (RTDST190702)
- RIE 2020 Advanced Manufacturing and Engineering (AME) programmatic grant (A19E2b0098)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- multimodal sentiment analysis, modality gap, modality-invariant representations, modality-specific representations, subspace projection, multimodal fusion, user-generated video, humor detection, MOSI, MOSEI
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
