The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
M3ER: Multiplicative Multimodal Emotion Recognition using Facial, Textual, and Speech Cues
Open accessAbstract
We present M3ER, a learning-based method for emotion recognition from multiple input modalities. Our approach combines cues from multiple co-occurring modalities (such as face, text, and speech) and also is more robust than other methods to sensor noise in any of the individual modalities. M3ER models a novel, data-driven multiplicative fusion method to combine the modalities, which learn to emphasize the more reliable cues and suppress others on a per-sample basis. By introducing a check step which uses Canonical Correlational Analysis to differentiate between ineffective and effective modalities, M3ER is robust to sensor noise. M3ER also generates proxy features in place of the ineffectual modalities. We demonstrate the efficiency of our network through experimentation on two benchmark datasets, IEMOCAP and CMU-MOSEI. We report a mean accuracy of 82.7% on IEMOCAP and 89.0% on CMU-MOSEI, which, collectively, is an improvement of about 5% over prior work.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Multimodal Machine Learning: A Survey and Taxonomy
Baltrusaitis, T., Ahuja, C., & Morency, L.-P. (2019) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2798607 · Cited by 5,018
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Ekman, P. (1993) · American Psychologist · doi:10.1037/0003-066x.48.4.384 · Cited by 3,441
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
Relations Between Two Sets of Variates
Hotelling, H. (1936) · Biometrika · doi:10.2307/2333955 · Cited by 1,645
Deep Canonical Correlation Analysis
Andrew, G., Arora, R., Bilmes, J., & Livescu, K. (2013) · International Conference on Machine Learning · Cited by 1,602
Multimodal Language Analysis in the Wild: CMU-MOSEI Dataset and Interpretable Dynamic Fusion Graph
Bagher Zadeh, A., Liang, P. P., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1208 · Cited by 1,433
Canonical analysis of several sets of variables
KETTENRING, J. R. (1971) · Biometrika · doi:10.1093/biomet/58.3.433 · Cited by 896
Facial and vocal expressions of emotion
Russell, J. A., Bachorowski, J.-A., & Fernández-Dols, J.-M. (2003) · Annual Review of Psychology · doi:10.1146/annurev.psych.54.101601.145102 · Cited by 873
Memory Fusion Network for Multi-view Sequential Learning
Zadeh, A., Liang, P. P., Mazumder, N., Poria, S., Cambria, E., & Morency, L.-P. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.12021 · Cited by 855
Body Cues, Not Facial Expressions, Discriminate Between Intense Positive and Negative Emotions
Aviezer, H., Trope, Y., & Todorov, A. (2012) · Science · doi:10.1126/science.1224313 · Cited by 786
Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions
Ringeval, F., Sonderegger, A., Sauer, J., & Lalanne, D. (2013) · 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG) · doi:10.1109/fg.2013.6553805 · Cited by 771
Multimodal Emotion Recognition in Response to Videos
Soleymani, M., Pantic, M., & Pun, T. (2012) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.37 · Cited by 722
Rapid perceptual integration of facial expression and emotional body language
Meeren, H. K. M., van Heijnsbergen, C. C. R. J., & de Gelder, B. (2005) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0507650102 · Cited by 675
Scherer, K. R., Johnstone, & Klasmeyer, G. (2002) · Handbook of Affective Sciences · doi:10.1093/oso/9780195126013.003.0023 · Cited by 426
Deep learning for robust feature generation in audiovisual emotion recognition
Kim, Y., Lee, H., & Provost, E. M. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638346 · Cited by 401
Multimodal sentiment analysis using hierarchical fusion with context modeling
Majumder, N., Hazarika, D., Gelbukh, A., Cambria, E., & Poria, S. (2018) · Knowledge-Based Systems · doi:10.1016/j.knosys.2018.07.041 · Cited by 383
Bi-modal emotion recognition from expressive face and body gestures
Gunes, H., & Piccardi, M. (2007) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2006.09.007 · Cited by 382
Face alignment through subspace constrained mean-shifts
Saragih, J. M., Lucey, S., & Cohn, J. F. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459377 · Cited by 368
Head and Body Cues in the Judgment of Emotion: A Reformulation
Ekman, P., & Friesen, W. V. (1967) · Perceptual and Motor Skills · doi:10.1177/003151256702400305 · Cited by 279
Emotion Recognition through Multiple Modalities: Face, Body Gesture, Speech
Castellano, G., Kessous, L., & Caridakis, G. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-85099-1_8 · Cited by 259
AFEW-VA database for valence and arousal estimation in-the-wild
Kossaifi, J., Tzimiropoulos, G., Todorovic, S., & Pantic, M. (2017) · Image and Vision Computing · doi:10.1016/j.imavis.2017.02.001 · Cited by 236
Affective multimodal human-computer interaction
Pantic, M., Sebe, N., Cohn, J. F., & Huang, T. (2005) · Proceedings of the 13th annual ACM international conference on Multimedia · doi:10.1145/1101149.1101299 · Cited by 234
Fear-type emotion recognition for future audio-based surveillance systems
Clavel, C., Vasilescu, I., Devillers, L., Richard, G., & Ehrette, T. (2008) · Speech Communication · doi:10.1016/j.specom.2008.03.012 · Cited by 180
Speech Emotion Recognition Using Multi-hop Attention Mechanism
Yoon, S., Byun, S., Dey, S., & Jung, K. (2019) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2019.8683483 · Cited by 159
Learn to combine modalities in multimodal deep learning
Liu, K., Li, Y., Xu, N., & Natarajan, P. (2018) · arXiv · doi:10.48550/arxiv.1805.11730 · Cited by 136
Beyond Facial Expressions: Learning Human Emotion from Body Gestures
Shan, C., Gong, S., & McOwan, P. W. (2007) · British Machine Vision Conference (BMVC) · doi:10.5244/c.21.43 · Cited by 127
Multiple kernel learning for emotion recognition in the wild
Sikka, K., Dykstra, K., Sathyanarayana, S., Littlewort, G., & Bartlett, M. (2013) · Proceedings of the 15th ACM on International conference on multimodal interaction · doi:10.1145/2522848.2531741 · Cited by 119
Convolutional Attention Networks for Multimodal Emotion Recognition from Speech and Text Data
Choi, W. Y., Song, K. Y., & Lee, C. W. (2018) · doi:10.18653/v1/w18-3304 · Cited by 92
Physiological Signals and Their Use in Augmenting Emotion Recognition for Human–Machine Interaction
Knapp, R. B., Kim, J., & André, E. (2010) · Cognitive technologies · doi:10.1007/978-3-642-15184-2_9 · Cited by 68
Multimodal Relational Tensor Network for Sentiment and Emotion Classification
Sahay, S., Kumar, S. H., Xia, R., Huang, J., & Nachman, L. (2018) · doi:10.18653/v1/w18-3303 · Cited by 48
Cited by
A systematic review on affective computing: emotion models, databases, and recent advances
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.009 · Cited by 615
Emotions Don't Lie: An Audio-Visual Deepfake Detection Method using Affective Cues
Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413570 · Cited by 319
Self-citationA Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face
Lian, H., Lu, C., Li, S., Zhao, Y., Tang, C., & Zong, Y. (2023) · Entropy · doi:10.3390/e25101440 · Cited by 224
Perception and deception: Exploring individual responses to deepfakes across different modalities
Ahmed, S., & Chua, H. W. (2023) · Heliyon · doi:10.1016/j.heliyon.2023.e20383 · Cited by 29
Gupta, S., Kumar, P., & Tekchandani, R. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-023-18021-x · Cited by 14
Metadata
- Authors
- Mittal, T., Bhattacharya, U., Chandra, R., Bera, A., & Manocha, D.
- Venue
- Proceedings of the AAAI Conference on Artificial Intelligence
- Published
- 2020-04-03
- Volume
- 34
- Issue
- 02
- Pages
- 1359-1367
- ISSN
- 2374-3468
- 2159-5399
- OpenAlex
- W2997258743
- Cited by
- 253
- References
- 33
- FWCI
- 33.08
- Type
- journal article
- Language
- en
- Open access
- green
- Licence
- www.aaai.org/ (unspecified)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.