Context-sensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index; none of the 3 sources its claims cite is checked for support yet.
Abstract
In this paper, we apply a context-sensitive technique for multimodal emotion recognition based on feature-level fusion of acoustic and visual cues. We use bidirectional Long Short-Term Memory (BLSTM) networks which, unlike most other emotion recognition approaches, exploit long-range contextual information for modeling the evolution of emotion within a conversation. We focus on recognizing dimensional emotional labels, which enables us to classify both prototypical and nonprototypical emotional expressions contained in a large audiovisual database. Subject-independent experiments on various classification tasks reveal that the BLSTM network approach generally prevails over standard classification techniques such as Hidden Markov Models or Support Vector Machines, and achieves F1-measures of the order of 72 %, 65 %, and 55 % for the discrimination of three clusters in emotional space and the distinction between three levels of valence and activation, respectively. Index Terms: emotion recognition, multimodality, long shortterm memory, hidden markov models, context modeling
Overview
CoChat Summary
TL;DR
The paper presents a context-sensitive multimodal emotion recognition framework that fuses acoustic and facial features and uses bidirectional LSTM networks to model long-range context for dimensional valence/activation and emotional-space clusters. In subject-independent experiments on IEMOCAP, BLSTM generally outperformed HMM and SVM, achieving F1-measures of 72.34% for three clusters, 65.18% for three valence levels, and 55.18% for three activation levels.
Key findings
- BLSTM achieved the best F1-measure for three-cluster discrimination (72.34%), four clusters (62.78%), and five clusters (54.84%); for three valence levels it reached 65.18% F1, versus 63.66% for unidirectional LSTM.
- For three activation levels, HMM+LM achieved the highest F1 (56.89%), while BLSTM reached 55.18%; the paper attributes this to class imbalance, since most utterances were labeled medium activation and training data for the remaining two activation classes was insufficient.
- Feature selection showed modality-specific patterns: for valence, on average 84% of selected features were facial, whereas for activation only 44% were facial; unimodal HMM results confirmed facial features were better for valence and acoustic features better for activation.
- Context modeling helped: HMM+LM improved over HMM for valence and activation, and BLSTM generally prevailed over HMM and SVM, except for the five-cluster task where LSTM and BLSTM showed no significant difference.
Methods
- Design: feature-level fusion of acoustic and visual cues; bidirectional Long Short-Term Memory networks for long-range contextual modeling, compared against fully-connected 3-state HMMs with a first-order language model and polynomial-kernel SVMs trained with SMO.
- Data: IEMOCAP, approximately 12 hours of audio-visual data from five mixed-gender pairs of actors (ten subjects total), with utterances annotated for valence, activation, and dominance on integer scales from one to five; experiments used cyclic leave-one-speaker-out cross validation, with mean test and training utterances of 498 ± 60 and 4475 ± 61 across folds.
- Features and analysis: 46 facial markers reduced via Principal Feature Analysis to 30 features plus first derivatives (60-dim), normalized per speaker; audio features included 12 MFCCs, 27 Mel Frequency Band coefficients, pitch, and energy plus first derivatives, extracted at 25 ms frame rate with 50 ms window; utterance-level statistical functionals were used for LSTM and SVM, with correlation-based feature subset selection yielding 66–224 features; BLSTM used 128 memory blocks per direction, learning rate 10−5, and zero-mean Gaussian input noise with standard deviation 0.6.
Limitations
- Stated: activation classification suffers from class imbalance, as the majority of utterances are labeled medium activation and the remaining two activation classes have insufficient training data.
- Stated future work: dynamic modeling of low-level features using a multimodal LSTM framework and inclusion of linguistic information were not yet examined.
- Evident from the text: the experiments use an acted IEMOCAP corpus with scripted and improvised scenarios, so real-life generalization remains untested despite the subject-independent setup.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 0 Not valid · 3 Not checked · 17 Own results and interpretations
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such data are a relatively large portion of the database (approximately 17 % of the total utterances).”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“BLSTM v+f 67.31 52.53 58.46 55.18”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Generally, multimodal classification outperforms unimodal AER.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For all cluster prediction tasks, we observe similar trends: LSTM modeling prevails over HMM and SVM classification”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“BLSTM network approach generally prevails over standard classification techniques”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
7 more claims
Create a free account to see all 20 claims and every source verdict.
- The quote only partly states this claim. 1
- Probably not a substantial claim. 5
- It may state two separate claims as one. 2
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4. Experimental Results and Discussion
“For four and five clusters they achieve F1-measures of 62.78 % and 55.00 % respectively.”
§ 4. Experimental Results and Discussion
“LSTM v+f 56.19 53.89 56.25 55.00 BLSTM v+f 56.31 53.76 56.13 54.84”
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
IEMOCAP: Interactive emotional dyadic motion capture database
Busso, C., Bulut, M., Lee, C.-C., Kazemzadeh, A., Mower, E., Kim, S., Chang, J. N., Lee, S., & Narayanan, S. S. (2008) · Language Resources and Evaluation · doi:10.1007/s10579-008-9076-6 · Cited by 3,869
Self-citationSupervised Sequence Labelling with Recurrent Neural Networks
Graves, A. (2012) · Studies in Computational Intelligence · doi:10.1007/978-3-642-24797-2 · Cited by 3,120
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
Mehrabian, A. (2017) · communication theory · doi:10.4324/9781315080918-15 · Cited by 879
Active and dynamic information fusion for facial expression understanding from image sequences
Zhang, Y., & Ji, Q. (2005) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2005.93 · Cited by 461
Mechrabian, A. (1968) · Medical Entomology and Zoology · Cited by 459
OpenEAR — Introducing the munich open-source emotion and affect recognition toolkit
Eyben, F., Wollmer, M., & Schuller, B. (2009) · 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops · doi:10.1109/acii.2009.5349350 · Cited by 418 · cited 1×
“All functionals were calculated using our openEAR toolkit [12].”
MethodSelf-citationFeature selection using principal feature analysis
Lu, Y., Cohen, I., Zhou, X. S., & Tian, Q. (2007) · Proceedings of the 15th ACM international conference on Multimedia · doi:10.1145/1291233.1291297 · Cited by 341
Wollmer, M., Schuller, B., Eyben, F., & Rigoll, G. (2010) · IEEE Journal of Selected Topics in Signal Processing · doi:10.1109/jstsp.2010.2057200 · Cited by 165
Self-citation
Research works printed but not linked: 1
- [14][14] S. Young, G. Evermann, M. Gales, T. Hain, D. Kershaw, X. Liu, G. Moore, J. Odell, D. Ollason, D. Povey, V. Valtchev, and P. Woodland, The HTK Book. Entropic Cambridge Research Laboratory, Cambridge, England, 2006. 2365
Cited by
Items 101–200 of 208
Affective Communication through Air Jet Stimulation: Evidence from Event-Related Potentials
Tsalamlal, M. Y., Rizer, W., Martin, J., Ammi, M., & Ziat, M. (2018) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2018.1424065 · Cited by 12
Acoustic-linguistic recognition of interest in speech with bottleneck-BLSTM nets
Wöllmer, M., Weninger, F. J., Eyben, F., & Schuller, B. W. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-20 · Cited by 13
Self-citationOn Modeling Context from Objects with a Long Short-Term Memory for Indoor Scene Recognition
Laranjeira, C., Lacerda, A. M., & Nascimento, E. R. (2019) · SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI) · doi:10.1109/sibgrapi.2019.00041 · Cited by 16
How You Feelin’? Learning Emotions and Mental States in Movie Scenes
Srivastava, D., Singh, A. K., & Tapaswi, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00248 · Cited by 15
Facial expression-based affective speech translation
Székely, É., Steiner, I., Ahmed, Z. A., & Carson-Berndsen, J. (2013) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-013-0128-x · Cited by 12
Combination of sequential class distributions from multiple channels using Markov fusion networks
Glodek, M., Schels, M., Schwenker, F., & Palm, G. (2014) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-014-0149-0 · Cited by 12
An Ensemble-Learning-Based Technique for Bimodal Sentiment Analysis
Shah, S., Ghomeshi, H., Vakaj, E., Cooper, E., & Mohammad, R. (2023) · Big Data and Cognitive Computing · doi:10.3390/bdcc7020085 · Cited by 26
Action Unit Models of Facial Expression of Emotion in the Presence of Speech
Shah, M., Cooper, D. G., Cao, H., Gur, R. C., Nenkova, A., & Verma, R. (2013) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2013.15 · Cited by 16
Context-sensitive learning for enhanced audiovisual emotion classification (Extended abstract)
Metallinou, A., Katsamanis, A., Wöllmer, M., Eyben, F., Schuller, B. W., & Narayanan, S. S. (2015) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2015.7344611 · Cited by 12
Self-citationA Self-Attentive Emotion Recognition Network
Partaourides, H., Papadamou, K., Kourtellis, N., Leontiadis, I., & Chatzis, S. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054762 · Cited by 14
Speech Emotion Recognition in Dyadic Dialogues with Attentive Interaction Modeling
Zhao, J., Chen, S., Liang, J., & Jin, Q. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-2103 · Cited by 11
Investigation of the Effect of Increased Dimension Levels in Speech Emotion Recognition
Wang, H., Zhao, X., & Zhao, Y. (2022) · IEEE Access · doi:10.1109/access.2022.3194039 · Cited by 11
DouFu: A Double Fusion Joint Learning Method for Driving Trajectory Representation
Wang, H., Huang, Z., Zhou, X., Yin, G., Bao, Y., & Zhang, Y. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.110035 · Cited by 9
Audio-Based Granularity-Adapted Emotion Classification
Shepstone, S. E., Tan, Z., & Jensen, S. H. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2598741 · Cited by 10
Schuller, B. W. (2016) · Human-computer interaction series · doi:10.1007/978-3-319-31413-6_4 · Cited by 8
Self-citationMultimodal Approach for Emotion Recognition Using a Formal Computational Model
Meftah, I. T., Thanh, N. L., & Amar, C. B. (2013) · International Journal of Applied Evolutionary Computation · doi:10.4018/jaec.2013070102 · Cited by 9
Multimodal Input for Perceptual User Interfaces
LaViola, J. J., Buchanan, S., & Pittman, C. R. (2014) · Wiley eBooks · doi:10.1002/9781118706237.ch9 · Cited by 14
OCR for bilingual documents using language modeling
(2015) · doi:10.1109/icdar.2015.7333965 · Cited by 7
Audio Visual Recognition of Spontaneous Emotions In-the-Wild
Xia, X., Guo, L., Jiang, D., Pei, E., Yang, L., & Sahli, H. (2016) · Communications in computer and information science · doi:10.1007/978-981-10-3005-5_57 · Cited by 7
Human Emotions Detection Using Hybrid CNN Approach
Sandhu, N., Malhotra, A., & Bedi, M. K. (2020) · International Journal of Computer Science and Mobile Computing · doi:10.47760/ijcsmc.2020.v09i10.001 · Cited by 9
Abnormal Heartbeat Detection Using Recurrent Neural Networks
(2018) · Cited by 7
Emojis Pictogram Classification for Semantic Recognition of Emotional Context
Atif, M., Franzoni, V., & Milani, A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86993-9_14 · Cited by 7
Recognizing emotions in dialogues with acoustic and lexical features
Tian, L., Moore, J. D., & Lai, C. (2015) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2015.7344651 · Cited by 6
Chao, G.-Y., Chang, C.-M., Li, J.-L., Wu, Y.-T., & Lee, C.-C. (2018) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/3242969.3242992 · Cited by 6
Single-Image Facial Expression Recognition Using Deep 3D Re-Centralization
Bao, Z., You, S., Gu, L., & Yang, Z. (2019) · IEEE International Conference on Computer Vision Workshops · doi:10.1109/iccvw.2019.00202 · Cited by 6
Automatic human facial expression recognition using Hidden Markov Model
Vijayalakshmi, M., & Senthil, T. S. (2014) · International Conference on Electronics and Communication Systems (ICECS) · doi:10.1109/ecs.2014.6892800 · Cited by 6
Jony, R. I. (2023) · Queensland University of Technology · doi:10.5204/thesis.eprints.238058 · Cited by 9
A hypothesize-and-verify framework for text recognition using deep recurrent neural networks
Ray, A., Rajeswar, S., & Chaudhury, S. (2015) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2015.7333899 · Cited by 6
Biondi, G., Franzoni, V., & Milani, A. (2022) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat55865.2022.00106 · Cited by 5
Multimodal Emotion Recognition
Abhang, P. A., Gawali, B. W., & Mehrotra, S. C. (2016) · Elsevier eBooks · doi:10.1016/b978-0-12-804490-2.00006-3 · Cited by 4
FedMEKT: Distillation-Based Embedding Knowledge Transfer for Multimodal Federated LearningPreprint
Le, H., Nguyen, M. N. H., Thwal, C. M., Qiao, Y., Zhang, C., & Hong, C. S. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4641403 · Cited by 6
Performance Analysis of Unimodal and Multimodal Models in Valence-Based Empathy Recognition
Mallol-Ragolta, A., Schmitt, M., Baird, A., Cummins, N., & Schuller, B. W. (2019) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2019.8756517 · Cited by 5
Self-citationAdaptive Graph Learning with Multimodal Fusion for Emotion Recognition in Conversation
Liu, J., Li, J., Dong, J., Mo, Z., Liu, N., Li, Q., & Yuan, Y. (2025) · Biomimetics · doi:10.3390/biomimetics10070414 · Cited by 10
Lang, C., Wachsmuth, S., Hanheide, M., & Wersing, H. (2013) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra.2013.6630572 · Cited by 8
Smart and Context-Aware System employing Emotions Recognition
Sehgal, S., Sharma, H., & Anand, A. (2021) · International Conference for Emerging Technology (INCET) · doi:10.1109/incet51464.2021.9456356 · Cited by 6
Real-Time Activity Detection in a Multi-Talker Reverberated Environment
Principi, E., Rotili, R., Wöllmer, M., Eyben, F., Squartini, S., & Schuller, B. W. (2012) · Cognitive Computation · doi:10.1007/s12559-012-9133-8 · Cited by 4
Self-citationMultimodal Approach for Emotion Recognition Using an Algebraic Representation of Emotional States
Meftah, I. T., Thanh, N. L., & Amar, C. B. (2012) · International Conference on Signal-Image Technology & Internet-Based Systems (SITIS) · doi:10.1109/sitis.2012.84 · Cited by 7
Adaptive facial expression recognition using inter-modal top-down context
Sarvadevabhatla, R. K., Benovoy, M., Musallam, S., & Ng‐Thow‐Hing, V. (2011) · INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION · doi:10.1145/2070481.2070488 · Cited by 5
Linking emotions to behaviors through deep transfer learningPreprint
Li, H., Baucom, B., & Georgiou, P. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.03641 · Cited by 4
Credit Card Fraud Detection Using Deep Learning Approach (LSTM) Under IoT Environment
Tayeb, B., Amine, A., Hamou, R. M., & Kumar, A. V. S. (2022) · International Journal of Organizational and Collective Intelligence · doi:10.4018/ijoci.305207 · Cited by 5
A Transparent Framework towards the Context-Sensitive Recognition of Conversational Engagement.
Heimerl, A., Baur, T., & André, E. (2020) · OPUS (Augsburg University) · Cited by 8
Multimodal recognition of emotions using a formal computational model
Meftah, I. T., Thanh, N. L., & Amar, C. B. (2012) · World Conference on Complex Systems · doi:10.1109/icocs.2012.6458511 · Cited by 4
Semi-supervised context adaptation: case study of audience excitement recognition
Vildjiounaite, E., Kyllönen, V., Mäkelä, S.-M., Vuorinen, O., Keränen, T., Peltola, J., & Gimel’farb, G. (2012) · Multimedia Systems · doi:10.1007/s00530-012-0257-1 · Cited by 4
Cazau, D., Duc, P. N. H., Druon, J., Matwins, S., & Fablet, R. (2021) · Machine Learning · doi:10.1007/s10994-021-06029-z · Cited by 6
Log Message Anomaly Detection and Classification Using Auto-B/LSTM and Auto-GRUPreprint
Farzad, A., & Gulliver, T. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.08744 · Cited by 22
MuBeFE: Multimodal Behavioural Features Extraction Method
D’Andrea, A., Caschera, M. C., Ferri, F., & Grifoni, P. (2021) · JUCS - Journal of Universal Computer Science · doi:10.3897/jucs.66375 · Cited by 3
Facial expression as an input annotation modality for affective speech-to-speech translation
Székely, É., Ahmed, Z. A., Steiner, I., & Carson-Berndsen, J. (2012) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 3
Kim, Y. (2015) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2015.7344653 · Cited by 4
Singhal, A., & Sharma, D. K. (2024) · Computer Systems Science and Engineering · doi:10.32604/csse.2023.046730 · Cited by 3
Multimodal user state and trait recognition: an overview
Schuller, B. W. (2018) · ACM eBooks · doi:10.1145/3107990.3107997 · Cited by 7
Self-citationComputational representation and analysis of emotion dynamics
Hakim, A., Marsland, S., & Guesgen, H. W. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-12490-2 · Cited by 3
Computational Assessment of Interest in Speech—Facing the Real-Life Challenge
Wöllmer, M., Weninger, F. J., Eyben, F., & Schuller, B. W. (2011) · Künstliche Intell. · doi:10.1007/s13218-011-0108-9 · Cited by 4
Self-citationBasha, S. A. K., & Vincent, P. M. D. R. (2024) · Journal of Advances in Information Technology · doi:10.12720/jait.15.7.853-861 · Cited by 4
Continuous emotion recognition using a particle swarm optimized NARX neural network
Banda, N., Engelbrecht, A. P., & Robinson, P. (2015) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2015.7344599 · Cited by 3
Rázuri, J. G. (2015) · DiVA (Stockholm University) · Cited by 4
Raju, J., Gaus, Y. F. A., & Breckon, T. P. (2021) · International Conference on Machine Learning and Applications · doi:10.1109/icmla52953.2021.00115 · Cited by 5
Error Reduction Network for DBLSTM-based Voice Conversion
(2018) · doi:10.48550/arxiv.1809.09841 · Cited by 3
GME-Dialogue-NET: Gated Multi-modal Sentiment Analysis Model Based on Fusion Mechanism
Yang, M. (2021) · Academic Journal of Computing & Information Science · doi:10.25236/ajcis.2021.040803 · Cited by 2
Analytical Review and Study on Emotion Recognition Strategies Using Multimodal Signals
Vala, J., & Jaliya, U. K. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-23092-9_21 · Cited by 3
Towards Interactive Agents that Infer Emotions from Voice and Context Information
Formolo, D., & Bosse, T. (2017) · EAI Endorsed Transactions on Creative Technologies · doi:10.4108/eai.4-9-2017.153054 · Cited by 3
Multi-Task Semi-Supervised Adversarial Autoencoding for Speech EmotionPreprint
Rana, R. K., Latif, S., Khalifa, S., Jurdak, R., & Epps, J. (2019) · arXiv (Cornell University) · Cited by 2
Polignano, M., de Gemmis, M., & Semeraro, G. M. (2019) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s19-2056 · Cited by 2
Lightweight Adaptation of Classifiers to Users and Contexts: Trends of the Emerging Domain
Vildjiounaite, E., Gimel’farb, G., Kyllönen, V., & Peltola, J. (2015) · The Scientific World JOURNAL · doi:10.1155/2015/434826 · Cited by 3
Sun, Z., Li, Y., Liu, H., Li, H., & Zhang, W. (2025) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2025.061920 · Cited by 1
Multimodal Dimensional and Continuous Emotion Recognition in Dyadic Video Interactions
Zhao, J., Chen, S., & Jin, Q. (2018) · Lecture notes in computer science · doi:10.1007/978-3-030-00776-8_28 · Cited by 2
RBi-LSTM Based CNN Model for Speech Emotion Recognition
Padhira, K. R., Gundla, N. R., Kumar, K., Sarangi, P. K., & Jabbar, M. A. (2025) · International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions) (ICRITO) · doi:10.1109/icrito66076.2025.11241506 · Cited by 1
FSE: a Powerful Feature Augmentation Technique for Classification Task
Liu, Y., Yang, Y., & Hossain, M. Z. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-92270-2_55 · Cited by 1
Vala, J., & Jaliya, U. K. (2023) · International Journal of Engineering Applied Sciences and Technology · doi:10.33564/ijeast.2023.v08i04.021 · Cited by 1
Deep learning approaches in face analysis
Çakır, D., Akay, S., & Arıca, N. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-822314-7.00010-9 · Cited by 1
Pikramenos, G., Kechagias, K., Psallidas, T., Smyrnis, G., Spyrou, E., & Perantonis, S. J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-66125-0_3 · Cited by 2
Vejvar, M. (2020) · Advances in intelligent systems and computing · doi:10.1007/978-3-030-63322-6_14
Multimodal Sentiment Analysis Data Sets and Preprocessing
Xu, H. (2023) · Springer eBooks · doi:10.1007/978-981-99-5776-7_2 · Cited by 3
Classification of Emotions Using a 2-Channel Convolution Neural Network
Mahajan, A., Singh, V., Srivastav, R., Kapoor, S., & Singh, E. (2021) · International Conference on Signal Processing and Integrated Networks (SPIN) · doi:10.1109/spin52536.2021.9566010 · Cited by 1
Tri-Modalities Fusion for Multimodal Sentiment Analysis
Wang, H., Liu, S., Zhao, D., Chen, Y., & Ji, W. (2023) · Asian Conference on Artificial Intelligence Technology (ACAIT) · doi:10.1109/acait60137.2023.10528533
Analyzing Post-Shopping Facial Expressions: Unraveling Emotions for Enhanced Consumer Insights
Rana, A., Goswami, A., Mazumder, S., Maji, P., Mondal, S. K., & Samanta, R. K. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-75157-8_10
Global-Local Fusion Transformer with Adaptive Window for Speech Emotion Recognition
Yang, X., Huang, H., Fan, Y., & Huang, K. (2026) · Circuits Systems and Signal Processing · doi:10.1007/s00034-026-03638-8
RESEARCH ON REAL TIME EMOTION RECOGNITION USING DIGITAL IMAGE PROCESSING USING ML
(2024) · International Research Journal of Modernization in Engineering Technology and Science · doi:10.56726/irjmets52043
Affective rating ranking based on face images in arousal-valence dimensional space
Xu, G., Lu, H.-T., Zhang, F., & Mao, Q. (2018) · Frontiers of Information Technology & Electronic Engineering · doi:10.1631/fitee.1700270
EFCC-IeT: Cross-Modal Electronic File Content Correlation via Image-Enhanced Text
Jing, P., Liu, J., Liu, C., et al. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-5492-2_17
Artificial Intelligence as a Support for Social Science Research
Nagy, K. K., & Ribni, F. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-18891-5_15
Investigating the Impact of Parameter Variation on Speech Emotion Analysis Using LSTM Models
Odedra, R., Bhoi, D., Bhatt, N., & Thakkar, A. R. (2025) · Lecture notes in electrical engineering · doi:10.1007/978-981-96-4391-2_8
Duc, P. N. H. (2020) · theses.fr (ABES) · doi:10.70675/17b51e87z4d64z4a42z834cz06bdd762a37a · Cited by 6
HERF: A Machine Learning Framework for Automatic Emotion Recognition from Audio
Basha, S. A. K., & Vincent, P. M. D. R. (2024) · Revue d intelligence artificielle · doi:10.18280/ria.380203
Application of Emotion Analysis in Deep Learning Techniques
Uysal, M., Demiral, M. F., & IŞIK, A. H. (2025) · DÜMF Mühendislik Dergisi · doi:10.24012/dumf.1757225
Schuller, B. W. (2013) · Signals and communication technology · doi:10.1007/978-3-642-36806-6_13
Self-citationSpeech Emotion Recognition Using Deep Learning Algorithm on RAVDESS Dataset
Barsainyan, N., & Singh, D. K. (2024) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-9554-7_33
Literature Survey and Datasets
Poria, S., Hussain, A., & Cambria, E. (2018) · Socio-affective computing · doi:10.1007/978-3-319-95020-4_3
Development of Techniques for Speech Emotion Recognition (SER) In the Context of Deep Learning
Triandi, B., Mawengkang, H., Efendi, S., & Syawaluddin (2022) · International Conference on Cybernetics and Intelligent System (ICORIS) · doi:10.1109/icoris56080.2022.10031385
Cohorte de réseaux de neurones récurrents pour la reconnaissance de l'écriture
Stuner, B. (2018) · theses.fr (ABES) · doi:10.70675/9c47340dzeec5z415bz8c85zf3a545a9c11e
Metadata
- Authors
- Wöllmer, M., Metallinou, A., Eyben, F., Schuller, B., & Narayanan, S. S.
- Affiliations
- Technical University of Munich (DE)
- University of Southern California (US)
- Venue
- Interspeech 2010
- Publisher
- ISCA
- Published
- 2010-09-26
- Pages
- 2362-2365
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W2149940198
- Semantic Scholar
- 923c20f9eacd1d76d95f5528654c9578906aa09b
- Cited by
- 201
- References
- 14
- Citations by year
- 2026: 3 · 2025: 7 · 2024: 12 · 2023: 13 · 2022: 20 · 2021: 17 · 2020: 18 · 2019: 20 · 2018: 19 · 2017: 10 · 2016: 8 · 2015: 14 · 2014: 4 · 2013: 10 · 2012: 17
- FWCI
- 21.93
- Type
- conference paper
- Language
- en
- Open access
- green
- mediatum.ub.tum.de
- Licence
- other-oa
- Locations
- Interspeech (USB) (publishedVersion)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion, other-oa)
- OPUS (Augsburg University) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- multimodal emotion recognition, bidirectional LSTM, feature-level fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.