Error Weighted Semi-Coupled Hidden Markov Model for Audio-Visual Emotion Recognition
The DOI matches its Crossref record; 75 of the 80 references it cites were found in a registry or an index.
Abstract
This paper presents an approach to the automatic recognition of human emotions from audio-visual bimodal signals using an error weighted semi-coupled hidden Markov model (EWSC-HMM). The proposed approach combines an SC-HMM with a state-based bimodal alignment strategy and a Bayesian classifier weighting scheme to obtain the optimal emotion recognition result based on audio-visual bimodal fusion. The state-based bimodal alignment strategy in SC-HMM is proposed to align the temporal relation between audio and visual streams. The Bayesian classifier weighting scheme is then adopted to explore the contributions of the SC-HMM-based classifiers for different audio-visual feature pairs in order to obtain the emotion recognition output. For performance evaluation, two databases are considered: the MHMC posed database and the SEMAINE naturalistic database. Experimental results show that the proposed approach not only outperforms other fusion-based bimodal emotion recognition methods for posed expressions but also provides satisfactory results for naturalistic expressions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Emotions precipitate physiological changes that directly impact speech production and, thus, significantly affect speech intonation [36].”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [36]
In the citing paper
“According to previous research findings, compared with neutral speech, the first formant frequency of happy and angry speech is slightly higher, while the first formant frequency of sad speech is significantly lower [36,42].”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [36]
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Rapid object detection using a boosted cascade of simple features
Viola, P., & Jones, M. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990517 · Cited by 18,300
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 1×
“Fleiss’ kappa can work for any number of raters giving categorical ratings to a fixed number of items [46].”
BackgroundPraat: Doing Phonetics by Computer
(2011) · Ear & Hearing · doi:10.1097/aud.0b013e31821473f7 · Cited by 7,863
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.927467 · Cited by 5,409
Handbook of Affective Sciences
(2002) · doi:10.1093/oso/9780195126013.001.0001 · Cited by 5,073
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · cited 2×
“scenario [40], [41] was developed from the ELIZA concept introduced by Weizenbaum [42]; it can be used to build a naturalistic emotion database through machine agent-human conversational elements [41].”
MethodEkman, P. (1993) · American Psychologist · doi:10.1037/0003-066x.48.4.384 · Cited by 3,441
Looking at pictures: Affective, facial, visceral, and behavioral reactions
LANG, P. J., GREENWALD, M. K., BRADLEY, M. M., & HAMM, A. O. (1993) · Psychophysiology · doi:10.1111/j.1469-8986.1993.tb03352.x · Cited by 3,190
Research works printed but not linked: 7
- brand, 1997, Coupled Hidden Markov Models for Modeling Interacting Processes MIT Media Lab Perceptual Computing/Learning and Common Sense Technical Report
- lucey, 2007, Face recognition, 275
- ivanov, 2005, Technical Report CBCL paper 258 Massachusetts Institute of Technology
- 1998, International Organization for Standardization Facial animation parameters Text for ISO/IEC FDIS 144962 Visual ISO/IET JTC1/SC29/WG11 N2502
- 10.1037/0033-2909.111.2.256 · DOI · Paper page
- ekman, 1978, The Facial Action Coding System A Technique for The Measurement of Facial Movement
- douglas-cowie, 2008, The sensitive artifical listener: An induction technique for generating emotionally coloured conversation, Proc Program Workshop Corpora for Research on Emotion and Affect
Cited by
Items 101–121 of 121
Riviello, M. T., & Esposito, A. A. (2016) · SpringerBriefs in cognitive computation · doi:10.1007/978-94-024-0887-4_1 · Cited by 1
Interaction Style Recognition Based on Multi-Layer Multi-View Profile Representation
Wei, W.-L., Lin, J., & Wu, C. (2016) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2553024 · Cited by 1
Self-citationComputing with Words Model for Emotion Recognition Using Interval Type-2 Fuzzy Sets
Kar, R., Halder, A., Konar, A., Chakraborty, A., & Nagar, A. K. (2015) · Advances in computational intelligence and robotics book series · doi:10.4018/978-1-4666-7278-9.ch014 · Cited by 1
Chiba, Y., Nose, T., & Ito, A. (2017) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-017-0238-y · Cited by 1
Human Emotion Interpreter: A Proposed Multi-dimension System for Emotion Recognition
Hamdy, S. (2017) · Lecture notes in networks and systems · doi:10.1007/978-3-319-56994-9_63 · Cited by 4
Literature Survey on Emotion Recognition for Social Signal Processing
Vijayalakshmi, A., & Mohanaiah, P. (2019) · Lecture notes in electrical engineering · doi:10.1007/978-981-15-0626-0_29 · Cited by 3
A primer on coupled state-switching models for multiple interacting time series
Pohle, J., Langrock, R., van der Schaar, M., King, R., & Jensen, F. H. (2020) · Statistical Modelling · doi:10.1177/1471082x20956423 · Cited by 1
Locality Preserved Joint Dictionary and Classifier Learning for Speech Emotion Recognition
Mathulaprangsan, S., & Sateanpattanakul, S. (2020) · Joint International Conference on Digital Arts, Media and Technology with ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI DAMT & NCON) · doi:10.1109/ectidamtncon48261.2020.9090708 · Cited by 1
Early detection of cardiac arrhythmia based on Bayesian methods from ECG data
Ghahjaverestan, N. M. (2015) · theses.fr (ABES) · doi:10.70675/7b7f29b1ze2e2z4507z8984za8cc85fcb4ee · Cited by 2
Multimodal Affect Recognition Using Boltzmann Zippers
Lu, K., & ZHANG, X. (2013) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.e96.d.2496
Audio-Visual Speech Emotion Recognition
Akputu, O. K., Seng, K. P., & Lee, Y. (2014) · Advances in information quality and management · doi:10.4018/978-1-4666-5888-2.ch011
Literature Survey and Datasets
Poria, S., Hussain, A., & Cambria, E. (2018) · Socio-affective computing · doi:10.1007/978-3-319-95020-4_3
Two-Way Affective Modeling for Hidden Movie Highlights’ Extraction
Wang, Z., Yan, X., Jiang, W., & Sun, M. (2018) · Sensors · doi:10.3390/s18124241
Locality Preserving Discriminative Complex-Valued Latent Variable Model
Chen, S.-H., Lee, Y.-S., & Wang, J. (2018) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2018.8545436
Locality Preserved Joint Nonnegative Matrix Factorization for Speech Emotion Recognition
Mathulaprangsan, S., Lee, Y.-S., & Wang, J. (2019) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2018dal0002
Development of User-friendly Smart Grid ArchitecturePreprint
Mishra, S. R. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.13677
A Review of Multimodal Sentiment Analysis for Measuring Visual Information
孙, 睿. (2023) · Statistics and Applications · doi:10.12677/sa.2023.121015
Feature parameters extraction and affective computing of voice message for social media environment
Jiang, P., Guo, C., & Dai, Y. (2023) · Computer Science and Information Systems · doi:10.2298/csis230509066j
A Comprehensive Investigation Into Speech Emotion Recognition Using Deep Learning
Zahoor, E. T. K. A. (2023) · Journal of Propulsion Technology · doi:10.52783/tjjpt.v44.i4.1097
Smart Cities as Affective Environments
Urbanik, P., & Mikulecký, P. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-73497-7_13
Wagh, N. R., Yadav, A. S., Lenka, R. M., Rane, S., Kulkarni, R. V., Vyas, B. K., & Yadav, S. T. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01461-3
Metadata
- Authors
- Lin, J.-C., Wu, C.-H., & Wei, W.-L.
- Affiliations
- National Cheng Kung University (TW)
- Venue
- IEEE Transactions on Multimedia
- Published
- 2012-02
- Volume
- 14
- Issue
- 1
- Pages
- 142-156
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W2038821742
- Cited by
- 121
- References
- 80
- Citations by year
- 2025: 2 · 2024: 5 · 2023: 11 · 2022: 7 · 2021: 11 · 2020: 9 · 2019: 7 · 2018: 12 · 2017: 13 · 2016: 10 · 2015: 10 · 2014: 9 · 2013: 14 · 2012: 1
- FWCI
- 9.88
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Multimedia (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- audio-visual emotion recognition, bimodal fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.