Error Weighted Semi-Coupled Hidden Markov Model for Audio-Visual Emotion Recognition
The DOI matches its Crossref record; 75 of the 80 references it cites were found in a registry or an index.
Abstract
This paper presents an approach to the automatic recognition of human emotions from audio-visual bimodal signals using an error weighted semi-coupled hidden Markov model (EWSC-HMM). The proposed approach combines an SC-HMM with a state-based bimodal alignment strategy and a Bayesian classifier weighting scheme to obtain the optimal emotion recognition result based on audio-visual bimodal fusion. The state-based bimodal alignment strategy in SC-HMM is proposed to align the temporal relation between audio and visual streams. The Bayesian classifier weighting scheme is then adopted to explore the contributions of the SC-HMM-based classifiers for different audio-visual feature pairs in order to obtain the emotion recognition output. For performance evaluation, two databases are considered: the MHMC posed database and the SEMAINE naturalistic database. Experimental results show that the proposed approach not only outperforms other fusion-based bimodal emotion recognition methods for posed expressions but also provides satisfactory results for naturalistic expressions.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Emotions precipitate physiological changes that directly impact speech production and, thus, significantly affect speech intonation [36].”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [36]
In the citing paper
“According to previous research findings, compared with neutral speech, the first formant frequency of happy and angry speech is slightly higher, while the first formant frequency of sad speech is significantly lower [36,42].”
From A Survey of Deep Learning-Based Multimodal Emotion Recognition: Speech, Text, and Face (2023) · cites this paper as [36]
Citations
Cites
The Measurement of Observer Agreement for Categorical Data
Landis, J. R., & Koch, G. G. (1977) · Biometrics · doi:10.2307/2529310 · Cited by 81,148
Rapid object detection using a boosted cascade of simple features
Viola, P., & Jones, M. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990517 · Cited by 18,300
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 1×
“Fleiss’ kappa can work for any number of raters giving categorical ratings to a fixed number of items [46].”
BackgroundPraat: Doing Phonetics by Computer
(2011) · Ear & Hearing · doi:10.1097/aud.0b013e31821473f7 · Cited by 7,863
Picard, R. W. (1997) · MIT Press · doi:10.7551/mitpress/1140.001.0001 · Cited by 7,382
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.927467 · Cited by 5,409
Handbook of Affective Sciences
(2002) · doi:10.1093/oso/9780195126013.001.0001 · Cited by 5,073
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452 · cited 2×
“scenario [40], [41] was developed from the ELIZA concept introduced by Weizenbaum [42]; it can be used to build a naturalistic emotion database through machine agent-human conversational elements [41].”
MethodEkman, P. (1993) · American Psychologist · doi:10.1037/0003-066x.48.4.384 · Cited by 3,441
Looking at pictures: Affective, facial, visceral, and behavioral reactions
LANG, P. J., GREENWALD, M. K., BRADLEY, M. M., & HAMM, A. O. (1993) · Psychophysiology · doi:10.1111/j.1469-8986.1993.tb03352.x · Cited by 3,190
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
ELIZA — a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1983) · Communications of the ACM · doi:10.1145/357980.357991 · Cited by 2,738
Emotion recognition in human-computer interaction
Cowie, R., Douglas-Cowie, E., Tsapatsoulis, N., Votsis, G., Kollias, S., Fellenz, W., & Taylor, J. G. (2001) · IEEE Signal Processing Magazine · doi:10.1109/79.911197 · Cited by 2,603
Emotion and motivation I: Defensive and appetitive reactions in picture processing.
Bradley, M. M., Codispoti, M., Cuthbert, B. N., & Lang, P. J. (2001) · Emotion · doi:10.1037/1528-3542.1.3.276 · Cited by 2,224 · cited 1×
“These studies have focused on the use of facial expressions [32] and static pictures [33]–[35] to show differences in some physiological signals between different discrete emotions.”
BackgroundAssessing Agreement on Classification Tasks: The Kappa Statistic
Carletta, J. (1996) · arXiv · doi:10.48550/arxiv.cmp-lg/9602004 · Cited by 2,088 · cited 1×
“Kappa statistic was widely used for measuring the amount of agreement among the raters [45].”
MethodThin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis.
Ambady, N., & Rosenthal, R. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.111.2.256 · Cited by 1,922
Recognizing action units for facial expression analysis
Tian, Y.-I., Kanade, T., & Cohn, J. F. (2001) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.908962 · Cited by 1,745 · cited 1×
“In previous studies [2]–[7], emotion recognition focused only on considering single facial or vocal modality.”
BackgroundCoupled hidden Markov models for complex action recognition
Brand, M., Oliver, N., & Pentland, A. (2002) · Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.1997.609450 · Cited by 1,052
Toward detecting emotions in spoken dialogs
Lee, C. M., & Narayanan, S. S. (2005) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.2004.838534 · Cited by 988
Mehrabian, A. (2017) · communication theory · doi:10.4324/9781315080918-15 · Cited by 879
Facial expression recognition from video sequences: temporal and static modeling
Cohen, I., Sebe, N., Garg, A., Chen, L. S., & Huang, T. S. (2003) · Computer Vision and Image Understanding · doi:10.1016/s1077-3142(03)00081-x · Cited by 874
Analysis of emotion recognition using facial expressions, speech and multimodal information
Busso, C., Deng, Z., Yildirim, S., Bulut, M., Lee, C. M., Kazemzadeh, A., Lee, S., Neumann, U., & Narayanan, S. (2004) · Proceedings of the 6th international conference on Multimodal interfaces · doi:10.1145/1027933.1027968 · Cited by 840 · cited 2×
“In terms of decision-level fusion [14], [17]–[20], multiple signals can be modeled by the corresponding classifier first and then the recognition results from each classifier are fused in the end.”
BackgroundMethodEmotional category data on images from the international affective picture system
Mikels, J. A., Fredrickson, B. L., Larkin, G. R., Lindberg, C. M., Maglio, S. J., & Reuter-Lorenz, P. A. (2005) · Behavior Research Methods · doi:10.3758/bf03192732 · Cited by 677
Describing the emotional states that are expressed in speech
Cowie, R., & Cornelius, R. R. (2003) · Speech Communication · doi:10.1016/s0167-6393(02)00071-7 · Cited by 604
Hidden Markov model-based speech emotion recognition
Schuller, B., Rigoll, G., & Lang, M. (2003) · 2003 International Conference on Multimedia and Expo. ICME '03. Proceedings (Cat. No.03TH8698) · doi:10.1109/icme.2003.1220939 · Cited by 570
Mechrabian, A. (1968) · Medical Entomology and Zoology · Cited by 459
Emotion recognition in human–computer interaction
Fragopanagos, N., & Taylor, J. G. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.03.006 · Cited by 414
Emotion recognition by speech signals
Kwon, O.-W., Chan, K., Hao, J., & Lee, T.-W. (2003) · 8th European Conference on Speech Communication and Technology (Eurospeech 2003) · doi:10.21437/eurospeech.2003-80 · Cited by 366
EMOTION RECOGNITION BY SPEECH SIGNAL
Kwon, O., Chan, K.-H., & Hao, J. (2003) · Interspeech (USB) · Cited by 338
Douglas-Cowie, E., Cowie, R., Sneddon, I., Cox, C., Lowry, O., McRorie, M., Martin, J.-C., Devillers, L., Abrilian, S., Batliner, A., et al. (2007) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-74889-2_43 · Cited by 321
Ensemble methods for spoken emotion recognition in call-centres
Morrison, D., Wang, R., & De Silva, L. C. (2007) · Speech Communication · doi:10.1016/j.specom.2006.11.004 · Cited by 320
The SEMAINE corpus of emotionally coloured character interactions
McKeown, G., Valstar, M. F., Cowie, R., & Pantic, M. (2010) · 2010 IEEE International Conference on Multimedia and Expo · doi:10.1109/icme.2010.5583006 · Cited by 278
Hidden Markov model-based speech emotion recognition
Schuller, B., Rigoll, G., & Lang, M. (2003) · 2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03). · doi:10.1109/icassp.2003.1202279 · Cited by 278
Wu, C.-H., & Liang, W.-B. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2010.16 · Cited by 266
Self-citationSpeech emotion recognition based on HMM and SVM
Lin, Y.-L., & Wei, G. (2005) · 2005 International Conference on Machine Learning and Cybernetics · doi:10.1109/icmlc.2005.1527805 · Cited by 260
Characterization of the Affective Norms for English Words by discrete emotional categories
Stevenson, R. A., Mikels, J. A., & James, T. W. (2007) · Behavior Research Methods · doi:10.3758/bf03192999 · Cited by 251
Emotion recognition from text using semantic labels and separable mixture models
Wu, C.-H., Chuang, Z.-J., & Lin, Y.-C. (2006) · ACM Transactions on Asian Language Information Processing · doi:10.1145/1165255.1165259 · Cited by 242
Self-citationEmotion recognition through facial expression analysis based on a neurofuzzy network
Ioannou, S. V., Raouzaiou, A. T., Tzouvaras, V. A., Mailis, T. P., Karpouzis, K. C., & Kollias, S. D. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.03.004 · Cited by 242
Recognition of facial expressions and measurement of levels of interest from video
Yeasin, M., Bullot, B., & Sharma, R. (2006) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2006.870737 · Cited by 235
Automatic emotional speech classification
Ververidis, D., Kotropoulos, C., & Pitas, I. (2004) · 2004 IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.2004.1326055 · Cited by 221
A coupled HMM for audio-visual speech recognition
Nefian, A. V., Liang, L., Pi, X., Xiaoxiang, L., Mao, C., & Murphy, K. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2002.5745027 · Cited by 217
Manifold based analysis of facial expression
Chang, Y., Hu, C., Feris, R., & Turk, M. (2006) · Image and Vision Computing · doi:10.1016/j.imavis.2005.08.006 · Cited by 215
Recognizing Human Emotional State From Audiovisual Signals
Wang, Y., & Guan, L. (2008) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2008.921734 · Cited by 214
Autonomic Specificity and Emotion
Levenson, R. W. (2002) · Handbook of Affective Sciences · doi:10.1093/oso/9780195126013.003.0011 · Cited by 207
Emotion Recognition Based on Joint Visual and Audio Cues
Sebe, N., Cohen, I., Gevers, T., & Huang, T. S. (2006) · 18th International Conference on Pattern Recognition (ICPR'06) · doi:10.1109/icpr.2006.489 · Cited by 202
Recognizing Human Emotional State From Audiovisual Signals*
Wang, Y., & Guan, L. (2008) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2008.927665 · Cited by 202
Face and 2-D mesh animation in MPEG-4
Tekalp, A. M., & Ostermann, J. (2000) · Signal Processing: Image Communication · doi:10.1016/s0923-5965(99)00055-7 · Cited by 200
Talking Machines: Theories, Models, and Designs
Day, E. T., Bailly, G., Benoı̂t, C., & Sawallis, T. R. (1992) · Medical Entomology and Zoology · Cited by 170
Trainable method of parametric shape description
Cootes, T. F., Cooper, D. H., Taylor, C. J., & Graham, J. (1992) · Image and Vision Computing · doi:10.1016/0262-8856(92)90044-4 · Cited by 165
Investigating Spontaneous Facial Action Recognition through AAM Representations of the Face
Lucey, S., Bilal, A., & F., J. (2007) · Face Recognition · doi:10.5772/4841 · Cited by 161
Exploring Fusion Methods for Multimodal Emotion Recognition with Missing Data
Wagner, J., Andre, E., Lingenfelser, F., & Kim, J. (2011) · IEEE Transactions on Affective Computing · doi:10.1109/t-affc.2011.12 · Cited by 156
Stress and Emotion Classification using Jitter and Shimmer Features
Li, X., Tao, J., Johnson, M. T., Soltis, J., Savage, A., Leong, K. M., & Newman, J. D. (2007) · 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07 · doi:10.1109/icassp.2007.367261 · Cited by 144
Automatic emotion recognition using prosodic parameters
Luengo, I., Navas, E., Hernáez, I., & Sánchez, J. (2005) · Interspeech 2005 · doi:10.21437/interspeech.2005-324 · Cited by 141
Audio-Visual Affect Recognition
Zeng, Z., Tu, J., Liu, M., Huang, T. S., Pianfetti, B., Roth, D., & Levinson, S. (2007) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2006.886310 · Cited by 136
Parameterized Facial Expression Synthesis Based on MPEG-4
Raouzaiou, A., Tsapatsoulis, N., Karpouzis, K., & Kollias, S. (2002) · EURASIP Journal on Advances in Signal Processing · doi:10.1155/s1110865702206149 · Cited by 111
Probabilistic combination of multiple modalities to detect interest
Kapoor, A., Picard, R. W., & Ivanov, Y. (2004) · Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004. · doi:10.1109/icpr.2004.1334690 · Cited by 99
Audio-Visual Emotion Recognition Using Gaussian Mixture Models for Face and Voice
Metallinou, A., Lee, S., & Narayanan, S. (2008) · 2008 Tenth IEEE International Symposium on Multimedia · doi:10.1109/ism.2008.40 · Cited by 96
Real-time Emotion Detection System using Speech: Multi-modal Fusion of Different Timescale Features
Kim, S., Georgiou, P., Lee, S., & Narayanan, S. S. (2007) · IEEE International Workshop on Multimedia Signal Processing (MMSP) · doi:10.1109/mmsp.2007.4412815 · Cited by 92
Metallinou, A., Lee, S., & Narayanan, S. (2010) · 2010 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2010.5494890 · Cited by 89
Bimodal Fusion of Emotional Data in an Automotive Environment
Hoch, S., Althoff, F., McGlaun, G., & Rigoll, G. (2006) · Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005. · doi:10.1109/icassp.2005.1415597 · Cited by 87
Audiovisual recognition of spontaneous interest within conversations
Schuller, B., Müeller, R., Höernler, B., Höethker, A., Konosu, H., & Rigoll, G. (2007) · Proceedings of the 9th international conference on Multimodal interfaces · doi:10.1145/1322192.1322201 · Cited by 85
Proceedings of the 13th annual ACM international conference on Multimedia
(2005) · doi:10.1145/1101149 · Cited by 85
Modeling Naturalistic Affective States Via Facial, Vocal, and Bodily Expressions Recognition
Karpouzis, K. C., Caridakis, G., Kessous, L., Amir, N., Raouzaiou, A. T., Malatesta, L., & Kollias, S. (2007) · Springer eBooks · doi:10.1007/978-3-540-72348-6_5 · Cited by 84
A coupled HMM approach to video-realistic speech animation
Xie, L., & Liu, Z. (2007) · Pattern Recognition · doi:10.1016/j.patcog.2006.12.001 · Cited by 77
Speech Emotion Recognition Using Segmental Level Prosodic Analysis
Koolagudi, S. G., Kumar, N., & Rao, K. S. (2011) · International Conference on Devices and Communications · doi:10.1109/icdecom.2011.5738536 · Cited by 68
A robust multimodal approach for emotion recognition
Song, M., You, M., Li, N., & Chen, C. (2008) · Neurocomputing · doi:10.1016/j.neucom.2007.07.041 · Cited by 58
Ananthakrishnan, S., & Narayanan, S. B. (2006) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2005.1415102 · Cited by 49
Automatic tracking, coding and reconstruction ofhuman faces, using flexible appearance models
Lanitis, A., Taylor, C., & Cootes, T. F. (1994) · Electronics Letters · doi:10.1049/el:19941110 · Cited by 36
Facial Expression Recognition using AAM and Local Facial Features
Tang, F., & Deng, B. (2007) · International Conference on Natural Computation (ICNC) · doi:10.1109/icnc.2007.373 · Cited by 32
Audio-Visual Affect Recognition in Activation-Evaluation Space
Zeng, Z., Zhang, Z., Pianfetti, B., Tu, J., & Huang, T. S. (2005) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme.2005.1521551 · Cited by 32
Emotion Perception and Recognition from Speech
Wu, C., Yeh, J., & Chuang, Z.-J. (2008) · Springer eBooks · doi:10.1007/978-1-84800-306-4_6 · Cited by 29
Self-citation3D active appearance model for aligning faces in 2D images
Chen, C.-W., & Wang, C.-C. (2008) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2008.4650788 · Cited by 25
Error weighted classifier combination for multi-modal human identification
Ivanov, Y. A., Serre, T., & Bouvrie, J. (2005) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 19
Choi, H., & Oh, S. (2006) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/icsmc.2006.384939 · Cited by 13
3D surface inspection using coupled HMMs
Pernkopf, F. (2004) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr.2004.1334508 · Cited by 11
Research works printed but not linked: 7
- brand, 1997, Coupled Hidden Markov Models for Modeling Interacting Processes MIT Media Lab Perceptual Computing/Learning and Common Sense Technical Report
- lucey, 2007, Face recognition, 275
- ivanov, 2005, Technical Report CBCL paper 258 Massachusetts Institute of Technology
- 1998, International Organization for Standardization Facial animation parameters Text for ISO/IEC FDIS 144962 Visual ISO/IET JTC1/SC29/WG11 N2502
- 10.1037/0033-2909.111.2.256 · DOI · Paper page
- ekman, 1978, The Facial Action Coding System A Technique for The Measurement of Facial Movement
- douglas-cowie, 2008, The sensitive artifical listener: An induction technique for generating emotionally coloured conversation, Proc Program Workshop Corpora for Research on Emotion and Affect
Cited by
A review of affective computing: From unimodal analysis to multimodal fusion
Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017) · Information Fusion · doi:10.1016/j.inffus.2017.02.003 · Cited by 1,644
Akçay, M. B., & Oğuz, K. (2020) · Speech Communication · doi:10.1016/j.specom.2019.12.001 · Cited by 721
A systematic review on affective computing: emotion models, databases, and recent advances
Wang, Y., Song, W., Tao, W., Liotta, A., Yang, D., Li, X., Gao, S., Sun, Y., Ge, W., Zhang, W., et al. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.03.009 · Cited by 615
A Review and Meta-Analysis of Multimodal Affect Detection Systems
D'mello, S. K., & Kory, J. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 628
Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition
Zhang, S., Zhang, S., Huang, T., Gao, W., & Tian, Q. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2719043 · Cited by 346
“Lin et al. [8] employ an error weighted semi-coupled Hidden Markov Models (HMM) to fuse audio and visual streams for emotion recognition.”
Background
Metadata
- Authors
- Lin, J.-C., Wu, C.-H., & Wei, W.-L.
- Affiliations
- National Cheng Kung University (TW)
- Venue
- IEEE Transactions on Multimedia
- Published
- 2012-02
- Volume
- 14
- Issue
- 1
- Pages
- 142-156
- ISSN
- 1520-9210 (ISSN-L)
- 1941-0077
- OpenAlex
- W2038821742
- Cited by
- 121
- References
- 80
- Citations by year
- 2025: 2 · 2024: 5 · 2023: 11 · 2022: 7 · 2021: 11 · 2020: 9 · 2019: 7 · 2018: 12 · 2017: 13 · 2016: 10 · 2015: 10 · 2014: 9 · 2013: 14 · 2012: 1
- FWCI
- 9.88
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- IEEE Transactions on Multimedia (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Keywords
- audio-visual emotion recognition, bimodal fusion
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.