A Combined Rule-Based & Machine Learning Audio-Visual Emotion Recognition Approach
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
This paper proposes an audio-visual emotion recognition system that uses a mixture of rule-based and machine learning techniques to improve the recognition efficacy in the audio and video paths. The visual path is designed using the Bi-directional Principal Component Analysis (BDPCA) and Least-Square Linear Discriminant Analysis (LSLDA) for dimensionality reduction and discrimination. The extracted visual features are passed into a newly designed Optimized Kernel-Laplacian Radial Basis Function (OKL-RBF) neural classifier. The audio path is designed using a combination of input prosodic features (pitch, log-energy, zero crossing rates and Teager energy operator) and spectral features (Mel-scale frequency cepstral coefficients). The extracted audio features are passed into an audio feature level fusion module that uses a set of rules to determine the most likely emotion contained in the audio signal. An audio visual fusion module fuses outputs from both paths. The performances of the proposed audio path, visual path, and the final system are evaluated on standard databases. Experiment results and comparisons reveal the good performance of the proposed system.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Lucey, P., Cohn, J. F., Kanade, T., Saragih, J., Ambadar, Z., & Matthews, I. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops · doi:10.1109/cvprw.2010.5543262 · Cited by 4,303
A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions
Zeng, Z., Pantic, M., Roisman, G. I., & Huang, T. S. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.52 · Cited by 2,797
Facial expression recognition based on Local Binary Patterns: A comprehensive study
Shan, C., Gong, S., & McOwan, P. W. (2009) · Image and Vision Computing · doi:10.1016/j.imavis.2008.08.005 · Cited by 2,300
The eNTERFACE’05 Audio-Visual Emotion Database
Martin, O., Kotsia, I., Macq, B., & Pitas, I. (2006) · 22nd International Conference on Data Engineering Workshops (ICDEW'06) · doi:10.1109/icdew.2006.145 · Cited by 615 · cited 1×
“Finally, the performance of the full audio-visual emotion recognition system was evaluated on the standard audio-visual emotion databases eNTERFACE’05 [43] and RML [20].”
MethodZhi, R., Flierl, M., Ruan, Q., & Kleijn, W. B. (2011) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2010.2044788 · Cited by 260
Towards an intelligent framework for multimodal affective data analysis
Poria, S., Cambria, E., Hussain, A., & Huang, G.-B. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.10.005 · Cited by 236
Class-level spectral features for emotion recognition
Bitouk, D., Verma, R., & Nenkova, A. (2010) · Speech Communication · doi:10.1016/j.specom.2010.02.010 · Cited by 219
De Silva, L. C., & Ng, P. C. (2002) · Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580) · doi:10.1109/afgr.2000.840655 · Cited by 196
Cited by
Audio-Visual Emotion Recognition in Video Clips
Noroozi, F., Marjanovic, M., Njegus, A., Escalera, S., & Anbarjafari, G. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2713783 · Cited by 266
Chen, L., Wang, K., Li, M., Wu, M., Pedrycz, W., & Hirota, K. (2023) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2022.3150097 · Cited by 113
A hybrid latent space data fusion method for multimodal emotion recognition
Nemati, S., Rohani, R., Basiri, M. E., Abdar, M., Yen, N. Y., & Makarenkov, V. (2019) · IEEE Access · doi:10.1109/access.2019.2955637 · Cited by 98
“Sensor-level and feature-level fusion methods belong to the first category, while scorelevel, rank-level, and decision-level methods are known as the fusion after matching methods [12], [13].”
BackgroundMethodSemi-supervised Multi-modal Emotion Recognition with Cross-Modal Distribution Matching
Liang, J., Li, R., & Jin, Q. (2020) · Proceedings of the 28th ACM International Conference on Multimedia · doi:10.1145/3394171.3413579 · Cited by 73
Metadata
- Authors
- Seng, K. P., Ang, L.-M., & Ooi, C. S.
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2018-01-01
- Volume
- 9
- Issue
- 1
- Pages
- 3-13
- ISSN
- 1949-3045 (ISSN-L)
- 2371-9850
- OpenAlex
- W2474425654
- Cited by
- 90
- References
- 8
- FWCI
- 8.06
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Combined Rule-Based & Machine Learning Audio-Visual Emotion Recognition Approach. https://app.cochat.ai/papers/10.1109/taffc.2016.2588488
- The paper
- Seng, K. P., Ang, L.-M., & Ooi, C. S. (2018). A Combined Rule-Based & Machine Learning Audio-Visual Emotion Recognition Approach. IEEE Transactions on Affective Computing. https://doi.org/10.1109/taffc.2016.2588488
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.