K-Means Clustering-Based Kernel Canonical Correlation Analysis for Multimodal Emotion Recognition in Human–Robot Interaction
The DOI matches its Crossref record; 32 of the 33 references it cites were found in a registry or an index.
Abstract
In this article,K-meansclustering-based Kernel canonical correlation analysis algorithm is proposed for multimodal emotion recognition in human–robot interaction (HRI). The multimodal features (gray pixels; time and frequency domain) extracted from facial expression and speech are fused based on Kernel canonical correlation analysis.K-means clustering is used to select features from multiple modalities and reduce dimensionality. The proposed approach can improve the heterogenicity among different modalities and make multiple modalities complementary to promote multimodal emotion recognition. Experiments on two datasets, namely SAVEE and eNTERFACE‘05, are conducted to evaluate the accuracy of the proposed method. The results show that the proposed method produces good recognition rates that are higher than the ones produced by the methods withoutK-means clustering; more specifically, they are 2.77% higher in SAVEE and 4.7% higher in eNTERFACE‘05.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Davies, D. L., & Bouldin, D. W. (1979) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.1979.4766909 · Cited by 9,145 · cited 1×
“This must ensure that the data of the same category have similar characteristics [25].”
BackgroundSurvey on speech emotion recognition: Features, classification schemes, and databases
El Ayadi, M., Kamel, M. S., & Karray, F. (2011) · Pattern Recognition · doi:10.1016/j.patcog.2010.09.020 · Cited by 2,073
EmotionMeter: A Multimodal Framework for Recognizing Human Emotions
Zheng, W.-L., Liu, W., Lu, Y., Lu, B.-L., & Cichocki, A. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2797176 · Cited by 1,276 · cited 1×
“Feature-based fusion refers to combining emotion features extracted from multiple modalities to form a fusion feature for emotion recognition [16].”
BackgroundToward detecting emotions in spoken dialogs
Lee, C. M., & Narayanan, S. S. (2005) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.2004.838534 · Cited by 988
Muda, L., Begam, M. H., & Elamvazuthi, I. (2010) · arXiv (Cornell University) · doi:10.48550/arxiv.1003.4083 · Cited by 803
A Review and Meta-Analysis of Multimodal Affect Detection Systems
D'mello, S. K., & Kory, J. (2015) · ACM Computing Surveys · doi:10.1145/2682899 · Cited by 628
Spatial–Temporal Recurrent Neural Network for Emotion Recognition
Zhang, T., Zheng, W., Cui, Z., Zong, Y., & Li, Y. (2019) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2788081 · Cited by 595
pyAudioAnalysis: An Open-Source Python Library for Audio Signal Analysis
Giannakopoulos, T. (2015) · PLOS ONE · doi:10.1371/journal.pone.0144610 · Cited by 454
Learning Affective Features With a Hybrid Deep Model for Audio–Visual Emotion Recognition
Zhang, S., Zhang, S., Huang, T., Gao, W., & Tian, Q. (2018) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2017.2719043 · Cited by 346 · cited 1×
“In the emotion recognition of multimodal information, emotion feature extraction and multimodal fusion are the most important steps [7], [8].”
BackgroundQin, J., Fu, W., Gao, H., & Zheng, W. X. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2016.2526683 · Cited by 333
A Survey of Decision Fusion and Feature Fusion Strategies for Pattern Classification
Mangai, U., Samanta, S., Das, S., & Chowdhury, P. (2010) · IETE Technical Review · doi:10.4103/0256-4602.64604 · Cited by 302
Audio-Visual Emotion Recognition in Video Clips
Noroozi, F., Marjanovic, M., Njegus, A., Escalera, S., & Anbarjafari, G. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2713783 · Cited by 266
A shape- and texture-based enhanced Fisher classifier for face recognition
Liu, C., & Wechsler, H. (2001) · IEEE Transactions on Image Processing · doi:10.1109/83.913594 · Cited by 261
Speaker Identification and Verification by Combining MFCC and Phase Information
Nakagawa, S., Wang, L., & Ohtsuka, S. (2012) · IEEE Transactions on Audio, Speech, and Language Processing · doi:10.1109/tasl.2011.2172422 · Cited by 233
Appearance models based on kernel canonical correlation analysis
Melzer, T., Reiter, M., & Bischof, H. (2003) · Pattern Recognition · doi:10.1016/s0031-3203(03)00058-x · Cited by 225
Chen, L., Zhou, M., Su, W., Wu, M., She, J., & Hirota, K. (2018) · Information Sciences · doi:10.1016/j.ins.2017.10.044 · Cited by 196
Self-citationChen, Z., Liu, C., Ding, S. X., Peng, T., Yang, C., Gui, W., & Shardt, Y. A. W. (2021) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2020.2989708 · Cited by 159
Online Robot Teaching With Natural Human–Robot Interaction
Du, G., Chen, M., Liu, C., et al. (2018) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2018.2823667 · Cited by 113
An Improved Mixture of Probabilistic PCA for Nonlinear Data-Driven Process Monitoring
Zhang, J., Chen, H., Chen, S., & Hong, X. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2771229 · Cited by 100
High-Performance Pixel-Level Grasp Detection Based on Adaptive Grasping and Grasp-Aware Network
Wang, D., Liu, C., Chang, F., Li, N., & Li, G. (2021) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2021.3120474 · Cited by 100
Facial Expression Recognition Using Local Transitional Pattern on Gabor Filtered Facial Images
Ahsan, T., Jabid, T., & Chong, U. (2013) · IETE Technical Review · doi:10.4103/0256-4602.107339 · Cited by 93
A Combined Rule-Based & Machine Learning Audio-Visual Emotion Recognition Approach
Seng, K. P., Ang, L.-M., & Ooi, C. S. (2018) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2016.2588488 · Cited by 90
A Cumulative Canonical Correlation Analysis-Based Sensor Precision Degradation Detection Method
Chen, Z., Yang, C., Peng, T., et al. (2018) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2018.2873100 · Cited by 85
Chen, L., Zhou, M., Wu, M., She, J., Liu, Z., Dong, F., & Hirota, K. (2018) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2018.2809691 · Cited by 73
Self-citationDeb, S., & Dandapat, S. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2787717 · Cited by 62
ISLA: Temporal Segmentation and Labeling for Audio-Visual Emotion Recognition
Kim, Y., & Provost, E. M. (2017) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2017.2702653 · Cited by 55
Jiang, Q., Gao, F., Yan, X., et al. (2018) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2018.2860571 · Cited by 53
Zeng, X., Li, Z., Gao, W., Ren, M., Zhang, J., Li, Z., & Zhang, B. (2018) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2018.2793196 · Cited by 41
An Effective Subsuperpixel-Based Approach for Background Subtraction
Chen, Y., Sun, Z.-L., & Lam, K. (2019) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2019.2893824 · Cited by 35
FaRoC: Fast and Robust Supervised Canonical Correlation Analysis for Multimodal Omics Data
Mandal, A., & Maji, P. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2685625 · Cited by 31
Improving Speech Related Facial Action Unit Recognition by Audiovisual Information Fusion
Meng, Z., Han, S., Liu, P., & Tong, Y. (2018) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2018.2840090 · Cited by 27
K-Means Clustering-based Kernel Canonical Correlation Analysis for Multimodal Emotion Recognition
Chen, L., Wang, K., Wu, M., et al. (2020) · IFAC-PapersOnLine · doi:10.1016/j.ifacol.2020.12.2756 · Cited by 11
Self-citation
Research works printed but not linked: 1
- 10.1109/TCSVT.2017.2719043 · DOI · Paper page
Cited by
Wu, Y., Mi, Q., & Gao, T. (2025) · Biomimetics · doi:10.3390/biomimetics10070418 · Cited by 183
“The feature extraction stage involves extracting relevant features from different modalities, such as speech, visual, and text data [22].”
Emotion recognition based on brain-like multimodal hierarchical perception
Zhu, X., Huang, Y., Wang, X., & Wang, R. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-17347-w · Cited by 142
Toward Explainable Affective Computing: A Review
Cortiñas-Lorenzo, K., & Lacey, G. (2023) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3270027 · Cited by 112
“Unsupervised methods such as clustering can not only help improve predictive performance by identifying the most relevant discriminant features [62], [63] but can also ease feature selection for semantic interpretation.”
MethodLearning facial expression and body gesture visual information for video emotion recognition
Wei, J., Hu, G., Yang, X., Tuan, L. A., & Dong, Y. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121419 · Cited by 73
V., J. P., S., A. A. V., P., G. K., & N.K., K. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.107977 · Cited by 47
Metadata
- Authors
- Chen, L., Wang, K., Li, M., Wu, M., Pedrycz, W., & Hirota, K.
- Affiliations
- China University of Geosciences (CN)
- University of Alberta (CA)
- Tokyo Institute of Technology (JP)
- Venue
- IEEE Transactions on Industrial Electronics
- Published
- 2023-01
- Volume
- 70
- Issue
- 1
- Pages
- 1016-1024
- ISSN
- 0278-0046 (ISSN-L)
- 1557-9948
- OpenAlex
- W4213102761
- Cited by
- 113
- References
- 33
- Citations by year
- 2026: 18 · 2025: 37 · 2024: 30 · 2023: 27
- FWCI
- 21.88
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Industrial Electronics (publishedVersion)
- Funders
- National Natural Science Foundation of China (61973286, 61603356, 61773353)
- Higher Education Discipline Innovation Project (B17040)
- Fundamental Research Funds for the Central Universities
- China University of Geosciences, Wuhan (2021063)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face and Expression Recognition
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- multimodal emotion recognition, human-robot interaction, kernel canonical correlation analysis, K-means clustering, feature selection, dimensionality reduction, facial expressions, speech emotion recognition, multimodal fusion, SAVEE
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.