The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Lightweight Multimodal Emotion Recognition for Companion Robots: A Deep Learning Framework Integrating Facial and Speech Features
Abstract
This paper presents a lightweight multimodal deep learning framework for real-time emotion recognition on resource-constrained companion robots, exemplified by Zenbo Junior II. The framework integrates a customized GhostNet with Triplet Attention Modules (TAM) and a Frame Attention Network (FAN) for spatio-temporal facial feature encoding, and employs a depth-optimized one-dimensional convolutional neural network (1D-CNN) for compact speech representation. Decision-level fusion based on the geometric mean enhances robustness to noisy modality predictions. The proposed model comprises 0.92 million parameters and requires 0.77 billion floating-point operations (GFLOPs), achieving 97.56% accuracy on the RAVDESS dataset and 82.33% on CREMA-D. In contrast to existing approaches that optimize accuracy at the expense of computational efficiency, the proposed method demonstrates a balance of accuracy, efficiency, and deployability. These results highlight both the novelty and the feasibility of the framework for real-time emotion monitoring in healthcare and human-robot interaction.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Rapid object detection using a boosted cascade of simple features
Viola, P., & Jones, M. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990517 · Cited by 18,300
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214
GhostNet: More Features From Cheap Operations
Han, K., Wang, Y., Tian, Q., Guo, J., Xu, C., & Xu, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00165 · Cited by 5,222
Rotate to Attend: Convolutional Triplet Attention Module
Misra, D., Nalamada, T., Arasanipalai, A. U., & Hou, Q. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00318 · Cited by 1,147
Assistive social robots in elderly care: a review
Broekens, J., Heerink, M., & Rosendal, H. (2024) · Gerontechnology · doi:10.4017/gt.2009.08.02.002.00 · Cited by 940
CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset
Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., & Verma, R. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2336244 · Cited by 914
The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS)
Livingstone, S. R., & Russo, F. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1188975 · Cited by 375
Emotion Recognition for Human-Robot Interaction: Recent Advances and Future Perspectives
Spezialetti, M., Placidi, G., & Rossi, S. (2020) · Frontiers in Robotics and AI · doi:10.3389/frobt.2020.532279 · Cited by 293
Multimodal Emotion Recognition with Deep Learning: Advancements, challenges, and future directions
A.V., G., T., M., D., P., & E., U. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102218 · Cited by 225
Frame Attention Networks for Facial Expression Recognition in Videos
Meng, D., Peng, X., Wang, K., & Qiao, Y. (2019) · 2019 IEEE International Conference on Image Processing (ICIP) · doi:10.1109/icip.2019.8803603 · Cited by 196
Mocanu, B., Țapu, R., & Zaharia, T. B. (2023) · Image and Vision Computing · doi:10.1016/j.imavis.2023.104676 · Cited by 181
An Active Learning Paradigm for Online Audio-Visual Emotion Recognition
Kansizoglou, I., Bampis, L., & Gasteratos, A. C. (2019) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2019.2961089 · Cited by 135
Multimodal Emotion Recognition From EEG Signals and Facial Expressions
Wang, S., Qu, J., Zhang, Y., & Zhang, Y. (2023) · IEEE Access · doi:10.1109/access.2023.3263670 · Cited by 124 · cited 1×
“Sharafi et al. [24] and Wang et al. [25] used CNNs to further extract features.”
Improving multimodal fusion with Main Modal Transformer for emotion recognition in conversation
Zou, S., Huang, X., Shen, X., & Liu, H. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109978 · Cited by 107
A novel spatio-temporal convolutional neural framework for multimodal emotion recognition
Sharafi, M., Yazdchi, M., Rasti, R., & Nasimi, F. (2022) · Biomedical Signal Processing and Control · doi:10.1016/j.bspc.2022.103970 · Cited by 73
Modeling Hierarchical Uncertainty for Multimodal Emotion Recognition in Conversation
Chen, F., Shao, J., Zhu, A., Ouyang, D., Liu, X., & Shen, H. T. (2022) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2022.3185119 · Cited by 57
User-generated video emotion recognition based on key frames
Wei, J., Yang, X., & Dong, Y. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-020-10203-1 · Cited by 52
Factors in Emotion Recognition With Deep Learning Models Using Speech and Text on Multiple Corpora
Braunschweiler, N., Doddipatla, R., Keizer, S., & Stoyanchev, S. (2022) · IEEE Signal Processing Letters · doi:10.1109/lsp.2022.3151551 · Cited by 42
Banskota, N., Alsadoon, A., Prasad, P. W. C., Dawoud, A., Rashid, T. A., & Alsadoon, O. H. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13567-8 · Cited by 41
Joint modelling of audio-visual cues using attention mechanisms for emotion recognition
Ghaleb, E., Niehues, J., & Asteriadis, S. (2022) · Multimedia Tools and Applications · doi:10.1007/s11042-022-13557-w · Cited by 40
Attending to Emotional Narratives
Wu, Z., Zhang, X., Zhi‐Xuan, T., Zaki, J., & Ong, D. C. (2019) · International Conference on Affective Computing and Intelligent Interaction and workshops · doi:10.1109/acii.2019.8925497 · Cited by 24
Video-based person-dependent and person-independent facial emotion recognition
Hajarolasvadi, N., Bashirov, E., & Demirel, H. (2021) · Signal Image and Video Processing · doi:10.1007/s11760-020-01830-0 · Cited by 22
He, J. Y. D., Yu, X., Sun, B., & Yu, L. (2021) · Journal on Multimodal User Interfaces · doi:10.1007/s12193-020-00363-7 · Cited by 21
Facial Expression Recognition using Distance Importance Scores Between Facial Landmarks
Рюмина, Е. В., & Karpov, A. A. (2020) · Proceedings of the 30th International Conference on Computer Graphics and Machine Vision (GraphiCon 2020). Part 2 · doi:10.51130/graphicon-2020-2-3-32 · Cited by 20
Sun, Q., Liang, L., Dang, X., & Chen, Y. (2022) · Computers & Electrical Engineering · doi:10.1016/j.compeleceng.2022.108469 · Cited by 16
Affective Computing for Social Companion Robots Using Fine-grained Speech Emotion Recognition
Ahuja, S., & Shabani, A. (2023) · 2023 IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai54212.2023.00146 · Cited by 12
A new context-based feature for classification of emotions in photographs
Krishnani, D., Shivakumara, P., Lü, T., Pal, U., Lopresti, D., & Kumar, G. H. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-020-10404-8 · Cited by 12
Kim, N., Cho, S., & Bae, B. (2022) · Sensors · doi:10.3390/s22155753 · Cited by 9
Emotion Recognition of College Students Based on Audio and Video Image
Zhu, C., Ding, T., & Min, X. (2022) · Traitement du signal · doi:10.18280/ts.390503 · Cited by 9
Lakshminarayana, N. N., Sankaran, N., Setlur, S., & Govindaraju, V. (2019) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2019.8756629 · Cited by 8
Ensemble Integration of Deep Learning Models for Gender-Based Speech Emotion Recognition
Swathi, G., Kumar, R. P., Bharathi Mohan, G., & Elakkiya, R. (2023) · International Conference on Computing, Communication and Networking Technologies (ICCCNT) · doi:10.1109/icccnt56998.2023.10306801 · Cited by 6
WellBe: An Intelligent Elderly Care and Well-Being Monitoring System Using Deep Learning
Raotole, A., Shirodkar, S., Shukla, R., Sisodia, J., & Devadkar, K. (2024) · International Conference on Intelligent Technologies (CONIT) · doi:10.1109/conit61985.2024.10626766 · Cited by 2
Research works printed but not linked: 2
- 10.1016/j.knosys.2022.109978 · DOI
- Hu, 2022, Speech emotion recognition model based on attention CNN Bi-GRU fusing visual information, Eng. Lett., 30
Cited by
Federated Deep Learning for IoMT-Based Retinal Fluid Segmentation in Optical Coherence Tomography
Lu, C., Hsu, T.-H., Lin, C. W., & Ng, H. W. (2026) · IEEE Internet of Things Journal · doi:10.1109/jiot.2026.3693061
“To provide a comprehensive assessment of deployment feasibility, hardware benchmarks were conducted in accordance with the evaluation guidelines established in [43].”
Self-citationBeyond Emotion Labels: Multimodal AI Approaches to Affective Computing and Sentiment ModelingPreprint
Armendáriz-Íñigo, J. E., Meza-Tudela, A., Ji, Z., & Vidaurreta-Apesteguia, P. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6890916
Adinath, M., Princy, J. S., Veni, A. K., M, C. B., Vijayalakskmi, R., Amurdhasri, G., Shamili, E., & Keerthanaa, L. (2026) · International Conference on AI-Driven Innovations & Applications (IC-AIDA) · doi:10.1109/ic-aida68291.2026.11564140
“Early work used speech-only or vision-only models[10], but noise and ambiguity motivated fusion approaches[11].”
Metadata
- Authors
- Lu, C.-K., Lu, C.-W., & Bo Lin, G.
- Affiliations
- National Taiwan Normal University (TW)
- Department of Electrical Engineering, National Taiwan Normal University, Taipei, Taiwan
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2026-01
- Volume
- 17
- Issue
- 1
- Pages
- 412-425
- ISSN
- 1949-3045 (ISSN-L)
- 2371-9850
- OpenAlex
- W4415125352
- Cited by
- 4
- References
- 33
- Citations by year
- 2026: 4
- FWCI
- 4.22
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Affective Computing (publishedVersion)
- Funders
- National Science and Technology Council (NSTC 111-2222-E-003-001, NSTC 112-2221-E-003-008, NSTC 113-2221-E-003-006-MY2)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- multimodal emotion recognition, companion robots, lightweight deep learning, facial expression recognition, speech emotion recognition, GhostNet, 1D convolutional neural network, decision fusion, real-time emotion monitoring, human-robot interaction, healthcare robotics, resource-constrained deployment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.