The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Lightweight Multimodal Emotion Recognition for Companion Robots: A Deep Learning Framework Integrating Facial and Speech Features
Abstract
This paper presents a lightweight multimodal deep learning framework for real-time emotion recognition on resource-constrained companion robots, exemplified by Zenbo Junior II. The framework integrates a customized GhostNet with Triplet Attention Modules (TAM) and a Frame Attention Network (FAN) for spatio-temporal facial feature encoding, and employs a depth-optimized one-dimensional convolutional neural network (1D-CNN) for compact speech representation. Decision-level fusion based on the geometric mean enhances robustness to noisy modality predictions. The proposed model comprises 0.92 million parameters and requires 0.77 billion floating-point operations (GFLOPs), achieving 97.56% accuracy on the RAVDESS dataset and 82.33% on CREMA-D. In contrast to existing approaches that optimize accuracy at the expense of computational efficiency, the proposed method demonstrates a balance of accuracy, efficiency, and deployability. These results highlight both the novelty and the feasibility of the framework for real-time emotion monitoring in healthcare and human-robot interaction.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Rapid object detection using a boosted cascade of simple features
Viola, P., & Jones, M. (2005) · Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001 · doi:10.1109/cvpr.2001.990517 · Cited by 18,300
A survey on Image Data Augmentation for Deep Learning
Shorten, C., & Khoshgoftaar, T. M. (2019) · Journal of Big Data · doi:10.1186/s40537-019-0197-0 · Cited by 13,203
Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks
Zhang, K., Zhang, Z., Li, Z., & Qiao, Y. (2016) · IEEE Signal Processing Letters · doi:10.1109/lsp.2016.2603342 · Cited by 6,214
GhostNet: More Features From Cheap Operations
Han, K., Wang, Y., Tian, Q., Guo, J., Xu, C., & Xu, C. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00165 · Cited by 5,222
Rotate to Attend: Convolutional Triplet Attention Module
Misra, D., Nalamada, T., Arasanipalai, A. U., & Hou, Q. (2021) · 2021 IEEE Winter Conference on Applications of Computer Vision (WACV) · doi:10.1109/wacv48630.2021.00318 · Cited by 1,147
Assistive social robots in elderly care: a review
Broekens, J., Heerink, M., & Rosendal, H. (2024) · Gerontechnology · doi:10.4017/gt.2009.08.02.002.00 · Cited by 940
CREMA-D: Crowd-Sourced Emotional Multimodal Actors Dataset
Cao, H., Cooper, D. G., Keutmann, M. K., Gur, R. C., Nenkova, A., & Verma, R. (2014) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2014.2336244 · Cited by 914
The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS)
Livingstone, S. R., & Russo, F. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1188975 · Cited by 375
Emotion Recognition for Human-Robot Interaction: Recent Advances and Future Perspectives
Spezialetti, M., Placidi, G., & Rossi, S. (2020) · Frontiers in Robotics and AI · doi:10.3389/frobt.2020.532279 · Cited by 293
Multimodal Emotion Recognition with Deep Learning: Advancements, challenges, and future directions
A.V., G., T., M., D., P., & E., U. (2024) · Information Fusion · doi:10.1016/j.inffus.2023.102218 · Cited by 225
Research works printed but not linked: 2
- 10.1016/j.knosys.2022.109978 · DOI
- Hu, 2022, Speech emotion recognition model based on attention CNN Bi-GRU fusing visual information, Eng. Lett., 30
Cited by
Federated Deep Learning for IoMT-Based Retinal Fluid Segmentation in Optical Coherence Tomography
Lu, C., Hsu, T.-H., Lin, C. W., & Ng, H. W. (2026) · IEEE Internet of Things Journal · doi:10.1109/jiot.2026.3693061
“To provide a comprehensive assessment of deployment feasibility, hardware benchmarks were conducted in accordance with the evaluation guidelines established in [43].”
Self-citationBeyond Emotion Labels: Multimodal AI Approaches to Affective Computing and Sentiment ModelingPreprint
Armendáriz-Íñigo, J. E., Meza-Tudela, A., Ji, Z., & Vidaurreta-Apesteguia, P. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6890916
Adinath, M., Princy, J. S., Veni, A. K., M, C. B., Vijayalakskmi, R., Amurdhasri, G., Shamili, E., & Keerthanaa, L. (2026) · International Conference on AI-Driven Innovations & Applications (IC-AIDA) · doi:10.1109/ic-aida68291.2026.11564140
“Early work used speech-only or vision-only models[10], but noise and ambiguity motivated fusion approaches[11].”
Metadata
- Authors
- Lu, C.-K., Lu, C.-W., & Bo Lin, G.
- Affiliations
- National Taiwan Normal University (TW)
- Department of Electrical Engineering, National Taiwan Normal University, Taipei, Taiwan
- Venue
- IEEE Transactions on Affective Computing
- Published
- 2026-01
- Volume
- 17
- Issue
- 1
- Pages
- 412-425
- ISSN
- 1949-3045 (ISSN-L)
- 2371-9850
- OpenAlex
- W4415125352
- Cited by
- 4
- References
- 33
- Citations by year
- 2026: 4
- FWCI
- 4.22
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Affective Computing (publishedVersion)
- Funders
- National Science and Technology Council (NSTC 111-2222-E-003-001, NSTC 112-2221-E-003-008, NSTC 113-2221-E-003-006-MY2)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Emotion and Mood Recognition
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- multimodal emotion recognition, companion robots, lightweight deep learning, facial expression recognition, speech emotion recognition, GhostNet, 1D convolutional neural network, decision fusion, real-time emotion monitoring, human-robot interaction, healthcare robotics, resource-constrained deployment
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.