Robot System Assistant (RoSA): Towards Intuitive Multi-Modal and Multi-Device Human-Robot Interaction
Open accessThe DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Abstract
This paper presents an implementation of RoSA, a Robot System Assistant, for safe and intuitive human-machine interaction. The interaction modalities were chosen and previously reviewed using a Wizard of Oz study emphasizing a strong propensity for speech and pointing gestures. Based on these findings, we design and implement a new multi-modal system for contactless human-machine interaction based on speech, facial, and gesture recognition. We evaluate our proposed system in an extensive study with multiple subjects to examine the user experience and interaction efficiency. It reports that our method achieves similar usability scores compared to the entirely human remote-controlled robot interaction in our Wizard of Oz study. Furthermore, our framework’s implementation is based on the Robot Operating System (ROS), allowing modularity and extendability for our multi-device and multi-user method.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures













![Figure 14 Pointing gesture: User points to cube [S], which is enveloped by the laser pointer.](https://figures.cochat.ai/paper-figures/f6240400bcb54895837e7f8f5cc9e408/robot-system-assistant-rosa-towards-intuitive-multi-modal-and-multi-device-human-figure-14.jpg)




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Studies [12,14,17,21] focus on users with high technical demands or operational autonomy, such as evaluators of language models [17], operators of automated vehicles [13], participants in spaceflight simulations [21], or users in automated driving simulators [5].”
From The Impact of User Interface and Experience (UI/UX) Design on Visual Ergonomics: A Technical Approach for Reducing Human Error in Industrial Settings (2026) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In collaborative or mobile environments, where interaction may be asynchronous or interrupted, solutions such as modular screens, simplified navigation systems, and functional colors to distinguish roles or tasks are highlighted [8,14,19,22].”
From The Impact of User Interface and Experience (UI/UX) Design on Visual Ergonomics: A Technical Approach for Reducing Human Error in Industrial Settings (2026) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Systems with projected or multimodal interfaces [2,7,14,15,19] also show notable performance improvements, minimizing repeated or incorrect actions through optimized information visualization.”
From The Impact of User Interface and Experience (UI/UX) Design on Visual Ergonomics: A Technical Approach for Reducing Human Error in Industrial Settings (2026) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Projected or multimodal interfaces in industrial workstations also contribute to reducing human errors [2,6,14,15,19].”
From The Impact of User Interface and Experience (UI/UX) Design on Visual Ergonomics: A Technical Approach for Reducing Human Error in Industrial Settings (2026) · cites this paper as [14]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In industrial environments with multitasking and operational pressure, robust and clear interfaces are necessary to prevent errors [2,6,12,14,15,19,22,30].”
From The Impact of User Interface and Experience (UI/UX) Design on Visual Ergonomics: A Technical Approach for Reducing Human Error in Industrial Settings (2026) · cites this paper as [14]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989
ArcFace: Additive Angular Margin Loss for Deep Face Recognition
Deng, J., Guo, J., Xue, N., & Zafeiriou, S. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00482 · Cited by 7,781 · cited 3×
“The network was trained on the refined MS-Celeb-1M dataset using the loss function Arcface [23].”
BackgroundMethodWIDER FACE: A Face Detection Benchmark
Yang, S., Luo, P., Loy, C. C., & Tang, X. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.596 · Cited by 1,830 · cited 1×
“The network was pre-trained on a WIDER FACE dataset [34].”
MethodRetinaFace: Single-Shot Multi-Level Face Localisation in the Wild
Deng, J., Guo, J., Ververas, E., Kotsia, I., & Zafeiriou, S. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.00525 · Cited by 1,681 · cited 5×
“detected faces Face Id 512 features ∈ [0, 1] 7 features ∈ [0, 1] 4 features for each ∈ (x, y) (in pixels) 1 features for each ∈ (x, y) (in pixels) 5 features for each ∈ (x, y) (in pixels) 1 feature ∈ Z>0 1 feature ∈ Z>0 ArcFace [23] Residual Masking Network [24] RetinaFace [25] post processed RetinaFace [25] post proc…”
BackgroundMethodInfluentialHead Pose Estimation in Computer Vision: A Survey
Murphy-Chutorian, E., & Trivedi, M. M. (2009) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2008.106 · Cited by 1,301
MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devices
Chen, S., Liu, Y., Gao, X., & Han, Z. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-97909-0_46 · Cited by 771
Psychometric Evaluation of the PSSUQ Using Data from Five Years of Usability Studies
Lewis, J. R. (2002) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2002.9669130 · Cited by 621 · cited 1×
“The questionnaires (SUS [55], UMUX [56], PSSUQ [57], and ASQ [58]) were completed after the experiment.”
MethodThe Usability Metric for User Experience
Finstad, K. (2010) · Interacting with Computers · doi:10.1016/j.intcom.2010.04.004 · Cited by 604 · cited 1×
“The questionnaires (SUS [55], UMUX [56], PSSUQ [57], and ASQ [58]) were completed after the experiment.”
MethodGaze360: Physically Unconstrained Gaze Estimation in the Wild
Kellnhofer, P., Recasens, A., Stent, S., Matusik, W., & Torralba, A. (2019) · 2019 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2019.00701 · Cited by 470
Automatic Speech Recognition: A Survey
Malik, M., Malik, M. K., Mehmood, K., & Makhdoom, I. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-020-10073-7 · Cited by 358 · cited 1×
“The performance of a speech recognition system is measured by the Word Error Rate (WER): where S is the number of words incorrectly replaced, I represents the number of additional words inserted, D is the number of words deleted, and N is the number of words correctly transcribed [59].”
Method
Research works printed but not linked: 4
- Hannun, A., Case, C., Casper, J., Catanzaro, B., Diamos, G., Elsen, E., Prenger, R., Satheesh, S., Sengupta, S., and Coates, A. (2014). Deepspeech: Scaling up end-to-end speech recognition. arXiv.
- Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., and Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv.
- Guo, X., Li, S., Yu, J., Zhang, J., Ma, J., Ma, L., Liu, W., and Ling, H. (2019). PFLD: A practical facial landmark detector. arXiv.
- Bocklisch, T., Faulkner, J., Pawlowski, N., and Nichol, A. (2017). Rasa: Open source language understanding and dialogue management. arXiv.
Cited by
Recent advancements in multimodal human–robot interaction
Su, H., Qi, W., Chen, J., Yang, C., Sandoval, J., & Laribi, M. A. (2023) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2023.1084000 · Cited by 181
L2CS-Net : Fine-Grained Gaze Estimation in Unconstrained Environments
Abdelrahman, A. A., Hempel, T., Khalifa, A., Al-Hamadi, A. K., & Dinges, L. (2023) · International Conference on Frontiers of Signal Processing (ICFSP) · doi:10.1109/icfsp59764.2023.10372944 · Cited by 127
“It indicates the user’s level of engagement in human-robot interaction [1,2], and open dialogue systems [3].”
BackgroundSelf-citationCobotics: The Evolving Roles and Prospects of Next‐Generation Collaborative Robots in Industry 5.0
Rahman, M. M., Khatun, F., Jahan, I., Devnath, R., & Bhuiyan, M. A.-A. (2024) · Journal of Robotics · doi:10.1155/2024/2918089 · Cited by 121
Chu, C., & Liu, Y.-L. (2023) · Journal of Manufacturing Systems · doi:10.1016/j.jmsy.2023.04.007 · Cited by 71
Research Progress of Human–Computer Interaction Technology Based on Gesture Recognition
Zhou, H., Wang, D., Yu, Y. W., & Zhang, Z. (2023) · Electronics · doi:10.3390/electronics12132805 · Cited by 91
Metadata
- Authors
- Strazdas, D., Hintz, J., Khalifa, A., Abdelrahman, A. A., Hempel, T., & Al-Hamadi, A.
- Affiliations
- Otto-von-Guericke-Universität Magdeburg (DE)
- Corresponding author
- Strazdas, Dominykas
- Al-Hamadi, Ayoub
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2022-01-25
- Volume
- 22
- Issue
- 3
- Pages
- 923
- ISSN
- 1424-8220 (ISSN-L)
- PMC
- PMC8838571
- OpenAlex
- W4210532635
- Cited by
- 53
- References
- 45
- Citations by year
- 2026: 11 · 2025: 10 · 2024: 12 · 2023: 16 · 2022: 4
- FWCI
- 9.92
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Sensors (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Federal Ministry of Education and Research (03ZZ0448L, 03ZZ04X02B)
- Topics
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Physical Sciences › Computer Science › Human-Computer Interaction › Interactive and Immersive Displays
- Keywords
- human-robot interaction, multimodal interaction, multi-device interaction, multi-user interaction, speech recognition, gesture recognition, face recognition, touchless interaction, Wizard-of-Oz study, Robot Operating System, robotic assistants, usability evaluation, interaction efficiency
- MeSH terms
- Gestures, Humans, Robotics, Software, Speech
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Robot System Assistant (RoSA): Towards Intuitive Multi-Modal and Multi-Device Human-Robot Interaction. https://app.cochat.ai/papers/10.3390/s22030923
- The paper
- Strazdas, D., Hintz, J., Khalifa, A., Abdelrahman, A. A., Hempel, T., & Al-Hamadi, A. (2022). Robot System Assistant (RoSA): Towards Intuitive Multi-Modal and Multi-Device Human-Robot Interaction. Sensors. https://doi.org/10.3390/s22030923
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.