The DOI matches its Crossref record; 36 of the 39 references it cites were found in a registry or an index; 3 of the 13 sources its claims cite were checked for support.
Modelling Multimodal Dialogues for Social Robots Using Communicative Acts
Open accessAbstract
Social Robots need to communicate in a way that feels natural to humans if they are to effectively bond with the users and provide an engaging interaction. Inline with this natural, effective communication, robots need to perceive and manage multimodal information, both as input and output, and respond accordingly. Consequently, dialogue design is a key factor in creating an engaging multimodal interaction. These dialogues need to be flexible enough to adapt to unforeseen circumstances that arise during the conversation but should also be easy to create, so the development of new applications gets simpler. In this work, we present our approach to dialogue modelling based on basic atomic interaction units called Communicative Acts. They manage basic interactions considering who has the initiative (the robot or the user), and what is his/her intention. The two possible intentions are either ask for information or give information. In addition, because we focus on one-to-one interactions, the initiative can only be taken by the robot or the user. Communicative Acts can be parametrised and combined in a hierarchical manner to fulfil the needs of the robot's applications, and they have been equipped with built-in functionalities that are in charge of low-level communication tasks. These tasks include communication error handling, turn-taking or user disengagement. This system has been integrated in Mini, a social robot that has been created to assist older adults with cognitive impairment. In a case of use, we demonstrate the operation of our system as well as its performance in real human-robot interactions.
Overview
CoChat Summary
## TL;DR
The paper proposes a two-level multimodal dialogue model for social robots, based on four basic Communicative Acts (CAs) and reusable Complex Communicative Acts (CCAs), and implements it in the Mini social robot. In a case of use with four seniors, the system handled 88.42% of 354 CAs without problems, and all measured response times stayed below one second.
## Key findings
- The authors define four basic CAs from two dimensions—initiative and intention: Robot Asks for Information, Robot Gives Information, User Asks for Information, and User Gives Information.
- They identify five CCAs for Mini’s applications: Right–Wrong Question, Question with Confirmation, Switching Mode Question, Manage Multimedia Content, and Communication Warning.
- In the case of use, four seniors interacted with Mini for 71 mins, and 354 CAs were used; 313/354 (88.42%) completed without problems, while 41/354 (11.58%) required recovery mechanisms.
- The paper reports 65 uses of recovery mechanisms overall, including 29 cases of ASR failures/low confidence, 19 cases where the system encouraged the user to answer, and 17 cases where the answer was not the expected one.
- All CAs used in the case of use had response times below one second, which the authors say satisfies the natural-interaction threshold they adopted.
## Methods
- The paper proposes a two-level dialogue architecture: applications manage task flow at the application level, while the HRI Manager handles CA execution, prioritization, concurrency, and recovery at the interaction level.
- The system is implemented in Mini, a tabletop social robot with 5 degrees of freedom, OLED eyes, a lighted heart, and an external tablet; the target users are older adults with mild cognitive impairment.
- Evaluation uses an objective case study with four senior participants interacting with Mini in real settings; the authors analyze recorded interactions to measure response time and use of recovery mechanisms.
## Limitations
- The paper states that the evaluation involved only 4 users and only one type of application, so broader generalization would require a larger and more heterogeneous sample.
- The authors note that the approach is limited to one-to-one interactions and cannot manage multiple users present at the same time.
- The paper states that multimodal fusion is not implemented yet, which limits interaction capabilities.
- The use of grammar-based ASR limits verbal input flexibility; this is stated in the paper and makes it evident that synonyms or out-of-grammar responses may be rejected.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures













Verification
Claims the paper makes
3 Valid · 2 Not valid · 6 Not checked · 97 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“From the related works, just three of them (Pealtson et al. [16], Alonso et al. [10] and Lison et al. [9]) consider the use of different communication channels. The rest of the works are designed for verbal communication exclusively.”
- Also stated in: § 2.8. Our Approach · plain
“Most of the works previously mentioned are exclusively focused on verbal communication, while multimodal communication has received considerably less attention. Only three of the reviewed approaches use multimodal inputs and outputs, while the rest of the solutions are either speech-based or text-based exclusively.”
§ Modelling Multimodal Dialogues for Social Robots Using Communicative Acts > 2. Related Work > 2.7. Comparison among Approaches
What the sources say
- Not checked — no full text of the sourcecites [16] Peltason, J., and Wrede, B. (2010, January 24–25). Pamini: A framework for assembling mixed-initiative human-robot interaction from generic interaction patterns. Proceedings of the SIGDIAL 2010 Conference: 11th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Tokyo, Japan.
- Supportedcites [10] Salichs, 2015, Augmented Robotics Dialog System for Enhancing Human-Robot Interaction, Sensors source
The citation identifies Alonso et al. as an approach using different communication channels. The Proof of Concept section says the robot responds through voice, gestures, and a tablet. The citing authors’ count of three approaches and comparison with the other works are not findings attributed specifically to this paper. (LLM, conf=0.92)
- Not checked — no full text of the sourcecites [9] Lison, 2015, A Hybrid Approach to Dialogue Management Based on Probabilistic Rules, Comput. Speech Lang. source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 2.8. Our Approach · plain
Every source the paper cites for this claim was checked and supports it.
In the paper
“In 2009, Shiwa et al. demonstrated that this rule can be applied to the field of Human–Robot Interaction [37].”
§ Modelling Multimodal Dialogues for Social Robots Using Communicative Acts > 6. Case of Use > 6.2. Response Time
What the sources say
- Supportedcites [37] Shiwa, 2009, How Quickly Should Communication Robots Respond?, J. Robot. Soc. Jpn. source
The introduction (p. 87) defines the two-second rule as not making users wait more than two seconds. The paper then tests whether that guideline applies to robot interaction, and its abstract concludes that a communication robot should react within two seconds. (LLM, conf=0.92)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After studying the different situations and needs that might arise during a dialogue, we came to the conclusion that all DAs have one of two goals: either the utterance aims to convey a certain information, without expecting anything back, or it aims to obtain information that the other speaker has.”
§ Modelling Multimodal Dialogues for Social Robots Using Communicative Acts > 3. Modelling Human–Robot Dialogues > 3.1. Communicative Acts
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, a wh-question and a yes–no question have different interaction goals, and so they are considered as different DAs. In the end, they both pursue a common goal: obtain a certain information from the other peer.”
§ Modelling Multimodal Dialogues for Social Robots Using Communicative Acts > 3. Modelling Human–Robot Dialogues
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The approach we have presented for modelling human–robot dialogues has been implemented in the social robot Mini.”
§ Modelling Multimodal Dialogues for Social Robots Using Communicative Acts > 4. The Robot Mini and Its HRI Architecture
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
32 more claims
Create a free account to see all 108 claims and every source verdict.
- Probably not a substantial claim. 53
- It may state two separate claims as one. 21
- The quote only partly states this claim. 2
Citations
Cites
Austin, J. L. (1975) · doi:10.1093/acprof:oso/9780198245537.001.0001 · Cited by 21,496
Socially intelligent robots: Dimensions of human-robot interaction
Dautenhahn, K. (2007) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2006.2004 · Cited by 1,210 · cited 1×
“For this to happen, robots need a set of skills that endows them with the functionalities required to interact with humans in a natural way [5,6].”
BackgroundResponse time in man-computer conversational transactions
Miller, R. B. (1968) · Proceedings of the December 9-11, 1968, fall joint computer conference, part I on - AFIPS '68 (Fall, part I) · doi:10.1145/1476589.1476628 · Cited by 942 · cited 1×
“Other researchers established the “two second” rule as a guideline for designing system response times [36].”
BackgroundA Network-based End-to-End Trainable Task-oriented Dialogue System
Wen, T.-H., Vandyke, D., Mrkšić, N., Gasic, M., Rojas Barahona, L. M., Su, P.-H., Ultes, S., & Young, S. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1042 · Cited by 778 · cited 1×
“[24] presented a neural network-based text-in, text-out dialogue system and a dialogue data collection method based on a Wizard-of-Oz framework.”
MethodKey-Value Retrieval Networks for Task-Oriented Dialogue
Eric, M., Krishnan, L., Charette, F., & Manning, C. D. (2017) · Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue · doi:10.18653/v1/w17-5506 · Cited by 388 · cited 1×
“[27] proposed a neural dialogue agent for sustaining grounded, multi-domain discourse through a key-value retrieval mechanism.”
BackgroundEnd-to-End Task-Completion Neural Dialogue SystemsPreprint
Li, X., Chen, Y.-N., Li, L. C., Gao, J., & Çelikyılmaz, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.01008 · Cited by 327
The RavenClaw dialog management framework: Architecture and systems
Bohus, D., & Rudnicky, A. I. (2009) · Computer Speech & Language · doi:10.1016/j.csl.2008.10.001 · Cited by 264 · cited 4×
“Plan-based [14] Design the domain-dependent aspects, using dialogue agents, while reusing the domain-independent aspects No Responses are implemented in the dialogue agents”
BackgroundMethodInfluentialExample-based dialog modeling for practical multi-domain dialog system
Lee, C., Jung, S., Kim, S., & Lee, G. G. (2009) · Speech Communication · doi:10.1016/j.specom.2009.01.008 · Cited by 139 · cited 2×
“[21], who proposed an example-based dialogue modelling approach for multi-domain DSs where the DM generates a structured query language statement based on the dialogue history and the current dialogue frame.”
BackgroundContinuously Learning Neural Dialogue ManagementPreprint
Su, P.-H., Gašić, M., Mrkšić, N., Rojas-Barahona, L. M., Ultes, S., Vandyke, D., Wen, T.-H., & Young, S. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.02689 · Cited by 133
ConvLab: Multi-Domain End-to-End Dialog System Platform
Lee, S., Zhu, Q., Takanobu, R., Zhang, Z., Zhang, Y., Li, X., Li, J., Peng, B., Li, X., Huang, M., et al. (2019) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/p19-3011 · Cited by 104
Research works printed but not linked: 2
- Fodor, P. (2007, January 5–7). Dialog management for decision processes. Proceedings of the 3rd Language and Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics, Poznan, Poland.
- Salichs, E., Castro-González, Á., Malfaz, M., and Salichs, M.A. (2016, January 2). Mini: A social assistive robot for people with mild cognitive impairment. Proceedings of the New Friends 2016—2nd International Conference on Social Robots in Therapy & and Education, Barcelona, Spain.
Cited by
Song, Y., & Luximon, Y. (2020) · Sensors · doi:10.3390/s20185087 · Cited by 115
Conversational Affective Social Robots for Ageing and Dementia Support
Lima, M. R., Wairagkar, M., Gupta, M., Rodriguez y Baena, F., Barnaghi, P., Sharp, D. J., & Vaidyanathan, R. (2022) · IEEE Transactions on Cognitive and Developmental Systems · doi:10.1109/tcds.2021.3115228 · Cited by 75
Irfan, B., Kuoppamäki, S., Hosseini, A., & Skantze, G. (2025) · Autonomous Robots · doi:10.1007/s10514-025-10190-y · Cited by 107
“Hence, spoken dialogue in companion robots is integral, and needs to be flexible enough to adapt to unforeseen circumstances during the conversation ( Fernández-Rodicio et al., 2020).”
Maroto-Gómez, M., Castro-González, Á., Castillo, J. C., Malfaz, M., & Salichs, M. Á. (2023) · User Modeling and User-Adapted Interaction · doi:10.1007/s11257-022-09321-2 · Cited by 37
“The human–robot interaction system (see (Fernández-Rodicio et al. 2020) for further details) controls the interaction with the user.”
Self-citationCompanion robots for older adults: Rodgers’ evolutionary concept analysis approach
Kim, J., Kim, S., Kim, S., Lee, E., Heo, Y., Hwang, C.-Y., Choi, Y.-Y., Kong, H., Ryu, H., & Lee, H. (2021) · Intelligent Service Robotics · doi:10.1007/s11370-021-00394-3 · Cited by 32
“To enjoy long-term and engaging multimodal interaction, the human–robot dialog is a key factor, which can be implemented through semantic analysis, understanding emotions from speech analysis, and the design of various robot behaviors [7, 13, 37, 38].”
Background
Metadata
- Authors
- Fernández-Rodicio, E., Castro-González, Á., Alonso-Martín, F., Maroto-Gómez, M., & Salichs, M. Á.
- Affiliations
- Robotics Lab, Universidad Carlos III de Madrid, Av. de la Universidad 30, Leganés, 28911 Madrid, Spain
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2020-06-18
- Volume
- 20
- Issue
- 12
- Pages
- 3440
- ISSN
- 1424-8220 (ISSN-L)
- PMC
- PMC7348960
- OpenAlex
- W3036971864
- Cited by
- 37
- References
- 39
- Citations by year
- 2026: 4 · 2025: 5 · 2024: 7 · 2023: 4 · 2022: 7 · 2021: 8 · 2020: 2
- FWCI
- 2.06
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Sensors (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- e-Archivo (Carlos III University of Madrid) (submittedVersion, cc-by)
- Funders
- Ministerio de Economía y Competitividad (DPI2014-57684-R)
- Comunidad Autónoma de Madrid (S2013/MIT-2748)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Keywords
- social robots, communicative acts, dialogue design, turn-taking, user disengagement, intent classification, human-robot interaction
- MeSH terms
- Social Interaction, Aged, Communication, Emotions, Female, Humans, Male, Robotics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Modelling Multimodal Dialogues for Social Robots Using Communicative Acts. https://app.cochat.ai/papers/10.3390/s20123440
- The paper
- Fernández-Rodicio, E., Castro-González, Á., Alonso-Martín, F., Maroto-Gómez, M., & Salichs, M. Á. (2020). Modelling Multimodal Dialogues for Social Robots Using Communicative Acts. Sensors. https://doi.org/10.3390/s20123440
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.