Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 36 of the 39 references it cites were found in a registry or an index; 3 of the 13 sources its claims cite were checked for support.

Modelling Multimodal Dialogues for Social Robots Using Communicative Acts

Open access
Fernández-Rodicio, Enrique, Castro-González, Álvaro, Alonso-Martín, Fernando, Maroto-Gómez, Marcos, Salichs, Miguel Á. · 2020 · Sensors

Abstract

Social Robots need to communicate in a way that feels natural to humans if they are to effectively bond with the users and provide an engaging interaction. Inline with this natural, effective communication, robots need to perceive and manage multimodal information, both as input and output, and respond accordingly. Consequently, dialogue design is a key factor in creating an engaging multimodal interaction. These dialogues need to be flexible enough to adapt to unforeseen circumstances that arise during the conversation but should also be easy to create, so the development of new applications gets simpler. In this work, we present our approach to dialogue modelling based on basic atomic interaction units called Communicative Acts. They manage basic interactions considering who has the initiative (the robot or the user), and what is his/her intention. The two possible intentions are either ask for information or give information. In addition, because we focus on one-to-one interactions, the initiative can only be taken by the robot or the user. Communicative Acts can be parametrised and combined in a hierarchical manner to fulfil the needs of the robot's applications, and they have been equipped with built-in functionalities that are in charge of low-level communication tasks. These tasks include communication error handling, turn-taking or user disengagement. This system has been integrated in Mini, a social robot that has been created to assist older adults with cognitive impairment. In a case of use, we demonstrate the operation of our system as well as its performance in real human-robot interactions.

Overview

CoChat Summary

## TL;DR

The paper proposes a two-level multimodal dialogue model for social robots, based on four basic Communicative Acts (CAs) and reusable Complex Communicative Acts (CCAs), and implements it in the Mini social robot. In a case of use with four seniors, the system handled 88.42% of 354 CAs without problems, and all measured response times stayed below one second.

## Key findings

  • The authors define four basic CAs from two dimensions—initiative and intention: Robot Asks for Information, Robot Gives Information, User Asks for Information, and User Gives Information.
  • They identify five CCAs for Mini’s applications: Right–Wrong Question, Question with Confirmation, Switching Mode Question, Manage Multimedia Content, and Communication Warning.
  • In the case of use, four seniors interacted with Mini for 71 mins, and 354 CAs were used; 313/354 (88.42%) completed without problems, while 41/354 (11.58%) required recovery mechanisms.
  • The paper reports 65 uses of recovery mechanisms overall, including 29 cases of ASR failures/low confidence, 19 cases where the system encouraged the user to answer, and 17 cases where the answer was not the expected one.
  • All CAs used in the case of use had response times below one second, which the authors say satisfies the natural-interaction threshold they adopted.

## Methods

  • The paper proposes a two-level dialogue architecture: applications manage task flow at the application level, while the HRI Manager handles CA execution, prioritization, concurrency, and recovery at the interaction level.
  • The system is implemented in Mini, a tabletop social robot with 5 degrees of freedom, OLED eyes, a lighted heart, and an external tablet; the target users are older adults with mild cognitive impairment.
  • Evaluation uses an objective case study with four senior participants interacting with Mini in real settings; the authors analyze recorded interactions to measure response time and use of recovery mechanisms.

## Limitations

  • The paper states that the evaluation involved only 4 users and only one type of application, so broader generalization would require a larger and more heterogeneous sample.
  • The authors note that the approach is limited to one-to-one interactions and cannot manage multiple users present at the same time.
  • The paper states that multimodal fusion is not implemented yet, which limits interaction capabilities.
  • The use of grammar-based ASR limits verbal input flexibility; this is stated in the paper and makes it evident that synonyms or out-of-grammar responses may be rejected.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.3390/s20123440

Cites

36 of 36 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 Mini, a social robot developed for interacting with older adults that suffer from mild cases of cognitive impairment.
Figure 1 Mini, a social robot developed for interacting with older adults that suffer from mild cases of cognitive impairment.CC BY · PubMed Central
Figure 2 Black-box view of the human–robot interaction (HRI) Architecture (Red box) and its communications with the rest of the robot’s software architecture. This paper focuses on the central piece of the architecture, the HRI Manager (blue box).
Figure 2 Black-box view of the human–robot interaction (HRI) Architecture (Red box) and its communications with the rest of the robot’s software architecture. This paper focuses on the central piece of the architecture, the HRI Manager (blue box).CC BY · PubMed Central
Figure 3 Flow diagram for the right–wrong question Complex Communicative Act (CCA). In this example, the CCA expects a sequence of answers in a given order.
Figure 3 Flow diagram for the right–wrong question Complex Communicative Act (CCA). In this example, the CCA expects a sequence of answers in a given order.CC BY · PubMed Central
Figure 4 Flow diagram for the question with confirmation CCA. This CCA asks the user for an explicit confirmation of the answer if the confidence of the answer recognition is below a given threshold.
Figure 4 Flow diagram for the question with confirmation CCA. This CCA asks the user for an explicit confirmation of the answer if the confidence of the answer recognition is below a given threshold.CC BY · PubMed Central
Figure 5 Flow diagram for the switching mode question CCA. This CCA can be configured with multiple input modes, and then it switches from mode to mode in case of communication problems.
Figure 5 Flow diagram for the switching mode question CCA. This CCA can be configured with multiple input modes, and then it switches from mode to mode in case of communication problems.CC BY · PubMed Central
Figure 6 Flow diagram for the manage multimedia content CCA. This CCA sends the content to the touch screen and allows the user to control the execution of the content with voice commands.
Figure 6 Flow diagram for the manage multimedia content CCA. This CCA sends the content to the touch screen and allows the user to control the execution of the content with voice commands.CC BY · PubMed Central
Figure 7 Flow diagram for the communication warning CCA. This CCA is requested if external peripherals used by the robot stop working. The CCA requests help from the user, waits for the problem to be solved and then thanks the user.
Figure 7 Flow diagram for the communication warning CCA. This CCA is requested if external peripherals used by the robot stop working. The CCA requests help from the user, waits for the problem to be solved and then thanks the user.CC BY · PubMed Central
Figure 8 An older adult interacting with Mini.
Figure 8 An older adult interacting with Mini.CC BY · PubMed Central
Figure 9 Second question of the awareness exercise. The boxes represent actions performed by the different agents and the arrows represent the connections between these actions. Numbers will be referenced in the text to help the reader.
Figure 9 Second question of the awareness exercise. The boxes represent actions performed by the different agents and the arrows represent the connections between these actions. Numbers will be referenced in the text to help the reader.CC BY · PubMed Central
Figure 10 First question of the monuments exercise. The boxes represent actions performed by the different agents and the arrows represent the connections between these actions. Numbers will be referenced in the text to help the reader.
Figure 10 First question of the monuments exercise. The boxes represent actions performed by the different agents and the arrows represent the connections between these actions. Numbers will be referenced in the text to help the reader.CC BY · PubMed Central
Figure 11 Example of a situation with several CAs with different priorities. The boxes represent actions performed by the different agents and the arrows represent the connections between these actions. Numbers will be referenced in the text to help the reader.
Figure 11 Example of a situation with several CAs with different priorities. The boxes represent actions performed by the different agents and the arrows represent the connections between these actions. Numbers will be referenced in the text to help the reader.CC BY · PubMed Central
Figure 12 Fourth question of the exercise. The boxes represent actions performed by the different agents and the arrows represent the connections between these actions. Numbers will be referenced in the text to help the reader.
Figure 12 Fourth question of the exercise. The boxes represent actions performed by the different agents and the arrows represent the connections between these actions. Numbers will be referenced in the text to help the reader.CC BY · PubMed Central
Figure 13 Evaluation of the CAs’ response time. Bars represent the mean value in seconds and whiskers represent standard deviation.
Figure 13 Evaluation of the CAs’ response time. Bars represent the mean value in seconds and whiskers represent standard deviation.CC BY · PubMed Central

Verification

Claims the paper makes

3 Valid · 2 Not valid · 6 Not checked · 97 Own results and interpretations

32 more claims

Create a free account to see all 108 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 53
  • It may state two separate claims as one. 21
  • The quote only partly states this claim. 2

Create a free account to see these claims.

Citations

Cites

Show all 39 references

Research works printed but not linked: 2

  1. Fodor, P. (2007, January 5–7). Dialog management for decision processes. Proceedings of the 3rd Language and Technology Conference: Human Language Technologies as a Challenge for Computer Science and Linguistics, Poznan, Poland.
  2. Salichs, E., Castro-González, Á., Malfaz, M., and Salichs, M.A. (2016, January 2). Mini: A social assistive robot for people with mild cognitive impairment. Proceedings of the New Friends 2016—2nd International Conference on Social Robots in Therapy & and Education, Barcelona, Spain.

Cited by

See all 38

Metadata

Authors
Fernández-Rodicio, E., Castro-González, Á., Alonso-Martín, F., Maroto-Gómez, M., & Salichs, M. Á.
Affiliations
Robotics Lab, Universidad Carlos III de Madrid, Av. de la Universidad 30, Leganés, 28911 Madrid, Spain
Venue
Sensors
Publisher
MDPI AG
Published
2020-06-18
Volume
20
Issue
12
Pages
3440
ISSN
1424-8220 (ISSN-L)
OpenAlex
W3036971864
Cited by
37
References
39
Citations by year
2026: 4 · 2025: 5 · 2024: 7 · 2023: 4 · 2022: 7 · 2021: 8 · 2020: 2
FWCI
2.06
Type
journal article
Language
en
Open access
Locations
  • Sensors (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • e-Archivo (Carlos III University of Madrid) (submittedVersion, cc-by)
Funders
  • Ministerio de Economía y Competitividad (DPI2014-57684-R)
  • Comunidad Autónoma de Madrid (S2013/MIT-2748)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
Keywords
social robots, communicative acts, dialogue design, turn-taking, user disengagement, intent classification, human-robot interaction
MeSH terms
Social Interaction, Aged, Communication, Emotions, Female, Humans, Male, Robotics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-09-30, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-30 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Modelling Multimodal Dialogues for Social Robots Using Communicative Acts. https://app.cochat.ai/papers/10.3390/s20123440
The paper
Fernández-Rodicio, E., Castro-González, Á., Alonso-Martín, F., Maroto-Gómez, M., & Salichs, M. Á. (2020). Modelling Multimodal Dialogues for Social Robots Using Communicative Acts. Sensors. https://doi.org/10.3390/s20123440

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.