Augmented Robotics Dialog System for Enhancing Human-Robot Interaction
Open accessThe DOI matches its Crossref record; 53 of the 57 references it cites were found in a registry or an index.
Abstract
Augmented reality, augmented television and second screen are cutting edge technologies that provide end users extra and enhanced information related to certain events in real time. This enriched information helps users better understand such events, at the same time providing a more satisfactory experience. In the present paper, we apply this main idea to human–robot interaction (HRI), to how users and robots interchange information. The ultimate goal of this paper is to improve the quality of HRI, developing a new dialog manager system that incorporates enriched information from the semantic web. This work presents the augmented robotic dialog system (ARDS), which uses natural language understanding mechanisms to provide two features: (i) a non-grammar multimodal input (verbal and/or written) text; and (ii) a contextualization of the information conveyed in the interaction. This contextualization is achieved by information enrichment techniques that link the extracted information from the dialog with extra information about the world available in semantic knowledge bases. This enriched or contextualized information (information enrichment, semantic enhancement or contextualized information are used interchangeably in the rest of this paper) offers many possibilities in terms of HRI. For instance, it can enhance the robot’s pro-activeness during a human–robot dialog (the enriched information can be used to propose new topics during the dialog, while ensuring a coherent interaction). Another possibility is to display additional multimedia content related to the enriched information on a visual device. This paper describes the ARDS and shows a proof of concept of its applications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures











Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“From the related works, just three of them (Pealtson et al. [16], Alonso et al. [10] and Lison et al. [9]) consider the use of different communication channels. The rest of the works are designed for verbal communication exclusively.”
From Modelling Multimodal Dialogues for Social Robots Using Communicative Acts (2020) · cites this paper as [10]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Knowledge vault: a web-scale approach to probabilistic knowledge fusion
Dong, X., Gabrilovich, E., Heitz, G., Horn, W., Lao, N., Murphy, K., Strohmann, T., Sun, S., & Zhang, W. (2014) · Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2623330.2623623 · Cited by 1,533 · cited 1×
“In contrast to Knowledge Graph, which needs human intervention to increase the knowledge base, Knowledge Vault is automatically created as explained in [69].”
BackgroundSemantic Sentiment Analysis of Twitter
Saif, H., He, Y., & Alani, H. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35176-1_32 · Cited by 521
Strategies for Natural Language Processing
(2014) · Psychology Press eBooks · doi:10.4324/9781315802671 · Cited by 517
Morris, C. (1946) · George Braziller eBooks · doi:10.1037/14607-000 · Cited by 463
User-level sentiment analysis incorporating social networks
Tan, C., Lee, L., Tang, J., Jiang, L., Zhou, M., & Li, P. (2011) · Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2020408.2020614 · Cited by 419
Socially Assistive Robots in Elderly Care: A Mixed-Method Systematic Literature Review
Kachouie, R., Sedighadeli, S., Khosla, R., & Chu, M.-T. (2014) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2013.873278 · Cited by 409
Exploiting social relations for sentiment analysis in microblogging
Hu, X., Tang, L., Tang, J., & Liu, H. (2013) · Proceedings of the sixth ACM international conference on Web search and data mining · doi:10.1145/2433396.2433465 · Cited by 384
Modelling, Design and Robust Control of a Remotely Operated Underwater Vehicle
García-Valdovinos, L. G., Salgado-Jiménez, T., Bandala-Sánchez, M., Nava-Balanzar, L., Hernández-Alvarado, R., & Cruz-Ledesma, J. A. (2014) · International Journal of Advanced Robotic Systems · doi:10.5772/56810 · Cited by 375
Knowledge base completion via search-based question answering
West, R., Gabrilovich, E., Murphy, K., Sun, S., Gupta, R., & Lin, D. (2014) · Proceedings of the 23rd international conference on World wide web · doi:10.1145/2566486.2568032 · Cited by 305 · cited 1×
“All of this extra information is obtained from knowledge databases, which relate an entity to information from the real world, as described in [65,66], and have been integrated into the ARDS.”
BackgroundUse of social commitment robots in the care of elderly people with dementia: A literature review
Mordoch, E., Osterreicher, A., Guse, L., Roger, K., & Thompson, G. (2013) · Maturitas · doi:10.1016/j.maturitas.2012.10.015 · Cited by 270
COLLAGEN: A Collaboration Manager for Software Interface Agents
Rich, C., & Sidner, C. L. (1998) · User Modeling and User-Adapted Interaction · doi:10.1023/a:1008204020038 · Cited by 242 · cited 1×
“The DM is based on collaborative task theory [72] and has been used in other scenarios, as shown in [73], where a non-expert user programs the robot by interaction.”
MethodClues for detecting irony in user-generated contents: oh...!! it's "so easy" ;-)
Carvalho, P., Sarmento, L., Silva, M. J., & de Oliveira, E. (2009) · Proceedings of the 1st international CIKM workshop on Topic-sentiment analysis for mass opinion · doi:10.1145/1651461.1651471 · Cited by 240
Building a multimodal human-robot interface
Perzanowski, D., Schultz, A. C., Adams, W., Marsh, E., & Bugajska, M. (2001) · IEEE Intelligent Systems · doi:10.1109/mis.2001.1183338 · Cited by 219
Spatial Language for Human–Robot Dialogs
Skubic, M., Perzanowski, D., Blisard, S., Schultz, A., Adams, W., Bugajska, M., & Brock, D. (2004) · IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews) · doi:10.1109/tsmcc.2004.826273 · Cited by 216
A study of speech recognition for children and the elderly
Wilpon, J. G., & Jacobsen, C. N. (2002) · 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings · doi:10.1109/icassp.1996.541104 · Cited by 190 · cited 1×
“Due to this communicative limitation and the cognitive problems associated with these groups (lack of attention, difficulties in reasoning, lack of memory or language comprehension), the communication can fail, and consequently, the HRI can be felt as unsatisfactory [16].”
BackgroundDeveloping Corpora for Sentiment Analysis: The Case of Irony and Senti-TUT
Bosco, C., Patti, V., & Bolioli, A. (2013) · IEEE Intelligent Systems · doi:10.1109/mis.2013.28 · Cited by 176
Enabling Multimodal Human–Robot Interaction for the Karlsruhe Humanoid Robot
Stiefelhagen, R., Ekenel, H. K., Fugen, C., Gieselmann, P., Holzapfel, H., Kraft, F., Nickel, K., Voit, M., & Waibel, A. (2007) · IEEE Transactions on Robotics · doi:10.1109/tro.2007.907484 · Cited by 136
Situated Dialogue and Spatial Organization: What, Where… and Why?
Kruijff, G.-J. M., Zender, H., Jensfelt, P., & Christensen, H. I. (2007) · International Journal of Advanced Robotic Systems · doi:10.5772/5701 · Cited by 134 · cited 2×
“Contextualized dialogs are mainly applied to mobile robotics with different purposes, for instance acquiring and learning how the environment is semantically structured [17], learning new routes in a constrained map [60], resolving deictic orders [61] or in collaborative tasks where the robot has to take into account…”
BackgroundA Multimodal Emotion Detection System during Human–Robot Interaction
Alonso-Martín, F., Malfáz, M., Sequeira, J. S., Gorostiza, J. F., & Salichs, M. A. (2013) · Sensors · doi:10.3390/s131115549 · Cited by 133 · cited 1×
“For instance, there is a skill that extracts some features of the user, analyzing the voice footprints, such as his/her name, gender [22], or the main emotion, using multimodal information [21]; also, there is another skill that localizes the external sound source in space [32], so it gives 2D information about where…”
BackgroundSelf-citationSpoken Language Understanding for Natural Interaction: The Siri Experience
Bellegarda, J. R. (2013) · Springer eBooks · doi:10.1007/978-1-4614-8280-2_1 · Cited by 128 · cited 1×
“In the area of ASR and dialog management, Bellegarda presented a statistical ASR system and a dialog manager (Siri) to perform natural interactions between a smart-phone and a user [33].”
BackgroundMaggie: A Robotic Platform for Human-Robot Social Interaction
Salichs, M., Barber, R., Khamis, A., Malfaz, M., Gorostiza, J., Pacheco, R., Rivas, R., Corrales, A., Delgado, E., & Garcia, D. (2006) · 2006 IEEE Conference on Robotics, Automation and Mechatronics · doi:10.1109/ramech.2006.252754 · Cited by 112 · cited 1×
“Maggie, as a robotic research platform, has been described with more detail in other papers ([6,18], and many others), and here, we shall focus on those components relevant to the present paper.”
Self-citationEnhanced Text Retrieval Using Natural Language Processing
Liddy, E. D. (1998) · Bulletin of the American Society for Information Science and Technology · doi:10.1002/bult.91 · Cited by 111 · cited 2×
“Furthermore, Liddy [27] and Feldman [28], based on the levels of Morris [29], suggested that in order to understand natural languages, it is important to distinguish between seven interdependent levels.”
Training personal robots using natural language instruction
Lauria, S., Bugmann, G., Kyriacou, T., Bos, J., & Klein, A. (2001) · IEEE Intelligent Systems · doi:10.1109/5254.956080 · Cited by 104
Children and robots learning to play hide and seek
Trafton, J. G., Schultz, A. C., Perznowski, D., Bugajska, M. D., Adams, W., Cassimatis, N. L., & Brock, D. (2006) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/1121241.1121283 · Cited by 81
Image Binarization for End-to-End Text Understanding in Natural Images
Milyaev, S., Barinova, O., Novikova, T., Kohli, P., & Lempitsky, V. (2013) · IAPR International Conference on Document Analysis and Recognition (ICDAR) · doi:10.1109/icdar.2013.33 · Cited by 70 · cited 1×
“[36] recognized text from a real-time video (see Figure 5).”
BackgroundSalichs, M. A., Encinar, I. P., Salichs, E., Castro-González, Á., & Malfaz, M. (2016) · International Journal of Social Robotics · doi:10.1007/s12369-015-0319-6 · Cited by 69
Self-citationMultimodal Human-Robot Interaction Framework for a Personal Robot
Gorostiza, J. F., Barber, R., Khamis, A. M., Malfaz, M., Pacheco, R., Rivas, R., Corrales, A., Delgado, E., & Salichs, M. A. (2006) · ROMAN 2006 - The 15th IEEE International Symposium on Robot and Human Interactive Communication · doi:10.1109/roman.2006.314392 · Cited by 63
Self-citationINTEGRATION OF A VOICE RECOGNITION SYSTEM IN A SOCIAL ROBOT
Alonso-Martín, F., & Salichs, M. A. (2011) · Cybernetics & Systems · doi:10.1080/01969722.2011.583593 · Cited by 57 · cited 1×
“Automatic speech recognition and optical character recognition techniques use this grammatical information to improve the accuracy of the transcription of the sentences; for this, they use statistical language models or handwritten grammars (for detailed information about ASR techniques in robotics, see [30]).”
MethodSelf-citationNLP Meets the Jabberwocky: Natural Language Processing in Information Retrieval.
Feldman, S. (1999) · Online · Cited by 51
Visuo-auditory Multimodal Emotional Structure to Improve Human-Robot-Interaction
Prado, J. A., Simplício, C., Lori, N. F., & Dias, J. M. M. (2011) · International Journal of Social Robotics · doi:10.1007/s12369-011-0134-7 · Cited by 49
Spoken Multimodal Human-Computer Dialogue in Mobile Environments
Minker, W., Dybkjær, L., & Bühler, D. (2005) · Text, speech and language technology · doi:10.1007/1-4020-3075-4 · Cited by 45 · cited 1×
“In robotics, a dialog system is intended to ease the communication between users and the robot [20].”
BackgroundCreating “companions” for children: the ethics of designing esthetic features for robots
Pearson, Y., & Borenstein, J. (2012) · AI & Society · doi:10.1007/s00146-012-0431-1 · Cited by 39
Knowledge bases in the age of big data analytics
Suchanek, F. M., & Weikum, G. (2014) · Proceedings of the VLDB Endowment · doi:10.14778/2733004.2733069 · Cited by 38
The Google Knowledge Graph: Information gatekeeper or a force to be reckoned with?
(2014) · Strategic Direction · doi:10.1108/sd-04-2014-0049 · Cited by 31
Integration of Intelligence Data through Semantic Enhancement
Salmen, D., Malyuta, T., Hansen, A. D., Cronen, S., & Smith, B. (2011) · PhilPapers (PhilPapers Foundation) · Cited by 26
Grounding the Interaction: Knowledge Management for Interactive Robots
Lemaignan, S. (2013) · Künstliche Intell. · doi:10.1007/s13218-013-0246-3 · Cited by 26
Companion robots for elderly people: Using theatre to investigate potential users' views
Walters, M. L., Koay, K. L., Syrdal, D. S., Campbell, A., & Dautenhahn, K. (2013) · IEEE RO-MAN · doi:10.1109/roman.2013.6628393 · Cited by 25
A multi-modal dialog system for a mobile robot
Toptsis, I., Li, S., Wrede, B., & Fink, G. A. (2004) · Interspeech (USB) · doi:10.21437/interspeech.2004-136 · Cited by 25
Robots to motivate elderly people: Present and future challenges
Pérez, J. G., Lohse, M., & Evers, V. (2013) · IEEE RO-MAN · doi:10.1109/roman.2013.6628392 · Cited by 22
A Study on the Education Assistant System Using Smartphones and Service Robots for Children
Jeong, G., Park, C., You, S., & Ji, S.-H. (2014) · International Journal of Advanced Robotic Systems · doi:10.5772/58389 · Cited by 21
SemRec: A Semantic Enhancement Framework for Tag Based Recommendation
Xu, G., Gu, Y., Dolog, P., Zhang, Y., & Kitsuregawa, M. (2011) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v25i1.8080 · Cited by 19
Speech recognition in Alzheimer's disease with personal assistive robots
Rudzicz, F., Wang, R. H., Begum, M., & Mihailidis, A. (2014) · Workshop on Speech and Language Processing for Assistive Technologies · doi:10.3115/v1/w14-1904 · Cited by 18 · cited 1×
“However, usually, these groups have severe difficulties in communicating with robots [15], and these scenarios are full of numerous challenges.”
BackgroundMultidomain Voice Activity Detection during Human-Robot Interaction
Alonso-Martín, F., Castro‐González, Á., Gorostiza, J. F., & Salichs, M. A. (2013) · Lecture notes in computer science · doi:10.1007/978-3-319-02675-6_7 · Cited by 16
Self-citationUsing verbal instructions for route learning: Instruction Analysis
Bugmann, G., Lauria, S., Kyriacou, T., Klein, E., Bos, J., & Coventry, K. R. (2001) · Cited by 15
User Localization During Human-Robot Interaction
Alonso-Martín, F., Gorostiza, J. F., Malfáz, M., & Salichs, M. A. (2012) · Sensors · doi:10.3390/s120709913 · Cited by 14 · cited 1×
“For instance, there is a skill that extracts some features of the user, analyzing the voice footprints, such as his/her name, gender [22], or the main emotion, using multimodal information [21]; also, there is another skill that localizes the external sound source in space [32], so it gives 2D information about where…”
BackgroundSelf-citationMULTIMODAL FUSION AS COMMUNICATIVE ACTS DURING HUMAN–ROBOT INTERACTION
Alonso-Martín, F., Gorostiza, J. F., Malfáz, M., & Salichs, M. A. (2013) · Cybernetics & Systems · doi:10.1080/01969722.2013.832096 · Cited by 13 · cited 2×
“The multimodal fusion module groups together the enriched information with the data provided by the other perception skills [24].”
BackgroundSelf-citationRealistic Language Comprehension
Riesbeck, C. K. (1982) · Cited by 13
Speaker identification using three signal voice domains during human-robot interaction
Alonso-Martín, F., Ramey, A., & Salichs, M. A. (2014) · ACM/IEEE International Conference on Human-Robot Interaction (HRI) · doi:10.1145/2559636.2563706 · Cited by 11
Self-citationCollective Collaborative Tagging System
Choi, J. Y., Rosen, J., Maini, S., Pierce, M., & Fox, G. (2008) · Grid Computing Environments · doi:10.1109/gce.2008.4738442 · Cited by 9
Pino, M., Boulay, M., & Rigaud, A. (2013) · Alzheimer s & Dementia · doi:10.1016/j.jalz.2013.04.205 · Cited by 7
Teaching sequences to a social robot by voice interaction
Gorostiza, J. F., & Salichs, M. A. (2009) · IEEE RO-MAN · doi:10.1109/roman.2009.5326334 · Cited by 5 · cited 1×
“The DM is based on collaborative task theory [72] and has been used in other scenarios, as shown in [73], where a non-expert user programs the robot by interaction.”
MethodSelf-citationCognitive Tools for Humanoid Robots in Space
Sofge, D., Perzanowski, D., Skubic, M., Bugajska, M. D., Trafton, J. G., Cassimatis, N., Brock, D., Adams, W., & Scbultz, A. (2004) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)32199-7 · Cited by 4 · cited 1×
“Contextualized dialogs are mainly applied to mobile robotics with different purposes, for instance acquiring and learning how the environment is semantically structured [17], learning new routes in a constrained map [60], resolving deictic orders [61] or in collaborative tasks where the robot has to take into account…”
Background(2009) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings
Research works printed but not linked: 3
- Gorostiza, J., Barber, R., Khamis, A., Malfaz, M., Pacheco, R., Rivas, R., Corrales, A., Delgado, E., and Salichs, M. (2006, January 6–8). Multimodal Human-Robot Interaction Framework for a Personal Robot. Hatfield, UK. · DOI · Paper page
- Alonso-Martin, F., Castro-González, A., Gorostiza, J.F., and Salichs, M.A. (2013). Social Robotics, Springer International Publishing.
- Bellegarda, J.R. (2014). Natural Interaction with Robots, Knowbots and Smartphones, Springer.
Cited by
A Survey on Dialogue Management in Human-robot Interaction
Reimann, M. M., Kunneman, F., Oertel, C., & Hindriks, K. V. (2024) · ACM Transactions on Human-Robot Interaction · doi:10.1145/3648605 · Cited by 63
“[41] presents such a system, where multiple slots can be illed at a time, for action requests for the robot Maggie.”
InfluentialPollmann, K., Loh, W., Fronemann, N., & Ziegler, D. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2023.122376 · Cited by 48
Intention Understanding in Human–Robot Interaction Based on Visual-NLP Semantics
Li, Z., Mu, Y., Sun, Z., Song, S., Su, J., & Zhang, J. (2021) · Frontiers in Neurorobotics · doi:10.3389/fnbot.2020.610139 · Cited by 45
“With the development of neural networks, some researchers tried to tackle the problem of natural language comprehension as a classification problem and connect the natural language representations of objects with objects in images (Matuszek et al., 2014; Alonso-Martín et al., 2015), although it turned out that classif…”
BackgroundAyanoğlu, H., & Sequeira, J. S. (2019) · Human-computer interaction series · doi:10.1007/978-3-319-96722-6_3 · Cited by 41
SeleCon: Scalable IoT Device Selection and Control Using Hand Gestures
Alanwar, A., Alzantot, M., Ho, B.-J., Martin, P. D., & Srivastava, M. B. (2017) · IEEE/ACM International Conference on Internet-of-Things Design and Implementation (IoTDI) · doi:10.1145/3054977.3054981 · Cited by 40
“Dierent forms of interaction have been proposed, such as speech recognition [10], face recognition [29], gaze/eye tracking [31], and hand gesture tracking [15].”
Background
Metadata
- Authors
- Alonso-Martín, F., Castro-González, A., Luengo, F., & Salichs, M.
- Affiliations
- Robotics Lab, Universidad Carlos III de Madrid, Av. de la Universidad 30, Leganés, Madrid 28911, Spain
- Universidad Carlos III de Madrid (ES)
- Venue
- Sensors
- Publisher
- MDPI AG
- Published
- 2015-07-03
- Volume
- 15
- Issue
- 7
- Pages
- 15799-15829
- ISSN
- 1424-8220 (ISSN-L)
- PMC
- PMC4541855
- OpenAlex
- W1500703277
- Cited by
- 27
- References
- 57
- Citations by year
- 2026: 2 · 2024: 3 · 2023: 1 · 2022: 1 · 2021: 1 · 2020: 6 · 2019: 3 · 2018: 3 · 2017: 4 · 2016: 2 · 2015: 1
- FWCI
- 3.26
- Type
- journal article
- Language
- en
- Open access
- gold
- dx.doi.org
- Licence
- Locations
- Sensors (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- e-Archivo (Carlos III University of Madrid) (submittedVersion, cc-by-nc-nd)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Social Sciences › Business, Management and Accounting › Marketing › AI in Service Interactions
- Keywords
- human-robot interaction, dialogue manager, augmented reality, semantic web, information enrichment, contextualization, natural language understanding, multimodal input, knowledge base, second screen
- MeSH terms
- Cybernetics, Humans, Natural Language Processing, Robotics, Speech, User-Computer Interface
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.