Embodied conversational agents for multimodal automated social skills training in people with autism spectrum disorders
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Social skills training, performed by human trainers, is a well-established method for obtaining appropriate skills in social interaction. Previous work automated the process of social skills training by developing a dialogue system that teaches social communication skills through interaction with a computer avatar. Even though previous work that simulated social skills training only considered acoustic and linguistic information, human social skills trainers take into account visual and other non-verbal features. In this paper, we create and evaluate a social skills training system that closes this gap by considering the audiovisual features of the smiling ratio and the head pose (yaw and pitch). In addition, the previous system was only tested with graduate students; in this paper, we applied our system to children or young adults with autism spectrum disorders. For our experimental evaluation, we recruited 18 members from the general population and 10 people with autism spectrum disorders and gave them our proposed multimodal system to use. An experienced human social skills trainer rated the social skills of the users. We evaluated the system's effectiveness by comparing pre- and post-training scores and identified significant improvement in their social skills using our proposed multimodal system. Computer-based social skills training is useful for people who experience social difficulties. Such a system can be used by teachers, therapists, and social skills trainers for rehabilitation and the supplemental use of human-based training anywhere and anytime.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Current agents are available via multiple digital platforms, including telephones, mobile phones, tablets, and computers, and in many virtual formats such as chatbots, embodied conversational agents, and three-dimensional avatars [2,7,8].”
From Effectiveness of Conversational Agents (Virtual Assistants) in Health Care: Protocol for a Systematic Review (2020) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Diagnostic and Statistical Manual of Mental Disorders(DSM)
(2008) · Encyclopedia of Counseling · doi:10.4135/9781412963978.n182 · Cited by 10,830
Baron-Cohen, S., Wheelwright, S., Skinner, R., Martin, J., & Clubley, E. (2001) · Journal of Autism and Developmental Disorders · doi:10.1023/a:1005653411471 · Cited by 6,450
Dual-task interference in simple tasks: data and theory
Pashler, H. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.116.2.220 · Cited by 3,112
Coding facial expressions with Gabor wavelets
Lyons, M., Akamatsu, S., Kamachi, M., & Gyoba, J. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670949 · Cited by 2,210
Baron-Cohen, S., Richler, J., Bisarya, D., Gurunathan, N., & Wheelwright, S. (2003) · Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rstb.2002.1206 · Cited by 870
Parsons, S., & Mitchell, P. (2002) · Journal of Intellectual Disability Research · doi:10.1046/j.1365-2788.2002.00425.x · Cited by 521
Bauminger, N. (2002) · Journal of Autism and Developmental Disorders · doi:10.1023/a:1016378718278 · Cited by 500
Embodied conversational agents in clinical psychology: a scoping review
Provoost, S., Lau, H. M., Ruwaard, J., & Riper, H. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6553 · Cited by 433
Face alignment through subspace constrained mean-shifts
Saragih, J. M., Lucey, S., & Cohn, J. F. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459377 · Cited by 368
MACH: my automated conversation coach
Hoque, M. (., Courgeon, M., Martin, J.-C., Mutlu, B., & Picard, R. W. (2013) · Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing · doi:10.1145/2493432.2493502 · Cited by 290
Wainer, A. L., & Ingersoll, B. R. (2011) · Research in Autism Spectrum Disorders · doi:10.1016/j.rasd.2010.08.002 · Cited by 267
Pedagogical Agent Design: The Impact of Agent Realism, Gender, Ethnicity, and Instructional Role
Baylor, A. L., & Kim, Y. (2004) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-30139-4_56 · Cited by 240
Computer-Aided Learning for People with Autism – a Framework for Research and Development
Moore, D., McGrath, P., & Thorpe, J. (2000) · Innovations in Education and Training International · doi:10.1080/13558000050138452 · Cited by 234
Technologies as Support Tools for Persons with Autistic Spectrum Disorder: A Systematic Review
Aresti-Bartolome, N., & Garcia-Zapirain, B. (2014) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph110807767 · Cited by 232
Social skills training for schizophrenia: A step-by-step guide
Corrigan, P. W. (1998) · Schizophrenia Research · doi:10.1016/s0920-9964(97)00145-x · Cited by 220
Social Skills Training for Schizophrenia: A Step-by-Step Guide
Bellack, A. S., Mueser, K. T., Gingerich, S. L., & Agresta, J. (1997) · Cited by 212
Abnormal Speech Spectrum and Increased Pitch Variability in Young Autistic Children
Bonneh, Y. S., Levanon, Y., Dean-Pardo, O., Lossos, L., & Adini, Y. (2011) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2010.00237 · Cited by 172
Interactive Technologies for Autism
Kientz, J. A., Goodwin, M. S., Hayes, G. R., & Abowd, G. D. (2013) · Synthesis lectures on assistive, rehabilitative, and health-preserving technologies · doi:10.2200/s00533ed1v01y201309arh004 · Cited by 154
Naim, I., Tanveer, M. I., Gildea, D., & Hoque, M. E. (2015) · International Conference on Automatic Face and Gesture Recognition · doi:10.1109/fg.2015.7163127 · Cited by 139
Exploring feedback strategies to improve public speaking: an interactive virtual audience framework
Chollet, M., Wörtwein, T., Morency, L.-P., Shapiro, A., & Scherer, S. (2015) · Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing · doi:10.1145/2750858.2806060 · Cited by 124
Rhema: A Real-Time In-Situ Intelligent Interface to Help People with Public Speaking
Tanveer, M. I., Lin, E., & Hoque, M. (. (2015) · Proceedings of the 20th International Conference on Intelligent User Interfaces · doi:10.1145/2678025.2701386 · Cited by 112
Essau, C. A., Olaya, B., Sasagawa, S., Pithia, J., Bray, D. E., & Ollendick, T. H. (2014) · Journal of Affective Disorders · doi:10.1016/j.jad.2014.05.056 · Cited by 84
The Internet for educating individuals with social impairments
Bishop, J. (2003) · Journal of Computer Assisted Learning · doi:10.1046/j.0266-4909.2003.00057.x · Cited by 70
Bone, D., Black, M. P., Lee, C.-C., et al. (2012) · Interspeech (USB) · doi:10.21437/interspeech.2012-307 · Cited by 58
Teaching Social Communication Skills Through Human-Agent Interaction
Tanaka, H., Sakti, S., Neubig, G., Toda, T., Negoro, H., Iwasaka, H., & Nakamura, S. (2016) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/2937757 · Cited by 40
Self-citationTanaka, H., Sakti, S., Neubig, G., Toda, T., & Nakamura, S. (2014) · Workshop on Computational Linguistics and Clinical Psychology (CLPsych) · doi:10.3115/v1/w14-3211 · Cited by 33
Self-citationSchuller, B. W., Marchi, E., Baron‐Cohen, S., O’Reilly, H., Pigat, D., Robinson, P., & Daves, I. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1403.5912 · Cited by 31
Nezu, S., Iwasaka, H., Saeki, K., Ishizuka, R., Goma, H., Okamoto, N., Makino, H., Tanimura, M., Yoshizaki, K., Obayashi, K., et al. (2014) · Environmental Health and Preventive Medicine · doi:10.1007/s12199-014-0427-1 · Cited by 22
Self-citationNOCOA+: Multimodal Computer-Based Training for Social and Communication Skills
Tanaka, H., Sakti, S., Neubig, G., Toda, T., & Nakamura, S. (2015) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2014edp7400 · Cited by 10
Self-citationAutomated social skills training with audiovisual information
Tanaka, H., Sakti, S., Neubig, G., et al. (2016) · doi:10.1109/embc.2016.7591180 · Cited by 5
Self-citation
Research works printed but not linked: 2
- Naim I, Tanveer MI, Gildea D, et al. Automated Analysis and Prediction of Job Interview Performance. arXiv preprint arXiv:150403425. 2015;.
- D Weschler, 2003, Wechsler intelligence scale for children-forth edition (WISC-IV) administration and scoring manual
Cited by
Conversational agents in healthcare: a systematic review
Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R., Surian, D., Gallego, B., Magrabi, F., Lau, A. Y. S., et al. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy072 · Cited by 1,388
“Out of 14 conversational agents, 5 were embodied conversational agents 20,22–24,35 The combination of architectures, initiatives, and dialogue management approaches are illustrated in Figure 2.”
BackgroundMethodAn overview of the features of chatbots in mental health: A scoping review
Abd-alrazaq, A. A., Alajlani, M., Alalwan, A. A., Bewick, B. M., Gardner, P., & Househ, M. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.103978 · Cited by 605
“This may be attributed to the fact that depression and anxiety are the most common mental disorder around the world [44, 45], and chatbots are reported to be effective in improving social skills for patients with autism [14, 17, 46-48].”
BackgroundThe Personalization of Conversational Agents in Health Care: Systematic Review
Kocaballi, A. B., Berkovsky, S., Quiroz, J. C., Laranjo, L., Tong, H. L., Rezazadegan, D., Briatore, A., & Coiera, E. (2019) · Journal of Medical Internet Research · doi:10.2196/15360 · Cited by 332
“Personalized content included: (1) feedback on mood states [35], narrative skills [36], symptom summaries [37], meditation practice [38], and current progress towards the goals set [39,40,46,47]; (2) reminders [37,47], warnings, and alerts [39,40,42,43]; (3) multimedia [35,46]; and (4) questions on…”
BackgroundMethodSurvey of conversational agents in health
Montenegro, J. L. Z., da Costa, C. A., & da Rosa Righi, R. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.03.054 · Cited by 322
Voice-Controlled Intelligent Personal Assistants in Health Care: International Delphi Study
(2021) · doi:10.2196/25312 · Cited by 78
Metadata
- Authors
- Tanaka, H., Negoro, H., Iwasaka, H., & Nakamura, S.
- Affiliations
- Nara Institute of Science and Technology (JP)
- Nara University of Education (JP)
- Corresponding author
- Tanaka, Hiroki
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-08-10
- Volume
- 12
- Issue
- 8
- Pages
- e0182151
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC5552034
- OpenAlex
- W2743233975
- Cited by
- 145
- References
- 30
- Citations by year
- 2026: 7 · 2025: 16 · 2024: 11 · 2023: 32 · 2022: 17 · 2021: 21 · 2020: 20 · 2019: 15 · 2018: 5
- FWCI
- 15.66
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Institutional Repositories DataBase (IRDB) (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- NAIST Digital Library (Nara Institute of Science and Technology) (submittedVersion, cc-by-nc-nd)
- Funders
- JSPS KAKEN (26540117, 16K16172)
- Topics
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Life Sciences › Neuroscience › Cognitive Neuroscience › Autism Spectrum Disorder Research
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- embodied conversational agents, social skills training, autism spectrum disorder, multimodal dialogue systems, audio-visual features, head orientation, nonverbal communication, social communication skills, children and young adults
- MeSH terms
- Autism Spectrum Disorder, Adult, Behavior Therapy, Communication, Female, Humans, Interpersonal Relations, Male, Young Adult, Social Skills
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.