Embodied conversational agents for multimodal automated social skills training in people with autism spectrum disorders
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Social skills training, performed by human trainers, is a well-established method for obtaining appropriate skills in social interaction. Previous work automated the process of social skills training by developing a dialogue system that teaches social communication skills through interaction with a computer avatar. Even though previous work that simulated social skills training only considered acoustic and linguistic information, human social skills trainers take into account visual and other non-verbal features. In this paper, we create and evaluate a social skills training system that closes this gap by considering the audiovisual features of the smiling ratio and the head pose (yaw and pitch). In addition, the previous system was only tested with graduate students; in this paper, we applied our system to children or young adults with autism spectrum disorders. For our experimental evaluation, we recruited 18 members from the general population and 10 people with autism spectrum disorders and gave them our proposed multimodal system to use. An experienced human social skills trainer rated the social skills of the users. We evaluated the system's effectiveness by comparing pre- and post-training scores and identified significant improvement in their social skills using our proposed multimodal system. Computer-based social skills training is useful for people who experience social difficulties. Such a system can be used by teachers, therapists, and social skills trainers for rehabilitation and the supplemental use of human-based training anywhere and anytime.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Current agents are available via multiple digital platforms, including telephones, mobile phones, tablets, and computers, and in many virtual formats such as chatbots, embodied conversational agents, and three-dimensional avatars [2,7,8].”
From Effectiveness of Conversational Agents (Virtual Assistants) in Health Care: Protocol for a Systematic Review (2020) · cites this paper as [8]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Diagnostic and Statistical Manual of Mental Disorders(DSM)
(2008) · Encyclopedia of Counseling · doi:10.4135/9781412963978.n182 · Cited by 10,830
Baron-Cohen, S., Wheelwright, S., Skinner, R., Martin, J., & Clubley, E. (2001) · Journal of Autism and Developmental Disorders · doi:10.1023/a:1005653411471 · Cited by 6,450
Dual-task interference in simple tasks: data and theory
Pashler, H. (1994) · Psychological Bulletin · doi:10.1037/0033-2909.116.2.220 · Cited by 3,112
Coding facial expressions with Gabor wavelets
Lyons, M., Akamatsu, S., Kamachi, M., & Gyoba, J. (2002) · Proceedings Third IEEE International Conference on Automatic Face and Gesture Recognition · doi:10.1109/afgr.1998.670949 · Cited by 2,210
Baron-Cohen, S., Richler, J., Bisarya, D., Gurunathan, N., & Wheelwright, S. (2003) · Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences · doi:10.1098/rstb.2002.1206 · Cited by 870
Parsons, S., & Mitchell, P. (2002) · Journal of Intellectual Disability Research · doi:10.1046/j.1365-2788.2002.00425.x · Cited by 521
Bauminger, N. (2002) · Journal of Autism and Developmental Disorders · doi:10.1023/a:1016378718278 · Cited by 500
Embodied conversational agents in clinical psychology: a scoping review
Provoost, S., Lau, H. M., Ruwaard, J., & Riper, H. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.6553 · Cited by 433
Face alignment through subspace constrained mean-shifts
Saragih, J. M., Lucey, S., & Cohn, J. F. (2009) · 2009 IEEE 12th International Conference on Computer Vision · doi:10.1109/iccv.2009.5459377 · Cited by 368
MACH: my automated conversation coach
Hoque, M. (., Courgeon, M., Martin, J.-C., Mutlu, B., & Picard, R. W. (2013) · Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing · doi:10.1145/2493432.2493502 · Cited by 290
Research works printed but not linked: 2
- Naim I, Tanveer MI, Gildea D, et al. Automated Analysis and Prediction of Job Interview Performance. arXiv preprint arXiv:150403425. 2015;.
- D Weschler, 2003, Wechsler intelligence scale for children-forth edition (WISC-IV) administration and scoring manual
Cited by
Conversational agents in healthcare: a systematic review
Laranjo, L., Dunn, A. G., Tong, H. L., Kocaballi, A. B., Chen, J., Bashir, R., Surian, D., Gallego, B., Magrabi, F., Lau, A. Y. S., et al. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy072 · Cited by 1,388
“Out of 14 conversational agents, 5 were embodied conversational agents 20,22–24,35 The combination of architectures, initiatives, and dialogue management approaches are illustrated in Figure 2.”
BackgroundMethodAn overview of the features of chatbots in mental health: A scoping review
Abd-alrazaq, A. A., Alajlani, M., Alalwan, A. A., Bewick, B. M., Gardner, P., & Househ, M. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.103978 · Cited by 605
“This may be attributed to the fact that depression and anxiety are the most common mental disorder around the world [44, 45], and chatbots are reported to be effective in improving social skills for patients with autism [14, 17, 46-48].”
BackgroundThe Personalization of Conversational Agents in Health Care: Systematic Review
Kocaballi, A. B., Berkovsky, S., Quiroz, J. C., Laranjo, L., Tong, H. L., Rezazadegan, D., Briatore, A., & Coiera, E. (2019) · Journal of Medical Internet Research · doi:10.2196/15360 · Cited by 332
“Personalized content included: (1) feedback on mood states [35], narrative skills [36], symptom summaries [37], meditation practice [38], and current progress towards the goals set [39,40,46,47]; (2) reminders [37,47], warnings, and alerts [39,40,42,43]; (3) multimedia [35,46]; and (4) questions on…”
BackgroundMethodSurvey of conversational agents in health
Montenegro, J. L. Z., da Costa, C. A., & da Rosa Righi, R. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2019.03.054 · Cited by 322
Voice-Controlled Intelligent Personal Assistants in Health Care: International Delphi Study
(2021) · doi:10.2196/25312 · Cited by 78
Metadata
- Authors
- Tanaka, H., Negoro, H., Iwasaka, H., & Nakamura, S.
- Affiliations
- Nara Institute of Science and Technology (JP)
- Nara University of Education (JP)
- Corresponding author
- Tanaka, Hiroki
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-08-10
- Volume
- 12
- Issue
- 8
- Pages
- e0182151
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC5552034
- OpenAlex
- W2743233975
- Cited by
- 145
- References
- 30
- Citations by year
- 2026: 7 · 2025: 16 · 2024: 11 · 2023: 32 · 2022: 17 · 2021: 21 · 2020: 20 · 2019: 15 · 2018: 5
- FWCI
- 15.66
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Institutional Repositories DataBase (IRDB) (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- NAIST Digital Library (Nara Institute of Science and Technology) (submittedVersion, cc-by-nc-nd)
- Funders
- JSPS KAKEN (26540117, 16K16172)
- Topics
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Life Sciences › Neuroscience › Cognitive Neuroscience › Autism Spectrum Disorder Research
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- embodied conversational agents, social skills training, autism spectrum disorder, multimodal dialogue systems, audio-visual features, head orientation, nonverbal communication, social communication skills, children and young adults
- MeSH terms
- Autism Spectrum Disorder, Adult, Behavior Therapy, Communication, Female, Humans, Interpersonal Relations, Male, Young Adult, Social Skills
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Embodied conversational agents for multimodal automated social skills training in people with autism spectrum disorders. https://app.cochat.ai/papers/10.1371/journal.pone.0182151
- The paper
- Tanaka, H., Negoro, H., Iwasaka, H., & Nakamura, S. (2017). Embodied conversational agents for multimodal automated social skills training in people with autism spectrum disorders. PLOS ONE. https://doi.org/10.1371/journal.pone.0182151
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.