CoChat
Sign Up
Cited by 47Social Robot Interaction and HRIIntelligent Tutoring Systems and Adaptive LearningAutism Spectrum Disorder Research

RAISE: Robotics & AI to improve STEM and social skills for elementary school students

Open access
Hughes, Charles E., Dieker, Lisa A., Glavey, Eileen M., Hines, Rebecca A., Wilkins, Ilene, Ingraham, Kathleen, Bukaty, Caitlyn A., Ali, Kamran +3 more·2022·Frontiers in Virtual Reality
DOI 10.3389/frvir.2022.968312
Open this paper in CoChat — free accountOpen PDF

Abstract

The authors present the design and implementation of an exploratory virtual learning environment that assists children with autism (ASD) in learning science, technology, engineering, and mathematics (STEM) skills along with improving social-emotional and communication skills. The primary contribution of this exploratory research is how educational research informs technological advances in triggering a virtual AI companion (AIC) for children in need of social-emotional and communication skills development. The AIC adapts to students’ varying levels of needed support. This project began by using puppetry control (human-in-the-loop) of the AIC, assisting students with ASD in learning basic coding, practicing their social skills with the AIC, and attaining emotional recognition and regulation skills for effective communication and learning. The student is given the challenge to program a robot, Dash™, to move in a square. Based on observed behaviors, the puppeteer controls the virtual agent’s actions to support the student in coding the robot. The virtual agent’s actions that inform the development of the AIC include speech, facial expressions, gestures, respiration, and heart color changes coded to indicate emotional state. The paper provides exploratory findings of the first 2 years of this 5-year scaling-up research study. The outcomes discussed align with a common approach of research design used for students with disabilities, called single case study research. This type of design does not involve random control trial research; instead, the student acts as her or his own control subject. Students with ASD have substantial individual differences in their social skill deficits, behaviors, communications, and learning needs, which vary greatly from the norm and from other individuals identified with this disability. Therefore, findings are reported as changes within subjects instead of across subjects. While these exploratory observations serve as a basis for longer term research on a larger population, this paper focuses less on student learning and more on evolving technology in AIC and supporting students with ASD in STEM environments.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3389/frvir.2022.968312

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI tools using NLP can alleviate the need for reinforcement from adults while helping increase self-regulation skills.

    “These types of AI tools using NLP can alleviate the need for reinforcement from adults, while helping increase self-regulation skills (Hughes et al., 2022).”

    From The Future of Artificial Intelligence in Special Education Technology (2023) · cites this paper as Hughes et al., 2022

    The Introduction (section 1) says the AI companion is intended to support students’ self-regulation instead of requiring constant monitoring by a teacher or instructional assistant. But the paper does not establish this as an achieved effect of an NLP tool: section 4.2.3 says the classroom phase still uses a human in the loop and its language patterns have yet to be analyzed for fully AI-driven behavior. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 48 · Cited by 47

Research works printed but not linked: 46

  1. Abualsamid, A., and Hughes, C. E. (2018). "Modeling augmentative communication with Amazon lex and polly," in Advances in usability, user experience and assistive technology. AHFE 2018. Advances in intelligent systems and computing. Editors T. Ahram and C. Falcão (Cham: Springer), 794, 871-879.
  2. Acar, C., Tekin-İftar, E., and Yıkmış, A. (2016). Effects of mother-delivered social stories and video modeling in teaching social skills to children with autism spectrum disorders. J. Spec. Educ. 50 (4), 215-226. doi:10.1177/ 0022466916649164 · DOI
  3. Ali, K., and Hughes, C. E. (2020). "Face reenactment based facial expression recognition," in Advances in visual computing. ISVC 2020. Lecture Notes in computer science 12509. Editors (Cham: Springer), 501-513.
  4. Ali, K., and Hughes, C. E. (2021). Facial expression recognition by using a disentangled identity-invariant expression representation. 25 th Int. Conf. Pattern Recognit. (ICPR 2020), 9460-9467. doi:10.1109/ICPR48806.2021. 941217 American Psychiatric Association (2013). Diagnostic and statistical manual of mental disorders. Available at: https://doi-org.ezproxy.frederick.edu/10.1176/appi. books.9780890425596. · DOI · Paper page
  5. Anderson, K., Dubiel, T., Tanaka, K., and Worsley, M. (2019). Proceedings of the 2019 international conference on multimodal interaction, 506-507. doi:10.1145/ 3340555.3358662 Chemistry pods: A mutlimodal real time and retrospective tool for the classroom. · DOI
  6. Bai, M., Xie, W., and Shen, L. (2019). Proceedings of 2019 IEEE international conference on image processing (ICIP), 31-35.Disentangled feature based adversarial learning for facial expression recognition.
  7. Baskoro, R. A. (2021). The comparison of numbered head together learning models and think pair share in terms of elementary school mathematics learning outcomes. International. J. Elem. Educ. 4 (4), 549-557. doi:10.23887/ijee.v414.32568 · DOI
  8. Beddiar, D. R., Nini, B., Sabokrou, M., and Hadid, A. (2020). Vision-based human activity recognition: A survey. Multimed. Tools Appl. 79 (41), 30509-30555. doi:10. 1007/s11042-020-09004-3 · DOI
  9. Belpaeme, T., Kennedy, J., Ramachandran, A., Scassellati, B., and Tanaka, F. (2018). Social robots for education: A review. Sci. Robot. 3 (21), eaat5954. doi:10. 1126/scirobotics.aat5954 · DOI · Paper page
  10. Ben-Ari, M., and Mondada, F. (2018). "Robots and their applications," in Elements of robotics (Cham: Springer). doi:10.1007/978-3-319-62533-1_1 · DOI
  11. Bers, M. U. (2010). Issue editor's notes. New media and technology: Youth as content creators. New Dir. Youth Dev. 12 (2), 1-4. doi:10.1002/yd.369 · DOI
  12. Bulagang, A. F., Mountstephens, J., and Teo, J. (2021). Multiclass emotion prediction using heart rate and virtual reality stimuli. J. Big Data 8 (1), 12. doi:10.1186/s40537-020-00401-x · DOI
  13. Editors G. Bebis, R. Boyle, B. Parvin, D. Koracin, A. Ushizima, S. Chai, et al. (Cham: Springer), 540-551.
  14. Dieker, L. A., Straub, C., Hynes, M., Hughes, C. E., Bukaty, C., Bousfield, T., et al. (2019). Using virtual rehearsal in a simulator to impact the performance of science teachers. Int. J. Gaming Comput. Mediat. Simul. 11 (4), 1-20. doi:10.4018/IJGCMS. 2019100101 · DOI
  15. Donnermann, M., Schaper, P., and Lugrin, B. (2022). Social robots in applied settings: A long-term study on adaptive robotic tutors in higher education. Front. Robot. AI 9, 831633. doi:10.3389/frobt.2022.831633 · DOI
  16. Geist, E. (2016). Robots, programming and coding, oh my. Child. Educ. 92, 298-304. doi:10.1080/00094056.2016.1208008 · DOI
  17. Ginosar, S., Bar, A., Kohavi, G., Chan, C., Owens, A., and Malik, J. (2019). Learning individual styles of conversational gesture. Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 3497-3506. doi:10.1109/CVPR.2019.00361 · DOI
  18. Editors G. Burnett, S. Coquillard, R. VanLiere, and G. Welch (Springer LNCS), 133-146.
  19. Humphrey, N., and Symes, W. (2011). Peer interaction patterns among adolescents with autistic spectrum disorders (ASDs) in mainstream school settings. Autism 15 (4), 397-419. doi:10.1177/1362361310387804 Individuals with Disabilities (2004). Individuals with disabilities education improvement act. Stat 118. · DOI
  20. Keating, C. T., and Cook, J. L. (2020). Facial expression production and recognition in autism spectrum disorders: A shifting landscape. Child Adolesc. Psychiatric Clin. N. Am. 29 (3), 557-571. doi:10.1016/j.chc.2020.02.006 · DOI · Paper page
  21. Khalil, A., Ahmed, S. G., Khattak, A. M., and Al-Qirim, N. (2020). Investigating bias in facial analysis systems: A systematic review. IEEE Access 8, 130751-130761. doi:10.1109/access.2020.3006051 · DOI
  22. Krzeminska, A., and Hawse, S. (2020). "Mainstreaming neurodiversity for an inclusive and sustainable future workforce: Autism-spectrum employees," in Industry and higher Education: Case studies for sustainable futures (Springer), 229-261.
  23. Laugeson, E., Gantman, A., Kapp, S., Orenski, K., and Ellingsen, R. (2015). A randomized controlled trial to improve social skills in young adults with autism spectrum disorder: The UCLA peers program. J. Autism Dev. Disord. 45 (12), 3978-3989. doi:10.1007/s10803-015-2504-8 · DOI
  24. Levenson, R. W., Ekman, P., and Friesen, W. V. (1990). Voluntary facial action generates emotion-specific autonomic nervous system Activity. Psychophysiology 27, 363-384. doi:10.1111/j.1469-8986.1990.tb02330.x · DOI · Paper page
  25. Lin, J., Jiang, F., and Shen, R. (2018). IEEE international conference on acoustics. New York, NY, United States: Speech & Signal Processing ICASSP, 6453-6457.
  26. Lye, S. Y., and Koh, J. H. L. (2014). Review on teaching and learning of computational thinking through programming: What is next for K-12? Comput. Hum. Behav. 41, 51-61. doi:10.1016/j.chb.2014.09.012 · DOI · Paper page
  27. Monkaresi, H., Bosch, N., Calvo, R., and D'Mello, S. (2016). Automated detection of engagement using video-based estimation of facial expressions and heart rate. IEEE Trans. Affect. Comput. 8 (1), 15-28. doi:10.1109/taffc. 2016.2515084 · DOI
  28. Mutluer, T., Aslan Genç, H., Özcan Morey, A., Yapici Eser, H., Ertinmaz, B., Can, M., et al. (2022).Population-based psychiatric comorbidity in children and adolescents with autism spectrum disorder: A meta-analysis, Front. Psychiatry , PMC9186340. PMID: 35693977. doi:10.3389/fpsyt.2022. 856208 · DOI
  29. Nezami, O. M., Dras, M., Hamey, L., Richards, D., Wan, S., and Paris, C. (2019). Joint European conference on machine learning and knowledge discovery in databases. Springer, 273-289.Automatic recognition of student engagement using deep learning and facial expression.
  30. Nojavanasghari, B., Baltrusaitis, T., Hughes, C. E., and Morency, L-P. (2016). EmoReact: A multimodal approach and dataset for recognizing emotional responses in children. Proc. Int. Conf. Multimodal Interact. (ICMI 2016), 137-144. doi:10.1145/2993148.2993168 · DOI
  31. Odom, S. L., Cox, A. W., and Brock, M. E. (2013). Implementation science, professional development, and autism spectrum disorders. Except. Child. 79 (2), 233-251. doi:10.1177/001440291307900207 · DOI
  32. Odom, S., Thompson, J., Hedges, S., Boyd, B., Dykstra, J., Duda, M., et al. (2015). Technology-aided interventions and instruction for adolescents with autism spectrum disorder. J. Autism Dev. Disord. 45 (12), 3805-3819. doi:10.1007/ s10803-014-2320-6 · DOI
  33. Ola, L., and Gullon-Scott, F. (2020). Facial emotion recognition in autistic adult females correlates with alexithymia, not autism. Autism 24 (8), 2021-2034. doi:10. 1177/1362361320932727 · DOI
  34. Rahayu, S., and Suningsih, A. (2018). The effects of type learning model numbered head together and think pair share. Int. J. Trends Math. Edu. Res. 1 (1), 19-21. doi:10.33122/ijtmer.v1i1. · DOI
  35. Rose, D. H., and Meyer, A. (2008). A practical reader in universal design for learning. Cambridge, MA: Harvard Education Press.
  36. Scassellati, B., Boccanfuso, L., Huang, C. M., Mademtzi, M., Qin, M., Salomons, N., et al. (2018). Improving social skills in children with ASD using a long-term, in-home social robot. Sci. Robot. 3 (21), eaat7544. doi:10. 1126/scirobotics.aat7544 · DOI
  37. Seraj, M., Katterfeldt, E-S., Bub, K., Autexier, S., and Drechsler, R. (2019). Scratch and google blockly: How girls' programming skills and attitudes are influenced, Proc. 19th Koli Call. Int. Conf. Comput. Educ. 19, 1-10. doi:10. 1145/3364510.3364515 · DOI
  38. Severson, H. H., Hill, M. W., Hope-Doolittle, J., Kratochwill, T. R., and Gresham, F. M. (2007). Proactive, early screening to detect behaviorally at-risk students: Issues, approaches, emerging innovations, and professional practices. J. Sch. Psychol. 45 (2), 193-223. doi:10.1016/j.jsp.2006.11.003 · DOI
  39. Shu, L., Yu, Y., Chen, W., Hua, H., Li, Q., Jin, J., et al. (2020). Wearable emotion recognition using heart rate data from a smart bracelet. Sensors 20 (3), 718. doi:10. 3390/s20030718 · DOI
  40. Strawhacker, A., and Bers, M. (2015). "I want my robot to look for food": Comparing Kindergartner's programming comprehension using tangible, graphic, and hybrid user interfaces. Int. J. Technol. Des. Educ. 25 (3), 293-319. doi:10.1007/ s10798-014-9287-7 · DOI
  41. Sullivan, F. R., and Heffernan, J. (2016). Robotic construction kits as computational manipulatives for learning in the STEM disciplines. J. Res. Technol. Educ. 48 (2), 105-128. doi:10.1080/15391523.2016.1146563 · DOI
  42. Syriopoulou-Delli, C. K., and Gkiolnta, E. (2022). Review of assistive technology in the training of children with autism spectrum disorders, International. J. Dev. Disabil. 68 (2), 73-85. doi:10.1080/20473869.2019. 1706333 · DOI
  43. Taylor, M. S. (2018). Computer programming with pre-K through first-grade students with intellectual disabilities. J. Spec. Educ. 52 (2), 78-88. doi:10.1177/ 0022466918761120 · DOI
  44. Taylor, M. S., Vasquez, E., and Donehower, C. (2017). Computer programming with early elementary students with Down syndrome. J. Spec. Educ. Technol. 32 (3), 149-159. doi:10.1177/0162643417704439 · DOI
  45. Valencia, K., Rusu, C., Quiñones, D., and Jamet, E. (2019). The impact of technology on people with autism spectrum disorder: A systematic literature review. Sensors 19 (20), 4485. doi:10.3390/s19204485 Wonder Workshop Wonder Workshop (n.d.) blockly. Retrieved from. Available at: https://home.makewonder.com/apps/blockly. · DOI
  46. Zhu, J., Ji, L., and Liu, C. (2019). Heart rate variability monitoring for emotion and disorders of emotion. Physiol. Meas. 40, 064004. doi:10.1088/ 1361-6579/ · DOI

Authors
Hughes, C. E., Dieker, L. A., Glavey, E. M., Hines, R. A., Wilkins, I., Ingraham, K., Bukaty, C. A., Ali, K., Shah, S., Murphy, J., & Taylor, M. S.
Affiliations
  • University of Central Florida (US)
  • University of South Florida (US)
  • Salve Regina University (US)
Corresponding author
Hughes, Charles E.
ORCID
Venue
Frontiers in Virtual Reality
Publisher
Frontiers Media SA
Published
2022-10-14
Volume
3
ISSN
2673-4192 (ISSN-L)
DOI
10.3389/frvir.2022.968312
OpenAlex
W4306253206
Cited by
47
References
48
Citations by year
2026: 9 · 2025: 13 · 2024: 15 · 2023: 10
FWCI
8.98
Type
journal article
Language
en
Open access
Licence
Locations
  • Frontiers in Virtual Reality (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
  • Directorate for Engineering (2120240 2114808 1725554)
  • Institute of Education Sciences (P116S210001 H327S210005 H327S200009H)
Topics
  • Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Autism Spectrum Disorder Research
Keywords
autism spectrum disorder, socioemotional skills, communication skills, STEM education, programming education, robot programming, emotion recognition, emotion regulation, human-in-the-loop, elementary school students

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review