CoChat
Sign Up
Cited by 27Speech Recognition and SynthesisPhonetics and Phonology ResearchSpeech and dialogue systems

Developing an AI-Assisted Low-Resource Spoken Language Learning App for Children

Open access
Getman, Yaroslav, Phan, Nhan, Al-Ghezi, Ragheb, Voskoboinik, Ekaterina, Singh, Mittul, Grósz, Tamás, Kurimo, Mikko, Salvi, Giampiero +4 more·2023·IEEE Access
DOI 10.1109/access.2023.3304274
Open this paper in CoChat — free accountOpen PDF

Abstract

Computer-assisted Language Learning (CALL) is a rapidly developing area accelerated by advancements in the field of AI. A well-designed and reliable CALL system allows students to practice language skills, like pronunciation, any time outside of the classroom. Furthermore, gamification via mobile applications has shown encouraging results on learning outcomes and motivates young users to practice more and perceive language learning as a positive experience. In this work, we adapt the latest speech recognition technology to be a part of an online pronunciation training system for small children. As part of our gamified mobile application, our models will assess the pronunciation quality of young Swedish children diagnosed with Speech Sound Disorder, and participating in speech therapy. Additionally, the models provide feedback to young non-native children learning to pronounce Swedish and Finnish words. Our experiments revealed that these new models fit into an online game as they function as speech recognizers and pronunciation evaluators simultaneously. To make our systems more trustworthy and explainable, we investigated whether the combination of modern input attribution algorithms and time-aligned transcripts can explain the decisions made by the models, give us insights into how the models work and provide a tool to develop more reliable solutions.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/access.2023.3304274

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The automatic assessment was based on acoustic models trained on global ratings of whole word accuracy, conducted by SLP listeners.

    “The automatic assessment was based on acoustic models trained on global ratings of whole word accuracy, conducted by SLP listeners; for technical details, see Getman et al. 2023).”

    From Self‐Supervised App‐Based Speech Training for Children With Speech Sound Disorder—A Single‐Case Experimental Design Study (2026) · cites this paper as Getman et al. 2023

    The Speech Rating System section describes models that assess recordings of children saying single words, and the Data section describes human ratings on a five-level scale. The SweSSD section says its recordings were rated by a speech-language pathologist, but the L2 Swedish and Finnish Data section says those recordings were rated by university students specializing in language teaching. Thus the SLP-listener description applies to one dataset, not all the ratings used to train the systems. (LLM, conf=0.91)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. James Emil Flege, Murray J. Munro, and Ian R. A. MacKay. Factors affecting strength of perceived foreign accent in a second language. The Journal of the Acoustical Society of America, 97(5):3125-3134, 1995.
  2. Yvonne Wren, Sam Harding, Juliet Goldbart, and Sue Roul- stone. A systematic review and classification of interventions for speech-sound disorder in preschool children. International Journal of Language & Communication Disorders, 53(3):446- 467, 2018.
  3. Alycia Cummings, Kristen Giesbrecht, and Janet Hallgrim- son. Intervention dose frequency: Phonological generalization is similar regardless of schedule. Child Language Teaching and Therapy, 37(1):99-115, 2021.
  4. Laleh Aghlara and Nasrin Hadidi Tamjid. The effect of digital games on Iranian children's vocabulary retention in foreign language acquisition. Procedia -Social and Behavioral Sciences, 29:552-560, 2011. The 2nd International Conference on Education and Educational Psychology 2011.
  5. Tsung-Yu Liu and Yu-Ling Chu. Using ubiquitous games in an English listening and speaking course: Impact on learning outcomes and motivation. Computers & Education, 55(2):630-643, 2010.
  6. Ping Li and Yu-Ju Lan. Digital language learning (DLL): In- sights from behavior, cognition, and the brain. Bilingualism: Language and Cognition, 25(3):361-378, 2022.
  7. Ping Li and Yu-Ju Lan. Understanding the interaction between technology and the learner: The case of DLL. Bilin- gualism: Language and Cognition, 25(3):402-405, 2022.
  8. Adam Hair, Kirrie J. Ballard, Constantina Markoulli, Pene- lope Monroe, Jacqueline Mckechnie, Beena Ahmed, and Ri- cardo Gutierrez-Osuna. A longitudinal evaluation of tablet- based child speech therapy with apraxia world. ACM Trans. Access. Comput., 14(1), mar 2021.
  9. Mor Nahum and Daphne Bavelier. Chapter 10 -video games as rich environments to foster brain plasticity. In Nick F. Ramsey and José del R. Millán, editors, Brain-Computer Interfaces, volume 168 of Handbook of Clinical Neurology, pages 117-136. Elsevier, 2020.
  10. Reima Karhila, Sari Ylinen, Seppo Enarvi, Kalle Palomäki, Aleksander Nikulin, Olli Rantula, Vertti Viitanen, Krupakar Dhinakaran, Anna-Riikka Smolander, Heini Kallio, Katja Junttila, Maria Uther, Perttu Hämäläinen, and Mikko Ku- rimo. SIAK -A game for foreign language pronunciation learning. In Proc. Interspeech 2017, pages 3429-3430, 2017.
  11. Cristian Tejedor-García, David Escudero-Mancebo, Valentín Cardeñoso-Payo, and César González-Ferreras. Using chal- lenges to enhance a learning game for pronunciation training of English as a second language. IEEE Access, 8:74250-74266, 2020.
  12. Muzakki Bashori, Roeland van Hout, Helmer Strik, and Catia Cucchiarini. 'Look, I can speak correctly': learning vocabulary and pronunciation through websites equipped with automatic speech recognition technology. Computer Assisted Language Learning, 0(0):1-29, 2022.
  13. Catia Cucchiarini, Ambra Neri, and Helmer Strik. Oral profi- ciency training in Dutch L2: The contribution of ASR-based corrective feedback. Speech Communication, 51(10):853-863, 2009. Spoken Language Technology for Education.
  14. Ewa M. Golonka, Anita R. Bowles, Victor M. Frank, Dorna L. Richardson, and Suzanne Freynik. Technologies for foreign language learning: a review of technology types and their ef- fectiveness. Computer Assisted Language Learning, 27(1):70- 105, 2014.
  15. Jacqueline McKechnie, Beena Ahmed, Ricardo Gutierrez- Osuna, Penelope Monroe, Patricia Mccabe, and Kirrie J. Bal- lard. Automated speech analysis tools for children's speech production: A systematic literature review. International Journal of Speech-Language Pathology, 20:583 -598, 2018.
  16. Silke M Witt and Steve J Young. Phone-level pronunciation scoring and assessment for interactive language learning. Speech communication, 30(2-3):95-108, 2000.
  17. Jiatong Shi, Nan Huo, and Qin Jin. Context-Aware Good- ness of Pronunciation for Computer-Assisted Pronunciation Training. In Proc. Interspeech 2020, pages 3057-3061, 2020.
  18. S. W. Lee, Alexandros Potamianos, and Shrikanth S. Narayanan. Acoustics of children's speech: developmental changes of temporal and spectral parameters. The Journal of the Acoustical Society of America, 105 3:1455-68, 1999.
  19. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural infor- mation processing systems, 33:12449-12460, 2020.
  20. Ragheb Al-Ghezi, Yaroslav Getman, Aku Rouhe, Raili Hildén, and Mikko Kurimo. Self-Supervised End-to-End ASR for Low Resource L2 Swedish. In Proc. Interspeech 2021, pages 1429-1433, 2021.
  21. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning -Volume 70, ICML'17, page 3319-3328. JMLR.org, 2017.
  22. Sofia Strömbergsson, Katarina Holm, Jens Edlund, Tove Lagerberg, and Anita McAllister. Audience response system- based evaluation of intelligibility of children's connected speech -validity, reliability and listener differences. Journal of Communication Disorders, 87:106037, 2020.
  23. Tove B. Lagerberg, Lena Hartelius, Jakob Åsberg Johnels, Anna-Karin Ahlman, Andrea Börjesson, and Christina Pers- son. Swedish test of intelligibility for children (STI-CH) - validity and reliability of a computer-mediated single word intelligibility test for children. Clinical Linguistics & Phonet- ics, 29(3):201-215, 2015. PMID: 25489674.
  24. Reima Karhila, Anna Riikka Smolander, Sari Ylinen, and Mikko Kurimo. Transparent pronunciation scoring using articulatorily weighted phoneme edit distance. In Proceed- ings of Interspeech, pages 1866-1870. International Speech Communication Association, 2019.
  25. Niklas Vanhainen and Giampiero Salvi. Free acoustic mod- els for large vocabulary continuous speech recognition in Swedish. In Proc. of the Language Resources and Evaluation Conference (LREC), 2014.
  26. Anton Batliner, Mats Blomberg, Shona D'Arcy, Daniel Ele- nius, Diego Giuliani, Matteo Gerosa, Christian Hacker, Mar- tin J. Russell, Stefan Steidl, and Michael Wong. The PF_STAR children's speech corpus. In INTERSPEECH 2005 -Eurospeech, 9th European Conference on Speech Com- munication and Technology, Lisbon, Portugal, September 4-8, 2005, pages 2761-2764. ISCA, 2005.
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkor- eit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 6000-6010, Red Hook, NY, USA, 2017.
  28. Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. Proceedings of the 23rd International Conference on Machine Learning, page 369-376, 2006.
  29. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why should I trust you?" Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144, 2016.
  30. Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung-Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Ben- gio. SpeechBrain: A general-purpose speech toolkit, 2021. arXiv:2106.04624. · arXiv
  31. Tamás Grósz, Dejan Porjazovski, Yaroslav Getman, Su- darsana Kadiri, and Mikko Kurimo. Wav2vec2-based par- alinguistic systems to recognise vocalised emotions and stut- tering. In Proceedings of the 30th ACM International Conference on Multimedia, MM '22, page 7026-7029, New York, NY, USA, 2022. Association for Computing Machinery.
  32. Yaroslav Getman, Ragheb Al-Ghezi, Katja Voskoboinik, Tamás Grósz, Mikko Kurimo, Giampiero Salvi, Torbjørn Svendsen, and Sofia Strömbergsson. wav2vec2-based Speech Rating System for Children with Speech Sound Disorder. In Proc. Interspeech 2022, pages 3618-3622, 2022.
  33. Ahmed Alqaraawi, Martin Schuessler, Philipp Weiß, Enrico Costanza, and Nadia Berthouze. Evaluating saliency map explanations for convolutional neural networks: A user study. In Proceedings of the 25th International Conference on Intel- ligent User Interfaces, IUI '20, page 275-285, New York, NY, USA, 2020. Association for Computing Machinery.
  34. Gherman Novakovsky, Nick Dexter, Maxwell W Libbrecht, Wyeth W Wasserman, and Sara Mostafavi. Obtaining ge- netics insights from deep learning via explainable artificial intelligence. Nature Reviews Genetics, 24(2):125-137, 2023.
  35. Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Sai Suggala, David I. Inouye, and Pradeep Ravikumar. On the (In)fidelity and Sensitivity of Explanations. In Neural Information Processing Systems, 2019.
  36. Hervé Bredin and Antoine Laurent. End-To-End Speaker Segmentation for Overlap-Aware Resegmentation. In Proc. Interspeech 2021, pages 3111-3115, 2021.
  37. Maxine Eskenazi. An overview of spoken language technology for education. Speech Communication, 51(10):832-844, 2009. Spoken Language Technology for Education.
  38. Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi- supervised learning and interpretation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, volume 1, pages 993-1003. Association for Computational Linguistics, August 2021.
  39. Frank H. Guenther. Speech sound acquisition, coarticulation, and rate effects in a neural network model of speech produc- tion. Psychological review, 102 3:594-621, 1995.
  40. Daphna Shohamy, Catherine E. Myers, Steven Grossman, Jake Sage, Mark A. Gluck, and Russell A. Poldrack. Cortico- striatal contributions to feedback-based learning: converging data from neuroimaging and neuropsychology. Brain : a journal of neurology, 127 Pt 4:851-9, 2004.

Cited by

Sorted by citations

Authors
Getman, Y., Phan, N., Al-Ghezi, R., Voskoboinik, E., Singh, M., Grósz, T., Kurimo, M., Salvi, G., Svendsen, T., Strömbergsson, S., Smolander, A., & Ylinen, S.
Affiliations
  • Aalto University (FI)
  • Norwegian University of Science and Technology (NO)
  • Karolinska Institutet (SE)
  • Tampere University (FI)
ORCID
Venue
IEEE Access
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Published
2023
Volume
11
Pages
86025-86037
ISSN
2169-3536 (ISSN-L)
DOI
10.1109/access.2023.3304274
OpenAlex
W4385731991
Cited by
27
References
8
Citations by year
2026: 9 · 2025: 11 · 2024: 7
FWCI
2.55
Type
journal article
Language
en
Open access
Licence
Locations
  • IEEE Access (publishedVersion, cc-by)
  • Aaltodoc (Aalto University) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Tampere University Institutional Repository (Tampere University) (submittedVersion, cc-by)
Funders
NordForsk through funding to technology-enhanced foreign and second-language learning of Nordic languages (103893)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
computer-assisted language learning, pronunciation teaching, speech recognition, speech sound disorders, speech therapy, gamification, mobile learning applications, low-resource languages, Swedish language, Finnish language, children, explainable AI, pronunciation assessment, non-native speakers

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review