“You don’t understand me!”: Comparing ASR Results for L1 and L2 Speakers of Swedish
Open accessAbstract
The performance of Automatic Speech Recognition (ASR) systems has constantly increased in state-of-the-art development. However, performance tends to decrease considerably in more challenging conditions (e.g., background noise, multiple speaker social conversations) and with more atypical speakers (e.g., children, non-native speakers or people with speech disorders), which signifies that general improvements do not necessarily transfer to applications that rely on ASR, e.g., educational software for younger students or language learners. In this study, we focus on the gap in performance between recognition results for native and non-native, read and spontaneous, Swedish utterances transcribed by different ASR services. We compare the recognition results using Word Error Rate and analyze the linguistic factors that may generate the observed transcription errors.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ASR performance varies across datasets and speaker groups, including by gender, race, and accent.
“Many sources show ASR's volatility with diferent datasets: ASR performs diferently for male or female speakers, shows racial biases or varying accuracy rates for speakers with diferent accents [Cumbal et al. 2021;Koenecke et al. 2020;Speechmatics 2023;Tadimeti et al. 2022;Tatman and Kasten 2017].”
From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Cumbal et al. 2021
Section 6.1 (page unavailable) reports different word error rates for native and non-native Swedish speakers and for speakers with different first languages; Table 2 also shows variation across datasets. These passages support the dataset and accent-related portions of the claim, but our copy does not substantiate the claims about gender differences or racial bias. (LLM, conf=0.88)
Experimental — accuracy not yet measuredPrior studies aim to show specific ASR biases affecting underrepresented groups but do not address overall engine accuracy.
“The studies aim to show that ASR has speciic biases that often afect speakers of underrepresented groups, but are not concerned with the overall accuracy of the engines.”
From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Cumbal et al. 2021
The Introduction states that the paper aims to examine the recognition gap between native and non-native Swedish speakers, supporting the claim about group-specific ASR performance. But §5.1 (Performance) says the authors evaluate each ASR using word error rate, and §6.1 compares the engines’ results. The claim that the study is not concerned with engine accuracy is therefore too absolute. (LLM, conf=0.84)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 17
- C.-C. Chiu, T. N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R. J. Weiss, K. Rao, E. Gonina et al., "State- of-the-art speech recognition with sequence-to-sequence models," in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 4774-4778.
- W. Xiong, J. Droppo, X. Huang, F. Seide, M. L. Seltzer, A. Stol- cke, D. Yu, and G. Zweig, "Toward Human Parity in Conver- sational Speech Recognition," IEEE/ACM Trans. Audio Speech Lang. Process., vol. 25, no. 12, pp. 2410-2423, 2017.
- G. Saon, G. Kurata, T. Sercu, K. Audhkhasi, S. Thomas, D. Dim- itriadis, X. Cui, B. Ramabhadran, M. Picheny, L. L. Lim, B. Roomi, and P. Hall, "English conversational telephone speech recognition by humans and machines," Proc. Annu. Conf. Int. Speech Commun. Assoc. INTERSPEECH, vol. 2017-August, pp. 132-136, 2017.
- V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, "Lib- rispeech: An asr corpus based on public domain audio books," in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206-5210.
- J. Kennedy, S. Lemaignan, C. Montassier, P. Lavalade, B. Irfan, F. Papadopoulos, E. Senft, and T. Belpaeme, "Child speech recognition in human-robot interaction: Eval- uations and recommendations," in Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interac- tion, ser. HRI '17. New York, NY, USA: Association for Computing Machinery, 2017, p. 82-90. [Online]. Available: https://doi.org/10.1145/2909824.3020229 · DOI
- T. Ashwell and J. R. Elam, "How accurately can the Google web speech API recognize and transcribe Japanese L2 English learn- ers' oral production?" vol. 13, 2017, pp. 59-76.
- K. Radzikowski, R. Nowak, L. Wang, and O. Yoshie, "Dual supervised learning for non-native speech recognition," EURASIP Journal on Audio, Speech, and Music Processing, 2019. [Online]. Available: https://doi.org/10.1186/s13636-018-0146-4 · DOI
- J. Y. Kim, C. Liu, R. A. Calvo, K. McCabe, S. C. R. Taylor, B. W. Schuller, and K. Wu, "A Comparison of Online Automatic Speech Recognition Systems and the Nonverbal Responses to Unintelligible Speech," pp. 1-13, 2019. [Online]. Available: http://arxiv.org/abs/1904.12403 · arXiv
- A. Khalifa, T. Kato, and S. Yamamoto, "Measuring effect of repet- itive queries and implicit learning with joining-in type robot as- sisted language learning system," in ISCA workshop on Speech and Language Technology in Education, 2017, pp. 13-17.
- O. Engwall, J. Lopes, and A. Åhlund, "Robot interaction styles for conversation practice in second language learning," International Journal of Social Robotics, 2020.
- F. Morbini, K. Audhkhasi, K. Sagae, R. Artstein, D. Can, P. Geor- giou, S. Narayanan, A. Leuski, and D. Traum, "Which ASR should i choose for my dialogue system?" SIGDIAL 2013 -14th Annu. Meet. Spec. Interes. Gr. Discourse Dialogue, Proc. Conf., no. August, pp. 394-403, 2013.
- X. Yao, P. Bhutada, K. Georgila, K. Sagae, R. Artstein, and D. Traum, "Practical evaluation of speech recognizers for virtual human dialogue systems," Proc. 7th Int. Conf. Lang. Resour. Eval. Lr. 2010, pp. 1597-1602, 2010.
- K. Georgila, A. Leuski, V. Yanov, and D. Traum, "Evaluation of Off-the-shelf Speech Recognizers Across Diverse Dialogue Do- mains," no. May, pp. 6469-6476, 2020.
- T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, "Transformers: State-of-the-art natural language processing," in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. Online: Association for Computational Linguistics, Oct. 2020, pp. 38-45. [Online]. Avail- able: https://www.aclweb.org/anthology/2020.emnlp-demos.6
- A. Baevski, H. Zhou, A. Mohamed, and M. Auli, "wav2vec 2.0: A framework for self-supervised learning of speech representa- tions," arXiv preprint arXiv:2006.11477, 2020. · arXiv
- J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre- training of deep bidirectional transformers for language under- standing," arXiv preprint arXiv:1810.04805, 2018. · arXiv
- R. Ardila, M. Branson, K. Davis, M. Henretty, M. Kohler, J. Meyer, R. Morais, L. Saunders, F. M. Tyers, and G. Weber, "Common voice: A massively-multilingual speech corpus," in Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), 2020, pp. 4211-4215.
Cited by
Sorted by citations
Measuring the Accuracy of Automatic Speech Recognition Solutions
Kuhn, K., Kersken, V., Reuter, B., Egger, N., & Zimmermann, G. (2023) · ACM Transactions on Accessible Computing · doi:10.1145/3636513 · Cited by 44 · see citing works
“…sources show ASR’s volatility with diferent datasets: ASR performs diferently for male or female speakers, shows racial biases or varying accuracy rates for speakers with diferent accents [Cumbal et al. 2021; Koenecke et al. 2020; Speechmatics 2023; Tadimeti et al. 2022; Tatman and Kasten 2017].”
Background
- Authors
- Cumbal, R., Moell, B., Lopes, J., & Engwall, O.
- Affiliations
- KTH Royal Institute of Technology (SE)
- Heriot-Watt University (GB)
- Venue
- Interspeech 2021
- Publisher
- ISCA
- Published
- 2021-08-30
- Pages
- 4463-4467
- ISSN
- 2308-457X (ISSN-L)
- OpenAlex
- W3198759167
- Cited by
- 20
- References
- 0
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 2 · 2023: 5 · 2022: 9 · 2021: 1
- FWCI
- 1.96
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- cc-by
- Locations
- Interspeech (USB) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- Diva portal (Dalarna University Library) (submittedVersion, other-oa)
- arXiv (Cornell University) (cc-by)
- Funders
- VINNOVA
- Vetenskapsrådet (2016-03698, 2019-02997)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Phonetics and Phonology Research
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- speech recognition, Swedish language, native and non-native speakers, second language speakers, word error rate, transcription errors, spontaneous speech, read speech, language learners, speech recognition performance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
