How Might We Create Better Benchmarks for Speech Recognition?
Open accessAbstract
The applications of automatic speech recognition (ASR) systems are proliferating, in part due to recent significant quality improvements.However, as recent work indicates, even state-of-the-art speech recognition systems -some which deliver impressive benchmark results, struggle to generalize across use cases.We review relevant work, and, hoping to inform future benchmark development, outline a taxonomy of speech recognition use cases, proposed for the next generation of ASR benchmarks.We also survey work on metrics, in addition to the de facto standard Word Error Rate (WER) metric, and we introduce a versatile framework designed to describe interactions between linguistic variation and ASR performance metrics.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Evaluation of an ASR system heavily depends on the use-case, and various factors have to be considered.
“Evaluation of an ASR system heavily depends on the use-case, and various factors have to be considered [Aksënova et al. 2021].”
From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Aksënova et al. 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Binary Codes capable of currecting deletions, insertions, and reversals
Levenshtein, V. I. (1965) · Soviet physics. Doklady · Cited by 10,387 · see citing works
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072 · see citing works
SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
Park, D. S., Chan, W., Zhang, Y., Chiu, C.-C., Zoph, B., Cubuk, E. D., & Le, Q. V. (2019) · Interspeech 2019 · doi:10.21437/interspeech.2019-2680 · Cited by 3,558 · see citing works
Audio Set: An ontology and human-labeled dataset for audio events
Gemmeke, J. F., Ellis, D. P. W., Freedman, D., Jansen, A., Lawrence, W., Moore, R. C., Plakal, M., & Ritter, M. (2017) · 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2017.7952261 · Cited by 3,134 · see citing works
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · see citing works
SWITCHBOARD: telephone speech corpus for research and development
Godfrey, J. J., Holliman, E. C., & McDaniel, J. (1992) · [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing · doi:10.1109/icassp.1992.225858 · Cited by 2,134 · cited 4× · see citing works
“Typical benchmark sets beyond Librispeech include TIMIT (Garofolo et al., 1993), Switchboard (Godfrey et al., 1992), WSJ (Paul and Baker, 1992), CALLHOME (Canavan et al., 1997), and Fisher (Cieri et al., 2004).”
BackgroundInfluentialPearce, D., & Hirsch, H. (2000) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.2000-743 · Cited by 1,779 · see citing works
The intersection of sex and social class in the course of linguistic change
Labov, W. (1990) · Language Variation and Change · doi:10.1017/s0954394500000338 · Cited by 1,472 · see citing works
Representativeness in Corpus Design
Biber, D. (1993) · Literary and Linguistic Computing · doi:10.1093/llc/8.4.243 · Cited by 1,274 · see citing works
The design for the wall street journal-based CSR corpus
Paul, D. B., & Baker, J. M. (1992) · DARPA Workshop on Speech and Natural Language (HLT) · doi:10.3115/1075527.1075614 · Cited by 1,142 · see citing works
Audio Adversarial Examples: Targeted Attacks on Speech-to-Text
Carlini, N., & Wagner, D. (2018) · IEEE Security and Privacy Workshops (SPW) · doi:10.1109/spw.2018.00009 · Cited by 1,071 · see citing works
The AMI Meeting Corpus: A Pre-announcement
Carletta, J., Ashby, S., Bourban, S., Flynn, M., Guillemot, M., Hain, T., Kadlec, J., Vasilis Karaiskos, B., Kraaij, W., Kronenthal, M., et al. (2006) · Lecture notes in computer science · doi:10.1007/11677482_3 · Cited by 939 · see citing works
Common Voice: A Massively-Multilingual Speech Corpus
Ardila, R., Branson, M., Davis, K., Henretty, M., Kohler, M., Meyer, J., Morais, R., Saunders, L., Tyers, F. M., & Weber, G. (2019) · arXiv · doi:10.48550/arxiv.1912.06670 · Cited by 905 · see citing works
Racial disparities in automated speech recognition
Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D., & Goel, S. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1915768117 · Cited by 765 · see citing works
A Handbook of Varieties of English
Kortmann, B., Schneider, E. W., Burridge, K., Mesthrie, R., & Upton, C. (2009) · De Gruyter eBooks · doi:10.1515/9783110197181 · Cited by 550 · see citing works
The Fisher Corpus: a Resource for the Next Generations of Speech-to-Text
Cieri, C., Miller, D., & Walker, K. P. (2004) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/5q862povzthr · Cited by 494 · see citing works
English Conversational Telephone Speech Recognition by Humans and Machines
Saon, G., Kurata, G., Sercu, T., Audhkhasi, K., Thomas, S., Dimitriadis, D., Cui, X., Ramabhadran, B., Picheny, M., Lim, L.-L., et al. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-405 · Cited by 364 · see citing works
Normalization of non-standard words
Sproat, R., Black, A. W., Chen, S. F., Kumar, S., Mari Ostendorf, E., & Richards, C. D. (2001) · Computer Speech & Language · doi:10.1006/csla.2001.0169 · Cited by 347 · see citing works
Gender and Dialect Bias in YouTube’s Automatic Captions
Tatman, R. (2017) · Proceedings of the First ACL Workshop on Ethics in Natural Language Processing · doi:10.18653/v1/w17-1606 · Cited by 338 · see citing works
Zue, V. W., Cole, R., & Ward, W. (1997) · Cited by 288 · see citing works
Toward Human Parity in Conversational Speech Recognition
Xiong, W., Droppo, J., Huang, X., Seide, F., Seltzer, M. L., Stolcke, A., Yu, D., & Zweig, G. (2017) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2017.2756440 · Cited by 245 · see citing works
Pushing the Limits of Semi-Supervised Learning for Automatic Speech RecognitionPreprint
Zhang, Y., Qin, J., Park, D., Han, W., Chiu, C., Pang, R., Le, Q. V., & Wu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10504 · Cited by 233 · see citing works
A Streaming On-Device End-To-End Model Surpassing Server-Side Conventional Model Quality and Latency
Sainath, T. N., He, Y., Li, B., Narayanan, A., Pang, R., Bruguier, A., Chang, S.-Y., Li, W., Álvarez, R., Chen, Z., et al. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054188 · Cited by 188 · see citing works
Effects of speaking rate and word frequency on pronunciations in convertional speech
Fosler‐Lussier, E., & Morgan, N. H. (1999) · Speech Communication · doi:10.1016/s0167-6393(99)00035-7 · Cited by 161 · see citing works
What size test set gives good error rate estimates?
Guyon, I., Makhoul, J., Schwartz, R. M., & Vapnik, V. N. (1998) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/34.655649 · Cited by 156 · see citing works
Is word error rate a good indicator for spoken language understanding accuracy
Wang, Y., Acero, A., & Chelba, C. I. (2004) · IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU) · doi:10.1109/asru.2003.1318504 · Cited by 149 · see citing works
The Third DIHARD Diarization Challenge
Ryant, N., Singh, P., Krishnamohan, V., Varma, R., Church, K., Cieri, C., Du, J., Ganapathy, S., & Liberman, M. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1208 · Cited by 136 · see citing works
Characterising Bias in Compressed ModelsPreprint
Hooker, S., Moorosi, N., Clark, G., Samy Bengio, A., & Denton, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.03058 · Cited by 132 · see citing works
Automatic Recognition of Spontaneous Speech for Access to Multilingual Oral History Archives
Byrne, B., Doermann, D., Franz, M., Gustman, S., Hajič, J., Oard, D. W., Picheny, M., Psutka, J., Ramabhadran, B., Soergel, D., et al. (2004) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.2004.828702 · Cited by 130 · see citing works
What Do Compressed Deep Neural Networks Forget?Preprint
Hooker, S., Courville, A. C., Clark, G., Dauphin, Y., & Frome, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.05248 · Cited by 125 · see citing works
The design for the wall street journal-based CSR corpus
Paul, D. B., & Baker, J. M. (1992) · International Conference on Spoken Language Processing (ICSLP) · doi:10.21437/icslp.1992-277 · Cited by 121 · see citing works
Personalizing ASR for Dysarthric and Accented Speech with Limited Data
Shor, J., Emanuel, D., Lang, O., Tuval, O., Brenner, M. P., Cattiau, J., Vieira, F. G., McNally, M., Charbonneau, T., Nollstadt, M., et al. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1427 · Cited by 117 · see citing works
FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN
Ansari, E., Axelrod, A. E., Bách, N., Bojar, O., Cattoni, R., Dalvi, F., Durrani, N., Federico, M., Federmann, C., Gu, J., et al. (2020) · International Conference on Spoken Language Translation (IWSLT) · doi:10.18653/v1/2020.iwslt-1.1 · Cited by 111 · see citing works
Andrus, M., Spitzer, E., Brown, J., & Xiang, A. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445888 · Cited by 111 · see citing works
Towards Fast and Accurate Streaming End-To-End ASR
Li, B., Chang, S.-Y., Sainath, T. N., Pang, R., He, Y., Strohman, T., & Wu, Y. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9054715 · Cited by 109 · see citing works
Het Corpus Gesproken Nederlands
Oostdijk, N. H. J. (2000) · Radboud Repository (Radboud University) · Cited by 104 · see citing works
Kinoshita, K., Ochiai, T., Delcroix, M., & Nakatani, T. (2020) · ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp40776.2020.9053266 · Cited by 104 · see citing works
Large vocabulary automatic speech recognition for children
Liao, H., Pundak, G., Siohan, O., Carroll, M. K., Coccaro, N., Jiang, Q.-M., Sainath, T. N., Senior, A., Beaufays, F., & Bacchiani, M. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-373 · Cited by 99 · see citing works
Bringing contextual information to google speech recognition
Aleksic, P. S., Ghodsi, M., Michaely, A. H., Allauzen, C., Hall, K., Brian Roark, A., Rybach, D., & Moreno, P. J. (2015) · Interspeech (USB) · doi:10.21437/interspeech.2015-177 · Cited by 91 · see citing works
On the effects of speech rate in large vocabulary speech recognition systems
Siegler, M. A., & Stern, R. M. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1995.479672 · Cited by 90 · see citing works
Effects of Talker Dialect, Gender & Race on Accuracy of Bing Speech and YouTube Automatic Captions
Tatman, R., & Kasten, C. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1746 · Cited by 89 · see citing works
Diversity and Inclusion Metrics in Subset Selection
Mitchell, M., Baker, D., Moorosi, N., Denton, R., Hutchinson, B., Hanna, A., Gebru, T., & Morgenstern, J. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375832 · Cited by 87 · see citing works
Assessing the accuracy of automatic speech recognition for psychotherapy
Miner, A. S., Haque, A., Fries, J., Fleming, S. L., Wilfley, D. E., Wilson, G. T., Milstein, A. S., Jurafsky, D., Arnow, B. A., Agras, W. S., et al. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0285-8 · Cited by 82 · see citing works
Towards Measuring Fairness in AI: The Casual Conversations Dataset
Hazirbas, C., Bitton, J., Dolhansky, B., Pan, J., Gordo, A., & Ferrer, C. C. (2022) · IEEE Transactions on Biometrics, Behavior, and Identity Science · doi:10.1109/tbiom.2021.3132237 · Cited by 79 · see citing works
Speech Recognition for Medical Conversations
Chiu, C., Tripathi, A., Chou, K., Co, C., Jaitly, N., Jaunzeikare, D., Kannan, A., Nguyen, P., Sak, H., Sankar, A., et al. (2018) · Interspeech (USB) · doi:10.21437/interspeech.2018-40 · Cited by 73 · see citing works
100,000 Podcasts: A Spoken English Document Corpus
Clifton, A., Reddy, S., Yu, Y., Pappu, A., Rezapour, R., Bonab, H., Eskevich, M., Jones, G. J. F., Jussi Karlgren, C., Carterette, B., et al. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.519 · Cited by 72 · see citing works
FastEmit: Low-Latency Streaming ASR with Sequence-Level Emission Regularization
Yu, J., Chiu, C., Li, B., Chang, S.-Y., Sainath, T. N., He, Y., Narayanan, A., Han, W., Gulati, A., Wu, Y., et al. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413803 · Cited by 65 · see citing works
Calibration of Confidence Measures in Speech Recognition
Yu, D., Li, J., & Deng, L. (2011) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2141988 · Cited by 64 · see citing works
Quantifying Bias in Automatic Speech RecognitionPreprint
Feng, S., Kudina, O., Halpern, B. M., & Scharenborg, O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.15122 · Cited by 61 · see citing works
ASR-based dictation practice for second language pronunciation improvement
McCrocklin, S. (2019) · Journal of Second Language Pronunciation · doi:10.1075/jslp.16034.mcc · Cited by 60 · see citing works
Rethinking Evaluation in ASR: Are Our Models Robust Enough?
Likhomanenko, T., Xu, Q., Pratap, V., Tomasello, P., Kahn, J., Avidov, G., Collobert, R., & Synnaeve, G. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1758 · Cited by 56 · see citing works
Designing an Online Infrastructure for Collecting AI Data From People With Disabilities
Park, J. S., Bragg, D., Kamar, E., & Morris, M. R. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445870 · Cited by 50 · see citing works
Cross Domain Automatic Transcription on the TC-STAR EPPS Corpus
Gollan, C., Bisani, M., Kanthak, S., Schlüter, R., & Ney, H. (2006) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2005.1415241 · Cited by 50 · see citing works
Speech Databases of Typical Children and Children with SLI
Grill, P., & Tučková, J. (2016) · PLoS ONE · doi:10.1371/journal.pone.0150365 · Cited by 43 · see citing works
Igraś-Cybulska, M., Ziółko, B., Żelasko, P., & Witkowski, M. (2016) · EURASIP Journal on Audio Speech and Music Processing · doi:10.1186/s13636-016-0096-7 · Cited by 43 · see citing works
WER we are and WER we think we are
Szymański, P., Żelasko, P., Morzy, M., Szymczak, A., Żyła-Hoppe, M., Banaszczak, J., Augustyniak, Ł., Mizgajski, J. T., & Carmiel, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.295 · Cited by 38 · see citing works
Knudsen, B., Creemers, A., & Meyer, A. S. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.593671 · Cited by 38 · see citing works
Improving Performance of End-to-End ASR on Numeric Sequences
Peyser, C., Zhang, H., Sainath, T. N., & Wu, Z. (2019) · Interspeech (USB) · doi:10.21437/interspeech.2019-1345 · Cited by 37 · see citing works
O’Neill, P. K., Lavrukhin, V., Majumdar, S., Noroozi, V., Zhang, Y., Kuchaiev, O., Balam, J., Dovzhenko, Y., Freyberg, K., Shulman, M. D., et al. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1860 · Cited by 36 · see citing works
Mattson, P. R., Cheng, C., Diamos, G., Coleman, C., Micikevicius, P., Patterson, D. A., Tang, H., Wei, G.-Y., Bailis, P. D., Bittorf, V., et al. (2020) · Proceedings of Machine Learning and Systems · Cited by 36 · see citing works
Towards Understanding ASR Error Correction for Medical Conversations
Mani, A., Palaskar, S., & Konam, S. (2020) · doi:10.18653/v1/2020.nlpmc-1.2 · Cited by 32 · see citing works
Effectively Building Tera Scale MaxEnt Language Models Incorporating Non-Linguistic Signals
Biadsy, F., Ghodsi, M., & Caseiro, D. (2017) · Interspeech (USB) · doi:10.21437/interspeech.2017-1203 · Cited by 30 · see citing works
Automatic speech recognition performance on a voicemail transcription task
Padmanabhan, M., Saon, G., Huang, J., Kingsbury, B. E. D., & Mangu, L. (2002) · IEEE Transactions on Speech and Audio Processing · doi:10.1109/tsa.2002.804303 · Cited by 27 · see citing works
Emitting Word Timings with End-to-End Models
Sainath, T. N., Pang, R., Rybach, D., García, B., & Strohman, T. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1059 · Cited by 27 · see citing works
Dissecting User-Perceived Latency of On-Device E2E Speech Recognition
Shangguan, Y., Prabhavalkar, R., Su, H., Mahadeokar, J., Shi, Y., Zhou, J., Wu, C., Le, D., Kalinli, O., Fuegen, C., et al. (2021) · Interspeech (USB) · doi:10.21437/interspeech.2021-1887 · Cited by 27 · see citing works
Sources of variation in the speech of African Americans: Perspectives from sociophonetics
Farrington, C., King, S., & Kohn, M. (2021) · Wiley Interdisciplinary Reviews Cognitive Science · doi:10.1002/wcs.1550 · Cited by 23 · see citing works
Shrinking Bigfoot: Reducing wav2vec 2.0 footprint
Peng, Z., Budhkar, A., Tuil, I., Levy, J., Sobhani, P., Cohen, R., & Nassour, J. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.14 · Cited by 22 · see citing works
End-to-End Speech Recognition and Disfluency Removal
Lou, P. J., & Johnson, M. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.186 · Cited by 20 · see citing works
Improved Robustness to Disfluencies in Rnn-Transducer Based Speech Recognition
Mendelev, V., Raissi, T., Camporese, G., & Giollo, M. (2021) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp39728.2021.9413618 · Cited by 19 · see citing works
Comparative analysis and review of interactive voice response systems
Inam, I. A., Azeta, A. A., & Daramola, O. J. (2017) · Conference on Information Communications Technology and Society (ICTAS) · doi:10.1109/ictas.2017.7920660 · Cited by 17 · see citing works
Learning Personalized Pronunciations for Contact Name Recognition
Bruguier, A., Peng, F., & Beaufays, F. (2016) · Interspeech (USB) · doi:10.21437/interspeech.2016-537 · Cited by 14 · see citing works
Analyzing the Quality and Stability of a Streaming End-to-End On-Device Speech Recognizer
Shangguan, Y., Knister, K., He, Y., Mcgraw, I. C., & Beaufays, F. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1194 · Cited by 11 · see citing works
Effect of different sampling rates and feature vector sizes on speech recognition performance
Ssnderson, C., & Paliwal, K. K. (2002) · IEEE Region 10 Conference (TENCON) · doi:10.1109/tencon.1997.647282 · Cited by 11 · see citing works
El corpus ESLORA de español oral: diseño, desarrollo y explotación
Barcala, M., Domínguez, E., Fernández, A., Rivas, R., del Río, M. P. S., Rozas, V. V., & Baltar, R. V. (2018) · CHIMERA Revista de Corpus de Lenguas Romances y Estudios Lingüísticos · doi:10.15366/chimera2018.5.2.003 · Cited by 10 · see citing works
Speech recognition of different sampling rates using fractal code descriptor
Hokking, R., Woraratpanya, K., & Kuroki, Y. (2016) · International Joint Conference on Computer Science and Software Engineering · doi:10.1109/jcsse.2016.7748895 · Cited by 9 · see citing works
Decision tree state clustering with word and syllable features
Liao, H., Alberti, C., Bacchiani, M., & Siohan, O. (2010) · Interspeech (USB) · doi:10.21437/interspeech.2010-28 · Cited by 9 · see citing works
Cohn, M., Sarian, M., Predeck, K., & Zellou, G. (2020) · Interspeech (USB) · doi:10.21437/interspeech.2020-1339 · Cited by 8 · see citing works
IGC-Parl: Icelandic Corpus of Parliamentary Proceedings
Steingrímsson, S., Barkarson, S., & Örnólfsson, G. T. (2020) · Cited by 7 · see citing works
Rethinking the corpus: moving towards dynamic linguistic resources
Rosenberg, A. (2012) · Interspeech (USB) · doi:10.21437/interspeech.2012-400 · Cited by 6 · see citing works
Initiation, Progression, and Conditioning of the Short-Front Vowel Shift in Australia
Grama, J., Travis, C. E., & Ochoa, S. G. (2019) · ANU Open Research (Australian National University) · Cited by 6 · see citing works
Attentional Speech Recognition Models Misbehave on Out-of-domain UtterancesPreprint
Keung, P., Niu, W., Lu, Y., Salazar, J., & Bhardwaj, V. S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.05150 · Cited by 5 · see citing works
Algorithmic Exploration of American English Dialects
Aksënova, A., Bruguier, A., Ritchart-Scott, A., & Mendlovic, U. (2020) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp40776.2020.9053751 · Cited by 4 · see citing works
Self-citation
Flagged first, then by citations
2 of the 84 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Jelke Bloem, Martijn Wieling, and John Nerbonne. 2016. The Future of Dialects: Automatically identifying characteristic features of non-native English accents, chapter 9. Language Science Press, Berlin, Germany.
- Naoyuki Kanda, Guoli Ye, Yu Wu, Yashesh Gaur, Xiaofei Wang, Zhong Meng, Zhuo Chen, and Takuya Yoshioka. 2021. Large-scale pre-training of end-to-end multi- talker ASR for meeting transcription with single distant microphone. In Proc. Interspeech 2021 (submitted), Brno, Czech Republic.
- William Labov. 1991. The three dialects of English. In New Ways of Analyzing Sound Change, pages 1-44.
- Jane W. S. Liu. 2000. Real-time systems. Prentice Hall, Upper Saddle River, NJ.
Cited by
Sorted by citations
Measuring the Accuracy of Automatic Speech Recognition Solutions
Kuhn, K., Kersken, V., Reuter, B., Egger, N., & Zimmermann, G. (2023) · ACM Transactions on Accessible Computing · doi:10.1145/3636513 · Cited by 44 · see citing works
“Evaluation of an ASR system heavily depends on the use-case, and various factors have to be considered [Aksënova et al. 2021].”
MethodA computer-assisted consecutive interpreting workflow: training and evaluation
Chen, S., & Kruger, J. (2024) · The Interpreter and Translator Trainer · doi:10.1080/1750399x.2024.2373553 · Cited by 19 · see citing works
Yang, C.-H. H., Park, T., Gong, Y., Li, Y., Chen, Z., Lin, Y., Chen, C., Hu, Y., Dhawan, K., Żelasko, P., et al. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832176 · Cited by 11 · see citing works
ASR4REAL: An extended benchmark for speech modelsPreprint
Rivière, M., Copet, J., & Synnaeve, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08583 · Cited by 15 · see citing works
Stoykova, R., Porter, K., & Beka, T. (2024) · Forensic Science International Synergy · doi:10.1016/j.fsisyn.2024.100563 · Cited by 7 · see citing works
Liesenfeld, A., Lopez, A., & Dingemanse, M. (2023) · SIGDIAL Conferences · doi:10.18653/v1/2023.sigdial-1.45 · Cited by 18 · see citing works
Accuracy of AI-generated Captions With Collaborative Manual Corrections in Real-Time
Kuhn, K., Kersken, V., & Zimmermann, G. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585724 · Cited by 7 · see citing works
Hutchinson, B., Louro, C. R., Collard, G. D., & Cooper, N. (2025) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3715275.3732010 · Cited by 4 · see citing works
Zhao, S., He, T., & Wen, J. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03450-2 · Cited by 4 · see citing works
A Data Perspective on Ethical Challenges in Voice Biometrics Research
Leschanowsky, A., Rusti, C., Quinlan, C., Pnacekova, M., Gorce, L., & Hutiri, W. T. (2024) · IEEE Transactions on Biometrics Behavior and Identity Science · doi:10.1109/tbiom.2024.3446846 · Cited by 7 · see citing works
Reflections on AI-Augmented Professional Development in Special Education
King, S. A. (2024) · Advances in educational technologies and instructional design book series · doi:10.4018/979-8-3693-5538-1.ch016 · Cited by 3 · see citing works
Voice Anonymization for All-Bias Evaluation of the Voice Privacy Challenge Baseline Systems
Leschanowsky, A., Gaznepoglu, Ü. E., & Peters, N. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10447137 · Cited by 2 · see citing works
Large Language Models Meet Speech Recognition
Chen, P., & Liu, S. (2025) · Springer eBooks · doi:10.1007/978-3-031-76770-8_9 · Cited by 2 · see citing works
A survey of Polish ASR speech datasets
Junczyk, M. (2024) · Poznań Studies in Contemporary Linguistics · doi:10.1515/psicl-2023-0019 · Cited by 1 · see citing works
Bahri, J. E., Kouissi, M., & Begdouri, M. A. (2025) · doi:10.3390/cmsf2025010006 · Cited by 1 · see citing works
BIGOS - Benchmark Intended Grouping of Open Speech Corpora for Polish Automatic Speech Recognition
Junczyk, M. (2023) · Annals of Computer Science and Information Systems · doi:10.15439/2023f1609 · Cited by 1 · see citing works
A new corpus of geolocated ASR transcripts from Germany
Coats, S. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09686-9 · Cited by 1 · see citing works
Ryan, A. M., Boyce, A. S., & Boyce, C. E. (2023) · doi:10.1093/oso/9780197611050.003.0002
Stoykova, R., Porter, K., & Beka, T. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4656884
Evaluating the impact of different acoustic contexts on German speech recognition
Pandya, D. A., & Stuckenschmidt, H. (2024) · IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops) · doi:10.1109/percomworkshops59983.2024.10503200
Integrating Speech Recognition and NLP for Efficient Transcription Solutions
Mehta, D., Upadhyay, R. S., & Jariwala, K. (2025) · International Journal of Scientific Research in Computer Science Engineering and Information Technology · doi:10.32628/cseit2526479
Investigating benchmarking techniques for unveiling large language model performance
Siri, D., Anuragini, S., Babu, B. S., Subbarayudu, Y., Hamza, L. H. J., Maarfi, F., & Dutt, A. (2025) · AIP conference proceedings · doi:10.1063/5.0261237
Bahri, J. E., Kouissi, M., & Begdouri, M. A. (2025) · Studies in systems, decision and control · doi:10.1007/978-3-032-04114-2_14
Louro, C. R., Collard, G. D., Narrier, H., Cox, K., Hayward, L., Colbung, D., & Hutchinson, B. (2026) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3772363.3799311
Gallardo, A., Klucinec, L., Bauer, L., & Cranor, L. F. (2026) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3817022
- Authors
- Aksënova, A., van Esch, D., Flynn, J., & Golik, P.
- Affiliations
- Google (United States) (US)
- ORCID
- Golik, Pavel
- Venue
- Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future
- Published
- 2021
- Pages
- 22-34
- OpenAlex
- W3184326107
- Cited by
- 25
- References
- 84
- Citations by year
- 2026: 2 · 2025: 6 · 2024: 8 · 2023: 7 · 2022: 1 · 2021: 1
- FWCI
- 1.96
- Type
- conference paper
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- speech recognition, benchmark development, evaluation metrics, word error rate, language variation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review