Cited by 25Speech Recognition and SynthesisSpeech and Audio ProcessingSpeech and dialogue systems

How Might We Create Better Benchmarks for Speech Recognition?

Open access
Aksënova, Alëna, van Esch, Daan, Flynn, James, Golik, Pavel·2021·Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future

Abstract

The applications of automatic speech recognition (ASR) systems are proliferating, in part due to recent significant quality improvements.However, as recent work indicates, even state-of-the-art speech recognition systems -some which deliver impressive benchmark results, struggle to generalize across use cases.We review relevant work, and, hoping to inform future benchmark development, outline a taxonomy of speech recognition use cases, proposed for the next generation of ASR benchmarks.We also survey work on metrics, in addition to the de facto standard Word Error Rate (WER) metric, and we introduce a versatile framework designed to describe interactions between linguistic variation and ASR performance metrics.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.18653/v1/2021.bppf-1.4

Cites

82 of 82 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Evaluation of an ASR system heavily depends on the use-case, and various factors have to be considered.

    “Evaluation of an ASR system heavily depends on the use-case, and various factors have to be considered [Aksënova et al. 2021].”

    From Measuring the Accuracy of Automatic Speech Recognition Solutions (2023) · cites this paper as Aksënova et al. 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

2 of the 84 references couldn’t be matched to a known paper.

Research works printed but not linked: 4

  1. Jelke Bloem, Martijn Wieling, and John Nerbonne. 2016. The Future of Dialects: Automatically identifying characteristic features of non-native English accents, chapter 9. Language Science Press, Berlin, Germany.
  2. Naoyuki Kanda, Guoli Ye, Yu Wu, Yashesh Gaur, Xiaofei Wang, Zhong Meng, Zhuo Chen, and Takuya Yoshioka. 2021. Large-scale pre-training of end-to-end multi- talker ASR for meeting transcription with single distant microphone. In Proc. Interspeech 2021 (submitted), Brno, Czech Republic.
  3. William Labov. 1991. The three dialects of English. In New Ways of Analyzing Sound Change, pages 1-44.
  4. Jane W. S. Liu. 2000. Real-time systems. Prentice Hall, Upper Saddle River, NJ.

Cited by

Sorted by citations

Authors
Aksënova, A., van Esch, D., Flynn, J., & Golik, P.
Affiliations
Google (United States) (US)
Venue
Proceedings of the 1st Workshop on Benchmarking: Past, Present and Future
Published
2021
Pages
22-34
OpenAlex
W3184326107
Cited by
25
References
84
Citations by year
2026: 2 · 2025: 6 · 2024: 8 · 2023: 7 · 2022: 1 · 2021: 1
FWCI
1.96
Type
conference paper
Language
en
Open access
Licence
cc-by
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
  • Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
  • Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
Keywords
speech recognition, benchmark development, evaluation metrics, word error rate, language variation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review