Measuring the Accuracy of Automatic Speech Recognition Solutions

May combine two claims.

Prior studies aim to show specific ASR biases affecting underrepresented groups but do not address overall engine accuracy.

“The studies aim to show that ASR has speciic biases that often afect speakers of underrepresented groups, but are not concerned with the overall accuracy of the engines.”

• cites Cumbal et al. 2021 sourcePartly supported

  The Introduction states that the paper aims to examine the recognition gap between native and non-native Swedish speakers, supporting the claim about group-specific ASR performance. But §5.1 (Performance) says the authors evaluate each ASR using word error rate, and §6.1 compares the engines’ results. The claim that the study is not concerned with engine accuracy is therefore too absolute. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites Koenecke et al. 2020 sourcePartly supported

  The Results report substantially higher word error rates for Black than white speakers across five commercial ASR systems, supporting the claim about bias affecting an underrepresented group. But the same section compares the systems’ overall performance, identifying Microsoft as best and Apple as worst, so the claim that the study was not concerned with engine accuracy is unsupported. No page number is available in our copy. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites Speechmatics 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Training and optimizing for specific datasets risks overfitting, so results may not indicate performance on novel datasets.

“Training and optimisation for speciic datasets bear the risk of overitting, and the obtained results might not be a good indicator of performance on novel datasets [Geirhos et al. 2020].”

• cites Geirhos et al. 2020 sourceSupported

May combine two claims.

ASR performance varies across datasets and speaker groups, including by gender, race, and accent.

“Many sources show ASR's volatility with diferent datasets: ASR performs diferently for male or female speakers, shows racial biases or varying accuracy rates for speakers with diferent accents [Cumbal et al. 2021;Koenecke et al. 2020;Speechmatics 2023;Tadimeti et al. 2022;Tatman and Kasten 2017].”

• cites Cumbal et al. 2021 sourcePartly supported

  Section 6.1 (page unavailable) reports different word error rates for native and non-native Swedish speakers and for speakers with different first languages; Table 2 also shows variation across datasets. These passages support the dataset and accent-related portions of the claim, but our copy does not substantiate the claims about gender differences or racial bias. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites Koenecke et al. 2020 sourcePartly supported

  The Results report different error rates by race and gender (passages 1 and 7), and associate error rates with AAVE dialect density and pronunciation differences (passages 4 and 6). The Discussion (passage 8) suggests that speakers with regional or nonnative-English accents may also face poor performance, but does not establish varying accuracy rates across those accent groups. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites Speechmatics 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Evaluation of an ASR system heavily depends on the use-case, and various factors have to be considered.

“Evaluation of an ASR system heavily depends on the use-case, and various factors have to be considered [Aksënova et al. 2021].”

• cites Aksënova et al. 2021 sourceSupported

May combine two claims.

Training a model on multiple public speech datasets results in comparable accuracy but higher robustness than models trained on a single source.

“Chan et al. [2021] show that training a model on multiple public speech datasets results in a comparable accuracy but higher robustness than models trained on a single source.”

• cites Chan et al. 2021 sourcePartly supported

  The Introduction (p. 1) describes training one model on combined speech datasets and reports results that compare favourably with prior work; the Experiments (p. 3) include single-domain results for comparison. The paper also reports strong transfer to a new dataset, but it does not establish that the mixed-data model is more robust than models trained on a single source. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Evaluating ASR systems on multiple public speech datasets results in an average End-to-End Speech Benchmark (ESB) score.

“Respectively [Gandhi et al. 2022] propose to evaluate ASR systems on multiple public speech datasets resulting in an average End-to-End Speech Benchmark (ESB) score.”

• cites Gandhi et al. 2022 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

The quality of ASR in poor acoustic environments has greatly improved, e.g. accuracy in noisy environments.

“The quality of ASR in poor acoustic environments has greatly improved, e.g. the accuracy in noisy environments [Kinoshita et al. 2020].”

• cites Kinoshita et al. 2020 sourceSupported

Classified as probably not a substantial claim.

The authors used the Whisper text normaliser, which transforms text to lowercase, removes all punctuation, and unifies common contractions, spelling differences and spoken numbers.

“We used the Whisper text normaliser [Radford et al. 2022], which transforms text to lowercase, removes all punctuation, and uniies common contractions, spelling diferences and spoken numbers.”

• cites Radford et al. 2022 sourcePartly supported

  Appendix C, Text Standardization (p. 21), describes the Whisper normalizer’s handling of contractions, British and American spellings, and spoken numbers. It does not list lowercasing among the English normalization steps, and its punctuation-removal rules preserve characters used in numeric expressions; the claim that it removes all punctuation is therefore too broad. Lowercasing is explicitly listed for non-English text. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

A prior study indicates that speakers adjust their behaviour in meetings with DHH peers when ASR is used.

“A prior study indicates that speakers adjust their behaviour in meetings with DHH peers when ASR is used [Seita et al. 2018].”

• cites Seita et al. 2018 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Focus groups can provide ideas on how to improve the use of ASR or help assess the acceptance of new features like automatic content summarisation.

“Focus groups can also provide ideas on how to improve the use of ASR [Seita et al. 2022], or help assess the acceptance of new features like automatic content summarisation [Alonzo et al. 2020].”

• cites Seita et al. 2022 sourceSupported
• cites Alonzo et al. 2020 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The CER may be useful for evaluating languages that do not separate words with spaces, such as Chinese.

“The CER may be useful for evaluating languages that do not separate words with spaces, such as Chinese [Radford et al. 2022].”

• cites Radford et al. 2022 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

A related study reported a speed increase of up to twelve times without sacrificing transcription accuracy.

“A related study reported a speed increase of up to twelve times without sacriicing transcription accuracy [Bain et al. 2023].”

• cites Bain et al. 2023 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

These findings are also reported by other studies.

“These indings are also reported by other studies [AssemblyAI 2023;Hughes 2023;Radford et al. 2022;Stephenson 2022].”

• cites AssemblyAI 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Hughes 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Radford et al. 2022 sourceCould not verify

  The citing sentence does not identify what “these findings” refers to, so there is no specific result to compare with Radford et al. Section 5 (p. 13) discusses a finding replicated across fields, and Section 4.3 (p. 11) reports transfer results, but the supplied sentence does not establish that either is the finding being cited. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Prior research reports WER measurements for several of the vendors included in this study.

“Summary of Word Error Rate measurements of diferent services from related research”

• cites Addlesee et al. 2020 sourceSupported
• cites Ballenger 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Catania et al. 2019 sourceNot checked — no full text of the source

May combine two claims.

The accuracy of the eleven ASR services varied widely between vendors and across individual audio samples.

“Our results show that accuracy ranges widely between vendors and for the individual audio samples.”

May combine two claims.

Some vendors used public datasets to test the accuracy of their ASR engines, while others referred to internal datasets.

“While some vendors used public datasets to test the accuracy of their ASR engines (SpeechText.AI [SpeechText.AI 2023], Rev AI [Jette 2020], Speechmatics [Hughes 2023], AssemblyAI [AssemblyAI 2023]), others referred to internal datasets (Deepgram [Stephenson 2022]).”

• cites SpeechText.AI 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Jette 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Stephenson 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Common cloud providers Amazon, Google, IBM and Microsoft appear in multiple studies, while specialised ASR providers are less represented.

“While the common cloud providers Amazon, Google, IBM and Microsoft appear in multiple studies, specialised ASR providers are less represented.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Some vendors require additional steps like uploading files to a cloud bucket, while others handle multiple steps within a single HTTP-Call.

“Some vendors require additional steps like uploading iles to a cloud bucket, while others handle multiple steps within a single HTTP-Call (e.g.transcription with a custom vocabulary).”

The quote only partly states this claim.

The average WER of the eight vendors offering custom vocabulary across the English datasets was 8.61% without a vocabulary.

“The average WER of these providers across the English datasets was 8.61%.”

May combine two claims.

On average, native English speakers achieved the lowest error rates and ESL speakers performed better compared to German Native Speakers.

“On average native English speakers achieved the lowest error rates and ESL speakers performed better compared to the results for German Native Speakers.”

May combine two claims.

Both the average accuracy and standard deviation of error rates varies between the vendors.

“The results show that both the average accuracy and standard deviation of error rates varies between the vendors.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some APIs offer the functionality to upload custom vocabularies before the actual transcription.

“Some APIs ofer the functionality to upload custom vocabularies before the actual transcription.”

May combine two claims.

For the 30 different samples, seven different vendors achieved the highest accuracy and three different vendors for the lowest accuracy.

“For the 30 diferent samples, we counted seven diferent vendors that achieved the highest accuracy and three diferent vendors for the lowest accuracy.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The average wpm is 148 (min 94, max 222).

“The average wpm is 148 (min 94, max 222).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The dataset has an even distribution of male and female speakers.

“The dataset has an even distribution of male and female speakers”

Classified as probably not a substantial claim.

All audio samples are around three minutes long.

“as all audio samples are around three minutes long.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The samples have an average duration of 184 seconds (min 174, max 209).

“The samples have an average duration of 184 seconds (min 174, max 209).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Some vendors did not provide all technical requirements e.g. custom vocabularies, streaming or specific combinations.

“Some vendors did not provide all technical requirements e.g.custom vocabularies, streaming or speciic combinations (e.g.German streaming with a custom vocabulary).”

May combine two claims.

Whisper has the highest accuracy and longest processing duration.

“Whisper has the highest accuracy and longest processing duration”

Probably not a substantial claim: may not pass S3 (load-bearing).

The number of spoken words is 455 on average (min 287, max 671).

“The number of spoken words is 455 on average (min 287, max 671).”

May combine two claims.

The lowest calculated WER value was 0% and the highest 53.8%.

“The lowest calculated value was 0% and the highest 53.8%.”

May combine two claims.

The average and standard deviation of Word Information Lost are around 30% higher than WER.

“they are around 30% higher for the Word Information Lost (M=0.152,SE=0.118).”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The samples do not contain strong accents.

“Even though the samples do not contain strong accents”

Probably not a substantial claim: may not pass S3 (load-bearing).

Providing a vocabulary increased the vocabulary-word hits for all vendors except Tencent.

“The hits increased for all vendors except Tencent when providing a vocabulary.”

May combine two claims.

Samples of non-native speakers show higher error rates for English, but still perform better compared to the German dataset.

“While the samples of non-native speakers show higher error rates for English, they still perform better compared to our German dataset.”

May combine two claims.

Whisper as an open-source model outperforms commercial services in many cases and reports quite low error rates in general.

“Whisper as an open-source model outperforms commercial services in many cases and reports quite low error rates in general.”

Probably not a substantial claim: may not pass S3 (load-bearing).

For all vendors, the majority of vocabulary-word hits were already counted without providing a vocabulary.

“For all vendors the majority of hits was already counted without a vocabulary.”

Probably not a substantial claim: may not pass S3 (load-bearing).

No vendor reached the maximum number of vocabulary-word hits according to the reference transcript.

“No vendor reached the maximum hits according to the reference transcript.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

ASR performs notably better for English and German compared to many other languages.

“Furthermore, ASR performs notably better for English and German compared to many other languages.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

Several transcription problems were encountered, including differences between UK and US spelling, common contractions, and currency writings.

“We encountered several problems during transcription, such as diferences between UK-and US-spelling, common contractions”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

Whisper (large-v2) had the lowest average WER on each dataset: 2.9 on English LibriSpeech, 3.3 on ESL, 3.3 on German, and 5.0 on average.

“| Whisper (large-v2) | 2.9 | 3.3 | 3.3 | 5.0 |”

May combine two claims.

The study confirmed very high accuracy rates in some cases, but found a wide range in quality across vendors, speakers and languages.

“We could conirm very high accuracy rates in some cases, but found a wide range in quality across vendors, speakers and languages.”

May combine two claims.

Alternative error metrics showed similar results compared to the WER and seem to be interchangeable.

“Alternative error metrics showed similar results compared to the WER and seem to be interchangeable.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Whisper with condition_on_previous_text enabled hallucinated and created longer transcripts than the actual file.

“as it hallucinated and created longer transcripts than the actual ile.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Tencent did not provide confidence values.

“Tencent did not provide conidence values.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Transcribing correct punctuation is difficult for spoken language and influences capitalization.

“Additionally, transcribing correct punctuation is diicult for spoken language and inluences capitalization”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The average WER across vendors was 7.0 for English LibriSpeech, 8.6 for ESL, 10.2 for German, and 15.3 overall.

“| Average | 7.0 | 8.6 | 10.2 | 15.3 |”

Probably not a substantial claim: may not pass S3 (load-bearing).

The NAD proposes to regulate the use of ASR and the metrics used to specify caption and transcript accuracy.

“The NAD proposes to regulate the use of ASR and the applied metrics to specify accuracy of captions and transcripts.”

• cites Blake 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

After an initial transcription using Whisper, the transcript was manually corrected to create an error-free reference and a vocabulary list.

“After an initial transcription using Whisper (4), the transcript was manually corrected (5) to create an error-free reference and a vocabulary list.”

Probably not a substantial claim: may not pass S2 (checkable).

Despite criticism, the WER is still useful to evaluate and compare ASR systems.

“Despite the criticism, the WER is still useful to evaluate and compare ASR systems.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

As ASR systems produce verbatim transcripts, a word-by-word comparison is suitable and an analysis in terms of content becomes less relevant.

“As these systems produce verbatim transcripts, a word by word comparison is suitable and an analysis in terms of the content becomes less relevant.”

• AssumptionRests on an assumption: AI is good at keeping up with the speed of speech and creates almost verbatim transcripts.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Speech can be up to ten times faster than typing, making manual transcription laborious.

“Manually creating accurate transcripts for spoken language is a laborious task, as speech is up to ten times faster than typing [Wald 2006].”

• cites Wald 2006 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

Heavily accented speech is challenging for ASR systems because it contains acoustic and linguistic markers of both the source and target language, and could thus be the source of transcription errors.

“We used these criteria because heavily accented speech is challenging for ASR systems, as it contains acoustic and linguistic markers of both the source and target language, and could thus be the source of transcription errors.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Transcription of real-time events is more challenging because there is no time to replay audio and only a small window to correct texts.

“Transcription of real-time events is even more challenging, as there is no time to replay audio, and there is only a small window of time to correct texts.”

Classified as probably not a substantial claim. May combine two claims.

The transcription process was fully automated using equivalent configurations across different vendors to provide high comparability.

“To provide a high comparability of results, the transcription process was fully automated, using equivalent conigurations across diferent vendors.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The FCC requires captions to match spoken words to the fullest extent possible and distinguishes between pre-recorded and live captioning rules.

“The Federal Communications Commission (FCC) requires captions to match the spoken words to the fullest extent possible, and additionally states that their rules on captioning distinguish between pre-recorded media and live transcriptions [Federal Communications Commission 2014].”

• cites Federal Communications Commission 2014Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

WER was calculated after extensive text normalisation to reduce errors caused by non-semantic differences.

“The WER was calculated after extensive text normalisation to reduce errors caused by non-semantic diferences.”

Classified as probably not a substantial claim. May combine two claims.

The dataset was created to reflect ASR performance with minimal bias and without targeting specific weaknesses.

“We created a novel dataset that attempts to relect the performance of ASR with minimal bias and without targeting speciic weaknesses.”

Classified as probably not a substantial claim.

A new dataset was created to minimise the risk that samples were part of an ASR model's training data, which could lead to artificially good performance.

“To achieve this objective, a new dataset was created to minimise the risk of samples being part of an ASR model's training data, which could lead to a particularly good performance.”

May combine two claims.

The WER values of different ASR studies are not comparable to each other because error rates depend on the tested dataset, the date of the test, and the degree of text normalisation used to calculate WER.

“The values of each study are not comparable to others, as the error rates depend on multiple factors: (1) the tested dataset, (2) the date of the test, and (3) the degree of text normalisation to calculate the WER.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Limited availability of transcription professionals is a big issue.

“Besides the complexity of this task and high operational costs, a big issue is the limited availability of these professionals.”

Classified as probably not a substantial claim. May combine two claims.

The quality of all speech recordings was measured with MOSNet and only recordings with an absolute value of 2 or higher were selected.

“We additionally measured the quality of all speech recordings with MOSNet and selected only recordings with an absolute value of 2 or higher [Lo et al. 2019].”

• cites Lo et al. 2019Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The majority of DHH individuals tend to favour the comprehensive access offered by verbatim texts.

“the majority of DHH individuals tend to favour the comprehensive access ofered by verbatim texts [National Institute on Deafness and Other Communication Disorders 2017].”

• cites National Institute on Deafness and Other Communication Disorders 2017Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Recordings with very professional setups or bad audio quality were avoided.

“We avoided recordings with either very professional setups, or bad audio quality (e.g.strong reverberation, noise interferences, unintelligible or barely audible speech).”

Classified as probably not a substantial claim.

Only videos with average audio quality representative of a Higher Education setting were selected.

“We selected only videos with an average audio quality, that (we think) is representative of a Higher Education setting (e.g.speakers using a clip-on microphone).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Within six years, the most accurate models could decrease the WER down to 2.5%.

“Within only six years, the most accurate models could decrease the WER down to 2.5% [Meta AI 2023].”

• cites Meta AI 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Accessible transcription incorporates issues beyond text accuracy, such as placement of captions, correct speaker identification, and presence of relevant non-speech information.

“Besides text accuracy, accessible transcription incorporates other issues like the placement of captions, correct speaker identiication or the presence of relevant non-speech information [Federal Communications Commission 2014;Lawton Henry et al. 2022].”

• cites Federal Communications Commission 2014Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites Lawton Henry et al. 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Prior to calculating WER, the ASR transcript and the reference transcript were normalised.

“Prior to calculating the WER, the ASR transcript and the reference transcript were normalised.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Human-generated transcripts can be easier to read than ASR results but can also miss important information.

“Even though the transcript might be easier to read than results from ASR, it can also miss important information.”

Classified as probably not a substantial claim.

The initial selection of videos was rated in a two step review process by three members of the team with regard to recording quality, accent, intelligibility, and High German without strong accents.

“This initial selection of videos was then rated in a two step review process by three members of the team with regard to the following criteria:”

Classified as probably not a substantial claim.

All videos that had manually created or corrected captions were excluded because they might be part of the training data of Whisper and other services.

“We also excluded all videos that had manually created or corrected captions, as these might be part of the training data of Whisper and other services.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The LibriSpeech corpus is a widely used benchmark in science and industry.

“The LibriSpeech corpus [Panayotov et al. 2015] is a widely used benchmark in science and industry.”

• cites Panayotov et al. 2015 sourceNot checked — no full text of the source

May combine two claims.

The WER does not reflect text understanding and only weakly correlates with human judgement of a transcript's quality.

“But the WER has been criticised, as it does not relect text understanding, and only weakly correlates with human judgement of a transcripts' quality [Favre et al. 2013;Mishra et al. 2011;Wang et al. 2003].”

• cites Favre et al. 2013 sourceNot checked — no full text of the source
• cites Mishra et al. 2011 sourceNot checked — no full text of the source
• cites Wang et al. 2003Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The FCC requires that online captions must be verbatim, and paraphrasing should be minimised for realtime transcription.

“The FCC also requires that oline captions must be verbatim, and paraphrasing should be minimised for realtime transcription [Federal Communications Commission 2014].”

• cites Federal Communications Commission 2014Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

ASR does not distinguish between skin colours, but reflects biases in the training data.

“Obviously, ASR does not distinguish between skin colours, but relects biases in the training data.”

Probably not a substantial claim: may not pass S3 (load-bearing).

With LibriSpeech release in 2015, first models reported a WER of 13.25%.

“With its release in 2015 the irst models reported a WER of 13.25%.”

• cites Panayotov et al. 2015 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

When companies publish data on the accuracy of their ASR services, it is not always clear whether these serve the purpose of advertising or a favourable comparison to other vendors.

“When companies publish data on the accuracy of their ASR services, it is not always clear whether these serve the purpose of advertising or a favourable comparison to other vendors.”

Classified as probably not a substantial claim.

W3C WAI states that ASR-generated captions do not meet accessibility requirements unless confirmed fully accurate.

“However the W3C Web Accessibility Initiative (WAI) is a bit more speciic by stating that ASR-generated captions do not meet accessibility requirements unless they can be conirmed to be fully accurate [Lawton Henry et al. 2022].”

• cites Lawton Henry et al. 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

The Higher Education scenario is suitable for evaluating ASR because it contains specialist vocabulary and lecturers are not professional speakers.

“The scenario of Higher Education is also suitable, as it contains specialist vocabulary and lecturers are used to presenting without being professional speakers.”

• AssumptionRests on an assumption: There is a high demand for affordable and scalable transcription of lectures, talks and other learning formats in Higher Education.
• AssumptionRests on an assumption: Higher Education institutions are required to offer equal access to all students, and transcription of lectures is essential for that, benefiting DHH students, language learners, and remote students.

Probably not a substantial claim: may not pass S3 (load-bearing).

Trained professionals are needed to accurately transcribe live events.

“Thus, trained professionals are needed to accurately transcribe live events.”

May combine two claims.

WCAG only specifies that captions are provided, without clearly defined accuracy requirements.

“The Web Content Accessibility Guidelines (WCAG) only specify that captions are provided, without clearly deined requirements for accuracy [Adams et al. 2022].”

• cites Adams et al. 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Both CART and respeaking are very demanding, requiring multiple professionals for longer events.

“As both methods are very demanding, multiple professionals are required for longer events to give the operators rest periods.”

Probably not a substantial claim: may not pass S3 (load-bearing).

ACE might not be suited for scenarios other than conversational speech, such as Higher Education or Entertainment.

“ACE for example is trained on conversational speech and thus might not be suited for other scenarios like Higher Education or Entertainment.”

• AssumptionRests on an assumption: ACE is trained on conversational speech.

Classified as probably not a substantial claim. May combine two claims.

The lower the WER is, the closer the produced transcript is to the actual text, and fewer errors should result in high levels of understandability.

“The lower the WER is, the closer the produced transcript is to the actual text -fewer errors should therefore result in high levels of understandability.”

Classified as probably not a substantial claim.

Only parts of recordings where only the presenter speaks were chosen, to avoid additional issues like speaker diarisation.

“We only chose parts of recordings, where only the presenter speaks, to avoid additional issues like speaker diarisation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Demand for transcription of events is increasing as more digital content is produced.

“At the same time, as more digital content is produced in the form of online presentations and remote meetings, the demand for transcription of events is increasing.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Bias in ASR applies to all kinds of dimensions.

“Bias in ASR applies to all kinds of dimensions.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The FCC requires that capitalisation and punctuation is included in accurate captions, as these influence an individual's understanding of the text.

“The FCC requires that capitalisation and punctuation is included in accurate captions, as these inluence an individual's understanding of the text [Federal Communications Commission 2014].”

• cites Federal Communications Commission 2014Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors selected content covering a wide area of topics to limit bias in the dataset.

“We try to limit this bias by selecting content that covered a wide area of topics from Computer Science, Electrical Engineering and Chemistry to History, Sociology, Politics, Linguistics, Religion and Neuroscience.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study has four different technical scenarios: batch or stream transcription and with or without an additional custom vocabulary.

“Additionally, we have four diferent technical scenarios: batch or stream transcription and with or without an additional custom vocabulary.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The study has four language groups: English, ESL, German and LibriSpeech (English).

“We have four language groups: English, ESL, German and LibriSpeech (English).”

Probably not a substantial claim: may not pass S3 (load-bearing).

Results might be different for more spontaneous, conversational, and colloquial speech.

“As our samples are recordings in the context of Higher Education, results might be diferent for more spontaneous, conversational, and colloquial speech.”

Classified as probably not a substantial claim.

The authors randomly selected a part of each video that had continuous and fluent speech.

“We randomly selected a part of each video that had continuous and luent speech (e.g.no interactions with the audience or writing on a board).”

Classified as probably not a substantial claim.

The audio was converted to a standardised format that all vendors supported (44,100 Hz, mono, PCM 16-bit, LE).

“The audio was converted to a standardised format that all vendors supported (44,100 Hz, mono, PCM 16-bit, LE).”

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors always used the closest region to their location if cloud vendors offered multiple regions.

“We always used the closest region to our location (e.g.eu-central), if cloud vendors ofered multiple regions.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

The lower results of AssemblyAI and Tencent could indicate an optimisation towards the LibriSpeech corpus.

“The lower results of AssemblyAI and Tencent could indicate an optimisation towards the LibriSpeech corpus.”

Classified as probably not a substantial claim.

Two independent coders manually corrected each transcript.

“Two independent coders manually corrected each transcript.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

As punctuation was removed for the calculation of the WER, the authors only used confidence outputs from an API on words and excluded punctuation marks.

“Consequently as punctuation was removed for the calculation of the WER, we only used conidence outputs from an API on words and excluded punctuation marks.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

It is difficult to assess ASR's ability to provide accessible transcripts, as it depends on a case-by-case basis.

“This makes it diicult to assess ASR's ability to provide accessible transcripts, as it depends on a case-by-case basis.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Our findings do not necessarily disprove other studies that report higher error rates, as WER is heavily dependent on the degree of text normalisation.

“However, our indings do not necessarily disprove other studies that report higher error rates (see Table 2), as the WER is heavily dependent on the degree of text normalisation.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors could not rule out the possibility that a repeated transcription of the same audio file influences the result of the second run (e.g., faster processing time through caching), except for Whisper.

“We could not rule out the possibility that a repeated transcription of the same audio ile inluences the result of the second run (e.g.faster processing time through caching), except for Whisper.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors did not select ESL speakers from countries or regions that might have English as an official government language (e.g., India or Nigeria), as they might be more fluent or completed their education in English.

“However, we did not select speakers for the ESL samples from countries or regions that might have English as an oicial government language (e.g.India or Nigeria), as they might be more luent or completed their education in English.”

Classified as probably not a substantial claim. May combine two claims.

The authors used the jiwer Python implementation to calculate the WER, and applied extensive text transformation to all transcripts to reduce the impact of formatting differences on the error rate.

“We used the jiwer [Vaessen 2023] Python implementation to calculate the WER, and applied extensive text transformation to all transcripts to reduce the impact of formatting diferences on the error rate.”

• cites Vaessen 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Previous research suggests using ASR as a basis for manual corrections to create more accurate transcripts compared to a fully manual approach.

“Previous research suggests using ASR as a basis for manual corrections to create more accurate transcripts compared to a fully manual approach [Che et al. 2017].”

• cites Che et al. 2017 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

The volume of the audio was normalised with sox.

“Additionally, the volume of the audio was normalised with sox [Bagwell 2023].”

• cites Bagwell 2023Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

ASR engines are only trained on limited datasets that do not include other variants of English, which could lead to or enforce existing biases.

“It is a general problem that ASR engines are only trained on limited datasets that do not include other variants of English as this could lead to or enforce existing biases.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors selected ESL samples with different accents (European, South American, Asian).

“We selected samples with ESL speakers who had diferent accents (European, South American, Asian).”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the case of Tencent, the authors had to exclude all words that exceeded the limit of ten characters.

“In the case of Tencent, we had to exclude all words that exceeded the limit of ten characters.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The different approaches might affect the measured timings and are considered for a comparative analysis.

“The diferent approaches might afect the measured timings and are considered for a comparative analysis.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

In a few cases, the authors had to restart a transcription job due to an HTTP or API error, and had no issues after a second try.

“In a few cases, we had to restart a transcription job, as an HTTP or API error occurred.We had no issues after a second try.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The dataset is not specifically designed to examine gender differences in ASR accuracy.

“but is not speciically designed to examine gender diferences in ASR accuracy.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors extended the normalisation for German according to common contractions and numbers appearing in the transcripts of their dataset.

“We extended the normalisation for German according to common contractions and numbers appearing in the transcripts of our dataset.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors chose 15 male and female speakers per language group.

“We chose 15 male and female speakers per language group.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

ASR engines are often trained on datasets with RP or GenAm, and samples with a large difference from this training set could produce more errors.

“Similarly, ASR engines are often trained on datasets with RP or GenAm.Samples with a large diference from this training set, could produce more errors.”

Probably not a substantial claim: may not pass S4 (based).

The control recordings fulfil the same criteria as the English videos with regard to speaker accents and audio length.

“The recordings fulill the same criteria as the English videos with regard to speaker accents and audio length.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Acceptance of ASR as an accessible accommodation is an important factor, as it can be perceived as a second-rate service by users.

“Acceptance of ASR as an accessible accommodation is an important factor, as it can be perceived as a second-rate service by users [National Deaf Center on Postsecondary Outcomes 2020].”

• cites National Deaf Center on Postsecondary Outcomes 2020Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors carefully looked out for Whisper-specific errors such as the removal of duplicate words.

“we carefully looked out for Whisper speciic errors like the removal of duplicate words.”

May combine two claims.

The quality most likely decreases for languages with a smaller corpus of training data and more complex languages.

“The quality most likely decreases for languages with a smaller corpus of training data, and more complex languages, e.g.languages with a number of grammatical cases, articles or capitalisation of nouns.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Results may differ in other environments or for more conversational speech.

“Results may difer in other environments or for more conversational speech.”

Classified as probably not a substantial claim.

All API parameters of each vendor were checked to assure comparability and to avoid unintentional default settings.

“We checked all API parameters of each vendor to assure comparability and to avoid unintentional default settings.”

May combine two claims.

The WER cannot be used exclusively, as text normalisation removes important aspects of punctuation or capitalisation, and other factors such as speaker diarisation or caption formatting are not covered.

“However, it cannot be used exclusively, as text normalisation removes important aspects of punctuation or capitalisation, and other factors such as speaker diarisation or caption formatting are not covered.”

Probably not a substantial claim: may not pass S2 (checkable).

Given the variety of ASR applications, a single metric is insufficient to comprehensively represent the quality of an ASR model.

“Due to the variety of applications of ASR, a single metric appears insuicient to comprehensively represent the quality of an ASR-model.”

May combine two claims.

Most of the tested ASR services seem to have a robust general performance, and do not only achieve particularly good results on a common dataset.

“Most of the tested ASR services seem to have a robust general performance, and do not only achieve particularly good results on a common dataset.”

Probably not a substantial claim: may not pass S3 (load-bearing).

If vendors provided multiple models, the model that promised the highest accuracy was selected.

“If vendors provided multiple models, we selected the model that promised the highest accuracy.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The findings are restricted to the date of the evaluation, as AI is a rapidly changing area of research.

“The indings are also restricted to the date of the evaluation, as AI is a rapidly changing area of research.”

• AssumptionRests on an assumption: AI is a rapidly changing area of research.

Probably not a substantial claim: may not pass S2 (checkable).

Despite criticism that the WER does not necessarily reflect the usefulness of a text, it still holds potential as a complementary accessibility metric.

“Despite criticism that the WER does not necessarily relect the usefulness of a text [Favre et al. 2013;Mishra et al. 2011;Wang et al. 2003], it still holds potential as a complementary accessibility metric.”

• cites Favre et al. 2013 sourceNot checked — no full text of the source
• cites Mishra et al. 2011 sourceNot checked — no full text of the source
• cites Wang et al. 2003Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: ASR continues to improve and error rates decrease, making metrics based on a string edit distance more robust.
• AssumptionRests on an assumption: The WER is objective, unbiased, and commonly used as an evaluation metric in ASR research.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The setting chosen, Higher Education, provided rather ideal conditions and avoided particular weaknesses of ASR.

“The setting we chose, Higher Education, provided rather ideal conditions and avoided particular weaknesses of ASR.”

Probably not a substantial claim: may not pass S2 (checkable), S4 (based). May combine two claims.

Only because ASR offers a solution for the complex task of transcription, it is not necessarily accessible and enables DHH individuals to participate in various events.

“But only because ASR ofers a solution for the complex task of transcription, it is not necessarily accessible and enables DHH individuals to participate in various events.”

May combine two claims.

Accuracy is not the only relevant factor for ASR, and E2E-models have shown weaknesses in streaming capabilities, latency and computational efficiency.

“However, accuracy is not the only relevant factor for ASR, and E2E-models have shown weaknesses in streaming capabilities, latency and computational eiciency [Li 2022].”

• cites Li 2022Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

It is very likely that Deepgram uses a split-and-merge approach to increase processing speed, given its very fast performance in the evaluation.

“It is very likely that Deepgram uses such an approach, given its very fast performance in our evaluation.”

May combine two claims.

The WER is particularly useful for verbatim transcription, and extensive text normalisation can reduce errors caused by non-semantic differences.

“The WER is particularly useful for verbatim transcription, and extensive text normalisation can reduce errors caused by non-semantic diferences.”

Classified as probably not a substantial claim.

Because text normalisation is used to reduce non-semantic differences, information about the correctness of punctuation and capitalisation is lost from WER.

“but because text normalisation is used to reduce non-semantic diferences, information about the correctness of punctuation and capitalisation is lost.”

Probably not a substantial claim: may not pass S4 (based).

The chosen scenario avoided many of the challenges of ASR, such as speaker diarisation, overlapping speech and mixed language.

“Our chosen scenario avoided many of the challenges of ASR, such as speaker diarisation, overlapping speech and mixed language.”

Probably not a substantial claim: may not pass S4 (based).

WER currently serves as the standard metric for reporting ASR accuracy.

“Currently, the WER serves as the standard metric for reporting accuracy”

Probably not a substantial claim: may not pass S3 (load-bearing).

To receive verbatim transcripts, all filters (e.g. profanity) were disabled.

“To receive verbatim transcripts, we disabled all ilters (e.g.profanity).”

Probably not a substantial claim: may not pass S3 (load-bearing).

There is no guarantee that any of these words will be correctly identified and transcribed by an ASR-model.

“even though there is no guarantee that any of these words will be correctly identiied and transcribed by an ASR-model.”

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

The dataset reflects biases inherent in the material available online and the over-representation of certain groups in Higher Education.

“Furthermore, our dataset relects biases inherent in the material available online and the over-representation of certain groups in Higher Education.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The analysis relies on the WER as the most common automated metric, which is a limited representation of ASR quality.

“Due to the large amount of data, the analysis relies on the WER as the most common automated metric, which is a limited representation of ASR quality.”

May combine two claims.

Even though ASR might meet accuracy requirements for caption users for English, it might fail to meet them for other languages and outside narrowly defined scenarios, which is a real problem for accessibility of events.

“Even though ASR might meet the accuracy requirements for caption users for English, it might fail to meet them for other languages and outside of narrowly deined scenarios, which is a real problem for the accessibility of events.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

These words might be fundamental for understanding the content and might consequently make the transcript more accessible.

“Even though adding the vocabulary did not signiicantly decrease the overall WER, these words might be fundamental for understanding the content and might consequently make the transcript more accessible.”

Classified as probably not a substantial claim.

To minimise the effects of audio processing that vendors might use, all samples were converted to a standard audio format and the volume was normalised.

“To minimise the efects of audio processing that vendors might use, we converted all samples to a standard audio format and normalised the volume.”

Classified as probably not a substantial claim.

The confidence values merely seem to represent the probabilities within an ASR-model, and not the actual correctness.

“The conidence values merely seem to represent the probabilities within an ASR-model, and not the actual correctness.”