Using Automatic Speech Recognition to Measure the Intelligibility of Speech Synthesized From Brain Signals
Open accessAbstract
Brain-computer interfaces (BCIs) can potentially restore lost function in patients with neurological injury. A promising new application of BCI technology has focused on speech restoration. One approach is to synthesize speech from the neural correlates of a person who cannot speak, as they attempt to do so. However, there is no established gold-standard for quantifying the quality of BCI-synthesized speech. Quantitative metrics, such as applying correlation coefficients between true and decoded speech, are not applicable to anarthric users and fail to capture intelligibility by actual human listeners; by contrast, methods involving people completing forced-choice multiple-choice questionnaires are imprecise, not practical at scale, and cannot be used as cost functions for improving speech decoding algorithms. Here, we present a deep learning-based "AI Listener" that can be used to evaluate BCI speech intelligibility objectively, rapidly, and automatically. We begin by adapting several leading Automatic Speech Recognition (ASR) deep learning models - Deepspeech, Wav2vec 2.0, and Kaldi - to suit our application. We then evaluate the performance of these ASRs on multiple speech datasets with varying levels of intelligibility, including: healthy speech, speech from people with dysarthria, and synthesized BCI speech. Our results demonstrate that the multiple-language ASR model XLSR-Wav2vec 2.0, trained to output phonemes, yields superior performance in terms of speech transcription accuracy. Notably, the AI Listener reports that several previously published BCI output datasets are not intelligible, which is consistent with human listeners.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ASR systems are typically trained to recognize natural waveforms, not artificially synthesized ones.
“ASR can also be used to transcribe synthesized speech into text to compute WERs; however, ASR systems are typically trained to recognize natural waveforms that are not artificially synthesized153.”
From The speech neuroprosthesis (2024) · cites this paper as [153]
The Methods (Automated speech recognition models; page unavailable) say that the Wav2vec 2.0 model was trained on speech corpora, and the paper applies ASR to BCI-synthesized speech. Our copy does not establish that ASR systems generally are trained on natural waveforms rather than artificially synthesized ones. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
“Librispeech: an ASR corpus based on public domain audio books,”
Panayotov, V., Chen, G., Povey, D., & Khudanpur, S. (2015) · 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2015.7178964 · Cited by 6,072 · see citing works
Kaldi Speech Recognition Toolkit
Povey, D. (2024) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · doi:10.57702/jb3fvbn9 · Cited by 4,837 · see citing works
High-performance brain-to-text communication via handwriting
Willett, F. R., Avansino, D. T., Hochberg, L. R., Henderson, J. M., & Shenoy, K. V. (2021) · Nature · doi:10.1038/s41586-021-03506-2 · Cited by 969 · cited 1× · see citing works
“Previous intracortical BCIs [1] successfully provided point-and-click and handwriting communication: paralyzed users attempted to move their hand, and algorithms decoded that neural data to make character selections (with speeds of 8 to 19 words per minute).”
BackgroundSpeech synthesis from neural decoding of spoken sentences
Anumanchipalli, G. K., Chartier, J., & Chang, E. F. (2019) · Nature · doi:10.1038/s41586-019-1119-1 · Cited by 883 · see citing works
Unsupervised Cross-Lingual Representation Learning for Speech Recognition
Conneau, A., Baevski, A., Collobert, R., Mohamed, A., & Auli, M. (2021) · Interspeech 2021 · doi:10.21437/interspeech.2021-329 · Cited by 637 · see citing works
Frenchay Dysarthria Assessment
Enderby, P. (1980) · International Journal of Language & Communication Disorders · doi:10.3109/13682828009112541 · Cited by 601 · see citing works
Dysarthric speech database for universal access research
Kim, H., Hasegawa-Johnson, M., Perlman, A., Gunderson, J., Huang, T. S., Watkin, K., & Frame, S. (2008) · Interspeech 2008 · doi:10.21437/interspeech.2008-480 · Cited by 397 · see citing works
The Nemours database of dysarthric speech
Menéndez-Pidal, X., Polikoff, J. B., Peters, S. M., et al. (2002) · International Conference on Spoken Language Processing (ICSLP) · doi:10.1109/icslp.1996.608020 · Cited by 167 · see citing works
Herff, C., Diener, L., Angrick, M., Mugler, E., Tate, M. C., Goldrick, M. A., Krusienski, D. J., Slutzky, M. W., & Schultz, T. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.01267 · Cited by 118 · see citing works
Decoding spoken English from intracortical electrode arrays in dorsal precentral gyrus
Wilson, G. H., Stavisky, S. D., Willett, F. R., Avansino, D. T., Kelemen, J. N., Hochberg, L. R., Henderson, J. M., Druckmann, S., & Shenoy, K. V. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/abbfef · Cited by 103 · see citing works
Self-citationPhonemizer: Text to Phones Transcription for Multiple Languages in Python
Bernard, M., & Titeux, H. (2021) · The Journal of Open Source Software · doi:10.21105/joss.03958 · Cited by 88 · see citing works
Simple and Effective Zero-shot Cross-lingual Phoneme Recognition
Xu, Q., Baevski, A., & Auli, M. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-60 · Cited by 80 · see citing works
Kao, J. C., Nuyujukian, P., Ryu, S. I., & Shenoy, K. V. (2016) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2016.2582691 · Cited by 78 · see citing works
Simple and Effective Zero-shot Cross-lingual Phoneme RecognitionPreprint
Xu, Q., Baevski, A., & Auli, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.11680 · Cited by 10 · see citing works
(2014) · doi:10.4230/oasics.slate.2021.16 · Cited by 1 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [5]Wairagkar Maitreyee, Hochberg Leigh R, Brandman David M and Stavisky Sergey D. “ Decoding Intracortical Neural Activity from Dorsal Motor Cortex.” In 2023 11th International IEEE/EMBS Conference on Neural Engineering (NER), (in press).
Cited by
Sorted by citations
Silva, A. B., Littlejohn, K. T., Liu, J. R., Moses, D. A., & Chang, E. F. (2024) · Nature Reviews Neuroscience · doi:10.1038/s41583-024-00819-9 · Cited by 103 · see citing works
- Authors
- Varshney, S., Farias, D., Brandman, D. M., Stavisky, S. D., & Miller, L. M.
- Affiliations
- University of California, Davis (US)
- Venue
- 2023 11th International IEEE/EMBS Conference on Neural Engineering (NER)
- Publisher
- IEEE
- Published
- 2023-04-24
- Volume
- 2023
- Pages
- 1-6
- ISSN
- 1948-3554 (ISSN-L)
- PMC
- PMC11424044
- OpenAlex
- W4377089543
- Cited by
- 6
- References
- 16
- Citations by year
- 2025: 3 · 2024: 1 · 2023: 2
- FWCI
- 1.06
- Type
- conference paper
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Licence
- Locations
- International IEEE/EMBS Conference on Neural Engineering (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion, public-domain)
- Funders
- National Institutes of Health (TR001860, UL1TR001860)
- University of California, Davis (NIH UL1-TR001860)
- National Center for Advancing Translational Sciences
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Health Sciences › Medicine › Physiology › Voice and Speech Disorders
- Keywords
- brain-computer interface, speech intelligibility, speech recognition, anarthria, dysarthria, speech recovery
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
