Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.

An accurate and rapidly calibrating speech neuroprosthesis

Open access

Preprint — published as 10.1056/nejmoa2314132 · Paper page

Abstract

Brain-computer interfaces can enable rapid, intuitive communication for people with paralysis by transforming the cortical activity associated with attempted speech into text on a computer screen. Despite recent advances, communication with brain-computer interfaces has been restricted by extensive training data requirements and inaccurate word output. A man in his 40’s with ALS with tetraparesis and severe dysarthria (ALSFRS-R = 23) was enrolled into the BrainGate2 clinical trial. He underwent surgical implantation of four microelectrode arrays into his left precentral gyrus, which recorded neural activity from 256 intracortical electrodes. We report a speech neuroprosthesis that decoded his neural activity as he attempted to speak in both prompted and unstructured conversational settings. Decoded words were displayed on a screen, then vocalized using text-to-speech software designed to sound like his pre-ALS voice. On the first day of system use, following 30 minutes of attempted speech training data, the neuroprosthesis achieved 99.6% accuracy with a 50-word vocabulary. On the second day, the size of the possible output vocabulary increased to 125,000 words, and, after 1.4 additional hours of training data, the neuroprosthesis achieved 90.2% accuracy. With further training data, the neuroprosthesis sustained 97.5% accuracy beyond eight months after surgical implantation. The participant has used the neuroprosthesis to communicate in self-paced conversations for over 248 hours. In an individual with ALS and severe dysarthria, an intracortical speech neuroprosthesis reached a level of performance suitable to restore naturalistic communication after a brief training period.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2023.12.26.23300110

Cites

64 of 64 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1. Electrode locations and speech decoding setup. a, Approximate microelectrode array locations, represented by black squares, superimposed on a 3d reconstruction of the participant’s brain. Colored regions correspond to the Human Connectome Project’s multi-modal atlas of cortical areas22 aligned to the participant’s brain using the Human Connectome Project’s MRI protocol scans before implantation, concordant with the precentral gyrus on a MNI template brain (Figure S11). b, Diagram of the brain-to-text speech neuroprosthesis. Cortical neural activity is measured from the left ventral precentral gyrus using four 64-electrode Utah arrays. Machine learning techniques decode the cortical neural activity into an English phoneme every 80 ms. Using a series of language models (LM), the predicted phoneme sequence is translated into a series of words that appear on a screen as the participant tries to speak. At the end of a sentence, an own-voice text-to-speech algorithm vocalizes the decoded sentence designed to emulate the participant’s voice prior to developing ALS (Section S5).
Figure 1. Electrode locations and speech decoding setup. a, Approximate microelectrode array locations, represented by black squares, superimposed on a 3d reconstruction of the participant’s brain. Colored regions correspond to the Human Connectome Project’s multi-modal atlas of cortical areas22 aligned to the participant’s brain using the Human Connectome Project’s MRI protocol scans before implantation, concordant with the precentral gyrus on a MNI template brain (Figure S11). b, Diagram of the brain-to-text speech neuroprosthesis. Cortical neural activity is measured from the left ventral precentral gyrus using four 64-electrode Utah arrays. Machine learning techniques decode the cortical neural activity into an English phoneme every 80 ms. Using a series of language models (LM), the predicted phoneme sequence is translated into a series of words that appear on a screen as the participant tries to speak. At the end of a sentence, an own-voice text-to-speech algorithm vocalizes the decoded sentence designed to emulate the participant’s voice prior to developing ALS (Section S5).CC BY · PubMed Central
Figure 2. Online speech decoding performance. Phoneme error rates (top) and word error rates (bottom) are shown for each session for two vocabulary sizes (50 versus 125,000 words). Reference error rates are plotted (horizontal dashed lines) for two previous speech neuroprosthesis studies19,21. The horizontal axis displays the research session number, the number of days since arrays implant, and the cumulative hours of neural data used to train the speech decoder for that session. Aggregate error rates across all evaluation sentences are shown for each session (mean ± 95% confidence interval). Vertical dashed lines represent when decoder improvements were introduced. Fig. S20 shows phoneme and word error rates for individual blocks.
Figure 2. Online speech decoding performance. Phoneme error rates (top) and word error rates (bottom) are shown for each session for two vocabulary sizes (50 versus 125,000 words). Reference error rates are plotted (horizontal dashed lines) for two previous speech neuroprosthesis studies19,21. The horizontal axis displays the research session number, the number of days since arrays implant, and the cumulative hours of neural data used to train the speech decoder for that session. Aggregate error rates across all evaluation sentences are shown for each session (mean ± 95% confidence interval). Vertical dashed lines represent when decoder improvements were introduced. Fig. S20 shows phoneme and word error rates for individual blocks.CC BY · PubMed Central
Figure 3. Extensive use of the neuroprosthesis for accurate self-initiated speech. a, Photograph of the participant and speech neuroprosthesis in Conversation Mode. The neuroprosthesis detected when he was trying to speak solely based on neural activity, and concluded either after 6 seconds of speech inactivity, or upon his optional activation of an on-screen button via eye tracking. After the decoded sentence was finalized, the participant used the on-screen confirmation buttons to indicate if the decoded sentence was correct. b, Sample transcript of our participant using the speech neuroprosthesis to speak to his daughter on the second day of use (Video 3). Additional transcripts are available in Table S4. c, Cumulative hours that the participant used the speech neuroprosthesis to communicate with those around him in structured research sessions and during personal use. For sessions represented by points outlined in red, decoding accuracy is quantified in (d). The distribution of self-reported decoding accuracy for each sentence across all Conversation Mode data (n = 21,829) is shown in the inset pie chart. Sentences where the participant did not self-report decoding accuracy within 30 seconds of sentence completion are excluded (n = 868). d, Evaluating speech decoding accuracy in conversations (n = 925 sentences with known true labels, sourced from red-labeled sessions in (c)). The average word error rate was 3.7% (95% CI, 3.3% to 4.3%).
Figure 3. Extensive use of the neuroprosthesis for accurate self-initiated speech. a, Photograph of the participant and speech neuroprosthesis in Conversation Mode. The neuroprosthesis detected when he was trying to speak solely based on neural activity, and concluded either after 6 seconds of speech inactivity, or upon his optional activation of an on-screen button via eye tracking. After the decoded sentence was finalized, the participant used the on-screen confirmation buttons to indicate if the decoded sentence was correct. b, Sample transcript of our participant using the speech neuroprosthesis to speak to his daughter on the second day of use (Video 3). Additional transcripts are available in Table S4. c, Cumulative hours that the participant used the speech neuroprosthesis to communicate with those around him in structured research sessions and during personal use. For sessions represented by points outlined in red, decoding accuracy is quantified in (d). The distribution of self-reported decoding accuracy for each sentence across all Conversation Mode data (n = 21,829) is shown in the inset pie chart. Sentences where the participant did not self-report decoding accuracy within 30 seconds of sentence completion are excluded (n = 868). d, Evaluating speech decoding accuracy in conversations (n = 925 sentences with known true labels, sourced from red-labeled sessions in (c)). The average word error rate was 3.7% (95% CI, 3.3% to 4.3%).CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

6 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 39 references

Research works printed but not linked: 11

  1. 2023, Interim Safety Profile From the Feasibility Study of the BrainGate Neural Interface System, Neurology, 100, e1177
  2. 2002, Timing of speech deterioration in people with amyotrophic lateral sclerosis, J Med Speech-Lang Pathol, 10, 231
  3. 10.1038/nature18933 · DOI · Paper page
  4. 10.1038/s41586-023-06377-x · DOI · Paper page
  5. 10.1056/nejmoa2027540 · DOI · Paper page
  6. 10.1038/s41467-022-33611-3 · DOI · Paper page
  7. 10.1523/JNEUROSCI.1614-22.2022 · DOI · Paper page
  8. Ali YH , Bodkin K , Rigotti-Thompson M , et al. BRAND: A platform for closed-loop experiments with deep network models [Internet]. 2023 [cited 2023 Dec 11];2023.08.08.552473. Available from: https://www.biorxiv.org/content/10.1101/2023.08.08.552473v1 · doi:10.1101/2023.08.08.552473v1
  9. Godfrey JJ , Holliman EC , McDaniel J. SWITCHBOARD: telephone speech corpus for research and development [Internet]. In: [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing. 1992 [cited 2023 Dec 11]. p. 517–20 vol.1.Available from: https://ieeexplore.ieee.org/document/225858 · DOI · Paper page
  10. Gao L , Biderman S , Black S , et al. The Pile: An 800GB Dataset of Diverse Text for Language Modeling [Internet]. 2020 [cited 2023 Dec 12];Available from: http://arxiv.org/abs/2101.00027 · arXiv
  11. 10.1038/ncomms13749 · DOI · Paper page

Cited by

See all 244

Metadata

Authors
Card, N. S., Wairagkar, M., Iacobacci, C., Hou, X., Singer-Clark, T., Willett, F. R., Kunz, E. M., Fan, C., Vahdati Nia, M., Deo, D. R., Srinivasan, A., Choi, E. Y., Glasser, M. F., Hochberg, L. R., Henderson, J. M., Shahlaie, K., Brandman, D. M., & Stavisky, S. D.
Affiliations
Howard Hughes Medical Institute (US)
Venue
New England Journal of Medicine
Publisher
openRxiv
Published
2023-12-26
Volume
391
Issue
7
Pages
609-618
ISSN
0028-4793 (ISSN-L)
OpenAlex
W4401560881
Cited by
244
References
39
Citations by year
2026: 105 · 2025: 109 · 2024: 29 · 2023: 1
FWCI
59.04
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • New England Journal of Medicine (publishedVersion)
  • medRxiv (acceptedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
  • eScholarship (California Digital Library) (submittedVersion)
  • PubMed Central (submittedVersion, cc-by)
  • Digital Commons@Becker (Washington University School of Medicine) (submittedVersion)
Funders
  • U.S. Department of Defense
  • Howard Hughes Medical Institute
  • U.S. Department of Veterans Affairs (I01 RX002295, A2295R)
  • Burroughs Wellcome Fund
  • Simons Foundation (872146SPI)
  • A.P. Giannini Foundation
  • Massachusetts General Hospital
  • Giannini Foundation of Agricultural Economics
  • Searle Scholars Program
  • National Institutes of Health (5u01dc017844-06)
  • Rehabilitation Research and Development Service
  • University of California, Davis
  • Wu Tsai Neurosciences Institute, Stanford University
  • National Institute of Mental Health (R01MH060974)
  • NIH Office of the Director
  • National Institute on Deafness and Other Communication Disorders (U01DC17844, U01DC019430, DP2DC021055, U01- DC017844)
  • Congressionally Directed Medical Research Programs (AL220043)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
  • Life Sciences › Neuroscience › Cellular and Molecular Neuroscience › Neuroscience and Neural Engineering
  • Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
Keywords
brain-computer interface, amyotrophic lateral sclerosis, tetraparesis, intracortical microelectrode arrays, text-to-speech, conversational communication

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: An accurate and rapidly calibrating speech neuroprosthesis. https://app.cochat.ai/papers/10.1101/2023.12.26.23300110
The paper
Card, N. S., Wairagkar, M., Iacobacci, C., Hou, X., Singer-Clark, T., Willett, F. R., Kunz, E. M., Fan, C., Vahdati Nia, M., Deo, D. R., Srinivasan, A., Choi, E. Y., Glasser, M. F., Hochberg, L. R., Henderson, J. M., Shahlaie, K., Brandman, D. M., & Stavisky, S. D. (2023). An accurate and rapidly calibrating speech neuroprosthesis. New England Journal of Medicine. https://doi.org/10.1101/2023.12.26.23300110

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.