A high-performance neuroprosthesis for speech decoding and avatar control

In healthy speakers, neural representations in the sensorimotor cortex encode articulatory movements of the orofacial musculature.

“In healthy speakers, neural representations in the SMC (comprising the precentral and postcentral gyri) encode articulatory movements of the orofacial musculature22,24,30.”

• cites [22] sourcePartly supported

  The Discussion reports that sensorimotor cortical activity encodes movements of the vocal-tract articulators, and the STAR Methods locate the analyzed electrodes on the precentral and postcentral gyri. However, the Participants section says the five speakers were undergoing clinical treatment for epilepsy. The study therefore does not establish the claim specifically for healthy speakers. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites [24] sourceSupported
  Experimental — accuracy not yet measured
• cites [30] sourceSupported
  Experimental — accuracy not yet measured

Initial attempts to decode text from brain activity in a person with anarthria during attempted speech had limitations such as slow decoding rates and small vocabulary sizes.

“Initial efforts to decode text from the brain activity ofa person with anarthria during attempted speech had various limitations, including slow decoding rates and small vocabulary sizes1,3.”

• cites [1] sourceSupported
• cites [3] sourcePartly supported

  The paper’s introductory passage (labeled Abstract in our copy; page unavailable) describes an earlier attempt to decode full words from cortical activity during attempted speech in a person with anarthria and identifies its preliminary 50-word vocabulary as a limitation. It calls that approach’s decoding speed “promising,” however, and does not characterize its decoding rate as slow. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Being able to use evolving technology to communicate with family and friends, participate in community and work, and engage in virtual social contexts can greatly expand access to meaningful interpersonal interactions and improve quality of life.

“The ability to interface with evolving technology to communicate with family and friends, facilitate community involvement and occupational participation, and engage in virtual, Internet-based social contexts (such as social media and metaverses) can vastly expand a person’s access to meaningful interpersonal interactions and ultimately improve their quality of life2,39.”

• cites [2] sourcePartly supported

  The Results report that potential BCI users wanted face-to-face communication, email, and social networking; the Discussion links assistive technology to social interaction and employment and describes improved communication and quality of life as clinical goals. The paper concerns people with disabilities, however, and does not address metaverses or establish that these technologies vastly expand interpersonal interactions or improve quality of life. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [39] sourceNot checked — no full text of the source

During 14 days of data collection shortly after implantation, text decoding performance exceeded previous BCIs' communication speeds by a factor of 4 or more.

“During 14 days of data collection shortly after device implantation, we achieved high-performance text decoding, exceeding communication speeds of previous BCIs by a factor of 4 or more1,3,9”

• cites [1] sourcePartly supported

  The Results (Sentence Decoding; page unavailable) report real-time decoding at a median 15.2 words per minute, supporting the text-decoding portion of the claim. The Methods (Word and Sentence Task Design; page unavailable) describe 50 sessions over 81 weeks, not 14 days shortly after implantation. Our copy does not report a fourfold speed advantage over previous BCIs. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites [3] sourcePartly supported

  The Decoding performance section reports real-time text decoding, but the passages do not support 14 days of data collection shortly after implantation or a speed advantage of at least fourfold over previous BCIs. The Discussion instead notes that a previous spelling BCI achieved 90 characters per minute. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [9] sourcePartly supported

  The Discussion reports high-performance text decoding at 90 characters per minute, exceeding previously reported BCI speeds. But it also reports prior speeds of 40 characters per minute for intracortical point-and-click typing and 60 for a visually evoked EEG speller, so it does not support a fourfold advantage over previous BCIs generally. The Daily decoder retraining section says the copy-typing data spanned 28 days, while the Discussion refers to functionality more than 1,000 days after implantation, not data collection shortly after implantation. (LLM, conf=0.89)

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: The participant's silent-speech attempts engage the same articulatory representations as natural speech production.

The vocabulary size was expanded by a factor of 20 relative to the previous direct-speech BCI.

“expanding the vocabulary size of our previous direct-speech BCI by a factor of 20 (ref. 1).”

• cites [1] sourcePartly supported

  The Methods (Word and Sentence Task Design; page unavailable) identify the cited direct-speech BCI as using a 50-word set. The paper does not report a subsequent vocabulary expansion or establish the claimed factor of 20. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

• AssumptionRests on an assumption: The participant's silent-speech attempts engage the same articulatory representations as natural speech production.

These results align with accumulating evidence for motor-movement encoding in the postcentral gyrus.

“These results are in line with growing evidence for motor-movement encoding in the postcentral gyrus33–35”

• cites [33] sourceSupported
• cites [34] sourceNot checked — no full text of the source
• cites [35] sourceNot checked — no full text of the source

People who have lost speech after severe paralysis rank faster, more accurate, and more natural communication among their most desired needs.

“Faster, more accurate, and more natural communication are among the most desired needs of people who have lost the ability to speak after severe paralysis2,38–40.”

• cites [2] sourcePartly supported

  The Results (page unavailable) identify slow communication as the most common complaint among interviewees using AAC; participants also sought conversation at a natural pace and raised accuracy concerns. The Methods (page unavailable) describe people with severe speech and/or physical impairments, not specifically people who lost speech after severe paralysis. The paper does not rank all three qualities among that population’s most desired needs. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites [38] sourcePartly supported

  The Discussion (page not provided) says that restrictive yes-no communication was associated with lower quality of life and that electronic devices enable real dialogue for people with locked-in syndrome. It does not report patients ranking faster, more accurate, or more natural communication among their most desired needs. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [39] sourceNot checked — no full text of the source
• cites [40] sourceNot checked — no full text of the source

Peak-activation times did not differ significantly between precentral and postcentral gyrus electrodes during silent speech attempts (P > 0.01, two-sided Mann-Whitney U-test).

“an analysis of peak-activation times that revealed no significant difference between electrodes in the precentral versus postcentral gyrus during silent attempts to speak (Supplementary Fig. 15; P > 0.01 two-sided Mann–Whitney U-test)33–35.”

• cites [33] sourceCould not verify

  The Results section on simultaneous recording compares M1 and S1 activity during a reaching-and-grasping task in monkeys, not silent speech attempts. Searches for precentral versus postcentral peak-activation times and the stated Mann–Whitney result found no passage addressing that comparison in our copy. (LLM, conf=0.77)

  Experimental — accuracy not yet measured
• cites [34] sourceNot checked — no full text of the source
• cites [35] sourceNot checked — no full text of the source

Using high-density surface recordings of speech cortex in a participant with severe limb and vocal paralysis, the study achieved high-performance real-time decoding across text, speech audio and facial-avatar animation.

“Here we use high-density surface recordings of the speech cortex in a clinical-trial participant with severe limb and vocal paralysis to achieve high-performance real-time decoding across three complementary speech-related output modalities: text, speech audio and facial-avatar animation.”

• AssumptionRests on an assumption: Spoken communication carries prosodic, expressive and identity-related information that can improve embodied communication beyond what text alone can convey.

The decoders reached high performance with less than two weeks of training data.

“The decoders reached high performance with less than two weeks of training.”

• Also stated in: § Text decoding · plain

  “We observed steadily declining error rates over the course of 13 days of training-data collection (Fig. 2f), during which we collected 9,506 sentence trials corresponding to about 1.6 h of training data per day.”

For the 1024-word-General text-decoding task, the median real-time decoding rate was about 78 words per minute.

“For text, we demonstrate accurate and rapid large-vocabulary decoding with a median rate of 78 words per minute and median word error rate of 25%.”

• Also stated in: § Text decoding · plain

  “We observed a median real-time decoding rate of 78.3 WPM (99% CI [75.5, 79.4]; Fig. 2f).”

For the 1024-word-General text-decoding task, the median word error rate was about 25%.

“For text, we demonstrate accurate and rapid large-vocabulary decoding with a median rate of 78 words per minute and median word error rate of 25%.”

• Also stated in: § Text decoding · plain

  “We achieved a median PER of 18.5% (99% confidence interval (CI) [14.1, 28.5]; Fig. 2b), a median WER of 25.5% (99% CI [19.3, 34.5]; Fig. 2c) and a median CER of 19.9% (99% CI [15.0, 30.1]; Fig. 2d; see Table 1 for example decodes; see Extended Data Fig. 1 for the relationship between decoded PER and WER).”

The study demonstrated control of virtual orofacial movements for speech and non-speech communicative gestures.

“For facial-avatar animation, we demonstrate the control of virtual orofacial movements for speech and non-speech communicative gestures.”

Performance degraded in an offline simulation of low-density electrode sampling.

“We also observed degraded performance during an offline simulation of low-density sampling (Supplementary Figs. 10 and 11 and Supplementary Table 8)”

The direct avatar-animation approach produced articulatory gestures that were strongly correlated with reference articulatory gestures across all datasets.

“We found that the direct approach produced articulatory gestures that were strongly correlated with reference articulatory gestures across all datasets (Supplementary Figs. 1 and 2 and Supplementary Table 4)”

There was a large degree of overlap between the distributions of avatar-healthy speaker correlations and healthy speaker-healthy speaker correlations.

“there was a large degree of overlap between the two distributions”

For real-time text decoding on the 1024-word-General set, the median phone error rate was 18.5% (99% CI 14.1 to 28.5).

“We achieved a median PER of 18.5% (99% confidence interval (CI) [14.1, 28.5]; Fig. 2b)”

Classification accuracy remained high after a 61-day hiatus in recording.

“Accuracy remained high after a 61-day hiatus in recording for the participant to travel.”

Avatar and healthy-speaker jaw-opening trajectories were correlated with a median value of 0.733 (99% CI 0.711 to 0.748).

“We found that the jaw opening, lip aperture and mouth width of the avatar and healthy speakers were well correlated with median values of 0.733 (99% CI [0.711, 0.748]), 0.690 (99% CI [0.663, 0.714]) and 0.446 (99% CI [0.417, 0.470]), respectively (Fig. 4c).”

• AssumptionRests on an assumption: The articulatory trajectories recorded from healthy speakers represent the expected natural articulatory trajectories.

Median classification accuracy for six articulatory movements was 87.8% (99% CI 85.1% to 90.5%) across 10 cross-validation folds.

“We observed a median classification accuracy of 87.8% (99% CI [85.1, 90.5]; across n = 10 cross-validation folds; Fig. 4d) when classifying between the six articulatory movements.”

For the 529-phrase-AAC set, mean mel-cepstral distortion was 4.49 dB (99% CI 4.07 to 4.67).

“We achieved mean MCDs of 3.45 (99% CI [3.25, 3.82]), 4.49 (99% CI [4.07, 4.67]) and 5.21 (99% CI [4.74, 5.51]) dB for the 50-phrase-AAC, 529-phrase-AAC and 1024-word-General sets, respectively (Fig. 3c).”

Many electrodes that were important for decoding were located next to one another.

“We noted that many important electrodes were adjacent”

Labial phonetic tuning was positively correlated with HGA magnitude during attempts to pucker the lips (P < 0.0001, r = 0.89, ordinary least-squares regression).

“We also observed a positive correlation between labial phonetic tuning and HGA magnitude during attempts to pucker the lips (P < 0.0001, r = 0.89, ordinary least-squares regression; Fig. 5h).”

• AssumptionRests on an assumption: The participant's silent-speech attempts engage the same articulatory representations as natural speech production.

For real-time text decoding on the 1024-word-General set, the median character error rate was 19.9% (99% CI 15.0 to 30.1).

“a median CER of 19.9% (99% CI [15.0, 30.1]; Fig. 2d”

With the 50-phrase-AAC set, the median decoding speed was 101 words per minute (99% CI 95.6 to 103).

“with median decoding speeds of 101 WPM (99% CI [95.6, 103])”

For the 50-phrase-AAC set, median forced-choice perceptual accuracy of the avatar was 85.7% (99% CI 79.0% to 92.0%).

“We obtained median accuracies of 85.7% (99% CI [79.0, 92.0]), 87.7% (99% CI [79.7, 93.7]) and 74.3% (99% CI [66.7, 80.8]) across the 50-phrase-AAC, 529-phrase-AAC and 1024-word-General sets”

For the 1024-word-General set, median forced-choice perceptual accuracy of the avatar was 74.3% (99% CI 66.7% to 80.8%).

“We obtained median accuracies of 85.7% (99% CI [79.0, 92.0]), 87.7% (99% CI [79.7, 93.7]) and 74.3% (99% CI [66.7, 80.8]) across the 50-phrase-AAC, 529-phrase-AAC and 1024-word-General sets”

Using only the RNN neural-decoding model without a language model or constrained vocabulary, the median phone error rate was 29.4% (99% CI 26.2 to 32.8).

“This yielded a median PER of 29.4% (99% CI [26.2, 32.8]; Fig. 2b)”

Kernel density estimates showed a somatotopic map in which hand-movement attempts, labial phones, and front-tongue phones were arranged along a dorsal-ventral axis.

“Kernel density estimates revealed a somatotopic map with encoding of attempted hand movements, labial phones and front-tongue phones organized along a dorsal–ventral axis.”

• AssumptionRests on an assumption: The participant's attempted hand movements, despite severe paralysis, produce neural activity that reflects intended hand movements.

For the 1024-word-General set, the median perceptual word error rate of synthesized speech was 54.4% (99% CI 50.5% to 65.2%).

“We achieved median WERs of 8.20% (99% CI [3.28, 14.5]), 28.2% (99% CI [18.6, 38.5]) and 54.4% (99% CI [50.5 65.2])”

Text-decoding performance remained robust as vocabulary size increased from 1,506 to 39,378 words.

“We observed robust decoding performance as vocabulary size grew (Fig. 2g; see Extended Data Fig. 2 for CER and PER).”

With the 529-phrase-AAC set, the median word error rate was 17.1% (99% CI 8.89% to 28.9%).

“With the 529-phrase-AAC set, we observed a median WER of 17.1% across sentences (99% CI [8.89%, 28.9%])”

For the 529-phrase-AAC set, the median perceptual word error rate of synthesized speech was 28.2% (99% CI 18.6% to 38.5%).

“We achieved median WERs of 8.20% (99% CI [3.28, 14.5]), 28.2% (99% CI [18.6, 38.5]) and 54.4% (99% CI [50.5 65.2])”

Brain areas with the strongest POA encoding, especially the SMC, were the most critical to decoding performance in leave-one-area-out offline analyses.

“the brain areas that most strongly encoded POA, notably the SMC, were the most critical to decoding performance in leave-one-area-out offline analyses (Extended Data Fig. 8, Supplementary Fig. 14 and Supplementary Table 8).”

For the 50-phrase-AAC set, the median perceptual word error rate of synthesized speech was 8.20% (99% CI 3.28% to 14.5%).

“We achieved median WERs of 8.20% (99% CI [3.28, 14.5]), 28.2% (99% CI [18.6, 38.5]) and 54.4% (99% CI [50.5 65.2])”

For the 50-phrase-AAC set, the median perceptual character error rate of synthesized speech was 6.64% (99% CI 2.71% to 10.6%).

“and median CERs of 6.64% (99% CI [2.71, 10.6]), 26.3% (99% CI [15.9, 29.7]) and 45.7% (99% CI [39.2, 51.6]) across test trials for the 50-phrase-AAC, 529-phrase-AAC and 1024-word-General sets, respectively”

Median classification accuracy for nine emotional expressions was 74.0% (99% CI 70.8% to 77.1%) across 15 cross-validation folds.

“We observed a median classification accuracy of 74.0% (99% CI [70.8, 77.1]; across n = 15 cross-validation folds; Fig. 4e) when classifying between the nine possible expressions”

Correlations among pairs of healthy speakers were higher than correlations between the avatar and healthy speakers (P < 0.0001).

“Although correlations among pairs of healthy speakers were higher than between the avatar and healthy speakers (all P < 0.0001, two-sided Mann–Whitney U-test with nine-way Holm–Bonferroni correction; Supplementary Table 3)”

Hierarchical clustering of phones revealed grouping by place of articulation (P < 0.0001 compared with chance).

“Hierarchical clustering of phones revealed grouping by POA (P < 0.0001 compared to chance, one-tailed permutation test; Fig. 5b).”

For the 1024-word-General set, the median perceptual character error rate of synthesized speech was 45.7% (99% CI 39.2% to 51.6%).

“and median CERs of 6.64% (99% CI [2.71, 10.6]), 26.3% (99% CI [15.9, 29.7]) and 45.7% (99% CI [39.2, 51.6]) across test trials for the 50-phrase-AAC, 529-phrase-AAC and 1024-word-General sets, respectively”

For the 1024-word-General set, mean mel-cepstral distortion was 5.21 dB (99% CI 4.74 to 5.51).

“We achieved mean MCDs of 3.45 (99% CI [3.25, 3.82]), 4.49 (99% CI [4.07, 4.67]) and 5.21 (99% CI [4.74, 5.51]) dB for the 50-phrase-AAC, 529-phrase-AAC and 1024-word-General sets, respectively (Fig. 3c).”

With the 50-phrase-AAC set, the median word error rate was 4.92% (99% CI 3.18% to 14.04%).

“With the 50-phrase-AAC set, we observed a median WER of 4.92% (99% CI [3.18, 14.04])”

With a 39,378-word vocabulary, the median offline word error rate was 27.6% (99% CI 20.0 to 34.7).

“With a vocabulary of 39,378 words, we achieved a median offline WER of 27.6% (99% CI [20.0 34.7]).”

Median classification accuracy for strong-intensity versions of the three expression types was 96.9% (99% CI 93.8% to 100%).

“and a median classification accuracy of 96.9% (99% CI [93.8,100]) when considering the classifier’s outputs for only the strong-intensity versions of the three expression types (Supplementary Fig. 6).”

With the 529-phrase-AAC set, the median decoding rate was 89.9 words per minute (99% CI 83.6 to 93.3).

“with a median decoding rate of 89.9 WPM (99% CI [83.6, 93.3])”

Spectrograms decoded in real time shared fine-grained and broad timescale information with corresponding reference spectrograms.

“We qualitatively observed that spectrograms decoded in real time shared both fine-grained and broad timescale information with corresponding reference spectrograms (Fig. 3b).”

In the NATO-motor task, electrodes encoding attempted finger flexions were largely orthogonal to electrodes encoding NATO code words.

“During the NATO-motor task, electrodes encoding attempted finger flexions were largely orthogonal to those encoding NATO code words”

• AssumptionRests on an assumption: The participant's attempted hand movements, despite severe paralysis, produce neural activity that reflects intended hand movements.

Perceptual word and character error rates improved as the number of unique words and sentences in the set decreased.

“As for the MCD results, WERs and CERs improved as the number of unique words and sentences in the sentence set decreased (all P < 0.0001, two-sided Wilcoxon rank-sum tests with 19-way Holm–Bonferroni correction; chance measured by shuffling the mapping between the transcriptions and the ground-truth sentence texts).”

For all text-decoding metrics, performance was better than chance computed with temporally shuffled neural data (P < 0.0001 for all three comparisons).

“For all metrics, performance was better than chance, which we computed by re-evaluating performance after using temporally shuffled neural data as the input to our decoding pipeline (P < 0.0001 for all three comparisons, two-sided Wilcoxon rank-sum tests with five-way Holm–Bonferroni correction).”

Avatar and healthy-speaker mouth-width trajectories were correlated with a median value of 0.446 (99% CI 0.417 to 0.470).

“We found that the jaw opening, lip aperture and mouth width of the avatar and healthy speakers were well correlated with median values of 0.733 (99% CI [0.711, 0.748]), 0.690 (99% CI [0.663, 0.714]) and 0.446 (99% CI [0.417, 0.470]), respectively (Fig. 4c).”

• AssumptionRests on an assumption: The articulatory trajectories recorded from healthy speakers represent the expected natural articulatory trajectories.

The model correctly classified 569 of 570 test trials as either finger flexion or silent speech.

“the model correctly classified 569 out of 570 test trials as either finger flexion or silent speech; Supplementary Fig. 9”

• AssumptionRests on an assumption: The participant's attempted hand movements, despite severe paralysis, produce neural activity that reflects intended hand movements.

The participant has used commercial head-tracking assistive technology to communicate at up to 14 words per minute.

“she has used commercial head-tracking assistive technology to communicate slowly to select letters at up to 14 words per minute (WPM; Supplementary Note 2).”

• cites Supplementary Note 2Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The system remained functional in a freeform setting where the participant spontaneously attempted to silently say unprompted sentences, with neural data aligned to speech onsets detected from neural features rather than go cues.

“We verified that our system remained functional in a freeform setting in which the participant volitionally and spontaneously attempted to silently say unprompted sentences, with the neural data aligned to speech onsets detected directly from the neural features instead of to go cues (Method 2 in Supplementary Methods and Supplementary Video 3).”

For the 529-phrase-AAC set, the median perceptual character error rate of synthesized speech was 26.3% (99% CI 15.9% to 29.7%).

“and median CERs of 6.64% (99% CI [2.71, 10.6]), 26.3% (99% CI [15.9, 29.7]) and 45.7% (99% CI [39.2, 51.6]) across test trials for the 50-phrase-AAC, 529-phrase-AAC and 1024-word-General sets, respectively”

Avatar and healthy-speaker lip-aperture trajectories were correlated with a median value of 0.690 (99% CI 0.663 to 0.714).

“We found that the jaw opening, lip aperture and mouth width of the avatar and healthy speakers were well correlated with median values of 0.733 (99% CI [0.711, 0.748]), 0.690 (99% CI [0.663, 0.714]) and 0.446 (99% CI [0.417, 0.470]), respectively (Fig. 4c).”

• AssumptionRests on an assumption: The articulatory trajectories recorded from healthy speakers represent the expected natural articulatory trajectories.

For the 50-phrase-AAC set, mean mel-cepstral distortion was 3.45 dB (99% CI 3.25 to 3.82).

“We achieved mean MCDs of 3.45 (99% CI [3.25, 3.82]), 4.49 (99% CI [4.07, 4.67]) and 5.21 (99% CI [4.74, 5.51]) dB for the 50-phrase-AAC, 529-phrase-AAC and 1024-word-General sets, respectively (Fig. 3c).”

Front-tongue phonetic encoding was positively correlated with HGA magnitude during attempts to raise the tongue (P < 0.0001, r = 0.84, ordinary least-squares regression).

“we found a positive correlation between front-tongue phonetic encoding and HGA magnitude during attempts to raise the tongue (P < 0.0001, r = 0.84, ordinary least-squares regression; Fig. 5g).”

• AssumptionRests on an assumption: The participant's silent-speech attempts engage the same articulatory representations as natural speech production.

For the 529-phrase-AAC set, median forced-choice perceptual accuracy of the avatar was 87.7% (99% CI 79.7% to 93.7%).

“We obtained median accuracies of 85.7% (99% CI [79.0, 92.0]), 87.7% (99% CI [79.7, 93.7]) and 74.3% (99% CI [66.7, 80.8]) across the 50-phrase-AAC, 529-phrase-AAC and 1024-word-General sets”

A new BCI control modality was introduced as a digital talking face: a personalized avatar capable of dynamic, realistic, and interpretable speech and non-verbal facial gestures.

“Finally, we introduced a modality of BCI control in the form of a digital ‘talking face’—a personalized avatar capable of dynamic, realistic and interpretable speech and non-verbal facial gestures.”

• AssumptionRests on an assumption: The participant's silent-speech attempts engage the same articulatory representations as natural speech production.

Discrete speech units derived from self-supervised learning served as effective intermediate representations for intelligible speech synthesis.

“self-supervised learning-derived discrete speech units, serving as effective intermediate representations for intelligible speech synthesis”

Advances in the neural interface provided denser and broader sampling of distributed orofacial and vocal-tract representations across the lateral SMC.

“advances in the neural interface, providing denser and broader sampling of the distributed orofacial and vocal-tract representations across the lateral SMC”

Custom sequence-learning neural-decoding models enabled training without aligning neural activity to output features.

“custom sequence-learning neural-decoding models, facilitating training without alignment of neural activity and output features”

A virtual face was controlled from brain activity to accompany synthesized speech and convey facial expressions.

“control of a virtual face from brain activity to accompany synthesized speech and convey facial expressions”

The participant's SMC contained hand and orofacial-motor somatotopy organized along a dorsal-ventral axis.

“including hand and orofacial-motor somatotopy organized along a dorsal–ventral axis”

• AssumptionRests on an assumption: The participant's attempted hand movements, despite severe paralysis, produce neural activity that reflects intended hand movements.

Temporal-lobe electrodes that contributed to decoding were not strongly activated during auditory perception (r < 0.1, P > 0.01, Pearson correlation permutation test).

“temporal-lobe electrodes contributing to decoding were not strongly activated during auditory perception (r < 0.1, P > 0.01, Pearson correlation permutation test; Supplementary Fig. 16)”

Intelligible speech can be synthesized from the brain activity of a person with paralysis.

“We also showed that intelligible speech can be synthesized from the brain activity of a person with paralysis.”

• AssumptionRests on an assumption: The participant's silent-speech attempts engage the same articulatory representations as natural speech production.

The participant's SMC contained phonetic tunings clustered by place of articulation.

“and phonetic tunings clustered by POA.”

• AssumptionRests on an assumption: The participant's silent-speech attempts engage the same articulatory representations as natural speech production.

Multimodal communication brain-computer interfaces have the potential to restore the ability to express meaningful orofacial gestures.

“illustrating the potential of multimodal communication BCIs to restore the ability to express meaningful orofacial gestures.”

High-density cortical recording provides a benefit for decoding performance.

“further highlighting the benefit of high-density cortical recording.”

It is possible to synthesize intelligible speech from the brain activity of a person with paralysis.

“Together, these results demonstrate that it is possible to synthesize intelligible speech from the brain activity of a person with paralysis.”

The avatar reasonably approximated expected articulatory trajectories relative to natural variance between healthy speakers.

“illustrating that the avatar reasonably approximated the expected articulatory trajectories relative to natural variances between healthy”

• AssumptionRests on an assumption: The articulatory trajectories recorded from healthy speakers represent the expected natural articulatory trajectories.

A major difficulty in learning mappings between ECoG features and phone or speech-sound sequences was the lack of clear timing information for words and phonemes in silently attempted speech.

“A major difficulty in learning statistical mappings between the ECoG features and the sequences of phones and speech-sound features in the sentences was caused by the absence of clear timing information of words and phonemes in the silently attempted speech.”

The primary contributor to phone-decoding performance was the neural-decoding RNN model, not the CTC beam search or language model.

“demonstrating that the primary contributor to phone-decoding performance was the neural-decoding RNN model and not the CTC beam search or language model (P < 0.0001 for all comparisons to chance and to the full model, two-sided Wilcoxon signed-rank tests with five-way Holm–Bonferroni correction; Extended Data Table 1).”

The adjacency of many important electrodes suggests that useful, non-redundant cortical information is sampled despite the electrodes' close spacing.

“suggesting sampling of useful, non-redundant information from the cortex despite the electrodes’ close proximity.”

Speech-decoding performance is driven by broad coverage of articulatory representations distributed throughout the sensorimotor cortex that persisted after years of paralysis.

“Additionally, we show that decoder performance is driven by broad coverage of articulatory representations distributed throughout the SMC that have persisted after years of paralysis.”

The 50-phrase-AAC and 529-phrase-AAC sentence sets closely mirror corpora preloaded on commercial AAC devices that are designed for expressing basic concepts and caregiving needs.

“The first two sets closely mirror corpora preloaded on commercially available augmentative and alternative communication (AAC) devices, designed to let patients express basic concepts and caregiving needs4.”

• cites [4]Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The avatar conveyed perceptually meaningful speech-related facial movements.

“demonstrating that the avatar conveyed perceptually meaningful speech-related facial movements (Fig. 4b).”

A speech neuroprosthesis that decodes articulatory cortical activity in real time into text, speech audio synchronized with a facial avatar, and facial expressions can address all of those communication needs.

“Here we have demonstrated that all of these needs can be addressed with a speech-neuroprosthetic system that decodes articulatory cortical activity into multiple output modalities in real time, including text, speech audio synchronized with a facial avatar, and facial expressions.”

• AssumptionRests on an assumption: The participant's silent-speech attempts engage the same articulatory representations as natural speech production.

The present proof-of-concept study's results come from only one participant, which is a limitation.

“A limitation of the present proof-of-concept study is that the results shown are from only one participant.”

The participant's SMC showed persistent articulatory encoding consistent with previous intact-speech characterizations despite more than 18 years of anarthria.

“persistent articulatory encoding in the SMC of our participant that is consistent with previous intact-speech characterizations despite more than 18 years of anarthria”

• AssumptionRests on an assumption: The participant's silent-speech attempts engage the same articulatory representations as natural speech production.

BCIs can restore this ability to patients through highly personalizable audio-visual synthesis that restores aspects of personhood and identity.

“We show here that BCIs can give this ability back to patients through highly personalizable audio-visual synthesis capable of restoring aspects of their personhood and identity.”

• AssumptionRests on an assumption: The participant's silent-speech attempts engage the same articulatory representations as natural speech production.