Speech synthesis from neural decoding of spoken sentences
Open accessPublished version of a preprint
Abstract
Technology that translates neural activity into speech would be transformative for people who are unable to communicate as a result of neurological impairments. Decoding speech from neural activity is challenging because speaking requires very precise and rapid multi-dimensional control of vocal tract articulators. Here we designed a neural decoder that explicitly leverages kinematic and sound representations encoded in human cortical activity to synthesize audible speech. Recurrent neural networks first decoded directly recorded cortical activity into representations of articulatory movement, and then transformed these representations into speech acoustics. In closed vocabulary tests, listeners could readily identify and transcribe speech synthesized from cortical activity. Intermediate articulatory dynamics enhanced performance even with limited data. Decoded articulatory representations were highly conserved across speakers, enabling a component of the decoder to be transferrable across participants. Furthermore, the decoder could synthesize speech when a participant silently mimed sentences. These findings advance the clinical viability of using speech neuroprosthetic technology to restore spoken communication.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Angrick et al. and Anumanchipalli et al. used a two-stage decoding process in which a deep-learning model regresses neural activity to acoustic features and a vocoder converts them to audible speech.
“Another approach, pursued by Angrick et al.111 and Anumanchipalli et al.110, involved a two-stage decoding process. In the first stage, a deep-learning model is trained to regress neural activity (recorded from SEEG111 or ECoG110) to a time series of acoustic features, such as the mel-spectrogram (Fig. 3b). In the second stage, a speech synthesizer — a vocoder — is used to convert the acoustic rep”
From The speech neuroprosthesis (2024) · cites this paper as [110]
The Speech decoder design and Methods (Neural decoder; page numbers unavailable) describe a different two-stage process: an ECoG-based neural network first decodes articulatory features, then another network converts those features to acoustic features. Audio is synthesized afterward, not as the second decoding stage. The paper treats direct neural-to-acoustic decoding as a separate comparator. (LLM, conf=0.89)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredPrior work has synthesized intelligible speech from neural activity during vocalized or mimed speech in speakers with intact speech, but not in a paralysed person.
“Previous work in speakers with intact speech has demonstrated that intelligible speech can be synthesized from neural activity during vocalized or mimed speech10,11, but this has not been shown with someone who is paralysed.”
From A high-performance neuroprosthesis for speech decoding and avatar control (2023) · cites this paper as [11]
The Abstract reports intelligible neurally synthesized speech from participants who spoke aloud, and the Methods identify them as people undergoing epilepsy monitoring rather than people with paralysis. The “Synthesizing mimed speech” section reports synthesis from silent mime, but assesses spectral similarity rather than listener intelligibility; the claim’s intelligibility qualifier is therefore not established for mimed speech. (LLM, conf=0.84)
Experimental — accuracy not yet measuredThe mel-cepstral distortion values are comparable to those of text-to-speech synthesizers and better than those in previous neural-decoding work with participants who could speak naturally.
“Furthermore, these MCDs are comparable to those observed with text-to-speech synthesizers16 and better than those in previous neural-decoding work with participants that were able to speak naturally11.”
From A high-performance neuroprosthesis for speech decoding and avatar control (2023) · cites this paper as [11]
Anumanchipalli et al. added a step decoding an articulatory representation from neural activity, then mapped it to acoustic representation and vocoded it into speech.
“Anumanchipalli et al. added a first step of decoding an articulatory representation from neural activity, which was then mapped to the intermediate acoustic representation and vocoded into speech110.”
From The speech neuroprosthesis (2024) · cites this paper as [110]
Anumanchipalli et al.'s approach further improved speech-synthesis performance by leveraging the underlying articulatory organization of the SMC.
“Their approach further improved speech-synthesis performance by leveraging the underlying articulatory organization of the SMC.”
From The speech neuroprosthesis (2024) · cites this paper as [110]
Previous work showed that first decoding articulatory features and transforming them to acoustic features improved synthesized speech quality compared with decoding acoustics directly from neural data.
“Previous work has demonstrated that first decoding articulatory features and transforming them to acoustic features improved the quality of synthesized speech (compared with decoding acoustics directly from neural data)110.”
From The speech neuroprosthesis (2024) · cites this paper as [110]
Experimental — accuracy not yet measuredSpeech-decoding BCIs show promise for rapid communication, but their accuracy and vocabulary size still need improvement for general-purpose use.
“Recently, speech-decoding BCIs have shown exciting promise for restoring rapid communication (e.g. 34,16,17), but their accuracies and vocabulary sizes require further improvement to support general-purpose use.”
From High-performance brain-to-text communication via handwriting (2021) · cites this paper as [16]
Efforts to develop a speech neuroprosthesis are built largely on studies using data that are retrospectively analyzed from able-bodied speakers undergoing electrophysiological monitoring for clinical purposes.
“Efforts to develop a speech neuroprosthesis are built largely on studies using data that are retrospectively analyzed from able-bodied speakers undergoing electrophysiological monitoring for clinical purposes 7-16 .”
From An Accurate and Rapidly Calibrating Speech Neuroprosthesis (2024) · cites this paper as [11]
Early speech BCI demonstrations have not yet achieved accuracies sufficiently high for communication of unconstrained sentences from a large vocabulary.
“Early demonstrations, although promising, have not yet achieved accuracies sufficiently high for communication of unconstrained sentences from a large vocabulary1–7.”
From A high-performance speech neuroprosthesis (2023) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Brain–computer interfaces for communication and control
Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., & Vaughan, T. M. (2002) · Clinical Neurophysiology · doi:10.1016/s1388-2457(02)00057-3 · Cited by 7,927 · see citing works
Framewise phoneme classification with bidirectional LSTM and other neural network architectures
Graves, A., & Schmidhuber, J. (2005) · Neural Networks · doi:10.1016/j.neunet.2005.06.042 · Cited by 5,882 · see citing works
Davis, S., & Mermelstein, P. (1980) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1980.1163420 · Cited by 5,465 · see citing works
Neuronal ensemble control of prosthetic devices by a human with tetraplegia
Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., Branner, A., Chen, D., Penn, R. D., & Donoghue, J. P. (2006) · Nature · doi:10.1038/nature04970 · Cited by 3,295 · see citing works
Farwell, L. A., & Donchin, E. (1988) · Electroencephalography and Clinical Neurophysiology · doi:10.1016/0013-4694(88)90149-6 · Cited by 3,228 · see citing works
(1994) · Knowledge Discovery and Data Mining · Cited by 3,137 · see citing works
Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461368 · Cited by 2,791 · see citing works
Articulatory Phonology: An Overview
Browman, C. P., & Goldstein, L. M. (1992) · Phonetica · doi:10.1159/000261913 · Cited by 2,004 · see citing works
Recognizing Spoken Words: The Neighborhood Activation Model
Luce, P. A., & Pisoni, D. B. (1998) · Ear and Hearing · doi:10.1097/00003446-199802000-00001 · Cited by 1,960 · see citing works
High-performance neuroprosthetic control by an individual with tetraplegia
Collinger, J. L., Wodlinger, B., Downey, J. E., Wang, W., Tyler-Kabara, E. C., Weber, D. J., McMorland, A. J., Velliste, M., Boninger, M. L., & Schwartz, A. B. (2013) · The Lancet · doi:10.1016/s0140-6736(12)61816-9 · Cited by 1,848 · see citing works
THE COMPARISON OF DENDROGRAMS BY OBJECTIVE METHODS
Sokal, R. R., & Rohlf, F. J. (1962) · Taxon · doi:10.2307/1217208 · Cited by 1,758 · see citing works
Direct Cortical Control of 3D Neuroprosthetic Devices
Taylor, D. M., Tillery, S. I. H., & Schwartz, A. B. (2002) · Science · doi:10.1126/science.1070291 · Cited by 1,542 · see citing works
Natural speech reveals the semantic maps that tile human cerebral cortex
Huth, A. G., de Heer, W. A., Griffiths, T. L., Theunissen, F. E., & Gallant, J. L. (2016) · Nature · doi:10.1038/nature17637 · Cited by 1,504 · see citing works
Real-time prediction of hand trajectory by ensembles of cortical neurons in primates
Wessberg, J., Stambaugh, C. R., Kralik, J. D., Beck, P. D., Laubach, M., Chapin, J. K., Kim, J., Biggs, S. J., Srinivasan, M. A., & Nicolelis, M. A. L. (2000) · Nature · doi:10.1038/35042582 · Cited by 1,491 · see citing works
Instant neural control of a movement signal
Serruya, M. D., Hatsopoulos, N. G., Paninski, L., Fellows, M. R., & Donoghue, J. P. (2002) · Nature · doi:10.1038/416141a · Cited by 1,415 · see citing works
The design for the wall street journal-based CSR corpus
Paul, D. B., & Baker, J. M. (1992) · DARPA Workshop on Speech and Natural Language (HLT) · doi:10.3115/1075527.1075614 · Cited by 1,142 · see citing works
Coding of intention in the posterior parietal cortex
Snyder, L. H., Batista, A. P., & Andersen, R. A. (1997) · Nature · doi:10.1038/386167a0 · Cited by 1,103 · see citing works
Phonetic Feature Encoding in Human Superior Temporal Gyrus
Mesgarani, N., Cheung, C., Johnson, K., & Chang, E. F. (2014) · Science · doi:10.1126/science.1245994 · Cited by 993 · see citing works
Self-citationAjiboye, A. B., Willett, F. R., Young, D. R., Memberg, W. D., Murphy, B. A., Miller, J. P., Walter, B. L., Sweet, J. A., Hoyen, H. A., Keith, M. W., et al. (2017) · The Lancet · doi:10.1016/s0140-6736(17)30601-3 · Cited by 862 · see citing works
Effect of temporal envelope smearing on speech reception
Drullman, R., Festen, J. Μ., & Plomp, R. (1994) · The Journal of the Acoustical Society of America · doi:10.1121/1.408467 · Cited by 805 · see citing works
Neural constraints on learning
Sadtler, P. T., Quick, K. M., Golub, M. D., Chase, S. M., Ryu, S. I., Tyler-Kabara, E. C., Yu, B. M., & Batista, A. P. (2014) · Nature · doi:10.1038/nature13665 · Cited by 713 · see citing works
Functional organization of human sensorimotor cortex for speech articulation
Bouchard, K. E., Mesgarani, N., Johnson, K., & Chang, E. F. (2013) · Nature · doi:10.1038/nature11911 · Cited by 686 · see citing works
Self-citationNeural Manifolds for the Control of Movement
Gallego, J. Á., Perich, M. G., Miller, L. E., & Solla, S. A. (2017) · Neuron · doi:10.1016/j.neuron.2017.05.025 · Cited by 676 · see citing works
Decoding motor imagery from the posterior parietal cortex of a tetraplegic human
Aflalo, T., Kellis, S., Klaes, C., Lee, B., Shi, Y., Pejsa, K., Shanfield, K., Hayes-Jackson, S., Aisen, M., Heck, C., et al. (2015) · Science · doi:10.1126/science.aaa5417 · Cited by 649 · see citing works
Reconstructing Speech from Human Auditory Cortex
Pasley, B. N., David, S. V., Mesgarani, N., Flinker, A., Shamma, S. A., Crone, N. E., Knight, R. T., & Chang, E. F. (2012) · PLoS Biology · doi:10.1371/journal.pbio.1001251 · Cited by 604 · see citing works
Self-citationComparison of different implementations of MFCC
Zheng, F., Zhang, G., & Song, Z. (2001) · Journal of Computer Science and Technology · doi:10.1007/bf02943243 · Cited by 594 · see citing works
Pandarinath, C., Nuyujukian, P., Blabe, C. H., Sorice, B. L., Saab, J., Willett, F. R., Hochberg, L. R., Shenoy, K. V., & Henderson, J. M. (2017) · eLife · doi:10.7554/elife.18554 · Cited by 566 · see citing works
Churchill, A., Gutiérrez, G. D., Singer, R. A., Lorberbaum, D. S., Fischer, K. A., & Sussel, L. (2017) · eLife · doi:10.7554/elife.20010 · Cited by 543 · see citing works
Redefining the role of Broca’s area in speech
Flinker, A., Korzeniewska, A., Shestyuk, A. Y., Franaszczuk, P. J., Dronkers, N. F., Knight, R. T., & Crone, N. E. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1414491112 · Cited by 487 · see citing works
Majaranta, P., & Räihä, K. (2002) · ACM Symposium on Eye Tracking Research and Applications (ETRA) · doi:10.1145/507072.507076 · Cited by 485 · see citing works
Pesaran, B., Vinck, M. A., Einevoll, G. T., Sirota, A., Fries, P., Siegel, M., Truccolo, W., Schroeder, C. E., & Srinivasan, R. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0171-8 · Cited by 365 · see citing works
Davis, S. B., & Mermelstein, P. G. (1990) · Elsevier eBooks · doi:10.1016/b978-0-08-051584-7.50010-3 · Cited by 341 · see citing works
A Wireless Brain-Machine Interface for Real-Time Speech Synthesis
Guenther, F. H., Brumberg, J. S., Wright, E. J., Nieto-Castanon, A., Tourville, J. A., Panko, M., Law, R., Siebert, S. A., Bartels, J. L., Andreasen, D. S., et al. (2009) · PLoS ONE · doi:10.1371/journal.pone.0008218 · Cited by 306 · see citing works
Electrocorticographic gamma activity during word production in spoken and sign language
Crone, N. E., Hao, L., Hart, J., Boatman, D. F., Lesser, R. P., Irizarry, R. A., & Gordon, B. (2001) · Neurology · doi:10.1212/wnl.57.11.2045 · Cited by 300 · see citing works
Brain-to-text: decoding spoken phrases from phone representations in the brain
Herff, C., Heger, D., de Pesters, A., Telaar, D., Brunner, P., Schalk, G., & Schultz, T. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00217 · Cited by 297 · see citing works
Learning by neural reassociation
Golub, M. D., Sadtler, P. T., Oby, E. R., Quick, K. M., Ryu, S. I., Tyler‐Kabara, E. C., Batista, A. P., Chase, S. M., & Yu, B. M. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0095-3 · Cited by 274 · see citing works
Towards reconstructing intelligible speech from the human auditory cortex
Akbari, H., Khalighinejad, B., Herrero, J. L., Mehta, A. D., & Mesgarani, N. (2019) · Scientific Reports · doi:10.1038/s41598-018-37359-z · Cited by 221 · see citing works
Huggins, J. E., Wren, P. A., & Gruis, K. L. (2011) · Amyotrophic Lateral Sclerosis · doi:10.3109/17482968.2011.572978 · Cited by 215 · see citing works
Encoding of Articulatory Kinematic Trajectories in Human Speech Sensorimotor Cortex
Chartier, J., Anumanchipalli, G. K., Johnson, K., & Chang, E. F. (2018) · Neuron · doi:10.1016/j.neuron.2018.04.031 · Cited by 208 · see citing works
Self-citationDecoding spectrotemporal features of overt and covert speech from the human cortex
Martin, S., Brunner, P., Holdgraf, C., Heinze, H.-J., Crone, N. E., Rieger, J., Schalk, G., Knight, R. T., & Pasley, B. N. (2014) · Frontiers in Neuroengineering · doi:10.3389/fneng.2014.00014 · Cited by 200 · see citing works
Mugler, E. M., Patton, J. L., Flint, R. D., Wright, Z. A., Schuele, S. U., Rosenow, J., Shih, J. J., Krusienski, D. J., & Slutzky, M. W. (2014) · Journal of Neural Engineering · doi:10.1088/1741-2560/11/3/035015 · Cited by 192 · see citing works
The Control of Vocal Pitch in Human Laryngeal Motor Cortex
Dichter, B. K., Breshears, J. D., Leonard, M. K., & Chang, E. F. (2018) · Cell · doi:10.1016/j.cell.2018.05.016 · Cited by 183 · see citing works
Self-citationMixed excitation for HMM-based speech synthesis
Yoshimura, T., Tokuda, K., Masuko, T., Kobayashi, T., & Kitamura, T. (2001) · European Conference on Speech Communication and Technology (Eurospeech) · doi:10.21437/eurospeech.2001-539 · Cited by 157 · see citing works
EMG-to-Speech: Direct Generation of Speech From Facial Electromyographic Signals
Janke, M., & Diener, L. (2017) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2017.2738568 · Cited by 137 · see citing works
Announcing the electromagnetic articulography (day 1) subset of the mngu0 articulatory corpus
Richmond, K., Hoole, P., & King, S. (2011) · Interspeech (USB) · doi:10.21437/interspeech.2011-316 · Cited by 130 · see citing works
The Emerging World Of Motor Neuroprosthetics: a Neurosurgical Perspective
Leuthardt, E. C., Schalk, G., Moran, D. W., & Ojemann, J. G. (2006) · Neurosurgery · doi:10.1227/01.neu.0000221506.06947.ac · Cited by 128 · see citing works
Hamilton, L. S., Chang, D. L., Lee, M. B., & Chang, E. F. (2017) · Frontiers in Neuroinformatics · doi:10.3389/fninf.2017.00062 · Cited by 111 · see citing works
Self-citationKoch Fager, S., Fried-Oken, M., Jakobs, T., & Beukelman, D. R. (2019) · Augmentative and Alternative Communication · doi:10.1080/07434618.2018.1556730 · Cited by 107 · see citing works
Brain–Computer Interfaces for Augmentative and Alternative Communication: A Tutorial
Brumberg, J. S., Pitt, K. M., Mantie-Kozlowski, A., & Burnison, J. D. (2018) · American Journal of Speech-Language Pathology · doi:10.1044/2017_ajslp-16-0244 · Cited by 100 · see citing works
Mugler, E. M., Tate, M. C., Livescu, K., Templer, J. W., Goldrick, M. A., & Slutzky, M. W. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.1206-18.2018 · Cited by 96 · see citing works
Moses, D. A., Mesgarani, N., Leonard, M. K., & Chang, E. F. (2016) · Journal of Neural Engineering · doi:10.1088/1741-2560/13/5/056004 · Cited by 92 · see citing works
Self-citationElectrocorticographic representations of segmental features in continuous speech
Lotte, F., Brumberg, J. S., Brunner, P., Gunduz, A., Ritaccio, A. L., Guan, C., & Schalk, G. (2015) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2015.00097 · Cited by 86 · see citing works
An Excitation Model for HMM-Based Speech Synthesis Based on Residual Modeling
Maia, R., Toda, T., Zen, H., et al. (2007) · Institutional Repositories DataBase (IRDB) · Cited by 78 · see citing works
Real-Time Control of an Articulatory-Based Speech Synthesizer for Brain Computer Interfaces
Bocquelet, F., Hueber, T., Laurent Girin, B., Savariaux, C., & Yvert, B. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1005119 · Cited by 77 · see citing works
Synthesizer Voice Quality of New Languages Calibrated with Mean Mel Cepstral Distortion
Kominek, J., Schultz, T., & Black, A. W. (2008) · Cited by 74 · see citing works
Brumberg, J. S., Krusienski, D. J., Chakrabarti, S., Gunduz, A., Brunner, P., Ritaccio, A. L., & Schalk, G. (2016) · PLoS ONE · doi:10.1371/journal.pone.0166872 · Cited by 71 · see citing works
Key considerations in designing a speech brain-computer interface
Bocquelet, F., Hueber, T., Laurent Girin, B., Chabardès, S., & Yvert, B. (2016) · Journal of Physiology-Paris · doi:10.1016/j.jphysparis.2017.07.002 · Cited by 70 · see citing works
Sub-Phonetic Modeling For Capturing Pronunciation Variations For Conversational Speech Synthesis
Prahallad, K., Black, A. W., & Mosur, R. (2006) · doi:10.1109/icassp.2006.1660155 · Cited by 66 · see citing works
The rôle of natural language processing in alternative and augmentative communication
Newell, A. F., LANGER, S., & Hickey, M. (1998) · Natural Language Engineering · doi:10.1017/s135132499800182x · Cited by 61 · see citing works
Evaluating speech synthesis intelligibility using Amazon Mechanical Turk
Wolters, M. K., Isaac, K. B., & Renals, S. J. (2010) · ERA · Cited by 45 · see citing works
Towards reconstructing intelligible speech from the human auditory cortexPreprint
Akbari, H., Khalighinejad, B., Herrero, J. L., Mehta, A. D., & Mesgarani, N. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/350124 · Cited by 44 · see citing works
Nourski, K. V., Steinschneider, M., Rhone, A. E., Oya, H., Kawasaki, H., Howard, M. A., & McMurray, B. (2015) · Brain and Language · doi:10.1016/j.bandl.2015.03.003 · Cited by 43 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [41]Anumanchipalli GK, Prahallad K, & Black AW (2011). Festvox: Tools for creation and analyses of large speech corpora, Workshop on Very Large Scale Phonetics Research, UPenn, Philadelphia. http://www.festvox.org
Cited by
Sorted by citations
An Integrated Brain-Machine Interface Platform With Thousands of Channels
Musk, E., & Neuralink (2019) · Journal of Medical Internet Research · doi:10.2196/16194 · Cited by 1,206 · see citing works
High-performance brain-to-text communication via handwriting
Willett, F. R., Avansino, D. T., Hochberg, L. R., Henderson, J. M., & Shenoy, K. V. (2021) · Nature · doi:10.1038/s41586-021-03506-2 · Cited by 969 · see citing works
“Recently, speech-decoding BCIs have shown exciting promise for restoring rapid communication 16,17,34 , but their accuracies and vocabulary sizes require further improvement to support general-purpose use.”
BackgroundMethodNeuroprosthesis for Decoding Speech in a Paralyzed Person with Anarthria
Moses, D. A., Metzger, S. L., Liu, J. R., Anumanchipalli, G. K., Makin, J. G., Sun, P. F., Chartier, J., Dougherty, M. E., Liu, P. M., Abrams, G. M., et al. (2021) · New England Journal of Medicine · doi:10.1056/nejmoa2027540 · Cited by 582 · see citing works
Self-citationA high-performance speech neuroprosthesis
Willett, F. R., Kunz, E. M., Fan, C., Avansino, D. T., Wilson, G. H., Choi, E. Y., Kamdar, F., Glasser, M. F., Hochberg, L. R., Druckmann, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06377-x · Cited by 581 · see citing works
A high-performance neuroprosthesis for speech decoding and avatar control
Metzger, S. L., Littlejohn, K. T., Silva, A. B., Moses, D. A., Seaton, M. P., Wang, R., Dougherty, M. E., Liu, J. R., Wu, P., Berger, M. A., et al. (2023) · Nature · doi:10.1038/s41586-023-06443-4 · Cited by 522 · see citing works
Self-citationSemantic reconstruction of continuous language from non-invasive brain recordings
Tang, J., LeBel, A., Jain, S., & Huth, A. G. (2023) · Nature Neuroscience · doi:10.1038/s41593-023-01304-9 · Cited by 442 · see citing works
An Accurate and Rapidly Calibrating Speech Neuroprosthesis
Card, N. S., Wairagkar, M., Iacobacci, C., Hou, X., Singer-Clark, T., Willett, F. R., Kunz, E. M., Fan, C., Vahdati Nia, M., Deo, D. R., et al. (2024) · New England Journal of Medicine · doi:10.1056/nejmoa2314132 · Cited by 238 · see citing works
Chiang, C.-H., Won, S. M., Orsborn, A. L., Yu, K. J., Trumpis, M., Bent, B., Wang, C., Xue, Y., Min, S., Woods, V., et al. (2020) · Science Translational Medicine · doi:10.1126/scitranslmed.aay4682 · Cited by 211 · see citing works
Evolving perspectives on the sources of the frequency-following response
Coffey, E. B. J., Nicol, T., White-Schwoch, T., Chandrasekaran, B., Krizman, J., Skoe, E., Zatorre, R. J., & Kraus, N. (2019) · Nature Communications · doi:10.1038/s41467-019-13003-w · Cited by 207 · see citing works
Imagined speech can be decoded from low- and cross-frequency intracranial EEG features
Proix, T., Delgado Saa, J., Christen, A., Martin, S., Pasley, B. N., Knight, R. T., Tian, X., Poeppel, D., Doyle, W. K., Devinsky, O., et al. (2022) · Nature Communications · doi:10.1038/s41467-021-27725-3 · Cited by 212 · see citing works
Metzger, S. L., Liu, J. R., Moses, D. A., Dougherty, M. E., Seaton, M. P., Littlejohn, K. T., Chartier, J., Anumanchipalli, G. K., Tu-Chan, A., Ganguly, K., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-33611-3 · Cited by 153 · see citing works
Self-citationHuman brain mapping with multithousand-channel PtNRGrids resolves spatiotemporal dynamics
Tchoe, Y., Bourhis, A. M., Cleary, D. R., Stedelin, B., Lee, J., Tonsfeldt, K. J., Brown, E. C., Siler, D. A., Paulk, A. C., Yang, J. C., et al. (2022) · Science Translational Medicine · doi:10.1126/scitranslmed.abj1441 · Cited by 135 · see citing works
Herff, C., Diener, L., Angrick, M., Mugler, E., Tate, M. C., Goldrick, M. A., Krusienski, D. J., Slutzky, M. W., & Schultz, T. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.01267 · Cited by 118 · see citing works
Neural ensemble dynamics in dorsal motor cortex during speech in people with paralysis
Stavisky, S. D., Willett, F. R., Wilson, G. H., Murphy, B. A., Rezaii, P., Avansino, D. T., Memberg, W. D., Miller, J. P., Kirsch, R. F., Hochberg, L. R., et al. (2019) · eLife · doi:10.7554/elife.46015 · Cited by 105 · see citing works
Angrick, M., Ottenhoff, M. C., Diener, L., Ivucic, D., Ivucic, G., Goulis, S., Saal, J., Colon, A. J., Wagner, L., Krusienski, D. J., et al. (2021) · Communications Biology · doi:10.1038/s42003-021-02578-0 · Cited by 125 · see citing works
Decoding spoken English from intracortical electrode arrays in dorsal precentral gyrus
Wilson, G. H., Stavisky, S. D., Willett, F. R., Avansino, D. T., Kelemen, J. N., Hochberg, L. R., Henderson, J. M., Druckmann, S., & Shenoy, K. V. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/abbfef · Cited by 103 · see citing works
Conformable hybrid systems for implantable bioelectronic interfaces
Fallegger, F., Schiavone, G., & Lacour, S. P. (2020) · Advanced Materials · doi:10.1002/adma.201903904 · Cited by 113 · see citing works
Brain-Computer Interface: Applications to Speech Decoding and Synthesis to Augment Communication
Luo, S., Rabbani, Q., & Crone, N. E. (2022) · Neurotherapeutics · doi:10.1007/s13311-022-01190-2 · Cited by 106 · see citing works
Silva, A. B., Littlejohn, K. T., Liu, J. R., Moses, D. A., & Chang, E. F. (2024) · Nature Reviews Neuroscience · doi:10.1038/s41583-024-00819-9 · Cited by 103 · see citing works
“Anumanchipalli et al. added a first step of decoding an articulatory representation from neural activity, which was then mapped to the intermediate acoustic representation and vocoded into speech 110 .”
InfluentialSelf-citationLuo, S., Angrick, M., Coogan, C., Candrea, D. N., Wyse‐Sookoo, K., Shah, S., Rabbani, Q., Milsap, G. W., Weiss, A. R., Anderson, W. S., et al. (2023) · Advanced Science · doi:10.1002/advs.202304853 · Cited by 90 · see citing works
Brain2Char: a deep architecture for decoding text from brain recordings
Sun, P., Anumanchipalli, G. K., & Chang, E. F. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/abc742 · Cited by 77 · see citing works
Self-citationRepresentation of internal speech by single neurons in human supramarginal gyrus
Wandelt, S. K., Bjånes, D. A., Pejsa, K., Lee, B., Liu, C., & Andersen, R. A. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01867-y · Cited by 74 · see citing works
High-resolution neural recordings improve the accuracy of speech decoding
Duraivel, S., Rahimpour, S., Chiang, C.-H., Trumpis, M., Wang, C., Barth, K., Harward, S. C., Lad, S. P., Friedman, A. H., Southwell, D. G., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-42555-1 · Cited by 75 · see citing works
Multi-Channel Neural Recording Implants: A Review
Hashemi Noshahr, F., Nabavi, M., & Sawan, M. (2020) · Sensors · doi:10.3390/s20030904 · Cited by 66 · see citing works
Inner speech in motor cortex and implications for speech neuroprostheses
Kunz, E. M., Abramovich Krasa, B., Kamdar, F., Avansino, D. T., Hahn, N., Yoon, S., Singh, A., Nason-Tomaszewski, S. R., Card, N. S., Jude, J. J., et al. (2025) · Cell · doi:10.1016/j.cell.2025.06.015 · Cited by 55 · see citing works
Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models
Berezutskaya, J., Freudenburg, Z. V., Vansteensel, M. J., Aarnoutse, E. J., Ramsey, N. F., & van Gerven, M. A. J. (2023) · Journal of Neural Engineering · doi:10.1088/1741-2552/ace8be · Cited by 42 · see citing works
Toward a Speech Neuroprosthesis
Chang, E. F., & Anumanchipalli, G. K. (2020) · JAMA · doi:10.1001/jama.2019.19813 · Cited by 35 · see citing works
Self-citationNeurorights, mental privacy, and mind reading
Brown, C. M. L. (2024) · Neuroethics · doi:10.1007/s12152-024-09568-z · Cited by 33 · see citing works
The Layer 7 Cortical Interface: A Scalable and Minimally Invasive Brain–Computer Interface Platform
Hettick, M., Ho, E., Poole, A. J., Monge, M., Papageorgiou, D., Takahashi, K., LaMarca, M., Trietsch, D., Reed, K., Murphy, M., et al. (2022) · doi:10.1101/2022.01.02.474656 · Cited by 31 · see citing works
Neural control of lexical tone production in human laryngeal motor cortex
Lu, J., Li, Y., Zhao, Z., Liu, Y., Zhu, Y., Mao, Y., Wu, J., & Chang, E. F. (2023) · Nature Communications · doi:10.1038/s41467-023-42175-9 · Cited by 26 · see citing works
Self-citationClassification of Articulator Movements and Movement Direction from Sensorimotor Cortex Activity
Salari, E., Freudenburg, Z. V., Branco, M. P., Aarnoutse, E. J., Vansteensel, M. J., & Ramsey, N. F. (2019) · Scientific Reports · doi:10.1038/s41598-019-50834-5 · Cited by 26 · see citing works
Algorithm and hardware considerations for real-time neural signal on-implant processing
Zhang, Z., Savolainen, O. W., & Constandinou, T. G. (2022) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac5268 · Cited by 25 · see citing works
Temporally resolved neural dynamics underlying handwriting
Pei, L., Longcamp, M., Leung, F. K.-S., & Ouyang, G. (2021) · NeuroImage · doi:10.1016/j.neuroimage.2021.118578 · Cited by 23 · see citing works
Speech-related dorsal motor cortex activity does not interfere with iBCI cursor control
Stavisky, S. D., Willett, F. R., Avansino, D. T., Hochberg, L. R., Shenoy, K. V., & Henderson, J. M. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab5b72 · Cited by 25 · see citing works
Angrick, M., Luo, S., Rabbani, Q., Candrea, D. N., Shah, S., Milsap, G. W., Anderson, W. S., Gordon, C. R., Rosenblatt, K. R., Clawson, L., et al. (2023) · doi:10.1101/2023.06.30.23291352 · Cited by 95 · see citing works
Berger, S., & Rossi, F. (2023) · Communications of the ACM · doi:10.1145/3529088 · Cited by 21 · see citing works
Tankus, A., Solomon, L., Aharony, Y., Faust-Socher, A., & Strauss, I. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac3315 · Cited by 20 · see citing works
An accurate and rapidly calibrating speech neuroprosthesis
Card, N. S., Wairagkar, M., Iacobacci, C., Hou, X., Singer-Clark, T., Willett, F. R., Kunz, E. M., Fan, C., Vahdati Nia, M., Deo, D. R., et al. (2023) · doi:10.1101/2023.12.26.23300110 · Cited by 244 · see citing works
Online internal speech decoding from single neurons in a human participant
Wandelt, S. K., Bjånes, D. A., Pejsa, K., Lee, B., Liu, C., & Andersen, R. A. (2022) · doi:10.1101/2022.11.02.22281775 · Cited by 17 · see citing works
Varshney, S., Farias, D., Brandman, D. M., Stavisky, S. D., & Miller, L. M. (2023) · 2023 11th International IEEE/EMBS Conference on Neural Engineering (NER) · doi:10.1109/ner52421.2023.10123751 · Cited by 6 · see citing works
Direct Speech Reconstruction from Sensorimotor Brain Activity with Optimized Deep Learning Models
Berezutskaya, J., Freudenburg, Z. V., Vansteensel, M. J., Aarnoutse, E. J., Ramsey, N. F., & van Gerven, M. A. J. (2022) · doi:10.1101/2022.08.02.502503 · Cited by 42 · see citing works
- Authors
- Anumanchipalli, G. K., Chartier, J., & Chang, E. F.
- Affiliations
- University of California, San Francisco (US)
- Corresponding author
- Chang, Edward F.
- Venue
- Nature
- Published
- 2019-04-24
- Volume
- 568
- Issue
- 7753
- Pages
- 493-498
- ISSN
- 0028-0836 (ISSN-L)
- 1476-4687
- PMC
- PMC9714519
- OpenAlex
- W2940585064
- Cited by
- 883
- References
- 63
- Citations by year
- 2026: 89 · 2025: 118 · 2024: 137 · 2023: 123 · 2022: 108 · 2021: 149 · 2020: 117 · 2019: 42
- FWCI
- 47.44
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- Nature (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- New York Stem Cell Foundation
- McKnight Foundation
- National Institutes of Health (1dp2od008627-01)
- National Institute of Neurological Disorders and Stroke (U01NS098971)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Health Sciences › Medicine › Physiology › Voice and Speech Disorders
- Keywords
- neural speech synthesis, neural decoding, cortical activity, articulatory kinematics, recurrent neural networks, silent speech, mimesis, communication restoration, neurological deficit
- MeSH terms
- Adult, Biomechanical Phenomena, Cerebral Cortex, Female, Humans, Male, Movement, Speech, Speech Acoustics, Speech Articulation Tests, Speech Intelligibility, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
