The DOI matches its Crossref record; 73 of the 76 references it cites were found in a registry or an index.
Speech synthesis from ECoG using densely connected 3D convolutional neural networks
Open accessPublished version of a preprint
Abstract
Objective . Direct synthesis of speech from neural signals could provide a fast and natural way of communication to people with neurological diseases. Invasively-measured brain activity (electrocorticography; ECoG) supplies the necessary temporal and spatial resolution to decode fast and complex processes such as speech production. A number of impressive advances in speech decoding using neural signals have been achieved in recent years, but the complex dynamics are still not fully understood. However, it is unlikely that simple linear models can capture the relation between neural activity and continuous spoken speech. Approach . Here we show that deep neural networks can be used to map ECoG from speech production areas onto an intermediate representation of speech (logMel spectrogram). The proposed method uses a densely connected convolutional neural network topology which is well-suited to work with the small amount of data available from each participant. Main results . In a study with six participants, we achieved correlations up to r = 0.69 between the reconstructed and original logMel spectrograms. We transfered our prediction back into an audible waveform by applying a Wavenet vocoder. The vocoder was conditioned on logMel features that harnessed a much larger, pre-existing data corpus to provide the most natural acoustic output. Significance . To the best of our knowledge, this is the first time that high-quality speech has been reconstructed from neural recordings during speech production using deep neural networks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Previous work in speakers with intact speech has demonstrated that intelligible speech can be synthesized from neural activity during vocalized or mimed speech10,11, but this has not been shown with someone who is paralysed.”
From A high-performance neuroprosthesis for speech decoding and avatar control (2023) · cites this paper as [10]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Another approach, pursued by Angrick et al.111 and Anumanchipalli et al.110, involved a two-stage decoding process. In the first stage, a deep-learning model is trained to regress neural activity (recorded from SEEG111 or ECoG110) to a time series of acoustic features, such as the mel-spectrogram (Fig. 3b). In the second stage, a speech synthesizer — a vocoder — is used to convert the acoustic rep”
From The speech neuroprosthesis (2024) · cites this paper as [111]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Densely Connected Convolutional Networks
Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.243 · Cited by 48,566
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Brain–computer interfaces for communication and control
Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., & Vaughan, T. M. (2002) · Clinical Neurophysiology · doi:10.1016/s1388-2457(02)00057-3 · Cited by 7,927
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911
Deep learning with convolutional neural networks for EEG decoding and visualization
Schirrmeister, R. T., Springenberg, J. T., Fiederer, L. D. J., Glasstetter, M., Eggensperger, K., Tangermann, M., Hutter, F., Burgard, W., & Ball, T. (2017) · Human Brain Mapping · doi:10.1002/hbm.23730 · Cited by 3,751
Neuronal ensemble control of prosthetic devices by a human with tetraplegia
Hochberg, L. R., Serruya, M. D., Friehs, G. M., Mukand, J. A., Saleh, M., Caplan, A. H., Branner, A., Chen, D., Penn, R. D., & Donoghue, J. P. (2006) · Nature · doi:10.1038/nature04970 · Cited by 3,295
BCI2000: A General-Purpose Brain-Computer Interface (BCI) System
Schalk, G., McFarland, D. J., Hinterberger, T., Birbaumer, N., & Wolpaw, J. R. (2004) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2004.827072 · Cited by 3,129
Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions
Shen, J., Pang, R., Weiss, R. J., Schuster, M., Jaitly, N., Yang, Z., Chen, Z., Zhang, Y., Wang, Y., Skerrv-Ryan, R., et al. (2018) · 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) · doi:10.1109/icassp.2018.8461368 · Cited by 2,791
Cited by
Sorted by citations
Items 201–227 of 227
Modeling of Perceived Musical Rhythms Using Electrocorticography
Dexheimer, M. G., Johnson, G. D., Shih, J. J., Herff, C., & Krusienski, D. J. (2023) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc53992.2023.10394660
Self-citationRoussel, P., Bocquelet, F., Chabardès, S., & Yvert, B. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.03.21.586130
Improvement of NSP and its application in speech imaging EEG
Li, N., Fu, D., Mo, J., & Hu, W. (2023) · International Conference on Internet of Things, Automation and Artificial Intelligence (IoTAAI) · doi:10.1145/3653081.3653108
Tracing Responsibility and Neuroprosthesis-Mediated Speech
Rainey, S. (2024) · Techno:Phil · doi:10.1007/978-3-662-68362-0_8
Cognition of Time and Thinking Beyond
Bi, Z. (2024) · Advances in experimental medicine and biology · doi:10.1007/978-3-031-60183-5_10
Deep multimodal representation learning for noninvasive neural speech decoding
Cooney, C., Folli, R., & Coyle, D. (2024) · Elsevier eBooks · doi:10.1016/b978-0-323-95437-2.00012-4
Bluemarble: Bridging Latent Uncertainty in Articulatory-to-Speech Synthesis with a Learned Codebook
Um, S., Kim, M., Kim, D., & Kang, H.-G. (2024) · Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) · doi:10.1109/apsipaasc63619.2025.10849007
Impact of Temporal Precision on Speech Synthesis Accuracy From Electrocorticographic Brain Signals
Rabbani, Q., Fifer, M. S., Crone, N. E., & Moro-Velázquez, L. (2025) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49660.2025.10888813 · Cited by 2
Expanding Human Capabilities with Neurotechnology
Chaudhary, U. (2025) · Springer eBooks · doi:10.1007/978-3-031-78991-5_10
Self-paced silent speech brain-computer interface for device controlPreprint
Luo, S., Angrick, M., Coogan, C. G., Candrea, D. N., Wyse‐Sookoo, K., Schippers, A., Ganji, R., Milsap, G., Anderson, W. S., Gordon, C. R., et al. (2025) · medRxiv · doi:10.1101/2025.04.09.25325542 · Cited by 1
Self-citationDaskalou, A., Delibasis, K. K., Nousias, G., & Maglogiannis, I. G. (2025) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-96239-4_21
Restoring communication with speech brain–computer interfaces
Prakash, P. R., Luo, S., Crone, N. E., & Slutzky, M. W. (2025) · Elsevier eBooks · doi:10.1016/b978-0-443-24824-5.00015-6 · Cited by 2
Self-citationA Review of Language BCIs: Comparing Input Methods and Machine Learning Models
Wu, Y. (2025) · Theoretical and Natural Science · doi:10.54254/2753-8818/2025.ld26088
Wu, X., Hu, K., Fu, Z., & Zhang, D. (2025) · Imaging Neuroscience · doi:10.1162/imag.a.146 · Cited by 1
Lee, H., Hwang, J., Kim, K., Lee, G., Chung, C. K., & Im, C. (2025) · Machine Learning with Applications · doi:10.1016/j.mlwa.2025.100805
Silent Speech Recognition using Electromyography Signals
Jain, D., & Pal, A. (2025) · IEEE International Symposium on the Internet of Sounds (IS2) · doi:10.1109/is264627.2025.11284651
Comparison of single-channel EEG decoding performance in imagined speech tasks
Ma, X., Ding, Y., Zhao, K., Zhang, P., Tang, Y., Shi, J., Chen, D., Wu, D., & Tang, Z. (2026) · Brain Hemorrhages · doi:10.1016/j.hest.2026.01.001
Soft Multiaxial Strain Mapping Interface with AI-Driven Decoding for Silent Speech in Noise
Hong, S., Yoo, J., & Park, S. (2026) · Cyborg and Bionic Systems · doi:10.34133/cbsystems.0536
Diffusion-Based Brain-to-Speech Synthesis: Decoding Intracranial EEG into Natural Speech
Akaaboune, F. E., & Al-Radhi, M. S. (2026) · Journal of Physics Conference Series · doi:10.1088/1742-6596/3190/1/012008
Liu, B., Chen, G., Yin, L., & Liu, J. (2026) · Sensors · doi:10.3390/s26092749
Jang, S.-N., Park, H.-Y., Kim, S.-W., Roh, H., Kim, A.-H., Kam, T., & Jeong, J.-H. (2026) · ACM Symposium on Applied Computing · doi:10.1145/3748522.3779851
Deep learning methods for EEG-based speech classification and decoding: A PRISMA review
Sbeih, A. H., García–Gutiérrez, J., Bhushan, M., & Benavides, D. (2026) · Computer Speech & Language · doi:10.1016/j.csl.2026.102020 · Cited by 1
Dong, P., Song, Y., Li, Y., Djurić, P. M., & Yao, S. (2026) · Soft Science · doi:10.20517/ss.2026.38
Brain2voice 2.0: High-performance voice synthesis brain-computer interfacePreprint
Wairagkar, M., Srinivasan, A., Card, N. S., Singer-Clark, T., Hou, X., Iacobacci, C., Miller, L. M., Hochberg, L. R., Brandman, D. M., & Stavisky, S. D. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.06.30.735633 · Cited by 2
Duhair, H. T. M., Ibrahim, M. B. M., Alsayaydeh, J. A. J., Bacarra, R., Al-Mhiqani, M., & AlZubi, A. A. (2026) · Array · doi:10.1016/j.array.2026.101073
Pretraining with Masked Autoencoding Improves Speech Decoding from ECoG
Samar, A. R., & Makin, J. G. (2025) · International IEEE/EMBS Conference on Neural Engineering · doi:10.1109/ner61569.2025.11588938
A Lightweight LSTM-Based iEEG-to-Speech Decoder with Event-Level Unseen Neural Pattern Detection
Memon, S., & Al-Radhi, M. S. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37870-5_13
Metadata
- Authors
- Angrick, M., Herff, C., Mugler, E., Tate, M. C., Slutzky, M. W., Krusienski, D. J., & Schultz, T.
- Affiliations
- University of Bremen (DE)
- Northwestern University (US)
- Virginia Commonwealth University (US)
- Corresponding author
- Angrick, Miguel
- Herff, Christian
- ORCID
- Venue
- Journal of Neural Engineering
- Publisher
- IOP Publishing
- Published
- 2019-04-16
- Volume
- 16
- Issue
- 3
- Pages
- 036019
- ISSN
- 1741-2552 (ISSN-L)
- 1741-2560
- PMC
- PMC6822609
- OpenAlex
- W2952609044
- Cited by
- 225
- References
- 76
- Citations by year
- 2026: 18 · 2025: 36 · 2024: 35 · 2023: 28 · 2022: 34 · 2021: 30 · 2020: 29 · 2019: 15
- FWCI
- 12.32
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Journal of Neural Engineering (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Research Publications (Maastricht University) (submittedVersion, other-oa)
- Funders
- National Science Foundation (1608140)
- National Institutes of Health (F32DC015708)
- National Center for Advancing Translational Sciences (UL1TR000150)
- Bundesministerium für Bildung und Forschung (01GQ1602)
- Doris Duke Charitable Foundation (2011039)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Keywords
- speech synthesis, electrocorticography, neural speech decoding, 3D convolutional neural networks, speech production, brain-computer interface, augmentative and alternative communication
- MeSH terms
- Electrocorticography, Cerebral Cortex, Communication Aids for Disabled, Communication Devices for People with Disabilities, Humans, Photic Stimulation, Speech, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
