Towards reconstructing intelligible speech from the human auditory cortex
Open accessPublished version of a preprint
The DOI matches its Crossref record; 82 of the 84 references it cites were found in a registry or an index.
Abstract
Auditory stimulus reconstruction is a technique that finds the best approximation of the acoustic stimulus from the population of evoked neural activity. Reconstructing speech from the human auditory cortex creates the possibility of a speech neuroprosthetic to establish a direct communication with the brain and has been shown to be possible in both overt and covert conditions. However, the low quality of the reconstructed speech has severely limited the utility of this method for brain-computer interface (BCI) applications. To advance the state-of-the-art in speech neuroprosthesis, we combined the recent advances in deep learning with the latest innovations in speech synthesis technologies to reconstruct closed-set intelligible speech from the human auditory cortex. We investigated the dependence of reconstruction accuracy on linear and nonlinear (deep neural network) regression methods and the acoustic representation that is used as the target of reconstruction, including auditory spectrogram and speech synthesis parameters. In addition, we compared the reconstruction accuracy from low and high neural frequency ranges. Our results show that a deep neural network model that directly estimates the parameters of a speech synthesizer from all neural frequencies achieves the highest subjective and objective scores on a digit recognition task, improving the intelligibility by 65% over the baseline method which used linear regression to reconstruct the auditory spectrogram. These results demonstrate the efficacy of deep learning and speech synthesis algorithms for designing the next generation of speech BCI systems, which not only can restore communications for paralyzed patients but also have the potential to transform human-computer interaction technologies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Neural activity in the STG encodes acoustic features — for example, recurring speech sound patterns in the mel-spectrogram — during speech perception53,54 and may be used to reconstruct perceived speech55,56.”
From The speech neuroprosthesis (2024) · cites this paper as [55]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
(2014) · Cited by 33,419
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
He, K., Zhang, X., Ren, S., & Sun, J. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.123 · Cited by 19,741
A fast learning algorithm for deep belief nets
Hinton, G. E., Osindero, S., & Teh, Y.-W. (2006) · Neural Computation · doi:10.1162/neco.2006.18.7.1527 · Cited by 16,336
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S. (2024) · arXiv (Cornell University) · doi:10.57702/o9raffed · Cited by 14,214
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A.-R., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., et al. (2012) · IEEE Signal Processing Magazine · doi:10.1109/msp.2012.2205597 · Cited by 10,260
Research works printed but not linked: 3
- Salza, P. L., Foti, E., Nebbia, L. & Oreglia, M. MOS and pair comparison combined methods for quality evaluation of text-to-speech systems. Acta Acust. united with Acust. 82, 650–656 (1996).
- Maas, A. L., Hannun, A. Y. & Ng, A. Y. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml 30, 3 (2013).
- Kingma, D. P. & Ba, J. Adam: A method for stochastic optimization. arXiv Prepr. arXiv1412.6980 (2014).
Cited by
Speech synthesis from neural decoding of spoken sentences
Anumanchipalli, G. K., Chartier, J., & Chang, E. F. (2019) · Nature · doi:10.1038/s41586-019-1119-1 · Cited by 883
“Meanwhile, decoding of auditory cortex responses has been more successful for speech sounds 30,31 in part because of the direct relationship between the auditory encoding of spectrotemporal information and the reconstructed spectrogram.”
BackgroundA Hybrid Deep Learning-Based Approach for Brain Tumor Classification
Raza, A., Ayub, H., Khan, J. A., Ahmad, I., S. Salama, A., Daradkeh, Y. I., Javeed, D., Ur Rehman, A., & Hamam, H. (2022) · Electronics · doi:10.3390/electronics11071146 · Cited by 354
Natural Language Processing for Smart Healthcare
Zhou, B., Yang, G., Shi, Z., & Ma, S. (2024) · IEEE Reviews in Biomedical Engineering · doi:10.1109/rbme.2022.3210270 · Cited by 230
Decoding speech perception from non-invasive brain recordings
Défossez, A., Caucheteux, C., Rapin, J., Kabeli, O., & King, J.-R. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00714-5 · Cited by 215
He, B., Yuan, H., Meng, J., & Gao, S. (2020) · Springer eBooks · doi:10.1007/978-3-030-43395-6_4 · Cited by 167
Metadata
- Authors
- Akbari, H., Khalighinejad, B., Herrero, J. L., Mehta, A. D., & Mesgarani, N.
- Affiliations
- Mortimer B. Zuckerman Mind Brain Behavior Institute (US)
- Feinstein Institute for Medical Research (US)
- Corresponding author
- Mesgarani, Nima
- Venue
- Scientific Reports
- Published
- 2019-01-29
- Volume
- 9
- Issue
- 1
- Pages
- 874-874
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC6351601
- OpenAlex
- W2913144232
- Cited by
- 221
- References
- 84
- Citations by year
- 2026: 11 · 2025: 25 · 2024: 33 · 2023: 28 · 2022: 28 · 2021: 35 · 2020: 29 · 2019: 32
- FWCI
- 18.01
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Scientific Reports (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- U.S. Department of Health & Human Services | NIH | National Institute on Deafness and Other Communication Disorders (DC014279)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Signal Processing › Speech and Audio Processing
- Keywords
- auditory cortex, speech reconstruction, brain-computer interface, deep neural network regression, speech intelligibility, digit recognition
- MeSH terms
- Deep Learning, Acoustic Stimulation, Algorithms, Auditory Cortex, Brain Mapping, Evoked Potentials, Auditory, Humans, Speech, Speech Intelligibility, Speech Perception, Neural Networks, Computer, Neural Prostheses
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Towards reconstructing intelligible speech from the human auditory cortex. https://app.cochat.ai/papers/10.1038/s41598-018-37359-z
- The paper
- Akbari, H., Khalighinejad, B., Herrero, J. L., Mehta, A. D., & Mesgarani, N. (2019). Towards reconstructing intelligible speech from the human auditory cortex. Scientific Reports. https://doi.org/10.1038/s41598-018-37359-z
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.