Neural speech recognition: continuous phoneme decoding using spatiotemporal representations of human cortical activity
Open accessThe DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
Abstract
OBJECTIVE: The superior temporal gyrus (STG) and neighboring brain regions play a key role in human language processing. Previous studies have attempted to reconstruct speech information from brain activity in the STG, but few of them incorporate the probabilistic framework and engineering methodology used in modern speech recognition systems. In this work, we describe the initial efforts toward the design of a neural speech recognition (NSR) system that performs continuous phoneme recognition on English stimuli with arbitrary vocabulary sizes using the high gamma band power of local field potentials in the STG and neighboring cortical areas obtained via electrocorticography. APPROACH: The system implements a Viterbi decoder that incorporates phoneme likelihood estimates from a linear discriminant analysis model and transition probabilities from an n-gram phonemic language model. Grid searches were used in an attempt to determine optimal parameterizations of the feature vectors and Viterbi decoder. MAIN RESULTS: The performance of the system was significantly improved by using spatiotemporal representations of the neural activity (as opposed to purely spatial representations) and by including language modeling and Viterbi decoding in the NSR system. SIGNIFICANCE: These results emphasize the importance of modeling the temporal dynamics of neural responses when analyzing their variations with respect to varying stimuli and demonstrate that speech recognition techniques can be successfully leveraged when decoding speech from neural signals. Guided by the results detailed in this work, further development of the NSR system could have applications in the fields of automatic speech recognition and neural prosthetics.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Random effects structure for confirmatory hypothesis testing: Keep it maximal
Barr, D. J., Levy, R., Scheepers, C., & Tily, H. J. (2013) · Journal of Memory and Language · doi:10.1016/j.jml.2012.11.001 · Cited by 10,328 · cited 1×
“In addition to the fi xed effect of phoneme time position, random intercepts and slopes were utilized for each stimulus [ 60 ] , which allowed the fi ts for each stimulus to vary in terms of overall classi fi cation accuracy and extent to which accuracy changes as a function of phoneme time position.”
MethodMixed-effects modeling with crossed random effects for subjects and items
Baayen, R. H., Davidson, D. J., & Bates, D. M. (2008) · Journal of Memory and Language · doi:10.1016/j.jml.2007.12.005 · Cited by 8,540
The cortical organization of speech processing
Hickok, G., & Poeppel, D. (2007) · Nature Reviews Neuroscience · doi:10.1038/nrn2113 · Cited by 5,495 · cited 1×
“The fewer number of relevant channels for subject C could be a result of electrode coverage over the language non-dominant hemisphere, although existing literature indicates that phonetic processing occurs bilaterally [ 38 ] .”
BackgroundDavis, S., & Mermelstein, P. (1980) · IEEE Transactions on Acoustics, Speech, and Signal Processing · doi:10.1109/tassp.1980.1163420 · Cited by 5,465
Categorical data analysis: Away from ANOVAs (transformation or not) and towards logit mixed models
Jaeger, T. F. (2008) · Journal of Memory and Language · doi:10.1016/j.jml.2007.11.007 · Cited by 3,317 · cited 1×
“For each feature type, we used the lme4 package within the R programming language [ 56, 57 ] to fi t a mixed effects logistic regression model with the associated data set to assess the effect that phoneme time position had on classi fi cation accuracy [ 58, 59 ] .”
MethodBrysbaert, M., & New, B. (2009) · Behavior Research Methods · doi:10.3758/brm.41.4.977 · Cited by 2,821
Human temporal lobe activation by speech and nonspeech sounds
Binder, J. R. (2000) · Cerebral Cortex · doi:10.1093/cercor/10.5.512 · Cited by 1,345
Phonetic Feature Encoding in Human Superior Temporal Gyrus
Mesgarani, N., Cheung, C., Johnson, K., & Chang, E. F. (2014) · Science · doi:10.1126/science.1245994 · Cited by 993 · cited 6×
“…conclusion is consistent with the theory that phonetic information is encoded more strongly in the STG than other information that is more variable between genders ( such as fundamental frequency ) [ 6 ] and suggests that data from multiple speakers can be used to effectively train an NSR system.”
BackgroundInfluentialSelf-citationFunctional organization of human sensorimotor cortex for speech articulation
Bouchard, K. E., Mesgarani, N., Johnson, K., & Chang, E. F. (2013) · Nature · doi:10.1038/nature11911 · Cited by 686 · cited 2×
“For these reasons, we applied eight bandpass Gaussian fi lters with logarithmically increasing center frequencies between 70 – 150 Hz and semi-logarithmically increasing bandwidths to the neural responses from each electrode channel [ 34 ] .”
BackgroundMethodSelf-citationReconstructing Speech from Human Auditory Cortex
Pasley, B. N., David, S. V., Mesgarani, N., Flinker, A., Shamma, S. A., Crone, N. E., Knight, R. T., & Chang, E. F. (2012) · PLoS Biology · doi:10.1371/journal.pbio.1001251 · Cited by 604 · cited 3×
“Previous research has shown that high gamma band activity ( 70 – 150 Hz ) correlates strongly with multi-unit fi ring processes in the brain [ 33 ] and is an effective representation of brain activity during speech processing [ 5, 6, 16, 34 ] .”
BackgroundSelf-citation
Cited by
Speech synthesis from neural decoding of spoken sentences
Anumanchipalli, G. K., Chartier, J., & Chang, E. F. (2019) · Nature · doi:10.1038/s41586-019-1119-1 · Cited by 883
Self-citationThe Cortical Organization of Syntax
Matchin, W., & Hickok, G. (2020) · Cerebral Cortex · doi:10.1093/cercor/bhz180 · Cited by 370
Speech synthesis from ECoG using densely connected 3D convolutional neural networks
Angrick, M., Herff, C., Mugler, E., Tate, M. C., Slutzky, M. W., Krusienski, D. J., & Schultz, T. (2019) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab0c59 · Cited by 225
Real-time decoding of question-and-answer speech dialogue using human cortical activity
Moses, D. A., Leonard, M. K., Makin, J. G., & Chang, E. F. (2019) · Nature Communications · doi:10.1038/s41467-019-10994-4 · Cited by 224
Self-citationTowards reconstructing intelligible speech from the human auditory cortex
Akbari, H., Khalighinejad, B., Herrero, J. L., Mehta, A. D., & Mesgarani, N. (2019) · Scientific Reports · doi:10.1038/s41598-018-37359-z · Cited by 221
Self-citation
Metadata
- Authors
- Moses, D. A., Mesgarani, N., Leonard, M. K., & Chang, E. F.
- Venue
- Journal of Neural Engineering
- Publisher
- IOP Publishing
- Published
- 2016-08-03
- Volume
- 13
- Issue
- 5
- Pages
- 056004
- ISSN
- 1741-2552 (ISSN-L)
- 1741-2560
- OpenAlex
- W2481854178
- Cited by
- 91
- References
- 18
- FWCI
- 7.75
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Neural speech recognition: continuous phoneme decoding using spatiotemporal representations of human cortical activity. https://app.cochat.ai/papers/10.1088/1741-2560/13/5/056004
- The paper
- Moses, D. A., Mesgarani, N., Leonard, M. K., & Chang, E. F. (2016). Neural speech recognition: continuous phoneme decoding using spatiotemporal representations of human cortical activity. Journal of Neural Engineering. https://doi.org/10.1088/1741-2560/13/5/056004
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.