Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
A fundamental challenge in neuroscience is to understand what structure in the world is represented in spatially distributed patterns of neural activity from multiple single-trial measurements. This is often accomplished by learning a simple, linear transformations between neural features and features of the sensory stimuli or motor task. While successful in some early sensory processing areas, linear mappings are unlikely to be ideal tools for elucidating nonlinear, hierarchical representations of higher-order brain areas during complex tasks, such as the production of speech by humans. Here, we apply deep networks to predict produced speech syllables from a dataset of high gamma cortical surface electric potentials recorded from human sensorimotor cortex. We find that deep networks had higher decoding prediction accuracy compared to baseline models. Having established that deep networks extract more task relevant information from neural data sets relative to linear models (i.e., higher predictive accuracy), we next sought to demonstrate their utility as a data analysis tool for neuroscience. We first show that deep network's confusions revealed hierarchical latent structure in the neural data, which recapitulated the underlying articulatory nature of speech motor control. We next broadened the frequency features beyond high-gamma and identified a novel high-gamma-to-beta coupling during speech production. Finally, we used deep networks to compare task-relevant information in different neural frequency bands, and found that the high-gamma band contains the vast majority of information relevant for the speech prediction task, with little-to-no additional contribution from lower-frequency amplitudes. Together, these results demonstrate the utility of deep networks as a data analysis tool for basic and applied neuroscience.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
The origin of extracellular fields and currents — EEG, ECoG, LFP and spikes
Buzsáki, G., Anastassiou, C. A., & Koch, C. (2012) · Nature Reviews Neuroscience · doi:10.1038/nrn3241 · Cited by 4,225 · cited 2×
“Being able to compare information across features is particularly useful for CSEPs which results from a variety of electrical processes in the brain [35].”
PENFIELD, W., & BOLDREY, E. (1937) · Brain · doi:10.1093/brain/60.4.389 · Cited by 4,188
Beta-band oscillations—Signalling the status quo?
Engel, A. K., & Fries, P. (2010) · Current Opinion in Neurobiology · doi:10.1016/j.conb.2010.02.015 · Cited by 2,953
Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning
Alipanahi, B., Delong, A., Weirauch, M. T., & Frey, B. J. (2015) · Nature Biotechnology · doi:10.1038/nbt.3300 · Cited by 2,873
Performance-optimized hierarchical models predict neural responses in higher visual cortex
Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1403112111 · Cited by 1,924
Learning to control a brain–machine interface for reaching and grasping by primates
Carmena, J. M., Lebedev, M. A., Crist, R. E., O'Doherty, J. E., Santucci, D. M., Dimitrov, D. F., Patil, P. G., Henriquez, C. S., & Nicolelis, M. A. L. (2003) · PLoS Biology · doi:10.1371/journal.pbio.0000042 · Cited by 1,761 · cited 1×
“Since deep networks are highly parameterized nonlinear models, their online interactions with learning may be more complex than typical methods [63].”
Different Origins of Gamma Rhythm and High-Gamma Activity in Macaque Visual Cortex
Ray, S., & Maunsell, J. H. R. (2011) · PLoS Biology · doi:10.1371/journal.pbio.1000610 · Cited by 1,012
Power-Law Scaling in the Brain Surface Electric Potential
Miller, K. J., Sorensen, L. B., Ojemann, J. G., & den Nijs, M. (2009) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000609 · Cited by 742
Michalareas, G., Vezoli, J., van Pelt, S., Schoffelen, J.-M., Kennedy, H., & Fries, P. (2016) · Neuron · doi:10.1016/j.neuron.2015.12.018 · Cited by 696
Functional organization of human sensorimotor cortex for speech articulation
Bouchard, K. E., Mesgarani, N., Johnson, K., & Chang, E. F. (2013) · Nature · doi:10.1038/nature11911 · Cited by 686 · cited 10×
“For each of the bands defined as: theta [4-7 Hz], alpha [8-14 Hz ], beta [ 15-29 15-29 Based on previous results [20–22], we focused on the electrodes in the ventral sensorimotor cortex (vSMC).”
InfluentialSelf-citationSingle-trial spike trains in parietal cortex reveal discrete steps during decision-making
Latimer, K. W., Yates, J. L., Meister, M. L. R., Huk, A. C., & Pillow, J. W. (2015) · Science · doi:10.1126/science.aaa4056 · Cited by 306 · cited 1×
“Indeed, in other contexts, aggregation has been shown to mask structure in neural signals [43].”
Brain-to-text: decoding spoken phrases from phone representations in the brain
Herff, C., Heger, D., de Pesters, A., Telaar, D., Brunner, P., Schalk, G., & Schultz, T. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00217 · Cited by 297 · cited 3×
“It has been shown that CSEPs contain information about motor control [20, 22, 23, 29, 31, 52] and variability [21].”
Oscillatory phase coupling coordinates anatomically dispersed functional cell assemblies
Canolty, R. T., Ganguly, K., Kennerley, S. W., Cadieu, C. F., Koepsell, K., Wallis, J. D., & Carmena, J. M. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1008306107 · Cited by 292
Pei, X., Barbour, D. L., Leuthardt, E. C., & Schalk, G. (2011) · Journal of Neural Engineering · doi:10.1088/1741-2560/8/4/046028 · Cited by 237
Decoding spoken words using local field potentials recorded from the cortical surface
Kellis, S., Miller, K., Thomson, K., Brown, R., House, P., & Greger, B. (2010) · Journal of Neural Engineering · doi:10.1088/1741-2560/7/5/056007 · Cited by 235 · cited 1×
“It has been shown that CSEPs contain information about motor control [20, 22, 23, 29, 31, 52] and variability [21].”
Cortical gamma responses: Searching high and low
Crone, N. E., Korzeniewska, A., & Franaszczuk, P. J. (2011) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2010.10.013 · Cited by 201
Top-Down Beta Enhances Bottom-Up Gamma
Richter, C. G., Thompson, W. H., Bosman, C. A., & Fries, P. (2017) · The Journal of Neuroscience · doi:10.1523/jneurosci.3771-16.2017 · Cited by 195
Mugler, E. M., Patton, J. L., Flint, R. D., Wright, Z. A., Schuele, S. U., Rosenow, J., Shih, J. J., Krusienski, D. J., & Slutzky, M. W. (2014) · Journal of Neural Engineering · doi:10.1088/1741-2560/11/3/035015 · Cited by 192 · cited 7×
“It has been shown that CSEPs contain information about motor control [20, 22, 23, 29, 31, 52] and variability [21].”
InfluentialMugler, E. M., Tate, M. C., Livescu, K., Templer, J. W., Goldrick, M. A., & Slutzky, M. W. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.1206-18.2018 · Cited by 96
Cited by
A comprehensive survey on applications of transformers for deep learning tasks
Islam, S., Elmekki, H., Elsebai, A., Bentahar, J., Drawel, N., Rjoub, G., & Pedrycz, W. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122666 · Cited by 487
Speech synthesis from ECoG using densely connected 3D convolutional neural networks
Angrick, M., Herff, C., Mugler, E., Tate, M. C., Slutzky, M. W., Krusienski, D. J., & Schultz, T. (2019) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab0c59 · Cited by 225
“To extract meaningful information from the ECoG signals, we calculated logarithmic power in the broadband-gamma frequency range (70–170 Hz, filter order 13 and 14, respectively), which is thought to largely reflect ensemble spiking [48] and contains localized information about movement and speech processes [49].”
MethodImagined speech can be decoded from low- and cross-frequency intracranial EEG features
Proix, T., Delgado Saa, J., Christen, A., Martin, S., Pasley, B. N., Knight, R. T., Tian, X., Poeppel, D., Doyle, W. K., Devinsky, O., et al. (2022) · Nature Communications · doi:10.1038/s41467-021-27725-3 · Cited by 212
Neural ensemble dynamics in dorsal motor cortex during speech in people with paralysis
Stavisky, S. D., Willett, F. R., Wilson, G. H., Murphy, B. A., Rezaii, P., Avansino, D. T., Memberg, W. D., Miller, J. P., Kirsch, R. F., Hochberg, L. R., et al. (2019) · eLife · doi:10.7554/elife.46015 · Cited by 105
“This observation is consistent with previous ECoG studies (Bouchard et al., 2013; Cheung et al., 2016; Livezey et al., 2019; Moses et al., 2019; Mugler et al., 2014), although the larger neural differences we observed between unvoiced k and p and the beginning of their voiced counterparts at the start of ga and ba sug…”
BackgroundResultDecoding spoken English from intracortical electrode arrays in dorsal precentral gyrus
Wilson, G. H., Stavisky, S. D., Willett, F. R., Avansino, D. T., Kelemen, J. N., Hochberg, L. R., Henderson, J. M., Druckmann, S., & Shenoy, K. V. (2020) · Journal of Neural Engineering · doi:10.1088/1741-2552/abbfef · Cited by 103
Metadata
- Authors
- Livezey, J. A., Bouchard, K. E., & Chang, E. F.
- Venue
- PLOS Computational Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2019-09-16
- Volume
- 15
- Issue
- 9
- Pages
- e1007091
- ISSN
- 1553-734X (ISSN-L)
- 1553-7358
- OpenAlex
- W2794504802
- Cited by
- 58
- References
- 20
- FWCI
- 4.73
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.