Brain2Char: a deep architecture for decoding text from brain recordings
Open accessAbstract
Objective. Decoding language representations directly from the brain can enable new brain–computer interfaces (BCIs) for high bandwidth human–human and human–machine communication. Clinically, such technologies can restore communication in people with neurological conditions affecting their ability to speak. Approach . In this study, we propose a novel deep network architecture Brain2Char, for directly decoding text (specifically character sequences) from direct brain recordings (called electrocorticography, ECoG). Brain2Char framework combines state-of-the-art deep learning modules—3D Inception layers for multiband spatiotemporal feature extraction from neural data and bidirectional recurrent layers, dilated convolution layers followed by language model weighted beam search to decode character sequences, and optimizing a connectionist temporal classification loss. Additionally, given the highly non-linear transformations that underlie the conversion of cortical function to character sequences, we perform regularizations on the network’s latent representations motivated by insights into cortical encoding of speech production and artifactual aspects specific to ECoG data acquisition. To do this, we impose auxiliary losses on latent representations for articulatory movements, speech acoustics and session specific non-linearities. Main results. In three (out of four) participants reported here, Brain2Char achieves 10.6%, 8.5%, and 7.0% word error rates respectively on vocabulary sizes ranging from 1200 to 1900 words. Significance. These results establish a new end-to-end approach on decoding text from brain signals and demonstrate the potential of Brain2Char as a high-performance communication BCI.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
By using CTC loss, Sun et al. trained an RNN to decode character sequences from brain activity during sentence production.
“By using the connectionist temporal classification (CTC) loss119, which scores the loss between predictions from an input sequence (such as neural activity) and an output sequence (such as linguistic subunits) without requiring their alignment, Sun et al.109 trained an RNN to decode character sequences from brain activity during sentence production.”
From The speech neuroprosthesis (2024) · cites this paper as [109]
The Methods (Brain2Char architecture; page not provided) describe Bi-LSTM recurrent layers in the neural-feature encoder and CTC loss for decoding character sequences from brain recordings. But the text decoder itself uses dilated CNN layers, so describing the trained decoder simply as an RNN misstates that part of the architecture. (LLM, conf=0.86)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · see citing works
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning
Szegedy, C., Ioffe, S., Vanhoucke, V., & Alemi, A. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11231 · Cited by 4,636 · see citing works
The String-to-String Correction Problem
Wagner, R. A., & Fischer, M. J. (1974) · Journal of the ACM · doi:10.1145/321796.321811 · Cited by 3,079 · see citing works
Natural speech reveals the semantic maps that tile human cerebral cortex
Huth, A. G., de Heer, W. A., Griffiths, T. L., Theunissen, F. E., & Gallant, J. L. (2016) · Nature · doi:10.1038/nature17637 · Cited by 1,504 · see citing works
Predicting Human Brain Activity Associated with the Meanings of Nouns
Mitchell, T. M., Shinkareva, S. V., Carlson, A., Chang, K.-M., Malave, V. L., Mason, R. A., & Just, M. A. (2008) · Science · doi:10.1126/science.1152876 · Cited by 1,180 · see citing works
Stimulus onset quenches neural variability: A widespread cortical phenomenon
Churchland, M. M., Yu, B. M., Cunningham, J. P., Sugrue, L. P., Cohen, M. R., Corrado, G. S., Newsome, W. T., Clark, A. M., Hosseini, P., Scott, B. B., et al. (2010) · Nature Neuroscience · doi:10.1038/nn.2501 · Cited by 1,084 · see citing works
Phonetic Feature Encoding in Human Superior Temporal Gyrus
Mesgarani, N., Cheung, C., Johnson, K., & Chang, E. F. (2014) · Science · doi:10.1126/science.1245994 · Cited by 993 · see citing works
Self-citationSpeech synthesis from neural decoding of spoken sentences
Anumanchipalli, G. K., Chartier, J., & Chang, E. F. (2019) · Nature · doi:10.1038/s41586-019-1119-1 · Cited by 883 · see citing works
Self-citationReconstructing Speech from Human Auditory Cortex
Pasley, B. N., David, S. V., Mesgarani, N., Flinker, A., Shamma, S. A., Crone, N. E., Knight, R. T., & Chang, E. F. (2012) · PLoS Biology · doi:10.1371/journal.pbio.1001251 · Cited by 604 · see citing works
Self-citationPandarinath, C., Nuyujukian, P., Blabe, C. H., Sorice, B. L., Saab, J., Willett, F. R., Hochberg, L. R., Shenoy, K. V., & Henderson, J. M. (2017) · eLife · doi:10.7554/elife.18554 · Cited by 566 · cited 1× · see citing works
“The performance of best communicationBCI is at eight wordsmin−1 (Pandarinath et al 2017).”
BackgroundRedefining the role of Broca’s area in speech
Flinker, A., Korzeniewska, A., Shestyuk, A. Y., Franaszczuk, P. J., Dronkers, N. F., Knight, R. T., & Crone, N. E. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1414491112 · Cited by 487 · see citing works
Scalable Modified Kneser-Ney Language Model Estimation
Heafield, K., Pouzyrevsky, I., Clark, J. H., & Koehn, P. (2013) · Cited by 442 · see citing works
High Gamma Activity in Response to Deviant Auditory Stimuli Recorded Directly From Human Cortex
Edwards, E., Soltani, M., Deouell, L. Y., Berger, M. S., & Knight, R. T. (2005) · Journal of Neurophysiology · doi:10.1152/jn.00324.2005 · Cited by 334 · see citing works
Toward a universal decoder of linguistic meaning from brain activation
Pereira, F., Lou, B., Pritchett, B., Ritter, S., Gershman, S. J., Kanwisher, N., Botvinick, M., & Fedorenko, E. (2018) · Nature Communications · doi:10.1038/s41467-018-03068-4 · Cited by 324 · see citing works
Machine translation of cortical activity to text with an encoder–decoder framework
Makin, J. G., Moses, D. A., & Chang, E. F. (2020) · Nature Neuroscience · doi:10.1038/s41593-020-0608-8 · Cited by 321 · see citing works
Self-citationPutting big data to good use in neuroscience
B, T. J. S. A., Churchland, P. S., & Movshon, J. A. (2014) · Nature Neuroscience · doi:10.1038/nn.3839 · Cited by 307 · see citing works
Brain-to-text: decoding spoken phrases from phone representations in the brain
Herff, C., Heger, D., de Pesters, A., Telaar, D., Brunner, P., Schalk, G., & Schultz, T. (2015) · Frontiers in Neuroscience · doi:10.3389/fnins.2015.00217 · Cited by 297 · see citing works
Wehbe, L., Murphy, B., Talukdar, P. P., Fyshe, A., Ramdas, A., & Mitchell, T. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0112575 · Cited by 256 · see citing works
Perge, J. A., Homer, M. L., Malik, W. Q., Cash, S., Eskandar, E., Friehs, G., Donoghue, J. P., & Hochberg, L. R. (2013) · Journal of Neural Engineering · doi:10.1088/1741-2560/10/3/036004 · Cited by 243 · see citing works
Wav2Letter: an End-to-End ConvNet-based Speech Recognition SystemPreprint
Collobert, R., Puhrsch, C., & Synnaeve, G. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.03193 · Cited by 230 · see citing works
Speech synthesis from ECoG using densely connected 3D convolutional neural networks
Angrick, M., Herff, C., Mugler, E., Tate, M. C., Slutzky, M. W., Krusienski, D. J., & Schultz, T. (2019) · Journal of Neural Engineering · doi:10.1088/1741-2552/ab0c59 · Cited by 225 · see citing works
Towards reconstructing intelligible speech from the human auditory cortex
Akbari, H., Khalighinejad, B., Herrero, J. L., Mehta, A. D., & Mesgarani, N. (2019) · Scientific Reports · doi:10.1038/s41598-018-37359-z · Cited by 221 · see citing works
Encoding of Articulatory Kinematic Trajectories in Human Speech Sensorimotor Cortex
Chartier, J., Anumanchipalli, G. K., Johnson, K., & Chang, E. F. (2018) · Neuron · doi:10.1016/j.neuron.2018.04.031 · Cited by 208 · see citing works
Self-citationCortical gamma responses: Searching high and low
Crone, N. E., Korzeniewska, A., & Franaszczuk, P. J. (2011) · International Journal of Psychophysiology · doi:10.1016/j.ijpsycho.2010.10.013 · Cited by 201 · see citing works
Mugler, E. M., Patton, J. L., Flint, R. D., Wright, Z. A., Schuele, S. U., Rosenow, J., Shih, J. J., Krusienski, D. J., & Slutzky, M. W. (2014) · Journal of Neural Engineering · doi:10.1088/1741-2560/11/3/035015 · Cited by 192 · see citing works
Word pair classification during imagined speech using direct brain recordings
Martin, S., Brunner, P., Iturrate, I., Millán, J. D. R., Schalk, G., Knight, R. T., & Pasley, B. N. (2016) · Scientific Reports · doi:10.1038/srep25803 · Cited by 171 · see citing works
Towards Large-Scale, Human-Based, Mesoscopic Neurotechnologies
Chang, E. F. (2015) · Neuron · doi:10.1016/j.neuron.2015.03.037 · Cited by 163 · see citing works
Self-citationMugler, E. M., Tate, M. C., Livescu, K., Templer, J. W., Goldrick, M. A., & Slutzky, M. W. (2018) · The Journal of Neuroscience · doi:10.1523/jneurosci.1206-18.2018 · Cited by 96 · see citing works
Moses, D. A., Mesgarani, N., Leonard, M. K., & Chang, E. F. (2016) · Journal of Neural Engineering · doi:10.1088/1741-2560/13/5/056004 · Cited by 92 · see citing works
Self-citationPerformance sustaining intracortical neural prostheses
Nuyujukian, P., Kao, J. C., Fan, J. M., Stavisky, S. D., Ryu, S. I., & Shenoy, K. V. (2014) · Journal of Neural Engineering · doi:10.1088/1741-2560/11/6/066003 · Cited by 66 · see citing works
Toward a Speech Neuroprosthesis
Chang, E. F., & Anumanchipalli, G. K. (2020) · JAMA · doi:10.1001/jama.2019.19813 · Cited by 35 · see citing works
Self-citationDecoder calibration with ultra small current sample set for intracortical brain–machine interface
Zhang, P., Ma, X., Chen, L., Zhou, J., Wang, C., Li, W., & He, J. (2018) · Journal of Neural Engineering · doi:10.1088/1741-2552/aaa8a4 · Cited by 26 · see citing works
Modeling neural dynamics during speech production using a state space variational autoencoderPreprint
Sun, P., Moses, D. A., & Chang, E. F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.04024 · Cited by 3 · see citing works
Self-citationModeling neural dynamics during speech production using a state space variational autoencoder
Sun, P., Moses, D. A., & Chang, E. F. (2019) · International IEEE/EMBS Conference on Neural Engineering · doi:10.1109/ner.2019.8716931 · Cited by 2 · see citing works
Self-citation(2014) · doi:10.4230/oasics.slate.2021.16 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Metzger, S. L., Liu, J. R., Moses, D. A., Dougherty, M. E., Seaton, M. P., Littlejohn, K. T., Chartier, J., Anumanchipalli, G. K., Tu-Chan, A., Ganguly, K., et al. (2022) · Nature Communications · doi:10.1038/s41467-022-33611-3 · Cited by 153 · see citing works
Self-citationBrain-Computer Interface: Applications to Speech Decoding and Synthesis to Augment Communication
Luo, S., Rabbani, Q., & Crone, N. E. (2022) · Neurotherapeutics · doi:10.1007/s13311-022-01190-2 · Cited by 106 · see citing works
Silva, A. B., Littlejohn, K. T., Liu, J. R., Moses, D. A., & Chang, E. F. (2024) · Nature Reviews Neuroscience · doi:10.1038/s41583-024-00819-9 · Cited by 103 · see citing works
“…loss 119 , which scores the loss between predictions from an input sequence (such as neural activity) and an output sequence (such as linguistic subunits) without requiring their alignment, Sun et al. 109 trained an RNN to decode character sequences from brain activity during sentence production.”
InfluentialSelf-citationHigh-resolution neural recordings improve the accuracy of speech decoding
Duraivel, S., Rahimpour, S., Chiang, C.-H., Trumpis, M., Wang, C., Barth, K., Harward, S. C., Lad, S. P., Friedman, A. H., Southwell, D. G., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-42555-1 · Cited by 75 · see citing works
Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models
Berezutskaya, J., Freudenburg, Z. V., Vansteensel, M. J., Aarnoutse, E. J., Ramsey, N. F., & van Gerven, M. A. J. (2023) · Journal of Neural Engineering · doi:10.1088/1741-2552/ace8be · Cited by 42 · see citing works
The Layer 7 Cortical Interface: A Scalable and Minimally Invasive Brain–Computer Interface Platform
Hettick, M., Ho, E., Poole, A. J., Monge, M., Papageorgiou, D., Takahashi, K., LaMarca, M., Trietsch, D., Reed, K., Murphy, M., et al. (2022) · doi:10.1101/2022.01.02.474656 · Cited by 31 · see citing works
Tankus, A., Solomon, L., Aharony, Y., Faust-Socher, A., & Strauss, I. (2021) · Journal of Neural Engineering · doi:10.1088/1741-2552/ac3315 · Cited by 20 · see citing works
Direct Speech Reconstruction from Sensorimotor Brain Activity with Optimized Deep Learning Models
Berezutskaya, J., Freudenburg, Z. V., Vansteensel, M. J., Aarnoutse, E. J., Ramsey, N. F., & van Gerven, M. A. J. (2022) · doi:10.1101/2022.08.02.502503 · Cited by 42 · see citing works
- Authors
- Sun, P., Anumanchipalli, G. K., & Chang, E. F.
- Affiliations
- University of California, San Francisco (US)
- Corresponding author
- Chang, Edward F
- Venue
- Journal of Neural Engineering
- Publisher
- IOP Publishing
- Published
- 2020-12-01
- Volume
- 17
- Issue
- 6
- Pages
- 066015
- ISSN
- 1741-2552 (ISSN-L)
- 1741-2560
- PMC
- PMC9591243
- OpenAlex
- W3095432487
- Cited by
- 77
- References
- 35
- Citations by year
- 2026: 5 · 2025: 19 · 2024: 18 · 2023: 10 · 2022: 12 · 2021: 9 · 2020: 4
- FWCI
- 4.70
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Journal of Neural Engineering (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- arXiv (Cornell University) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Facebook (Sponsored Academic Research Agreement)
- National Institutes of Health (U01 NS098971)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Physical Sciences › Computer Science › Artificial Intelligence › Speech Recognition and Synthesis
- Physical Sciences › Computer Science › Human-Computer Interaction › Gaze Tracking and Assistive Technology
- Keywords
- brain-computer interface, electrocorticography, text decoding, bidirectional recurrent neural network, dilated convolution, connectionist temporal classification, articulatory movements, speech acoustics, speech production, word error rate
- MeSH terms
- Electrocorticography, Brain, Humans, Language, Speech, Brain-Computer Interfaces
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
