Generating Text from Functional Brain Images
Open accessThe DOI matches its Crossref record; 8 of the 8 references it cites were found in a registry or an index.
Abstract
Recent work has shown that it is possible to take brain images acquired during viewing of a scene and reconstruct an approximation of the scene from those images. Here we show that it is also possible to generate text about the mental content reflected in brain images. We began with images collected as participants read names of concrete items (e.g., "Apartment'') while also seeing line drawings of the item named. We built a model of the mental semantic representation of concrete concepts from text data and learned to map aspects of such representation to patterns of activation in the corresponding brain image. In order to validate this mapping, without accessing information about the items viewed for left-out individual brain images, we were able to generate from each one a collection of semantically pertinent words (e.g., "door," "window" for "Apartment''). Furthermore, we show that the ability to generate such words allows us to perform a classification task and thus validate our method quantitatively.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Beyond mind-reading: multi-voxel pattern analysis of fMRI data
Norman, K. A., Polyn, S. M., Detre, G. J., & Haxby, J. V. (2006) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2006.07.005 · Cited by 2,271
Self-citationDetecting awareness in the vegetative state
Owen, A. M., Coleman, M. R., Boly, M., Davis, M. H., Laureys, S., & Pickard, J. D. (2006) · Science · doi:10.1126/science.1130197 · Cited by 1,978
Machine learning classifiers and fMRI: A tutorial overview
Pereira, F., Mitchell, T., & Botvinick, M. (2009) · NeuroImage · doi:10.1016/j.neuroimage.2008.11.007 · Cited by 1,624
Self-citationPredicting Human Brain Activity Associated with the Meanings of Nouns
Mitchell, T. M., Shinkareva, S. V., Carlson, A., Chang, K.-M., Malave, V. L., Mason, R. A., & Just, M. A. (2008) · Science · doi:10.1126/science.1152876 · Cited by 1,180 · cited 8×
“More recently (Mitchell et al., 2008) and (Kay et al., 2008) have extended this approach applying sophisticated forward models to predict brain activation patterns induced by stimuli from outside the initial training set, thus enabling a more-open ended form of classification.”
BackgroundMethodInfluentialTopics in semantic representation
Griffiths, T. L., Steyvers, M., & Tenenbaum, J. B. (2007) · Psychological Review · doi:10.1037/0033-295x.114.2.211 · Cited by 1,122
Breaking the silence: Brain-computer interfaces (BCI) for communication and motor control
Birbaumer, N. (2006) · Psychophysiology · doi:10.1111/j.1469-8986.2006.00456.x · Cited by 722
Miyawaki, Y., Uchida, H., Yamashita, O., Sato, M.-A., Morito, Y., Tanabe, H. C., Sadato, N., & Kamitani, Y. (2008) · Neuron · doi:10.1016/j.neuron.2008.11.004 · Cited by 543 · cited 1×
“Concrete examples include reconstructing a simple visual stimulus (Miyawaki et al., 2008), a pattern of dots mentally visualized by the subject (Thirion et al., 2006) and generating a structural and semantic description of a stimulus scene, which could then be used to find a similar scene in a very…”
Decoding spoken words using local field potentials recorded from the cortical surface
Kellis, S., Miller, K., Thomson, K., Brown, R., House, P., & Greger, B. (2010) · Journal of Neural Engineering · doi:10.1088/1741-2560/7/5/056007 · Cited by 235 · cited 1×
“…over previous efforts to decode patterns of neural activation into language outputs, either letter-by-letter (Birbaumer, 2006) or word-by-word (Kellis et al., 2010), with possible clinical implications for conditions such as locked-in syndrome (Owen et al., 2006; this would require showing…”
Background
Cited by
Inferring mental states from neuroimaging data: from reverse inference to large-scale decoding
Poldrack, R. A. (2011) · Neuron · doi:10.1016/j.neuron.2011.11.001 · Cited by 821
Toward a universal decoder of linguistic meaning from brain activation
Pereira, F., Lou, B., Pritchett, B., Ritter, S., Gershman, S. J., Kanwisher, N., Botvinick, M., & Fedorenko, E. (2018) · Nature Communications · doi:10.1038/s41467-018-03068-4 · Cited by 324
Self-citationDistributional Semantics: Meaning Through Culture and Interaction
Contreras Kallens, P., & Christiansen, M. H. (2025) · Topics in Cognitive Science · doi:10.1111/tops.12771 · Cited by 17
“Pereira et al. (2011) reproduced these results using a Wikipedia-trained Topic Model, and Huth et al. (2016) used a co-occurrence matrix with a set of basic words from Wikipedia to construct a semantic map of the brain, showing what regions are activated when processing particular meanings.”
Metadata
- Authors
- Pereira, F., Detre, G., & Botvinick, M.
- Venue
- Frontiers in Human Neuroscience
- Publisher
- Frontiers Media SA
- Published
- 2011
- Volume
- 5
- Pages
- 72-72
- ISSN
- 1662-5161 (ISSN-L)
- OpenAlex
- W1990561581
- Cited by
- 49
- References
- 8
- FWCI
- 2.11
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.