Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 54 of the 54 references it cites were found in a registry or an index.

Toward a universal decoder of linguistic meaning from brain activation

Open access

Abstract

Prior work decoding linguistic meaning from imaging data has been largely limited to concrete nouns, using similar stimuli for training and testing, from a relatively small number of semantic categories. Here we present a new approach for building a brain decoding system in which words and sentences are represented as vectors in a semantic space constructed from massive text corpora. By efficiently sampling this space to select training stimuli shown to subjects, we maximize the ability to generalize to new meanings from limited imaging data. To validate this approach, we train the system on imaging data of individual concepts, and show it can decode semantic vector representations from imaging data of sentences about a wide variety of both concrete and abstract topics from two separate datasets. These decoded representations are sufficiently detailed to distinguish even semantically similar sentences, and to capture the similarity structure of meaning relationships between sentences.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41467-018-03068-4

Cites

54 of 54 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Visualization of semantic space. Two-dimensional visualization of 80 regions of 300-dimensional semantic space, containing points for 80 of the 180 target words used in experiment 1 (in red), surrounded by 5 words from the same cluster (in black). The blue lines are an approximation of region boundaries as determined by the clustering algorithm
Fig. 1 Visualization of semantic space. Two-dimensional visualization of 80 regions of 300-dimensional semantic space, containing points for 80 of the 180 target words used in experiment 1 (in red), surrounded by 5 words from the same cluster (in black). The blue lines are an approximation of region boundaries as determined by the clustering algorithmCC BY · PubMed Central
Fig. 2 Decoder schematic. a The decoder is trained to take a brain image as input and output the corresponding text semantic vector, for many different image/vector pairs. b, c The decoder is applied to new brain images and outputs decoded semantic vectors, which are then evaluated against text semantic vectors. b A pairwise evaluation, which is correct if vectors decoded from two images are more similar to the text semantic vectors for their respective stimuli than to the alternative. c A rank evaluation, where the decoded vector is compared to the text semantic vectors for a range of stimuli
Fig. 2 Decoder schematic. a The decoder is trained to take a brain image as input and output the corresponding text semantic vector, for many different image/vector pairs. b, c The decoder is applied to new brain images and outputs decoded semantic vectors, which are then evaluated against text semantic vectors. b A pairwise evaluation, which is correct if vectors decoded from two images are more similar to the text semantic vectors for their respective stimuli than to the alternative. c A rank evaluation, where the decoded vector is compared to the text semantic vectors for a range of stimuliCC BY · PubMed Central
Fig. 3 Examples of stimuli from experiments 1, 2 and 3. a Examples of stimuli used for sample words (a noun, a verb, and an adjective) in experiment 1. Each word was presented in three paradigms (in a sentence, with an image, or with five related words), with multiple repetitions in each paradigm. Cardinal image courtesy of Stephen Wolfe. Barn swallow image courtesy of Malene Thyssen. b Examples of stimuli used in experiments 2 and 3. In experiment 2, each passage belongs to a subtopic (e.g., “clarinet”) of one of the 24 main topics (e.g. “musical instruments”); in experiment 3, there are three different passages for each one of the 24 main topics. Note that sentences are presented on the screen one at a time
Fig. 3 Examples of stimuli from experiments 1, 2 and 3. a Examples of stimuli used for sample words (a noun, a verb, and an adjective) in experiment 1. Each word was presented in three paradigms (in a sentence, with an image, or with five related words), with multiple repetitions in each paradigm. Cardinal image courtesy of Stephen Wolfe. Barn swallow image courtesy of Malene Thyssen. b Examples of stimuli used in experiments 2 and 3. In experiment 2, each passage belongs to a subtopic (e.g., “clarinet”) of one of the 24 main topics (e.g. “musical instruments”); in experiment 3, there are three different passages for each one of the 24 main topics. Note that sentences are presented on the screen one at a timeCC BY · PubMed Central
Fig. 4 Classification results. a Pairwise accuracy results in experiments 1 (n = 16, left), 2 (n = 8, middle), and 3 (n = 6, right). For experiment 1, we report the results for each of the three paradigms separately, as well as averaged across paradigms. For experiments 2 and 3, we report three measures: classifying sentences from (i) different topics (left), (ii) different passages within the same topic (middle), and (iii) the same passage (right). Each dot represents pairwise accuracy for an individual subject. Dots of the same color refer to the same individual across experiments. Filled dots (cf. empty dots) represent significant results. b Rank accuracy results in experiments 1 (180 word choices), 2 (384 sentence choices), and 3 (243 sentence choices). c The distribution of word-spotting results in experiment 2, on average across subjects, contrasted with the distribution under a simulated null model. d The same for experiment 3
Fig. 4 Classification results. a Pairwise accuracy results in experiments 1 (n = 16, left), 2 (n = 8, middle), and 3 (n = 6, right). For experiment 1, we report the results for each of the three paradigms separately, as well as averaged across paradigms. For experiments 2 and 3, we report three measures: classifying sentences from (i) different topics (left), (ii) different passages within the same topic (middle), and (iii) the same passage (right). Each dot represents pairwise accuracy for an individual subject. Dots of the same color refer to the same individual across experiments. Filled dots (cf. empty dots) represent significant results. b Rank accuracy results in experiments 1 (180 word choices), 2 (384 sentence choices), and 3 (243 sentence choices). c The distribution of word-spotting results in experiment 2, on average across subjects, contrasted with the distribution under a simulated null model. d The same for experiment 3CC BY · PubMed Central
Fig. 5 Distribution of correlation values between semantic vectors for stimulus sentences in experiments 2 and 3. a The distribution of correlation values between the text-derived semantic vectors for pairs of sentences from different topics (red), from different passages within the same topic (blue), and from the same passage (green) for experiments 2. b Same for experiment 3
Fig. 5 Distribution of correlation values between semantic vectors for stimulus sentences in experiments 2 and 3. a The distribution of correlation values between the text-derived semantic vectors for pairs of sentences from different topics (red), from different passages within the same topic (blue), and from the same passage (green) for experiments 2. b Same for experiment 3CC BY · PubMed Central
Fig. 6 Similarity structure between text-derived and decoded semantic vectors. a Experiment 2 (384 sentences): correlations between the text-derived vectors (left) and between text-derived and decoded vectors, averaged across subjects (right). The sentences are sorted so that sentences in the same passage are adjacent, passages in the same topic are adjacent, and related topics are adjacent (only topic labels are shown for readability). b Corresponding plots for experiment 3 (243 sentences)
Fig. 6 Similarity structure between text-derived and decoded semantic vectors. a Experiment 2 (384 sentences): correlations between the text-derived vectors (left) and between text-derived and decoded vectors, averaged across subjects (right). The sentences are sorted so that sentences in the same passage are adjacent, passages in the same topic are adjacent, and related topics are adjacent (only topic labels are shown for readability). b Corresponding plots for experiment 3 (243 sentences)CC BY · PubMed Central
Fig. 7 Distribution of informative voxels across the brain. a The fraction of subjects, out of the 16 subjects used in experiment 1, where each voxel was among the 5000 informative voxels that were used to train a decoder. b The fraction of the 5000 informative voxels belonging to each of the four networks described in the text, as well as the rest of the brain
Fig. 7 Distribution of informative voxels across the brain. a The fraction of subjects, out of the 16 subjects used in experiment 1, where each voxel was among the 5000 informative voxels that were used to train a decoder. b The fraction of the 5000 informative voxels belonging to each of the four networks described in the text, as well as the rest of the brainCC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 52 references

Cited by

See all 442

Metadata

Authors
Pereira, F., Lou, B., Pritchett, B., Ritter, S., Gershman, S. J., Kanwisher, N., Botvinick, M., & Fedorenko, E.
Affiliations
  • Siemens (United States) (US)
  • Massachusetts Institute of Technology (US)
  • Google DeepMind (United Kingdom) (GB)
  • Harvard University (US)
  • McGovern Institute for Brain Research (US)
Corresponding author
  • Pereira, Francisco
  • Fedorenko, Evelina
Venue
Nature Communications
Published
2018-03-06
Volume
9
Issue
1
Pages
963-963
ISSN
2041-1723 (ISSN-L)
OpenAlex
W2782213998
Cited by
324
References
52
Citations by year
2026: 23 · 2025: 49 · 2024: 45 · 2023: 46 · 2022: 38 · 2021: 39 · 2020: 39 · 2019: 37 · 2018: 8
FWCI
18.28
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Digital Access to Scholarship at Harvard (DASH) (Harvard University) (submittedVersion, cc-by)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
  • OSF Preprints (OSF Preprints) (submittedVersion)
Funders
  • Johns Hopkins University
  • Office of the Director of National Intelligence
  • National Institutes of Health (5k99hd057522-02)
  • Intelligence Advanced Research Projects Activity (FA8650-14-C-7358)
  • McGovern Institute for Brain Research, Massachusetts Institute of Technology
  • NIH Office of the Director
  • Eunice Kennedy Shriver National Institute of Child Health and Human Development (K99 HD-057522, DP1HD091947, R00HD057522)
  • Air Force Research Laboratory (FA8650-14-C-7358)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Neurobiology of Language and Bilingualism
  • Social Sciences › Psychology › Developmental and Educational Psychology › Reading and Literacy Development
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
semantic decoding, brain decoding, linguistic meaning, semantic space, semantic vectors, text corpora, sentence meaning
MeSH terms
Adult, Brain Mapping, Female, Humans, Magnetic Resonance Imaging, Male, Physical Stimulation, Semantics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Toward a universal decoder of linguistic meaning from brain activation. https://app.cochat.ai/papers/10.1038/s41467-018-03068-4
The paper
Pereira, F., Lou, B., Pritchett, B., Ritter, S., Gershman, S. J., Kanwisher, N., Botvinick, M., & Fedorenko, E. (2018). Toward a universal decoder of linguistic meaning from brain activation. Nature Communications. https://doi.org/10.1038/s41467-018-03068-4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.