Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 77 of the 77 references it cites were found in a registry or an index.

A 7T fMRI dataset of synthetic images for out-of-distribution modeling of vision

Open access

Abstract

Large-scale visual neural datasets such as the Natural Scenes Dataset (NSD) are enabling models of the brain with performances beyond what was possible just a decade ago. However, because the stimuli of these datasets typically live within a common naturalistic visual distribution, they make it challenging to implement out-of-distribution (OOD) generalization tests crucial for the development of robust brain models. Here, we address this by releasing NSD-synthetic, a dataset of 7T fMRI responses from the same eight NSD participants for 284 synthetic images. We show that NSD-synthetic’s fMRI responses reliably encode stimulus-related information and are OOD with respect to NSD; that OOD generalization tests on NSD-synthetic reveal differences between brain models that are not detected in-distribution; and that the degree of OOD (quantified as the test data distance from the training data) is predictive of the magnitude of model failures. Together, NSD-synthetic enables OOD generalization tests that facilitate the development of more robust models of visual processing.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1038/s41467-026-69345-9

Cites

77 of 77 references verified0 citation errors · 0 not checkable

Statistics

Figures

Fig. 1 Stimuli and experimental design. a Example natural scenes from NSD-core. b Example stimulus images in NSD-synthetic. The images are divided into 8 classes, indicated by colored boxes. Every class is further divided into a varying number of subclasses, where each subclass consists of 4 images. For each subclass, we visualize one of the four images along with the subclass name. For visualization purposes, the images with the most peripheral words are not square cropped (Word4 Pos1, Word4 Pos5, Word6 Pos1, Word6 Pos5). c Trial design. Images were presented using a 2-s ON, 2-s OFF trial structure. A small central fixation dot changed luminance every 1.4 s. d Tasks. Participants performed two different tasks in alternating runs. In the fixation task, participants reported increments and decrements of the luminance of the fixation dot. In the one-back task, participants judged whether the currently presented image was identical to the previously presented image.
Fig. 1 Stimuli and experimental design. a Example natural scenes from NSD-core. b Example stimulus images in NSD-synthetic. The images are divided into 8 classes, indicated by colored boxes. Every class is further divided into a varying number of subclasses, where each subclass consists of 4 images. For each subclass, we visualize one of the four images along with the subclass name. For visualization purposes, the images with the most peripheral words are not square cropped (Word4 Pos1, Word4 Pos5, Word6 Pos1, Word6 Pos5). c Trial design. Images were presented using a 2-s ON, 2-s OFF trial structure. A small central fixation dot changed luminance every 1.4 s. d Tasks. Participants performed two different tasks in alternating runs. In the fixation task, participants reported increments and decrements of the luminance of the fixation dot. In the one-back task, participants judged whether the currently presented image was identical to the previously presented image.CC BY · PubMed Central
Fig. 2 Noise ceiling, univariate, and multivariate analyses reveal robust visual signals in NSD-synthetic. a Participant-wise noise ceiling signal-to-noise ratio (NCSNR), plotted on flattened cortical surfaces. White contours indicate regions based on the ‘streams’ region of interest (ROI) collection as provided in the NSD data release. b ROI-wise participant-average univariate responses for noise, natural/manipulated scenes, contrast modulation, and phase-coherence modulation (groupings indicated by vertical white lines). c ROI-wise responses for chromatic noise (Hue01–Hue16) and corresponding achromatic noise (Hue00). d ROI-wise responses for single words. For visualization purposes, the images with the most peripheral words are not square cropped (Word4 Pos1, Word4 Pos5, Word6 Pos1, Word6 Pos5). e ROI-wise responses for spiral gratings. f Representational similarity analysis (RSA) scores indicating the similarity of multivariate fMRI responses between participants and ROIs. Thin white lines separate groups of the eight participants within the same ROI. Cyan boxes indicate cross-participant comparisons within the same ROI.
Fig. 2 Noise ceiling, univariate, and multivariate analyses reveal robust visual signals in NSD-synthetic. a Participant-wise noise ceiling signal-to-noise ratio (NCSNR), plotted on flattened cortical surfaces. White contours indicate regions based on the ‘streams’ region of interest (ROI) collection as provided in the NSD data release. b ROI-wise participant-average univariate responses for noise, natural/manipulated scenes, contrast modulation, and phase-coherence modulation (groupings indicated by vertical white lines). c ROI-wise responses for chromatic noise (Hue01–Hue16) and corresponding achromatic noise (Hue00). d ROI-wise responses for single words. For visualization purposes, the images with the most peripheral words are not square cropped (Word4 Pos1, Word4 Pos5, Word6 Pos1, Word6 Pos5). e ROI-wise responses for spiral gratings. f Representational similarity analysis (RSA) scores indicating the similarity of multivariate fMRI responses between participants and ROIs. Thin white lines separate groups of the eight participants within the same ROI. Cyan boxes indicate cross-participant comparisons within the same ROI.CC BY · PubMed Central
Fig. 3 Multidimensional scaling (MDS) confirms brain responses to NSD-synthetic’s artificial stimuli are out-of-distribution with respect to natural scenes. a MDS embedding of single-trial fMRI responses from the NSD-synthetic scan session and two NSD-core scan sessions. Colored dots represent trials from the NSD-synthetic session, color-coded according to image class. Black and gray dots represent trials from two NSD-core sessions, respectively. b MDS embedding of trial-averaged fMRI responses to each of the 284 NSD-synthetic images. Colored borders indicate image class.
Fig. 3 Multidimensional scaling (MDS) confirms brain responses to NSD-synthetic’s artificial stimuli are out-of-distribution with respect to natural scenes. a MDS embedding of single-trial fMRI responses from the NSD-synthetic scan session and two NSD-core scan sessions. Colored dots represent trials from the NSD-synthetic session, color-coded according to image class. Black and gray dots represent trials from two NSD-core sessions, respectively. b MDS embedding of trial-averaged fMRI responses to each of the 284 NSD-synthetic images. Colored borders indicate image class.CC BY · PubMed Central
Fig. 4 Brain encoding models exhibit reduced performance when tested out-of-distribution on NSD-synthetic. We built AlexNet-based encoding models and tested them in-distribution (ID) on NSD-core as well as out-of-distribution (OOD) on NSD-synthetic. This figure shows results averaged across participants on flattened cortical surfaces. a Model explained variance (r2). b Noise ceiling (r2). c Model explained variance normalized by the noise ceiling. To reduce false positives, for each vertex, we averaged results from only those participants with a noise ceiling greater than 0.3 (vertices for which no participant has a noise ceiling above 0.3 are not shown). d Difference between ID and OOD noise-ceiling-normalized explained variance. For each vertex, we averaged results from only those participants with noise ceiling greater than 0.3 for both NSD-core and NSD-synthetic. The asterisk indicates a significant difference between the ID and OOD noise-ceiling-normalized explained variance scores (P = 10−7, paired sample t-test, one-sided, N = 8 participants). e Scatterplots of vertex-wise explained variance scores against their noise ceilings for ID (NSD-core) and OOD (NSD-synthetic) generalization tests, color coded by the visual stream that each vertex belongs to. For each vertex, we plotted results from only those participants with noise ceiling greater than 0.3 for both NSD-core and NSD-synthetic (same as in d). Black crosses indicate median scores across all vertices.
Fig. 4 Brain encoding models exhibit reduced performance when tested out-of-distribution on NSD-synthetic. We built AlexNet-based encoding models and tested them in-distribution (ID) on NSD-core as well as out-of-distribution (OOD) on NSD-synthetic. This figure shows results averaged across participants on flattened cortical surfaces. a Model explained variance (r2). b Noise ceiling (r2). c Model explained variance normalized by the noise ceiling. To reduce false positives, for each vertex, we averaged results from only those participants with a noise ceiling greater than 0.3 (vertices for which no participant has a noise ceiling above 0.3 are not shown). d Difference between ID and OOD noise-ceiling-normalized explained variance. For each vertex, we averaged results from only those participants with noise ceiling greater than 0.3 for both NSD-core and NSD-synthetic. The asterisk indicates a significant difference between the ID and OOD noise-ceiling-normalized explained variance scores (P = 10−7, paired sample t-test, one-sided, N = 8 participants). e Scatterplots of vertex-wise explained variance scores against their noise ceilings for ID (NSD-core) and OOD (NSD-synthetic) generalization tests, color coded by the visual stream that each vertex belongs to. For each vertex, we plotted results from only those participants with noise ceiling greater than 0.3 for both NSD-core and NSD-synthetic (same as in d). Black crosses indicate median scores across all vertices.CC BY · PubMed Central
Fig. 5 Out-of-distribution generalization tests reveal differences between brain encoding models not detected by in-distribution tests. We built deep-neural-network-based encoding models and tested them in-distribution (ID) on NSD-core as well as out-of-distribution (OOD) on NSD-synthetic. This figure shows results averaged across participants on flattened cortical surfaces. To reduce false positives, for each vertex we averaged results from only those participants with a noise ceiling greater than 0.3 (vertices for which no participant has a noise ceiling above 0.3 are not shown). a AlexNet encoding models’ explained variance normalized by the noise ceiling. b vit_b_32 encoding models’ explained variance normalized by the noise ceiling. c Difference between AlexNet and vit_b_32’s noise-ceiling-normalized explained variance. d Difference between MoCo and ResNet50’s noise-ceiling-normalized explained variance.
Fig. 5 Out-of-distribution generalization tests reveal differences between brain encoding models not detected by in-distribution tests. We built deep-neural-network-based encoding models and tested them in-distribution (ID) on NSD-core as well as out-of-distribution (OOD) on NSD-synthetic. This figure shows results averaged across participants on flattened cortical surfaces. To reduce false positives, for each vertex we averaged results from only those participants with a noise ceiling greater than 0.3 (vertices for which no participant has a noise ceiling above 0.3 are not shown). a AlexNet encoding models’ explained variance normalized by the noise ceiling. b vit_b_32 encoding models’ explained variance normalized by the noise ceiling. c Difference between AlexNet and vit_b_32’s noise-ceiling-normalized explained variance. d Difference between MoCo and ResNet50’s noise-ceiling-normalized explained variance.CC BY · PubMed Central
Fig. 6 The degree of out-of-distribution is informative of brain model failures. We trained vision-transformer-based encoding models on NSD-core, and tested them independently for each NSD-synthetic image class. a MDS embedding of trial-averaged fMRI responses (aggregated across participants) from NSD-core (gray dots) and from NSD-synthetic (colored dots, color-coded according to image class). b Encoding models’ explained variance for each NSD-synthetic image class, normalized by the noise ceiling. This figure shows results averaged across participants on flattened cortical surfaces. To reduce false positives, for each vertex we averaged results from only those participants with a noise ceiling greater than 0.3 (vertices for which no participant has a noise ceiling above 0.3 are not shown). c Scatterplot indicating the relationship between the degree of out-of-distribution (OOD)–that is, the degree of distributional distance–of fMRI responses and the encoding accuracy scores, across NSD-synthetic image classes. The x-axis represents the Euclidean distance in MDS embedding space between fMRI responses for each NSD-synthetic image class and NSD-core (i.e., the distances in a). The y-axis represents the noise-ceiling normalized encoding accuracy of encoding models tested on each NSD-synthetic image class (i.e., the mean scores in b). d Encoding models’ zero-shot identification scores of the recorded fMRI response stimulus images, for each NSD-synthetic image class, on violin plots. The y-axis represents the correct image rank, that is, the position at which the correct image appears among the models’ choices (choosing among 8 randomly selected images per NSD-synthetic image class, for a total of 64 images; the final identification scores reflect the average across 1000 analysis iterations, each with a different random selection of 8 images per image class), with lower ranks indicating a more robust image identification. The colored violin shapes indicate the distribution of observations, with wider sections corresponding to higher density of values. Small transparent dots indicate the correct image rank for single images and participants. Large opaque dots indicate the average correct image rank across all participants and images within each NSD-synthetic image class, with the corresponding score reported above the dot. Horizontal dashed lines indicate chance identification scores. e Scatterplot indicating the relationship between the degree of OOD of fMRI responses and the zero-shot image identification scores, across NSD-synthetic image classes. The x-axis represents the Euclidean distance in MDS embedding space between fMRI responses for each NSD-synthetic image class and NSD-core (i.e., the distances in a). The y-axis represents the average correct image rank across all images within each NSD-synthetic image class (i.e., the average scores in d). Large opaque and small transparent dots indicate participant-average and single participant scores, respectively. c, e The asterisk indicates a significant correlation between response distance scores and encoding model performance (P < 0.001, one-sample t-test, one-sided, N = 8 participants).
Fig. 6 The degree of out-of-distribution is informative of brain model failures. We trained vision-transformer-based encoding models on NSD-core, and tested them independently for each NSD-synthetic image class. a MDS embedding of trial-averaged fMRI responses (aggregated across participants) from NSD-core (gray dots) and from NSD-synthetic (colored dots, color-coded according to image class). b Encoding models’ explained variance for each NSD-synthetic image class, normalized by the noise ceiling. This figure shows results averaged across participants on flattened cortical surfaces. To reduce false positives, for each vertex we averaged results from only those participants with a noise ceiling greater than 0.3 (vertices for which no participant has a noise ceiling above 0.3 are not shown). c Scatterplot indicating the relationship between the degree of out-of-distribution (OOD)–that is, the degree of distributional distance–of fMRI responses and the encoding accuracy scores, across NSD-synthetic image classes. The x-axis represents the Euclidean distance in MDS embedding space between fMRI responses for each NSD-synthetic image class and NSD-core (i.e., the distances in a). The y-axis represents the noise-ceiling normalized encoding accuracy of encoding models tested on each NSD-synthetic image class (i.e., the mean scores in b). d Encoding models’ zero-shot identification scores of the recorded fMRI response stimulus images, for each NSD-synthetic image class, on violin plots. The y-axis represents the correct image rank, that is, the position at which the correct image appears among the models’ choices (choosing among 8 randomly selected images per NSD-synthetic image class, for a total of 64 images; the final identification scores reflect the average across 1000 analysis iterations, each with a different random selection of 8 images per image class), with lower ranks indicating a more robust image identification. The colored violin shapes indicate the distribution of observations, with wider sections corresponding to higher density of values. Small transparent dots indicate the correct image rank for single images and participants. Large opaque dots indicate the average correct image rank across all participants and images within each NSD-synthetic image class, with the corresponding score reported above the dot. Horizontal dashed lines indicate chance identification scores. e Scatterplot indicating the relationship between the degree of OOD of fMRI responses and the zero-shot image identification scores, across NSD-synthetic image classes. The x-axis represents the Euclidean distance in MDS embedding space between fMRI responses for each NSD-synthetic image class and NSD-core (i.e., the distances in a). The y-axis represents the average correct image rank across all images within each NSD-synthetic image class (i.e., the average scores in d). Large opaque and small transparent dots indicate participant-average and single participant scores, respectively. c, e The asterisk indicates a significant correlation between response distance scores and encoding model performance (P < 0.001, one-sample t-test, one-sided, N = 8 participants).CC BY · PubMed Central
Fig. 7 NSD enables out-of-distribution tests of brain models across both naturalistic and synthetic stimulus images. We trained vision-transformer-based encoding models on NSD-core, and tested their generalization across three conditions: in-distribution (ID) on NSD-core, naturalistic out-of-distribution (OOD) on NSD-core, and synthetic OOD on NSD-synthetic. a MDS embedding of trial-averaged fMRI responses (aggregated across participants) for the train images (gray dots), for the ID test images (orange dots), for the naturalistic OOD test images (yellow dots), and for the synthetic OOD test images (turquoise dots). b Encoding models’ explained variance for the three test conditions, normalized by the noise ceiling. This figure shows results averaged across participants on flattened cortical surfaces. To reduce false positives, for each vertex we averaged results from only those participants with a noise ceiling greater than 0.3 for both NSD-core and NSD-synthetic (vertices for which no participant has a noise ceiling above 0.3 are not shown). c Encoding models’ zero-shot identification scores of the recorded fMRI response stimulus images, for each of the three test conditions, on violin plots. The y-axis represents the correct image rank, that is, the position at which the correct image appears among the models’ choices (choosing among all 852 images from the three test splits), with lower ranks indicating a more robust image identification. The colored violin shapes indicate the distribution of observations, with wider sections corresponding to higher density of values. Small transparent dots indicate the correct image rank for single images and participants. Large opaque dots indicate the average correct image rank across all participants and images within each test condition, with the corresponding score reported above the dot. Horizontal dashed lines indicate chance identification scores.
Fig. 7 NSD enables out-of-distribution tests of brain models across both naturalistic and synthetic stimulus images. We trained vision-transformer-based encoding models on NSD-core, and tested their generalization across three conditions: in-distribution (ID) on NSD-core, naturalistic out-of-distribution (OOD) on NSD-core, and synthetic OOD on NSD-synthetic. a MDS embedding of trial-averaged fMRI responses (aggregated across participants) for the train images (gray dots), for the ID test images (orange dots), for the naturalistic OOD test images (yellow dots), and for the synthetic OOD test images (turquoise dots). b Encoding models’ explained variance for the three test conditions, normalized by the noise ceiling. This figure shows results averaged across participants on flattened cortical surfaces. To reduce false positives, for each vertex we averaged results from only those participants with a noise ceiling greater than 0.3 for both NSD-core and NSD-synthetic (vertices for which no participant has a noise ceiling above 0.3 are not shown). c Encoding models’ zero-shot identification scores of the recorded fMRI response stimulus images, for each of the three test conditions, on violin plots. The y-axis represents the correct image rank, that is, the position at which the correct image appears among the models’ choices (choosing among all 852 images from the three test splits), with lower ranks indicating a more robust image identification. The colored violin shapes indicate the distribution of observations, with wider sections corresponding to higher density of values. Small transparent dots indicate the correct image rank for single images and participants. Large opaque dots indicate the average correct image rank across all participants and images within each test condition, with the corresponding score reported above the dot. Horizontal dashed lines indicate chance identification scores.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 77 references

Research works printed but not linked: 7

  1. Davison, M. L. & Sireci, S. G. Multidimensional Scaling. Handb. Appl. Multivar. Stat. Math. Modeling 323, 352 (2000).
  2. He, K., Zhang, X., Ren, S. & Sun, J. Deep Residual Learning for Image Recognition. Proc. IEEE Conf. Computer Vis. Pattern Recognit. 770, 778 (2016).
  3. Zbontar, J., Jing, L., Misra, I., LeCun, Y. & Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. in Proceedings of the 38th international conference on machine learning (eds Meila, M. & Zhang, T.) vol. 139, 12310–12320 (PMLR, 2021).
  4. Schaeffer, R. et al. Does maximizing neural regression scores teach us about the brain? in UniReps: 2nd edition of the workshop on unifying representations in neural models (2024).
  5. Cheng, F. L. et al. Reconstructing visual illusory experiences from human brain activity. Sci. Adv. 9, eadj3906 (2023). · DOI
  6. Singer, J. J. D., Cichy, R. M. & Hebart, M. N. The spatiotemporal neural dynamics of object recognition for natural images and line drawings. J. Neurosci. 43, 484–500 (2023). · doi:10.1523/jneurosci.1546-22.2022
  7. Kay, K. N. & Yeatman, J. D. Bottom-up and top-down computations in word- and face-selective cortex. eLife 6, e22341 (2017). · DOI

Cited by

See all 6

Metadata

Authors
Gifford, A. T., Cichy, R. M., Naselaris, T., & Kay, K.
Affiliations
  • Humboldt-Universität zu Berlin (DE)
  • University of Minnesota (US)
Corresponding author
  • Gifford, Alessandro T.
  • Kay, Kendrick
Venue
Nature Communications
Published
2026-02-09
Volume
17
Issue
1
Pages
1589-1589
ISSN
2041-1723 (ISSN-L)
OpenAlex
W7128426427
Cited by
6
References
77
Citations by year
2026: 6
FWCI
24.71
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Communications (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PubMed Central (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University)
Funders
National Eye Institute (R01EY034118)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
  • Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Advanced MRI Techniques and Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Visual Attention and Saliency Detection
Keywords
out-of-distribution generalization, naturalistic visual stimuli, 7T fMRI, synthetic images, visual cortex responses, stimulus encoding
MeSH terms
Brain, Brain Mapping, Humans, Magnetic Resonance Imaging, Models, Neurological, Photic Stimulation, Vision, Ocular, Visual Cortex, Visual Perception

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A 7T fMRI dataset of synthetic images for out-of-distribution modeling of vision. https://app.cochat.ai/papers/10.1038/s41467-026-69345-9
The paper
Gifford, A. T., Cichy, R. M., Naselaris, T., & Kay, K. (2026). A 7T fMRI dataset of synthetic images for out-of-distribution modeling of vision. Nature Communications. https://doi.org/10.1038/s41467-026-69345-9

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.