Generative Feedback Explains Distinct Brain Activity Codes for Seen and Mental Images
Open accessThe DOI matches its Crossref record; 29 of the 29 references it cites were found in a registry or an index.
Abstract
SUMMARY The relationship between mental imagery and vision is a long-standing problem in neuroscience. Currently, it is not known whether differences between the activity evoked during vision and reinstated during imagery reflect different codes for seen and mental images. To address this problem, we modeled mental imagery in the human brain as feedback in a hierarchical generative network. Such networks synthesize images by feeding abstract representations from higher to lower levels of the network hierarchy. When higher processing levels are less sensitive to stimulus variation than lower processing levels, as in the human brain, activity in low-level visual areas should encode variation in mental images with less precision than seen images. To test this prediction, we conducted an fMRI experiment in which subjects imagined and then viewed hundreds of spatially varying naturalistic stimuli. To analyze these data, we developed imagery-encoding models. These models accurately predicted brain responses to imagined stimuli and enabled accurate decoding of their position and content. They also allowed us to compare, for every voxel, tuning to seen and imagined spatial frequencies, as well as the location and size of receptive fields in visual and imagined space. We confirmed our prediction, showing that, in low-level visual areas, imagined spatial frequencies in individual voxels are reduced relative to seen spatial frequencies and that receptive fields in imagined space are larger than in visual space. These findings reveal distinct codes for seen and mental images and link mental imagery to the computational abilities of generative networks.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Jenkinson, M., Beckmann, C. F., Behrens, T. E. J., Woolrich, M. W., & Smith, S. M. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2011.09.015 · Cited by 11,314
Fischl, B. (2012) · NeuroImage · doi:10.1016/j.neuroimage.2012.01.021 · Cited by 9,526
Emergence of simple-cell receptive field properties by learning a sparse code for natural images
Olshausen, B. A., & Field, D. J. (1996) · Nature · doi:10.1038/381607a0 · Cited by 5,694 · cited 2×
“Previous results relating vision to inference have supplied evidence that representations in biological visual systems are adapted to the structure of the visual environment [68, 69, 73].”
BackgroundRao, R. P. N., & Ballard, D. H. (1999) · Nature Neuroscience · doi:10.1038/4580 · Cited by 5,483 · cited 1×
“In visual cortex, spontaneous dynamics [68], stimulus response nonlinearities [64, 69, 70], the encoding of prediction error [71, 72], and the structure of visual representations in low-level [73] and high-level [74] visual areas can be explained by modeling vision as forward inference in a generative network that emb…”
BackgroundA theory of cortical responses.
Friston, K. (2005) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2005.1622 · Cited by 4,578 · cited 1×
“One influential theory of vision holds that the brain uses such a world model to infer the probability of abstract features in the environment, given a visual stimulus [26, 27].”
BackgroundSUN database: Large-scale scene recognition from abbey to zoo
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., & Torralba, A. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5539970 · Cited by 3,283
Performance-optimized hierarchical models predict neural responses in higher visual cortex
Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1403112111 · Cited by 1,924
How does the brain solve visual object recognition?
DiCarlo, J. J., Zoccolan, D., & Rust, N. C. (2012) · Neuron · doi:10.1016/j.neuron.2012.01.010 · Cited by 1,790
Hierarchical Bayesian inference in the visual cortex
Lee, T. S., & Mumford, D. (2003) · Journal of the Optical Society of America A · doi:10.1364/josaa.20.001434 · Cited by 1,473 · cited 1×
“One influential theory of vision holds that the brain uses such a world model to infer the probability of abstract features in the environment, given a visual stimulus [26, 27].”
BackgroundMental Imagery: Functional Mechanisms and Clinical Applications
Pearson, J., Naselaris, T., Holmes, E. A., & Kosslyn, S. M. (2015) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2015.08.003 · Cited by 1,087 · cited 2×
“low-level visual areas is consistently weaker than in high-level visual areas [2, 17, 46, 47].”
BackgroundSelf-citation
Cited by
A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence
Allen, E. J., St-Yves, G., Wu, Y., Breedlove, J. L., Prince, J. S., Dowdle, L. T., Nau, M., Caron, B., Pestilli, F., Charest, I., et al. (2022) · Nature Neuroscience · doi:10.1038/s41593-021-00962-x · Cited by 577
Self-citationThe Border Between Seeing and Thinking
Block, N. (2023) · doi:10.1093/oso/9780197622223.001.0001 · Cited by 367
(2021) · doi:10.1093/texcom/tgab035 · Cited by 270
Subjective signal strength distinguishes reality from imagination
Dijkstra, N., & Fleming, S. M. (2023) · Nature Communications · doi:10.1038/s41467-023-37322-1 · Cited by 115
Why do imagery and perception look and feel so different?
Koenig-Robert, R., & Pearson, J. (2021) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2019.0703 · Cited by 104
Tasks and their role in visual neuroscience
Kay, K., Bonnen, K., Denison, R. N., Arcaro, M. J., & Barack, D. L. (2023) · Neuron · doi:10.1016/j.neuron.2023.03.022 · Cited by 61
Li, S., Zeng, X., Shao, Z., & Yu, Q. (2023) · The Journal of Neuroscience · doi:10.1523/jneurosci.0592-23.2023 · Cited by 23
The brain as image processor and generator: towards function-restoring brain-computer-interfaces
van Hoof, R. (2022) · doi:10.26481/dis.20220603rh · Cited by 1
Metadata
- Authors
- Breedlove, J. L., St-Yves, G., Olman, C. A., & Naselaris, T.
- Venue
- Current Biology
- Publisher
- Elsevier BV
- Published
- 2020-06
- Volume
- 30
- Issue
- 12
- Pages
- 2211-2224.e6
- ISSN
- 0960-9822 (ISSN-L)
- OpenAlex
- W3022265947
- Cited by
- 108
- References
- 29
- FWCI
- 7.76
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.