Reconstructing Visual Experiences from Brain Activity Evoked by Natural Movies
Open accessAbstract
Quantitative modeling of human brain activity can provide crucial insights about cortical representations [1, 2] and can form the basis for brain decoding devices [3-5]. Recent functional magnetic resonance imaging (fMRI) studies have modeled brain activity elicited by static visual patterns and have reconstructed these patterns from brain activity [6-8]. However, blood oxygen level-dependent (BOLD) signals measured via fMRI are very slow [9], so it has been difficult to model brain activity elicited by dynamic stimuli such as natural movies. Here we present a new motion-energy [10, 11] encoding model that largely overcomes this limitation. The model describes fast visual information and slow hemodynamics by separate components. We recorded BOLD signals in occipitotemporal visual cortex of human subjects who watched natural movies and fit the model separately to individual voxels. Visualization of the fit models reveals how early visual areas represent the information in movies. To demonstrate the power of our approach, we also constructed a Bayesian decoder [8] by combining estimated encoding models with a sampled natural movie prior. The decoder provides remarkable reconstructions of the viewed movies. These results demonstrate that dynamic brain activity measured under naturalistic conditions can be decoded using current fMRI technology.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Neurotechnologies can very crudely reconstruct conscious and unconscious mental imagery.
“They can also very crudely reconstruct conscious and unconscious mental imagery. 24”
From AI and Neurotechnology (2023) · cites this paper as [24]
The Discussion reports reconstructions of dynamic perceptual experiences from brain activity while participants viewed natural movies. It says reconstruction of visual imagery might be possible and decoding dreams or hallucinations could potentially be possible; it does not report having reconstructed conscious or unconscious mental imagery. (LLM, conf=0.88)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Regression shrinkage and selection via the lasso
Tibshirani, R. (1996) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1996.tb02080.x · Cited by 51,482 · see citing works
Decoding the visual and subjective contents of the human brain
Kamitani, Y., & Tong, F. (2005) · Nature Neuroscience · doi:10.1038/nn1444 · Cited by 1,829 · see citing works
Jones, J. P., & Palmer, L. A. (1987) · Journal of Neurophysiology · doi:10.1152/jn.1987.58.6.1233 · Cited by 1,658 · see citing works
Visual field maps in human cortex.
Wandell, B. A., Dumoulin, S. O., & Brewer, A. A. (2007) · Neuron · doi:10.1016/j.neuron.2007.10.012 · Cited by 1,231 · cited 1× · see citing works
“The spatial receptive fields of individual voxels were spatially localized (Figures 2F and 2G, left) and were organized retinotopically (Figures 2H and 2I), as reported in previous fMRI studies [12, 19–23].”
Flagged first, then by citations
Research works printed but not linked: 42
- [1]Wu MC David SV Gallant JL 2006 Complete functional characterization of sensory neurons by system identification Annu Rev Neurosci 29 477 505 16776594
- [2]Naselaris T Kay KN Nishimoto S Gallant JL 2011 Encoding and decoding in fMRI Neuroimage 56 400 410 20691790 PMC3037423
- [4]Haynes JD Rees G 2006 Decoding mental states from brain activity in humans Nat Rev Neurosci 7 523 534 16791142
- [5]Kay KN Gallant JL 2009 I can see what you see Nat Neurosci 12 245 19238184
- [6]Thirion B Duchesnay E Hubbard E Dubois J Poline JB Lebihan D Dehaene S 2006 Inverse retinotopy: inferring the visual content of images from brain activation patterns Neuroimage 33 1104 1116 17029988
- [7]Miyawaki Y Uchida H Yamashita O Sato MA Morito Y Tanabe HC Sadato N Kamitani Y 2008 Visual image reconstruction from human brain activity using a combination of multiscale local image decoders Neuron 60 915 929 19081384
- [8]Naselaris T Prenger RJ Kay KN Oliver M Gallant JL 2009 Bayesian reconstruction of natural images from human brain activity Neuron 63 902 915 19778517 PMC5553889
- [10]Adelson EH Bergen JR 1985 Spatiotemporal energy models for the perception of motion J Opt Soc Am A 2 284 299 3973762
- [11]Watson AB Ahumada Jr 1985 Model of human visual-motion sensing J Opt Soc Am A 2 322 341 3973764
- [12]Engel SA Rumelhart DE Wandell BA Lee AT Glover GH Chichilnisky EJ Shadlen MN 1994 fMRI of human visual cortex Nature 369 525 8031403
- [13]Logothetis NK 2008 What we can do and what we cannot do with fMRI Nature 453 869 878 18548064
- [14]Kriegeskorte N Simmons WK Bellgowan PS Baker CI 2009 Circular analysis in systems neuroscience: the dangers of double dipping Nat Neurosci 12 535 540 19396166 PMC2841687
- [17]Friston KJ Frith CD Turner R Frackowiak RS 1995 Characterizing evoked hemodynamics with fMRI Neuroimage 2 157 165 9343598
- [18]Boynton GM Engel SA Glover GH Heeger DJ 1996 Linear systems analysis of functional magnetic resonance imaging in human V1 J Neurosci 16 4207 4221 8753882 PMC6579007
- [19]Kay KN Naselaris T Prenger RJ Gallant JL 2008 Identifying natural images from human brain activity Nature 452 352 355 18322462 PMC3556484
- [20]Sereno MI Dale AM Reppas JB Kwong KK Belliveau JW Brady TJ Rosen BR Tootell RBH 1995 Borders of multiple visual areas in humans revealed by functional magnetic resonance imaging Science 268 889 893 7754376
- [21]DeYoe EA Carman GJ Bandettini P Glickman S Wieser J Cox R Miller D Neitz J 1996 Mapping striate and extrastriate visual areas in human cerebral cortex Proc Natl Acad Sci U S A 93 2382 2386 8637882 PMC39805
- [23]Dumoulin SO Wandell BA 2008 Population receptive field estimates in human visual cortex Neuroimage 39 647 660 17977024 PMC3073038
- [24]Singh KD Smith AT Greenlee MW 2000 Spatiotemporal frequency and direction sensitivities of human visual areas measured using fMRI Neuroimage 12 550 564 11034862
- [25]Henriksson L Nurminen L Hyvarinen A Vanni S 2008 Spatial frequency tuning in human retinotopic visual areas J Vis 8 5 1 13 19146347
- [26]Kelly DH 1984 Retinal inhomogeneity. I. Spatiotemporal contrast sensitivity J Opt Soc Am A 1 107 113 6699746
- [27]McKee SP Nakayama K 1984 The detection of motion in the peripheral visual field Vision Res 24 25 32 6695503
- [28]Orban GA Van Calenbergh F De Bruyn B Maes H 1985 Velocity discrimination in central and peripheral visual field J Opt Soc Am A 2 1836 1847 4067694
- [29]Orban GA Kennedy H Bullier J 1986 Velocity sensitivity and direction selectivity of neurons in areas V1 and V2 of the monkey: influence of eccentricity J Neurophysiol 56 462 480 3760931
- [30]Yu HH Verma R Yang Y Tibballs HA Lui LL Reser DH Rosa MG 2010 Spatial and temporal frequency tuning in striate cortex: functional uniformity and specializations related to receptive field eccentricity Eur J Neurosci 31 1043 1062 20377618
- [31]Domingos P Heckerman D Mannila H Pregibon D Uthurusamy R 1997 Why does bagging work? A Bayesian account and its implications Proceedings of the Third International Conference on Knowledge Discovery and Data Mining 155 158
- [32]Eckert MP Buchsbaum G 1993 Efficient coding of natural time varying images in the early visual system Philos Trans R Soc Lond B Biol Sci 339 385 395 8098870
- [33]Lu HD Chen G Tanigawa H Roe AW 2010 A motion direction map in macaque v2 Neuron 68 1002 1013 21145011 PMC3391546
- [34]Moeller S Yacoub E Olman CA Auerbach E Strupp J Harel N Ugurbil K 2010 Multiband multislice GE-EPI at 7 tesla, with 16-fold acceleration using partial parallel imaging with application to high spatial and temporal whole-brain fMRI Magn Reson Med 63 1144 1153 20432285 PMC2906244
- [35]Feinberg DA Moeller S Smith SM Auerbach E Ramanna S Glasser MF Miller KL Ugurbil K Yacoub E 2010 Multiplexed echo planar imaging for sub-second whole brain FMRI and fast diffusion imaging PLoS One 5 e15710 21187930 PMC3004955
- [36]Fine I MacLeod DI Boynton GM 2003 Surface segmentation based on the luminance and color statistics of natural scenes J Opt Soc Am A Opt Image Sci Vis 20 1283 1291 12868634
- [37]Zhou C Mel BW 2008 Cue combination and color edge detection in natural scenes J Vis 8 4 1 25 18484843
- [38]Hansen T Gegenfurtner KR 2009 Independence of color and luminance edges in natural scenes Vis Neurosci 26 35 49 19152717
- [39]Brouwer GJ Heeger DJ 2009 Decoding and reconstructing color from responses in human visual cortex J Neurosci 29 13992 14003 19890009 PMC2799419
- [40]Oliva A Schyns PG 2000 Diagnostic colors mediate scene recognition Cogn Psychol 41 176 210 10968925
- [41]Bartels A Zeki S Logothetis NK 2008 Natural vision reveals regional specialization to local motion and to contrast-invariant, global flow in the human brain Cereb Cortex 18 705 717 17615246
- [42]Harrison SA Tong F 2009 Decoding reveals the contents of visual working memory in early visual areas Nature 458 632 635 19225460 PMC2709809
- [43]Kamitani Y Tong F 2006 Decoding seen and attended motion directions from activity in the human visual cortex Curr Biol 16 1096 1102 16753563 PMC1635016
- [44]Reddy L Tsuchiya N Serre T 2010 Reading the mind's eye: decoding category information during mental imagery Neuroimage 50 818 825 20004247 PMC2823980
- [45]Mitchell TM Shinkareva SV Carlson A Chang KM Malave VL Mason RA Just MA 2008 Predicting human brain activity associated with the meanings of nouns Science 320 1191 1195 18511683
- [46]Li LJ Socher R Fei-Fei L 2009 Towards total scene understanding: Classification, annotation and segmentation in an automatic framework IEEE Conference on Computer Vision and Pattern Recognition 2036 2043
- [47]Hansen KA David SV Gallant JL 2004 Parametric reverse correlation reveals spatial linearity of retinotopic human V1 BOLD response Neuroimage 23 233 241 15325370
Cited by
Sorted by citations
Natural speech reveals the semantic maps that tile human cerebral cortex
Huth, A. G., de Heer, W. A., Griffiths, T. L., Theunissen, F. E., & Gallant, J. L. (2016) · Nature · doi:10.1038/nature17637 · Cited by 1,504 · see citing works
Self-citationBuilding better biomarkers: brain models in translational neuroimaging
Woo, C.-W., Chang, L. J., Lindquist, M. A., & Wager, T. D. (2017) · Nature Neuroscience · doi:10.1038/nn.4478 · Cited by 1,056 · see citing works
Cross-validation failure: Small sample sizes lead to large error bars
Varoquaux, G. (2018) · NeuroImage · doi:10.1016/j.neuroimage.2017.06.061 · Cited by 897 · see citing works
“…the neural support of cognition (Haynes and Rees, 2006), information mapping (Kriegeskorte et al., 2006), prediction of individual differences –behavioral or clinical– (Smith et al., 2015), rich encoding models (Nishimoto et al., 2011), principled reverse inferences (Poldrack et al., 2009), etc .”
BackgroundDecoding Neural Representational Spaces Using Multivariate Pattern Analysis
Haxby, J. V., Connolly, A. C., & Guntupalli, J. S. (2014) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-062012-170325 · Cited by 717 · see citing works
Boly, M., Massimini, M., Tsuchiya, N., Postle, B. R., Koch, C., & Tononi, G. (2017) · The Journal of Neuroscience · doi:10.1523/jneurosci.3218-16.2017 · Cited by 557 · see citing works
No-Report Paradigms: Extracting the True Neural Correlates of Consciousness
Tsuchiya, N., Wilke, M., Frässle, S., & Lamme, V. A. F. (2015) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2015.10.002 · Cited by 551 · see citing works
A Primer on Pattern-Based Approaches to fMRI: Principles, Pitfalls, and Perspectives
Haynes, J.-D. (2015) · Neuron · doi:10.1016/j.neuron.2015.05.025 · Cited by 402 · see citing works
Semantic reconstruction of continuous language from non-invasive brain recordings
Tang, J., LeBel, A., Jain, S., & Huth, A. G. (2023) · Nature Neuroscience · doi:10.1038/s41593-023-01304-9 · Cited by 442 · see citing works
Toward a universal decoder of linguistic meaning from brain activation
Pereira, F., Lou, B., Pritchett, B., Ritter, S., Gershman, S. J., Kanwisher, N., Botvinick, M., & Fedorenko, E. (2018) · Nature Communications · doi:10.1038/s41467-018-03068-4 · Cited by 324 · see citing works
Attention during natural vision warps semantic representation across the human brain
Çukur, T., Nishimoto, S., Huth, A. G., & Gallant, J. L. (2013) · Nature Neuroscience · doi:10.1038/nn.3381 · Cited by 351 · see citing works
Self-citationThe present and future of deep learning in radiology
Saba, L., Biswas, M., Kuppili, V., Cuadrado Godia, E., Suri, H. S., Edla, D. R., Omerzu, T., Laird, J. R., Khanna, N. N., Mavrogeni, S., et al. (2019) · European Journal of Radiology · doi:10.1016/j.ejrad.2019.02.038 · Cited by 353 · see citing works
Seeing it all: Convolutional network layers map the function of the human visual system
Eickenberg, M., Gramfort, A., Varoquaux, G., & Thirion, B. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2016.10.001 · Cited by 338 · see citing works
HOGgles: Visualizing Object Detection Features
Vondrick, C., Khosla, A., Malisiewicz, T., & Torralba, A. (2013) · 2013 IEEE International Conference on Computer Vision · doi:10.1109/iccv.2013.8 · Cited by 282 · see citing works
Deniz, F., Nunez-Elizalde, A. O., Huth, A. G., & Gallant, J. L. (2019) · The Journal of Neuroscience · doi:10.1523/jneurosci.0675-19.2019 · Cited by 214 · see citing works
“…to characterize the semantic selectivity of each voxel in each presentation modality and for each individual participant (see Materials and Methods and Naselaris et al., 2011; Nishimoto et al., 2011; Huth et al., 2012, 2016; Çukur et al., 2013; Stansbury et al., 2013; Lescroart et al., 2015).”
BackgroundSelf-citationBrain–Computer Interface Technologies in the Coming Decades
Lance, B. J., Kerick, S. E., Ries, A. J., Oie, K. S., & McDowell, K. (2012) · Proceedings of the IEEE · doi:10.1109/jproc.2012.2184830 · Cited by 231 · see citing works
A Direct Brain-to-Brain Interface in Humans
Rao, R. P. N., Stocco, A., Bryan, M., Sarma, D., Youngquist, T. M., Wu, J., & Prat, C. S. (2014) · PLoS ONE · doi:10.1371/journal.pone.0111332 · Cited by 210 · see citing works
Are there islands of awareness?
Bayne, T., Seth, A. K., & Massimini, M. (2020) · Trends in Neurosciences · doi:10.1016/j.tins.2019.11.003 · Cited by 145 · see citing works
Memory reactivation in healthy aging: evidence of stimulus-specific dedifferentiation
St-Laurent, M., Abdi, H., Bondad, A., & Buchsbaum, B. R. (2014) · The Journal of Neuroscience · doi:10.1523/jneurosci.3054-13.2014 · Cited by 126 · see citing works
Yu, B. (2013) · Bernoulli · doi:10.3150/13-bejsp14 · Cited by 113 · see citing works
Self-citationDetecting and interpreting conscious experiences in behaviorally non-responsive patients
Naci, L., Sinai, L., & Owen, A. M. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2015.11.059 · Cited by 105 · see citing works
Progress toward openness, transparency, and reproducibility in cognitive neuroscience
Gilmore, R. O., Diaz, M. T., Wyble, B. A., & Yarkoni, T. (2017) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.13325 · Cited by 95 · see citing works
Estimating average single-neuron visual receptive field sizes by fMRI
Keliris, G. A., Li, Q., Papanikolaou, A., Logothetis, N. K., & Smirnakis, S. M. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1809612116 · Cited by 48 · see citing works
Clark, I. A., Niehaus, K. E., Duff, E. P., Di Simplicio, M. C., Clifford, G. D., Smith, S. M., Mackay, C. E., Woolrich, M. W., & Holmes, E. A. (2014) · Behaviour Research and Therapy · doi:10.1016/j.brat.2014.07.010 · Cited by 37 · see citing works
Levin, M. (2025) · BioEssays · doi:10.1002/bies.202400196 · Cited by 34 · see citing works
Berger, S., & Rossi, F. (2023) · Communications of the ACM · doi:10.1145/3529088 · Cited by 21 · see citing works
- Authors
- Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., & Gallant, J. L.
- Affiliations
- University of California, Berkeley (US)
- Corresponding author
- Gallant, Jack L.
- Venue
- Current Biology
- Publisher
- Elsevier BV
- Published
- 2011-10
- Volume
- 21
- Issue
- 19
- Pages
- 1641-1646
- ISSN
- 0960-9822 (ISSN-L)
- PMC
- PMC3326357
- OpenAlex
- W2126810579
- Cited by
- 933
- References
- 4
- Citations by year
- 2026: 55 · 2025: 81 · 2024: 52 · 2023: 72 · 2022: 60 · 2021: 71 · 2020: 68 · 2019: 76 · 2018: 91 · 2017: 68 · 2016: 52 · 2015: 55 · 2014: 51 · 2013: 51 · 2012: 25
- FWCI
- 20.03
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Current Biology (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Funders
- National Institutes of Health (5r01ey019684-08)
- National Eye Institute (R01-EY019684)
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Advanced MRI Techniques and Applications
- Life Sciences › Neuroscience › Cognitive Neuroscience › Functional Brain Connectivity Studies
- Keywords
- visual cortex, BOLD signal, brain dynamics, visual reconstruction, hemodynamic response, occipitotemporal cortex, fMRI decoding
- MeSH terms
- Adult, Bayes Theorem, Brain Mapping, Humans, Image Processing, Computer-Assisted, Magnetic Resonance Imaging, Male, Models, Neurological, Motion Perception, Neurophysiology, Oxygen, Photic Stimulation, Regression Analysis, Time Factors, Visual Cortex, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
