Visual image reconstruction from human brain activity using a combination of multiscale local image decoders.
Open accessAbstract
Perceptual experience consists of an enormous number of possible states. Previous fMRI studies have predicted a perceptual state by classifying brain activity into prespecified categories. Constraint-free visual image reconstruction is more challenging, as it is impractical to specify brain activity for all possible images. In this study, we reconstructed visual images by combining local image bases of multiple scales, whose contrasts were independently decoded from fMRI activity by automatically selecting relevant voxels and exploiting their correlated patterns. Binary-contrast, 10 x 10-patch images (2(100) possible states) were accurately reconstructed without any image prior on a single trial or volume basis by measuring brain activity only for several hundred random images. Reconstruction was also used to identify the presented image among millions of candidates. The results suggest that our approach provides an effective means to read out complex perceptual states from brain activity while discovering information representation in multivoxel patterns.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Non-invasive neural BMI interface strategies include EEG, MEG, fMRI, and optical strategies including fNIRS.
“Non-invasive neural BMI interface strategies include the use of EEG, magnetoencephalography (MEG), fMRI (Miyawaki et al., 2008) and optical strategies, including fNIRS (Naseer and Hong, 2015).”
From Human Brain/Cloud Interface (2019) · cites this paper as Miyawaki et al., 2008
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Decoding the visual and subjective contents of the human brain
Kamitani, Y., & Tong, F. (2005) · Nature Neuroscience · doi:10.1038/nn1444 · Cited by 1,829 · see citing works
Self-citationNeural correlations, population coding and computation
Averbeck, B. B., Latham, P. E., & Pouget, A. (2006) · Nature reviews. Neuroscience · doi:10.1038/nrn1888 · Cited by 1,743 · see citing works
Cortical magnification within human primary visual cortex correlates with acuity thresholds
Duncan, R. O., & Boynton, G. M. (2003) · Neuron · doi:10.1016/s0896-6273(03)00265-4 · Cited by 385 · see citing works
Spatio-temporal point-spread function of fMRI signal in human gray matter at 7 Tesla
Shmuel, A., Yacoub, E., Chaimow, D., Logothetis, N. K., & Ugurbil, K. (2007) · NeuroImage · doi:10.1016/j.neuroimage.2006.12.030 · Cited by 294 · see citing works
Flagged first, then by citations
Research works printed but not linked: 17
- [1]1. Arsenault, A.S. ∙ Wilkinson, F. ∙ Kingdom, F.A. Modulation frequency and orientation tuning of second-order texture mechanisms J. Opt. Soc. Am. A Opt. Image Sci. Vis. 1999; 16:427-435 Crossref Scopus (46) PubMed Google Scholar
- [3]3. Bishop, C.M. Pattern Recognition and Machine Learning Springer, New York, 2006 Google Scholar
- [4]4. Campbell, F.W. ∙ Robson, J.G. Application of Fourier analysis to the visibility of gratings J. Physiol. 1968; 197:551-566 Crossref Scopus (2365) PubMed Google Scholar
- [5]5. Chen, Y. ∙ Geisler, W.S. ∙ Seidemann, E. Optimal decoding of correlated neural population responses in the primate visual cortex Nat. Neurosci. 2006; 9:1412-1420 Crossref Scopus (155) PubMed Google Scholar
- [6]6. Cox, D.D. ∙ Savoy, R.L. Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex Neuroimage. 2003; 19:261-270 Crossref Scopus (835) PubMed Google Scholar
- [7]7. De Valois, R.L. ∙ Albrecht, D.G. ∙ Thorell, L.G. Spatial frequency selectivity of cells in macaque visual cortex Vision Res. 1982; 22:545-559 Crossref Scopus (1186) PubMed Google Scholar
- [8]8. Donoghue, J.P. Connecting cortex to machines: recent advances in brain interfaces Nat. Neurosci. 2002; 5:1085-1088 Crossref Scopus (558) PubMed Google Scholar
- [9]9. Dougherty, R.F. ∙ Koch, V.M. ∙ Brewer, A.A. ... Visual field representations and locations of visual areas V1/2/3 in human visual cortex J. Vis. 2003; 3:586-598 Crossref Scopus (429) PubMed Google Scholar
- [10]10. Dumoulin, S.O. ∙ Wandell, B.A. Population receptive field estimates in human visual cortex Neuroimage. 2008; 39:647-660 Crossref Scopus (753) PubMed Google Scholar
- [12]12. Engel, S.A. ∙ Rumelhart, D.E. ∙ Wandell, B.A. ... fMRI of human visual cortex Nature. 1994; 369:525 Crossref Scopus (769) PubMed Google Scholar
- [13]13. Engel, S.A. ∙ Glover, G.H. ∙ Wandell, B.A. Retinotopic organization in human visual cortex and the spatial precision of functional MRI Cereb. Cortex. 1997; 7:181-192 Crossref Scopus (1264) PubMed Google Scholar
- [14]14. Hansen, K.A. ∙ David, S.V. ∙ Gallant, J.L. Parametric reverse correlation reveals spatial linearity of retinotopic human V1 BOLD response Neuroimage. 2004; 23:233-241 Crossref Scopus (61) PubMed Google Scholar
- [15]15. Haxby, J.V. ∙ Gobbini, M.I. ∙ Furey, M.L. ... Distributed and overlapping representations of faces and objects in ventral temporal cortex Science. 2001; 293:2425-2430 Crossref Scopus (2908) PubMed Google Scholar
- [16]16. Haynes, J.D. ∙ Rees, G. Predicting the orientation of invisible stimuli from activity in human primary visual cortex Nat. Neurosci. 2005; 8:686-691 Crossref Scopus (658) PubMed Google Scholar
- [17]17. Hubel, D.H. ∙ Wiesel, T.N. Receptive fields and functional architecture of monkey striate cortex J. Physiol. 1968; 195:215-243 Crossref Scopus (4705) PubMed Google Scholar
- [19]19. Kamitani, Y. ∙ Tong, F. Decoding seen and attended motion directions from activity in the human visual cortex Curr. Biol. 2006; 16:1096-1102 Full Text Full Text (PDF) Scopus (285) PubMed Google Scholar
- [20]20. Kapadia, M.K. ∙ Ito, M. ∙ Gilbert, C.D. ... Improvement in visual sensitivity by changes in local context: parallel studies in human observers and in V1 of alert monkeys Neuron. 1995; 15:843-856
Cited by
Sorted by citations
Building better biomarkers: brain models in translational neuroimaging
Woo, C.-W., Chang, L. J., Lindquist, M. A., & Wager, T. D. (2017) · Nature Neuroscience · doi:10.1038/nn.4478 · Cited by 1,056 · see citing works
Decoding Neural Representational Spaces Using Multivariate Pattern Analysis
Haxby, J. V., Connolly, A. C., & Guntupalli, J. S. (2014) · Annual Review of Neuroscience · doi:10.1146/annurev-neuro-062012-170325 · Cited by 717 · see citing works
Neural decoding of visual imagery during sleep
Horikawa, T., Tamaki, M., Miyawaki, Y., & Kamitani, Y. (2013) · Science · doi:10.1126/science.1234330 · Cited by 498 · see citing works
Self-citationA Primer on Pattern-Based Approaches to fMRI: Principles, Pitfalls, and Perspectives
Haynes, J.-D. (2015) · Neuron · doi:10.1016/j.neuron.2015.05.025 · Cited by 402 · see citing works
Decoding Patterns of Human Brain Activity
Tong, F., & Pratte, M. S. (2012) · Annual Review of Psychology · doi:10.1146/annurev-psych-120710-100412 · Cited by 375 · see citing works
Toward a universal decoder of linguistic meaning from brain activation
Pereira, F., Lou, B., Pritchett, B., Ritter, S., Gershman, S. J., Kanwisher, N., Botvinick, M., & Fedorenko, E. (2018) · Nature Communications · doi:10.1038/s41467-018-03068-4 · Cited by 324 · see citing works
Distributed Representations in Memory: Insights from Functional Brain Imaging
Rissman, J., & Wagner, A. D. (2012) · Annual Review of Psychology · doi:10.1146/annurev-psych-120710-100344 · Cited by 299 · see citing works
Prior expectations bias sensory representations in visual cortex
Kok, P., Brouwer, G. J., van Gerven, M. A. J., & de Lange, F. P. (2013) · The Journal of Neuroscience · doi:10.1523/jneurosci.0742-13.2013 · Cited by 260 · see citing works
Against hyperacuity in brain reading: Spatial smoothing does not hurt multivariate fMRI analyses?
Op de Beeck, H. P. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.02.047 · Cited by 275 · see citing works
Nonstimulated early visual areas carry information about surrounding context
Smith, F. W., & Muckli, L. (2010) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1000233107 · Cited by 238 · see citing works
Mind Reading and Writing: The Future of Neurotechnology
Roelfsema, P. R., Denys, D., & Klink, P. C. (2018) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2018.04.001 · Cited by 107 · see citing works
Martins, N. R. B., Angelica, A., Chakravarthy, K., Svidinenko, Y., Boehm, F. J., Opris, I., Lebedev, M. A., Swan, M., Garan, S. A., Rosenfeld, J. V., et al. (2019) · Frontiers in Neuroscience · doi:10.3389/fnins.2019.00112 · Cited by 91 · see citing works
Model-based feature construction for multivariate decoding
Brodersen, K. H., Haiss, F., Ong, C. S., Jung, F., Tittgemeyer, M., Buhmann, J. M., Weber, B., & Stephan, K. E. (2011) · NeuroImage · doi:10.1016/j.neuroimage.2010.04.036 · Cited by 36 · see citing works
Simistira Liwicki, F., Gupta, V., Saini, R., De, K., Abid, N., Rakesh, S., Wellington, S., Wilson, H., Liwicki, M., & Eriksson, J. (2023) · Scientific Data · doi:10.1038/s41597-023-02286-w · Cited by 28 · see citing works
Mackey, S. C. (2013) · The Journal of Pain · doi:10.1016/j.jpain.2013.01.001 · Cited by 24 · see citing works
- Authors
- Miyawaki, Y., Uchida, H., Yamashita, O., Sato, M.-A., Morito, Y., Tanabe, H. C., Sadato, N., & Kamitani, Y.
- Affiliations
- Advanced Telecommunications Research Institute International (JP)
- The Graduate University for Advanced Studies, SOKENDAI (JP)
- Corresponding author
- Kamitani, Yukiyasu
- ORCID
- Venue
- Neuron
- Publisher
- Elsevier BV
- Published
- 2008-12
- Volume
- 60
- Issue
- 5
- Pages
- 915-929
- ISSN
- 0896-6273 (ISSN-L)
- OpenAlex
- W2097267381
- Cited by
- 543
- References
- 4
- Citations by year
- 2026: 24 · 2025: 26 · 2024: 24 · 2023: 32 · 2022: 29 · 2021: 27 · 2020: 32 · 2019: 36 · 2018: 37 · 2017: 31 · 2016: 23 · 2015: 32 · 2014: 30 · 2013: 34 · 2012: 37
- FWCI
- 15.38
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- www.elsevier.com/open-access/userlicense/1.0/ (vor, 1725 d)
- doi.org/10.15223/policy-017 (stm-asf)
- doi.org/10.15223/policy-037 (stm-asf)
- doi.org/10.15223/policy-012 (stm-asf)
- doi.org/10.15223/policy-029 (stm-asf)
- doi.org/10.15223/policy-004 (stm-asf)
- Locations
- Neuron (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Institute for Physiological Sciences
- Topics
- Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
- Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
- Life Sciences › Neuroscience › Cognitive Neuroscience › EEG and Brain-Computer Interfaces
- Keywords
- fMRI decoding, single-trial decoding, image recognition, voxel selection
- MeSH terms
- Brain, Brain Mapping, Female, Humans, Image Processing, Computer-Assisted, Magnetic Resonance Imaging, Male, Oxygen, Pattern Recognition, Visual, Photic Stimulation, Contrast Sensitivity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
