The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
Generic decoding of seen and imagined objects using hierarchical visual features
Open accessAbstract
Object recognition is a key function in both human and machine vision. While brain decoding of seen and imagined objects has been achieved, the prediction is limited to training examples. We present a decoding approach for arbitrary objects using the machine vision principle that an object category is represented by a set of features rendered invariant through hierarchical processing. We show that visual features, including those derived from a deep convolutional neural network, can be predicted from fMRI patterns, and that greater accuracy is achieved for low-/high-level features with lower-/higher-level visual areas, respectively. Predicted features are used to identify seen/imagined object categories (extending beyond decoder training) from a set of computed features for numerous object images. Furthermore, decoding of imagined objects reveals progressive recruitment of higher-to-lower visual representations. Our results demonstrate a homology between human and machine vision and its utility for brain-based information retrieval.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
ImageNet: A large-scale hierarchical image database
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206848 · Cited by 65,333
Object recognition from local scale-invariant features
Lowe, D. G. (1999) · Proceedings of the Seventh IEEE International Conference on Computer Vision · doi:10.1109/iccv.1999.790410 · Cited by 16,472
Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex
Hubel, D. H., & Wiesel, T. N. (1962) · The Journal of Physiology · doi:10.1113/jphysiol.1962.sp006837 · Cited by 13,680
WordNet: An Electronic Lexical Database
Kilgarriff, A., & Fellbaum, C. (2000) · Language · doi:10.2307/417141 · Cited by 11,464
Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R., & Bengio, Y. (2015) · arXiv · doi:10.48550/arxiv.1502.03044 · Cited by 8,248
The fusiform face area: a module in human extrastriate cortex specialized for face perception
Kanwisher, N., McDermott, J., & Chun, M. M. (1997) · The Journal of Neuroscience · doi:10.1523/jneurosci.17-11-04302.1997 · Cited by 7,738
Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope
Oliva, A., & Torralba, A. (2001) · International Journal of Computer Vision · doi:10.1023/a:1011139631724 · Cited by 6,319
Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
Simonyan, K., Vedaldi, A., & Zisserman, A. (2013) · arXiv · doi:10.48550/arxiv.1312.6034 · Cited by 5,370
WordNet: An Electronic Lexical Database
(1998) · The MIT Press eBooks · doi:10.7551/mitpress/7287.001.0001 · Cited by 4,857
Visual categorization with bags of keypoints
Csurka, G. (2004) · Cited by 4,379
Distributed and overlapping representations of faces and objects in ventral temporal cortex
Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten, J. L., & Pietrini, P. (2001) · Science · doi:10.1126/science.1063736 · Cited by 3,940
Representational similarity analysis-connecting the branches of systems neuroscience
Kriegeskorte, N. (2008) · Frontiers in Systems Neuroscience · doi:10.3389/neuro.06.004.2008 · Cited by 3,355
Hierarchical models of object recognition in cortex
Riesenhuber, M., & Poggio, T. (1999) · Nature Neuroscience · doi:10.1038/14819 · Cited by 3,309
A cortical representation of the local visual environment
Epstein, R., & Kanwisher, N. (1998) · Nature · doi:10.1038/33402 · Cited by 2,984
Vlfeat: an open and portable library of computer vision algorithms
Vedaldi, A., & Fulkerson, B. (2010) · Proceedings of the 18th ACM international conference on Multimedia · doi:10.1145/1873951.1874249 · Cited by 2,664
Borders of Multiple Visual Areas in Humans Revealed by Functional Magnetic Resonance Imaging
Sereno, M. I., Dale, A. M., Reppas, J. B., Kwong, K. K., Belliveau, J. W., Brady, T. J., Rosen, B. R., & Tootell, R. B. H. (1995) · Science · doi:10.1126/science.7754376 · Cited by 2,658
RECEPTIVE FIELDS AND FUNCTIONAL ARCHITECTURE IN TWO NONSTRIATE VISUAL AREAS (18 AND 19) OF THE CAT
Hubel, D. H., & Wiesel, T. N. (1965) · Journal of Neurophysiology · doi:10.1152/jn.1965.28.2.229 · Cited by 2,603
Performance-optimized hierarchical models predict neural responses in higher visual cortex
Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1403112111 · Cited by 1,924
Decoding the visual and subjective contents of the human brain
Kamitani, Y., & Tong, F. (2005) · Nature Neuroscience · doi:10.1038/nn1444 · Cited by 1,829
Self-citationUnderstanding deep image representations by inverting them
Mahendran, A., & Vedaldi, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7299155 · Cited by 1,827
Robust Object Recognition with Cortex-Like Mechanisms
Serre, T., Wolf, L., Bileschi, S., Riesenhuber, M., & Poggio, T. (2007) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2007.56 · Cited by 1,667
Decoding reveals the contents of visual working memory in early visual areas
Harrison, S. A., & Tong, F. (2009) · Nature · doi:10.1038/nature07832 · Cited by 1,347
Identifying natural images from human brain activity
Kay, K. N., Naselaris, T., Prenger, R. J., & Gallant, J. L. (2008) · Nature · doi:10.1038/nature06713 · Cited by 1,286
Deep supervised, but not unsupervised, models may explain IT cortical representation
Khaligh-Razavi, S.-M., & Kriegeskorte, N. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003915 · Cited by 1,204
Predicting Human Brain Activity Associated with the Meanings of Nouns
Mitchell, T. M., Shinkareva, S. V., Carlson, A., Chang, K.-M., Malave, V. L., Mason, R. A., & Just, M. A. (2008) · Science · doi:10.1126/science.1152876 · Cited by 1,180
Mental Imagery: Functional Mechanisms and Clinical Applications
Pearson, J., Naselaris, T., Holmes, E. A., & Kosslyn, S. M. (2015) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2015.08.003 · Cited by 1,087
Engel, S. A., Rumelhart, D. E., Wandell, B. A., Lee, A. T., Glover, G. H., Chichilnisky, E.-J., & Shadlen, M. N. (1994) · Nature · doi:10.1038/369525a0 · Cited by 990
Guclu, U., & van Gerven, M. A. J. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.5023-14.2015 · Cited by 961
A Toolbox for Representational Similarity Analysis
Nili, H., Wingfield, C., Walther, A., Su, L., Marslen-Wilson, W., & Kriegeskorte, N. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003553 · Cited by 937
Representational geometry: integrating cognition, computation, and the brain
Kriegeskorte, N., & Kievit, R. A. (2013) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2013.06.007 · Cited by 911
Cadieu, C. F., Hong, H., Yamins, D. L. K., Pinto, N., Ardila, D., Solomon, E. A., Majaj, N. J., & DiCarlo, J. J. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003963 · Cited by 824
Kobatake, E., & Tanaka, K. (1994) · Journal of Neurophysiology · doi:10.1152/jn.1994.71.3.856 · Cited by 807
How does nature program neuron types?
Borst, A. (2008) · Frontiers in Neuroscience · doi:10.3389/neuro.01.016.2008 · Cited by 705
Cortical Regions Involved in Perceiving Object Shape
Kourtzi, Z., & Kanwisher, N. (2000) · The Journal of Neuroscience · doi:10.1523/jneurosci.20-09-03310.2000 · Cited by 673
Columns for visual features of objects in monkey inferotemporal cortex
Fujita, I., Tanaka, K., Ito, M., & Cheng, K. (1992) · Nature · doi:10.1038/360343a0 · Cited by 623
Visual features of intermediate complexity and their use in classification
Ullman, S., Vidal-Naquet, M., & Sali, E. (2002) · Nature Neuroscience · doi:10.1038/nn870 · Cited by 600
Miyawaki, Y., Uchida, H., Yamashita, O., Sato, M.-A., Morito, Y., Tanabe, H. C., Sadato, N., & Kamitani, Y. (2008) · Neuron · doi:10.1016/j.neuron.2008.11.004 · Cited by 543
Self-citationShared representations for working memory and mental imagery in early visual cortex
Albers, A. M., Kok, P., Toni, I., Dijkerman, H. C., & de Lange, F. P. (2013) · Current Biology · doi:10.1016/j.cub.2013.05.065 · Cited by 528
Bayesian Reconstruction of Natural Images from Human Brain Activity
Naselaris, T., Prenger, R. J., Kay, K. N., Oliver, M., & Gallant, J. L. (2009) · Neuron · doi:10.1016/j.neuron.2009.09.006 · Cited by 511
Neural decoding of visual imagery during sleep
Horikawa, T., Tamaki, M., Miyawaki, Y., & Kamitani, Y. (2013) · Science · doi:10.1126/science.1234330 · Cited by 498
Self-citationVisualizing Deep Convolutional Neural Networks Using Natural Pre-images
Mahendran, A., & Vedaldi, A. (2016) · International Journal of Computer Vision · doi:10.1007/s11263-016-0911-8 · Cited by 496
Deficient approaches to human neuroimaging
Stelzer, J., Lohmann, G., Mueller, K., Buschmann, T., & Turner, R. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00462 · Cited by 412
Online Model Selection Based on the Variational Bayes
Sato, M.-A. (2001) · Neural Computation · doi:10.1162/089976601750265045 · Cited by 375
Object Class Recognition and Localization Using Sparse Features with Limited Receptive Fields
Mutch, J., & Lowe, D. G. (2008) · International Journal of Computer Vision · doi:10.1007/s11263-007-0118-0 · Cited by 373
A voxel-wise encoding model for early visual areas decodes mental images of remembered scenes
Naselaris, T., Olman, C. A., Stansbury, D. E., Ugurbil, K., & Gallant, J. L. (2015) · NeuroImage · doi:10.1016/j.neuroimage.2014.10.018 · Cited by 356
Nguyen, A. M., Yosinski, J., & Clune, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.03616 · Cited by 336
Sparse estimation automatically selects voxels relevant for the decoding of fMRI activity patterns
Yamashita, O., Sato, M.-A., Yoshioka, T., Tong, F., & Kamitani, Y. (2008) · NeuroImage · doi:10.1016/j.neuroimage.2008.05.050 · Cited by 327
Self-citationTop-down activation of shape-specific population codes in visual cortex during mental imagery
Stokes, M., Thompson, R., Cusack, R., & Duncan, J. (2009) · The Journal of Neuroscience · doi:10.1523/jneurosci.4657-08.2009 · Cited by 323
Elipot, Y., Hinaux, H., Callebert, J., & Rétaux, S. (2013) · Current Biology · doi:10.1016/j.cub.2012.10.044 · Cited by 321
The heterogeneity of mental representation: Ending the imagery debate
Pearson, J., & Kosslyn, S. M. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1504933112 · Cited by 315
Reading the mind's eye: Decoding category information during mental imagery
Reddy, L., Tsuchiya, N., & Serre, T. (2010) · NeuroImage · doi:10.1016/j.neuroimage.2009.11.084 · Cited by 304
Imagine all the people: how the brain creates and uses personality models to predict behavior
Hassabis, D., Spreng, R. N., Rusu, A. A., Robbins, C. A., Mar, R. A., & Schacter, D. L. (2014) · Cerebral Cortex · doi:10.1093/cercor/bht042 · Cited by 247
Detecting Representations of Recent and Remote Autobiographical Memories in vmPFC and Hippocampus
Bonnici, H. M., Chadwick, M. J., Lutti, A., Hassabis, D., Weiskopf, N., & Maguire, E. A. (2012) · The Journal of Neuroscience · doi:10.1523/jneurosci.2475-12.2012 · Cited by 243
Imagery and perception share cortical representations of content and location
Cichy, R. M., Heinzle, J., & Haynes, J.-D. (2012) · Cerebral Cortex · doi:10.1093/cercor/bhr106 · Cited by 242
Decoding Individual Episodic Memory Traces in the Human Hippocampus
Chadwick, M. J., Hassabis, D., Weiskopf, N., & Maguire, E. A. (2010) · Current Biology · doi:10.1016/j.cub.2010.01.053 · Cited by 219
Rice, G. E., Watson, D. M., Hartley, T., & Andrews, T. J. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.5265-13.2014 · Cited by 139
Decoding individual natural scene representations during perception and imagery
Johnson, M. R., & Johnson, M. K. (2014) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2014.00059 · Cited by 113
Relating population-code representations between man, monkey, and computational models
Kriegeskorte, N. (2009) · Frontiers in Neuroscience · doi:10.3389/neuro.01.035.2009 · Cited by 98
Decoding Working Memory of Stimulus Contrast in Early Visual Cortex
Xing, Y., Ledgeway, T., McGraw, P. V., & Schluppeck, D. (2013) · Journal of Neuroscience · doi:10.1523/jneurosci.3754-12.2013 · Cited by 84
Comparing visual representations across human fMRI and computational vision
Leeds, D., Seibert, D., Pyles, J. A., & Tarr, M. J. (2013) · Journal of Vision · doi:10.1167/13.13.25 · Cited by 32
Investigation of the pseudo-shock wave in a two-dimensional supersonic inlet
Lee, H. J., Lee, B. J., Kim, S.-D., & Jeung, I. (2009) · Journal of Visualization · doi:10.1007/s12650-009-0008-3 · Cited by 20
Cited by
Deep Learning for Spatio-Temporal Data Mining: A Survey
Wang, S., Cao, J., & Yu, P. S. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.3025580 · Cited by 822
The human imagination: the cognitive neuroscience of visual mental imagery
Pearson, J. (2019) · Nature Reviews Neuroscience · doi:10.1038/s41583-019-0202-9 · Cited by 820
A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence
Allen, E. J., St-Yves, G., Wu, Y., Breedlove, J. L., Prince, J. S., Dowdle, L. T., Nau, M., Caron, B., Pestilli, F., Charest, I., et al. (2022) · Nature Neuroscience · doi:10.1038/s41593-021-00962-x · Cited by 577
Deep Neural Networks as Scientific Models
Cichy, R. M., & Kaiser, D. (2019) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2019.01.009 · Cited by 454
The blind mind: No sensory visual imagery in aphantasia
Keogh, R., & Pearson, J. (2018) · Cortex · doi:10.1016/j.cortex.2017.10.012 · Cited by 352
Metadata
- Authors
- Horikawa, T., & Kamitani, Y.
- Venue
- Nature Communications
- Published
- 2017-05-22
- Volume
- 8
- Issue
- 1
- Pages
- 15037-15037
- ISSN
- 2041-1723 (ISSN-L)
- OpenAlex
- W2963341661
- Cited by
- 516
- References
- 62
- FWCI
- 21.33
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.