CoChat
Sign Up
Cited by 139

An ecologically motivated image dataset for deep learning yields better models of human vision

Mehrer, Johannes, Spoerer, Courtney J., Jones, Emer C., Kriegeskorte, Nikolaus, Kietzmann, Tim C.·2021·Proceedings of the National Academy of Sciences
DOI 10.1073/pnas.2011417118PMC PMC7923360
Open this paper in CoChat — free account

Abstract

Significance: Inspired by core principles of information processing in the brain, deep neural networks (DNNs) have demonstrated remarkable success in computer vision applications. At the same time, networks trained on the task of object classification exhibit similarities to representations found in the primate visual system. This result is surprising because the datasets commonly used for training are designed to be engineering challenges. Here, we use linguistic corpus statistics and human concreteness ratings as guiding principles to design a resource that more closely mirrors categories that are relevant to humans. The result is ecoset, a collection of 1.5 million images from 565 basic-level categories. We show that ecoset-trained DNNs yield better models of human higher-level visual cortex and human behavior.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1073/pnas.2011417118

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Artificial vision models such as deep convolutional neural networks are already both highly successful and often claimed to be good models of human vision.

    “artificial vision models such as deep convolutional neural networks (DCNNs), are already both highly successful and often claimed to be good models of human vision (Kietzmann et al. 2019, Lindsay 2021, Mehrer et al. 2021, Zhuang et al. 2021).”

    From Consciousness in Artificial Intelligence: Insights from the Science of Consciousness (2023) · cites this paper as Mehrer et al. 2021

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 43

  1. [1]S.-M. Khaligh-Razavi, N. Kriegeskorte, Deep supervised, but not unsupervised, models may explain IT cortical representation. PLoS Comput. Biol. 10, e1003915 (2014). 25375136 PMC4222664
  2. [2]U. Güçlü, M. A. J. van Gerven, Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. J. Neurosci. 35, 10005– 10014 (2015). 26157000 PMC6605414
  3. [3]M. Schrimpf., Brain-Score: Which artificial neural network for object recognition is most brain-like? bioRxiv [Preprint] (2020). 10.1101/407007. (Accessed 17 October 2020). · DOI
  4. [5]K. S. Weiner, K. Grill-Spector, Neural representations of faces and limbs neighbor in human high-level visual cortex: Evidence for a new organization principle. Psychol. Res. 77, 74– 97 (2013). 22139022 PMC3535411
  5. [7]K. R. Storrs, N. Kriegeskorte, Deep learning for cognitive neuroscience. arXiv [Preprint] (2019). https://arxiv.org/abs/1903.01458. (Accessed 7 October 2019). · arXiv
  6. [8]B. A. Richards., A deep learning framework for neuroscience. Nat. Neurosci. 22, 1761– 1770 (2019). 31659335 PMC7115933
  7. [9]R. M. Cichy, D. Kaiser, Deep neural networks as scientific models. Trends Cogn. Sci. 23, 305– 317 (2019). 30795896
  8. [10]M. Brysbaert, B. New, Moving beyond Kucera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behav. Res. Methods 41, 977– 990 (2009). 19897807
  9. [11]M. Brysbaert, A. B. Warriner, V. Kuperman, Concreteness ratings for 40 thousand generally known English word lemmas. Behav. Res. Methods 46, 904– 911 (2014). 24142837
  10. [12]T. Horikawa, Y. Kamitani, Generic decoding of seen and imagined objects using hierarchical visual features. Nat. Commun. 8, 15037 (2017). 28530228 PMC5458127
  11. [13]R. M. Cichy, D. Pantazis, A. Oliva, Resolving human object recognition in space and time. Nat. Neurosci. 17, 455– 462 (2014). 24464044 PMC4261693
  12. [14]M. Mur., Human object-similarity judgments reflect and transcend the primate-IT object representation. Front. Psychol. 4, 128 (2013). 23525516 PMC3605517
  13. [15]N. Kriegeskorte, Representational similarity analysis–Connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2, 4 (2008). 19104670 PMC2605405
  14. [16]N. Kriegeskorte, J. Diedrichsen, Peeling the onion of brain representations. Annu. Rev. Neurosci. 42, 407– 432 (2019). 31283895
  15. [17]K. R. Storrs, T. C. Kietzmann, A. Walther, J. Mehrer, N. Kriegeskorte, Diverse deep neural networks all predict human IT well, after training and fitting. bioRxiv [Preprint] (2020). 10.1101/2020.05.07.082743. (Accessed 11 May 2020). 34272948 · DOI
  16. [18]A. Krizhevsky, One weird trick for parallelizing convolutional neural networks. arXi [Preprint] (2014). https://arxiv.org/pdf/1404.5997v1.pdf. (Accessed 9 April 2014). · arXiv
  17. [19]B. A. Wandell, J. Winawer, Computational neuroimaging and population receptive fields. Trends Cogn. Sci. 19, 349– 357 (2015). 25850730 PMC4484758
  18. [20]K. Grill-Spector, K. S. Weiner, K. Kay, J. Gomez, The functional neuroanatomy of human face perception. Annu. Rev. Vis. Sci. 3, 167– 196 (2017). 28715955 PMC6345578
  19. [22]P. Bao, L. She, M. McGill, D. Y. Tsao, A map of object space in primate inferotemporal cortex. Nature 583, 103– 108 (2020). 32494012 PMC8088388
  20. [24]K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition. arXiv [Preprint] (2014). http://export.arxiv.org/abs/1409.1556v6. (Accessed 17 October 2014). · arXiv
  21. [25]G. Huang, Z. Liu, L. van der Maaten, K. Q. Weinberger, Densely connected convolutional networks. arXiv [Preprint] (2017). https://arxiv.org/abs/1608.06993. (Accessed 17 October 2019). · arXiv
  22. [26]N. Kriegeskorte., Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron 60, 1126– 1141 (2008). 19109916 PMC3143574
  23. [27]T. Konkle, A. Caramazza, Tripartite organization of the ventral stream by animacy and object size. J. Neurosci. 33, 10235– 10242 (2013). 23785139 PMC3755177
  24. [28]N. Kriegeskorte, M. Mur, M. D. S. Inverse, Inverse MDS: Inferring dissimilarity structure from multiple item arrangements. Front. Psychol. 3, 245 (2012). 22848204 PMC3404552
  25. [29]C. R. Pernet, R. Wilcox, G. A. Rousselet, Robust correlation analyses: False positive and power validation using a new open source matlab toolbox. Front. Psychol. 3, 606 (2013). 23335907 PMC3541537
  26. [31]K. Kar, J. Kubilius, K. Schmidt, E. B. Issa, J. J. DiCarlo, Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior. Nat. Neurosci. 22, 974– 983 (2019). 31036945 PMC8785116
  27. [32]C. J. Spoerer, T. C. Kietzmann, J. Mehrer, I. Charest, N. Kriegeskorte, Recurrent neural networks can explain flexible trading of speed and accuracy in biological vision. PLoS Comput. Biol. 16, e1008215 (2020). 33006992 PMC7556458
  28. [33]D. Linsley, J. Kim, A. Ashok, T. Serre, “ Recurrent neural circuits for contour detection” in 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia (ICLR, 2020), p. 23.
  29. [34]R. S. van Bergen, N. Kriegeskorte, Going in circles is the way forward: The role of recurrence in visual inference. arXiv [Preprint] (2020). https://arxiv.org/abs/2003.12128 (Accessed 7 October 2020). 33279795 · arXiv
  30. [35]T. Konkle, G. A. Alvarez, Instance-level contrastive learning yields human brain-like representation without category-supervision. bioRxiv [Preprint] (2020). https://www.biorxiv.org/content/10.1101/2020.06.15.153247v1. (Accessed 16 September 2020). · DOI
  31. [36]C. Zhuang., Unsupervised neural network models of the ventral visual stream. bioRxiv [Preprint] (2021). https://www.biorxiv.org/content/10.1101/2020.06.16.155556v1. (Accessed 16 September 2020). PMC7826371 33431673 · DOI
  32. [37]K. R. Storrs, R. W. Fleming, Unsupervised learning predicts human perception and misperception of gloss. bioRxiv [Preprint] (2020). https://www.biorxiv.org/content/10.1101/2020.04.07.026120v2. (Accessed 16 September 2020). PMC8526360 33958744 · DOI
  33. [38]D. L. K. Yamins., Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proc. Natl. Acad. Sci. U.S.A. 111, 8619– 8624 (2014). 24812127 PMC4060707
  34. [39]R. Geirhos., Comparing deep neural networks against humans: Object recognition when the signal gets weaker. arXiv [Preprint] (2017). https://arxiv.org/abs/1706.06969v2. (Accessed 14 May 2018). · arXiv
  35. [40]M. N. Hebart, C. Zheng, F. Pereira, C. I. Baker, Revealing the multidimensional mental representations of natural objects underlying human similarity judgments. Nat. Hum. Behav. 4, 1173– 1185 (2020). 33046861 PMC7666026
  36. [41]R. Geirhos, K. Meding, F. A. Wichmann, Beyond accuracy: Quantifying trial-by-trial behaviour of CNNs and humans by measuring error consistency. arXiv [Preprint] (2020). https://arxiv.org/abs/2006.16736v3. (Accessed 7 October 2020). · arXiv
  37. [42]R. Rajalingham., Large-scale, high-resolution comparison of the core visual object recognition behavior of humans, monkeys, and state-of-the-art deep artificial neural networks. J. Neurosci. 38, 7255– 7269 (2018). 30006365 PMC6096043
  38. [43]M. N. Hebart., THINGS: A database of 1,854 object concepts and more than 26,000 naturalistic object images. PLoS One 14, e0223792 (2019). 31613926 PMC6793944
  39. [46]J. Sullivan, M. Mei, A. Perfors, E. H. Wojcik, M. C. Frank, SAYCam: A large, longitudinal audiovisual dataset recorded from the infant’s perspective PsyArXiv [Preprint] (2020). 10.31234/osf.io/fy8zx. (Accessed 6 January 2021). PMC8412186 34485795 · DOI
  40. [47]N. A. Sugden, M. C. Moulson, The infant’s face diet: Data on 3-month-old infant-perspective experience with faces video-recorded in their typical, daily environment. Data Brief 29, 105070 (2019). 32071956 PMC7013344
  41. [48]M. A. Lambon Ralph, K. Sage, R. W. Jones, E. J. Mayberry, Coherent concepts are computed in the anterior temporal lobes. Proc. Natl. Acad. Sci. U.S.A. 107, 2717– 2722 (2010). 20133780 PMC2823909
  42. [49]D. Y. Tsao, S. Moeller, W. A. Freiwald, Comparing face patch systems in macaques and humans. Proc. Natl. Acad. Sci. U.S.A. 105, 19514– 19519 (2008). 19033466 PMC2614792
  43. [50]R. Rajimehr, J. C. Young, R. B. H. Tootell, An anterior temporal face patch in human cortex, predicted by macaque maps. Proc. Natl. Acad. Sci. U.S.A. 106, 1995– 2000 (2009). 19179278 PMC2632713

Cited by

Sorted by citations

Authors
Mehrer, J., Spoerer, C. J., Jones, E. C., Kriegeskorte, N., & Kietzmann, T. C.
Affiliations
  • MRC Cognition and Brain Sciences Unit, University of Cambridge, CB2 7EF Cambridge, United Kingdom;
  • Department of Psychology, Zuckerman Institute, Columbia University, New York, NY 10027;
ORCID
Venue
Proceedings of the National Academy of Sciences
Publisher
National Academy of Sciences
Published
2021-02-15
Volume
118
Issue
8
ISSN
  • 0027-8424
  • 1091-6490
DOI
10.1073/pnas.2011417118
PMC
PMC7923360
Cited by
139
References
5
Language
en
Licence
creativecommons.org/licenses/by/4.0/ (vor, 1 d)
Funders
  • Cambridge Trust (Vice Chancellor's Award)
  • RCUK | Biotechnology and Biological Sciences Research Council (BB/M011194/1)
  • Deutsche Forschungsgemeinschaft (DynaVision)
  • EC | Horizon 2020 (720270, 785907)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review