CoChat
Sign Up
PreprintCited by 160Visual perception and processing mechanismsDomain Adaptation and Few-Shot LearningFace Recognition and Perception

The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.

What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines?

Open access
Conwell, Colin, Prince, Jacob S., Kay, Kendrick N., Alvarez, George A., Konkle, Talia·2022·Nature Communications
DOI 10.1101/2022.03.28.485868PMC PMC11526138
Open this paper in CoChat — free accountOpen PDF

Abstract

The rapid development and open-source release of highly performant computer vision models offers new potential for examining how different inductive biases impact representation learning and emergent alignment with the high-level human ventral visual system. Here, we assess a diverse set of 224 models, curated to enable controlled comparison of different model properties, testing their brain predictivity using large-scale functional magnetic resonance imaging data. We find that models with qualitatively different architectures (e.g. CNNs versus Transformers) and markedly different task objectives (e.g. purely visual contrastive learning versus vision-language alignment) achieve near equivalent degrees of brain predictivity, when other factors are held constant. Instead, variation across model visual training diets yields the largest, most consistent effect on emergent brain predictivity. Overarching model properties commonly suspected to increase brain predictivity (e.g. greater effective dimensionality; learnable parameter count) were not robust indicators across this more extensive survey. We highlight that standard model-to-brain linear re-weighting methods may be too flexible, as most performant models have very similar brain-predictivity scores, despite significant variation in their underlying representations. Broadly, our findings point to the importance of visual diet, challenge common assumptions about the methods used to link models to brains, and more concretely outline future directions for leveraging the full diversity of existing open-source models as tools to probe the common computational principles underlying biological and artificial visual systems.

Overview

Integrity signals

Registry

Crossref · 10.1101/2022.03.28.485868

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Training data plays a large role in alignment with biological representations.

    “Further, Conwell et al. (2022) showed that training data plays a large role in alignment.”

    From The Platonic Representation Hypothesis (2024) · cites this paper as Conwell et al. (2022)

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 106

  1. 2023, Semdedup: Data-efficient learning at web-scale through semantic deduplication, arXiv preprint arXiv, 2303, 09540
  2. Achlioptas, D . (2001). Database-friendly random projections. In Proceedings of the twentieth ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems, pages 274–281. · DOI
  3. 10.1038/s41593-021-00962-x · DOI
  4. 10.1038/s41583-021-00490-4 · DOI
  5. Bashivan, P. , Kar, K. , and DiCarlo, J . (2019). Neural population control via deep image synthesis. Science, 364. · DOI
  6. 10.18637/jss.v067.i01 · DOI · Paper page
  7. 10.1073/pnas.2112566119 · DOI
  8. Bowers, J. S. , Malhotra, G. , Dujmović, M., Montero, M. L. , Tsvetkov, C. , Biscione, V. , Puebla, G. , Adolfi, F. , Hummel, J. E. , Heaton, R. F. , et al. (2022). Deep problems with neural network models of human vision. Behavioral and Brain Sciences, pages 1–74. · DOI · Paper page
  9. 2019, Deep convolutional models improve predictions of macaque V1 responses to natural images, PLoS computational biology, 15, e1006897, 10.1371/journal.pcbi.1006897 · DOI
  10. Cadena, S. A. , Willeke, K. F. , Restivo, K. , Denfield, G. , Sinz, F. H. , Bethge, M. , Tolias, A. S. , and Ecker, A. S. (2022). Diverse task-driven modeling of macaque V4 reveals functional specialization towards semantic tasks. bioRxiv, pages 2022–05. Publisher: Cold Spring Harbor Laboratory.
  11. 2014, Deep neural networks rival the representation of primate IT cortex for core visual object recognition, PLoS computational biology, 10, e1003963, 10.1371/journal.pcbi.1003963 · DOI · Paper page
  12. 2021, Explanatory models in neuroscience: Part 1–taking mechanistic abstraction seriously, arXiv preprint arXiv, 2104, 01490
  13. Chen, T. , Kornblith, S. , Norouzi, M. , and Hinton, G. (2020). A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597–1607. PMLR. arXiv preprint arXiv:2002.05709. · arXiv
  14. 2019, The algonauts project: A platform for communication between the sciences of biological and artificial intelligence, arXiv preprint arXiv, 1905, 05675
  15. 2020, Separability and geometry of object manifolds in deep neural networks, Nature communications, 11, 746, 10.1038/s41467-020-14578-5 · DOI
  16. 2021, Neural regression, representational similarity, model zoology & neural taskonomy at scale in rodent visual cortex, Advances in Neural Information Processing Systems, 34, 5590
  17. 2023, Scaling vision transformers to 22 billion parameters, arXiv preprint arXiv, 2302, 05442
  18. 10.1080/00273171.2020.1743631 · DOI
  19. 10.1016/j.neuron.2012.01.010 · DOI · Paper page
  20. 10.1016/j.tics.2007.06.010 · DOI
  21. 2023, Cortical topographic motifs emerge in a self-organized map of object space, Science Advances, 9, eade8187, 10.1126/sciadv.ade8187 · DOI
  22. 2021, Unveiling functions of the visual cortex using task-specific deep neural networks, PLoS computational biology, 17, e1009267, 10.1371/journal.pcbi.1009267 · DOI · Paper page
  23. Elmoznino, E. and Bonner, M. F. (2022). High-performing neural network models of visual cortex benefit from high latent dimensionality. bioRxiv, pages 2022–07. Publisher: Cold Spring Harbor Laboratory.
  24. Gallicchio, C. and Scardapane, S . (2020). Deep randomized neural networks. In Recent Trends in Learning From Data: Tutorials from the INNS Big Data and Deep Learning Conference (INNSBDDL2019), pages 43–68. Springer.
  25. 2022, Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank, arXiv preprint arXiv, 2210, 02885
  26. Gatys, L. A. , Ecker, A. S. , Bethge, M. , Hertzmann, A. , and Shechtman, E . (2017). Controlling perceptual factors in neural style transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3985–3993. · DOI
  27. 2020, On the surprising similarities between supervised and self-supervised models, arXiv preprint arXiv, 2010, 08377
  28. 2021, Partial success in closing the gap between human and machine vision, Advances in Neural Information Processing Systems, 34, 23885
  29. 2022, Distinguishing representational geometries with controversial stimuli: Bayesian experimental design and its application to face dissimilarity judgments, arXiv preprint arXiv, 2211, 15053
  30. 10.1073/pnas.1912334117 · DOI
  31. 10.1093/biomet/53.3-4.325 · DOI
  32. 2022, Vision models are more robust and fair when pretrained on uncurated images without supervision, arXiv preprint arXiv, 2202, 08360
  33. 10.1523/JNEUROSCI.5023-14.2015 · DOI · Paper page
  34. 10.1523/JNEUROSCI.5023-14.2015 · DOI · Paper page
  35. Han, Y. , Poggio, T. , and Cheung, B . (2023). System identification of neural systems: If we got it right, would we know? arXiv preprint arXiv:2302.06677. · arXiv
  36. 10.1016/S0896-6273(02)00662-1 · DOI · Paper page
  37. 10.1113/jphysiol.1968.sp008455 · DOI
  38. 2019, Neural style transfer: A review, IEEE transactions on visualization and computer graphics, 26, 3365
  39. 2022, Feature-reweighted representational similarity analysis: A method for improving the fit between computational models, brains, and behavior, NeuroImage, 257, 119294, 10.1016/j.neuroimage.2022.119294 · DOI
  40. Kanwisher, N. , Khosla, M. , and Dobs, K . (2023). Using artificial neural networks to ask ‘why’ questions of minds and brains. Trends in Neurosciences. · DOI
  41. 2018, Principles for models of neural information processing, NeuroImage, 180, 101, 10.1016/j.neuroimage.2017.08.016 · DOI
  42. 10.1016/j.jmp.2016.10.007 · DOI
  43. 10.1016/j.jmp.2016.10.007 · DOI
  44. 2022, A self-supervised domain-general learning framework for human ventral stream representation, Nature Communications, 13, 1
  45. 10.1016/j.neuron.2016.12.041 · DOI · Paper page
  46. 2015, Deep neural networks: a new framework for modeling biological vision and brain information processing, Annual review of vision science, 1, 417, 10.1146/annurev-vision-082114-035447 · DOI · Paper page
  47. 2008, Representational similarity analysis-connecting the branches of systems neuroscience, Frontiers in systems neuroscience, 2, 4
  48. 10.1016/j.neuron.2008.10.043 · DOI
  49. Leclerc, G. , Ilyas, A. , Engstrom, L. , Park, S. M. , Salman, H. , and Mądry, A. (2023). Ffcv: Accelerating training by removing data bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12011–12020. · DOI
  50. 10.1038/nature14539 · DOI · Paper page
  51. Lee, H. , Margalit, E. , Jozwik, K. M. , Cohen, M. A. , Kanwisher, N. , Yamins, D. L. , and DiCarlo, J. J . (2020). Topographic deep artificial neural networks reproduce the hallmarks of the primate inferior temporal cortex face processing network. bioRxiv, pages 2020–07. · DOI
  52. Lin, T.-Y. , Maire, M. , Belongie, S. , Hays, J. , Perona, P. , Ramanan, D. , Dollár, P. , and Zitnick, C. L. (2014). Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer. · DOI · Paper page
  53. 2023, Performance-optimized deep neural networks are evolving into worse models of inferotemporal visual cortex, arXiv preprint arXiv, 2306, 03779
  54. Liu, Z. , Mao, H. , Wu, C.-Y. , Feichtenhofer, C. , Darrell, T. , and Xie, S . (2022). A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11976–11986. · DOI · Paper page
  55. 10.1073/pnas.1719616115 · DOI
  56. 2020, Extracting, computing and exploring the parameters of statistical models using R, Journal of Open Source Software, 5, 2445, 10.21105/joss.02445 · DOI
  57. 10.1016/j.tics.2011.01.004 · DOI
  58. Margalit, E. , Lee, H. , Finzi, D. , DiCarlo, J. J. , Grill-Spector, K. , and Yamins, D. L . (2023). A unifying principle for the functional organization of visual cortex. bioRxiv, pages 2023–05. · DOI
  59. Marques, T. , Schrimpf, M. , and DiCarlo, J. J. (2021). Multi-scale hierarchical neural network models that bridge from single neurons in the primate primary visual cortex to object recognition behavior. bioRxiv. Publisher: Cold Spring Harbor Laboratory.
  60. McGreivy, N. and Hakim, A . (2022). Convolutional layers are not translation equivariant. arXiv preprint arXiv:2206.04979. · arXiv
  61. 10.3758/s13414-012-0291-2 · DOI
  62. 2021, SLIP: Self-supervision meets Language-Image Pre-training, arXiv preprint arXiv, 2112, 12750
  63. 10.1002/sim.1545 · DOI
  64. 2008, segmented: an r package to fit regression models with broken-line relationships, R News, 8, 20
  65. 2017, Interval estimation for the breakpoint in segmented regression: a smoothed score-based approach, Australian New Zealand Journal of Statistics, 59, 311, 10.1111/anzs.12200 · DOI
  66. Muttenthaler, L. , Dippel, J. , Linhardt, L. , Vandermeulen, R. A. , and Kornblith, S . (2022). Human alignment of neural network representations. arXiv preprint arXiv:2211.01201. · arXiv
  67. Nayebi, A. , Kong, N. C. , Zhuang, C. , Gardner, J. L. , Norcia, A. M. , and Yamins, D. L. (2021). Unsupervised Models of Mouse Visual Cortex. bioRxiv. Publisher: Cold Spring Harbor Laboratory.
  68. 2021, Brain hierarchy score: Which deep neural networks are hierarchically brain-like?, IScience, 24, 103013, 10.1016/j.isci.2021.103013 · DOI
  69. 10.1038/nrn2314 · DOI
  70. Paszke, A. , Gross, S. , Massa, F. , Lerer, A. , Bradbury, J. , Chanan, G. , Killeen, T. , Lin, Z. , Gimelshein, N. , Antiga, L. , Desmaison, A. , Kopf, A. , Yang, E. , DeVito, Z. , Raison, M. , Tejani, A. , Chilamkurthy, S. , Steiner, B. , Fang, L. , Bai, J. , and Chintala, S. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Wallach, H. , Larochelle, H. , Beygelzimer, A. , Alché-Buc, F. d. , Fox, E. , and Garnett, R. , editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc.
  71. 10.1524/auto.2011.0951 · DOI
  72. 10.1038/s41593-021-00921-6 · DOI
  73. 10.7554/eLife.77599 · DOI · Paper page
  74. 10.1167/jov.20.11.1577 · DOI
  75. Prince, J. S. and Konkle, T. (2023). Lesioning category-selective units in silico yields functionally specialized deficits. Vision Sciences Society. · DOI
  76. 2020, Scalable transfer learning with expert models, arXiv preprint arXiv, 2009, 13239
  77. Radford, A. , Kim, J. W. , Hallacy, C. , Ramesh, A. , Goh, G. , Agarwal, S. , Sastry, G. , Askell, A. , Mishkin, P. , Clark, J. , and others (2021). Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. tex.organization: PMLR arXiv preprint arXiv:2103.00020. · arXiv
  78. 2021, Computational models of category-selective brain regions enable high-throughput tests of selectivity, Nature communications, 12, 5540, 10.1038/s41467-021-25409-6 · DOI
  79. Ren, Y. and Bashivan, P . (2023). How well do models of visual cortex generalize to out of distribution samples? bioRxiv, pages 2023–05. · DOI
  80. 10.1038/s41593-019-0520-2 · DOI · Paper page
  81. 2021, Imagenet-21k pretraining for the masses, arXiv preprint arXiv, 2104, 10972
  82. 10.1038/nn1606 · DOI
  83. Sax, A. , Zhang, J. O. , Emi, B. , Zamir, A. , Savarese, S. , Guibas, L. , and Malik, J. (2019). Learning to Navigate Using Mid-Level Visual Priors. arXiv:1912.11121 [cs]. arXiv: 1912.11121. · arXiv
  84. Schrimpf, M. , Kubilius, J. , Hong, H. , Majaj, N. J. , Rajalingham, R. , Issa, E. B. , Kar, K. , Bashivan, P. , Prescott-Roy, J. , Geiger, F. , Schmidt, K. , Yamins, D. L. K. , and DiCarlo, J. J . (2018a). Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like? bioRxiv preprint. · DOI
  85. Schrimpf, M. , Kubilius, J. , Hong, H. , Majaj, N. J. , Rajalingham, R. , Issa, E. B. , Kar, K. , Bashivan, P. , Prescott-Roy, J. , Geiger, F. , Schmidt, K. , Yamins, D. L. K. , and DiCarlo, J. J . (2018b). Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like? bioRxiv preprint. · DOI
  86. 2019, Deep learning: the good, the bad, and the ugly, Annual Review of Vision Science, 5, 399, 10.1146/annurev-vision-091718-014951 · DOI
  87. 10.1073/PNAS.2200800119/SUPPL{\_}FILE/PNAS.2200800119.SAPP.PDF · DOI
  88. 2018, The feature-weighted receptive field: an interpretable encoding model for complex feature spaces, NeuroImage, 180, 188, 10.1016/j.neuroimage.2017.06.035 · DOI
  89. 10.1162/jocn\_a\_01755 · DOI
  90. Tang, J. , Du, M. , Vo, V. A. , Lal, V. , and Huth, A. G . (2023). Brain encoding models based on multimodal transformers can transfer across language and vision. arXiv preprint arXiv:2305.12248. · arXiv
  91. 2020, Reliability-based voxel selection, NeuroImage, 207, 116350, 10.1016/j.neuroimage.2019.116350 · DOI
  92. Wang, A. , Tarr, M. , and Wehbe, L . (2019). Neural taskonomy: Inferring the similarity of task-derived representations from brain activity. Advances in Neural Information Processing Systems, 32. · DOI
  93. Wang, A. Y. , Kay, K. , Naselaris, T. , Tarr, M. J. , and Wehbe, L. (2022). Incorporating natural language into vision models improves prediction and understanding of higher visual cortex. BioRxiv, pages 2022–09. Publisher: Cold Spring Harbor Laboratory.
  94. 2018, Deep residual network predicts cortical representation and organization of visual features for rapid categorization, Scientific reports, 8, 1
  95. 2021, Resnet strikes back: An improved training procedure in timm. arxiv 2021, arXiv preprint arXiv, 2110, 00476
  96. Willeke, K. F. , Fahey, P. G. , Bashiri, M. , Pede, L. , Burg, M. F. , Blessing, C. , Cadena, S. A. , Ding, Z. , Lurz, K.-K. , Ponder, K. , and others (2022). The Sensorium competition on predicting large-scale mouse primary visual cortex activity. arXiv preprint arXiv:2206.08666. · arXiv
  97. Wortsman, M. , Ilharco, G. , Kim, J. W. , Li, M. , Kornblith, S. , Roelofs, R. , Lopes, R. G. , Hajishirzi, H. , Farhadi, A. , Namkoong, H. , et al. (2022). Robust fine-tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959–7971. · DOI · Paper page
  98. 2020, XDream: Finding preferred stimuli for visual neurons using generative networks and gradient-free optimization, PLoS computational biology, 16, e1007973, 10.1371/journal.pcbi.1007973 · DOI
  99. 10.1038/nn.4244 · DOI · Paper page
  100. 10.1073/pnas.1403112111 · DOI · Paper page
  101. 2023, Learning efficient coding of natural images with maximum manifold capacity representations, arXiv preprint arXiv, 2303, 03307
  102. Yun, S. , Han, D. , Oh, S. J. , Chun, S. , Choe, J. , and Yoo, Y . (2019). Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023–6032. · DOI · Paper page
  103. Zamir, A. R. , Sax, A. , Shen, W. , Guibas, L. J. , Malik, J. , and Savarese, S . (2018). Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3712–3722. · DOI · Paper page
  104. Zhang, H. , Cisse, M. , Dauphin, Y. N. , and Lopez-Paz, D . (2017). mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412. · arXiv
  105. Zhou, H.-Y. , Lu, C. , Yang, S. , and Yu, Y . (2021). ConvNets vs. Transformers: Whose visual representations are more transferable? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2230–2238.
  106. Zhuang, C. , Yan, S. , Nayebi, A. , Schrimpf, M. , Frank, M. C. , DiCarlo, J. J. , and Yamins, D. L . (2021). Unsupervised neural network models of the ventral visual stream. Proceedings of the National Academy of Sciences, 118(3). Publisher: National Acad Sciences.

Cited by

Sorted by citations

Metadata

Authors
Conwell, C., Prince, J. S., Kay, K. N., Alvarez, G. A., & Konkle, T.
Affiliations
  • Harvard University (US)
  • University of Minnesota (US)
Corresponding author
  • Conwell, Colin
  • Konkle, Talia
ORCID
Venue
Nature Communications
Publisher
openRxiv
Published
2022-03-29
Volume
15
Issue
1
Pages
9383-9383
ISSN
2041-1723 (ISSN-L)
DOI
10.1101/2022.03.28.485868
PMC
PMC11526138
OpenAlex
W4403924340
Cited by
160
References
11
Citations by year
2026: 54 · 2025: 65 · 2024: 33 · 2023: 8
FWCI
33.15
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • Nature Communications (publishedVersion, cc-by-nc-nd)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc)
  • PubMed Central (publishedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • National Science Foundation (1822683, 1822929, 1942438, CAREER)
  • National Institutes of Health
  • National Defense Science and Engineering Graduate
  • National Eye Institute (R01EY034118)
Topics
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Visual perception and processing mechanisms
  • Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
  • Life Sciences › Neuroscience › Cognitive Neuroscience › Face Recognition and Perception
Keywords
inductive bias, representational similarity analysis, vision models, convolutional neural networks, vision transformer, contrastive learning, vision-language alignment, model architecture
MeSH terms
Brain, Humans, Models, Neurological, Visual Perception, Neural Networks, Computer

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review