A comparative biology approach to DNN modeling of vision: A focus on differences, not similarities
Open accessThe DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
Deep neural networks (DNNs) have revolutionized computer science and are now widely used for neuroscientific research. A hot debate has ensued about the usefulness of DNNs as neuroscientific models of the human visual system; the debate centers on to what extent certain shortcomings of DNNs are real failures and to what extent they are redeemable. Here, we argue that the main problem is that we often do not understand which human functions need to be modeled and, thus, what counts as a falsification. Hence, not only is there a problem on the DNN side, but there is also one on the brain side (i.e., with the explanandum-the thing to be explained). For example, should DNNs reproduce illusions? We posit that we can make better use of DNNs by adopting an approach of comparative biology by focusing on the differences, rather than the similarities, between DNNs and humans to improve our understanding of visual information processing in general.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2010.11929 · Cited by 47,472
One Pixel Attack for Fooling Deep Neural Networks
Su, J., Vargas, D. V., & Sakurai, K. (2019) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2019.2890858 · Cited by 2,406 · cited 5×
“Finally, DNNs are prone to overfitting and adversarial attacks (e.g., Goodfellow, Shlens, & Szegedy, 2014; Su et al., 2019; Dujmović, Malhotra, & Bowers, 2020).”
BackgroundInfluentialMLP-Mixer: An all-MLP Architecture for Vision
Tolstikhin, I., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.01601 · Cited by 2,087 · cited 1×
“…Cichy, Hauk, & Kriegeskorte, 2019b), with (e.g., Dosovitskiy et al., 2020; Radford et al., 2021) or without (e.g., Krizhevsky et al., 2012) attention and even with very different architectures, such as transformers (Dosovitskiy et al., 2020) and multilayer perceptrons (Tolstikhin et al., 2021).”
BackgroundPerformance-optimized hierarchical models predict neural responses in higher visual cortex
Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1403112111 · Cited by 1,924
Intersubject synchronization of cortical activity during natural vision
Hasson, U., Nir, Y., Levy, I., Fuhrmann, G., & Malach, R. (2004) · Science · doi:10.1126/science.1089506 · Cited by 1,839
How does the brain solve visual object recognition?
DiCarlo, J. J., Zoccolan, D., & Rust, N. C. (2012) · Neuron · doi:10.1016/j.neuron.2012.01.010 · Cited by 1,790
Deep supervised, but not unsupervised, models may explain IT cortical representation
Khaligh-Razavi, S.-M., & Kriegeskorte, N. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003915 · Cited by 1,204
A deep learning framework for neuroscience
Richards, B. A., Lillicrap, T. P., Beaudoin, P., Bengio, Y., Bogacz, R., Christensen, A., Clopath, C., Costa, R. P., de Berker, A., Ganguli, S., et al. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0520-2 · Cited by 1,027 · cited 2×
“This excellent performance and the analogy between DNNs and the primate visual cortex have caused a fierce discussion about the use of DNNs as neuroscientific models of the brain (e.g., Rosenholtz, 2017; VanRullen, 2017; Majaj & Pelli, 2018; Kubilius, 2018; Cichy & Kaiser, 2019; Kietzmann, McClure, & Kriegeskorte, 201…”
Adversarial Examples Are Not Bugs, They Are Features
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., & Madry, A. (2019) · arXiv · doi:10.48550/arxiv.1905.02175 · Cited by 971
Inter-Brain Synchronization during Social Interaction
Dumas, G., Nadel, J., Soussignan, R., Martinerie, J., & Garnero, L. (2010) · PLoS ONE · doi:10.1371/journal.pone.0012166 · Cited by 966
Olah, C., Mordvintsev, A., & Schubert, L. (2017) · Distill · doi:10.23915/distill.00007 · Cited by 880
Cadieu, C. F., Hong, H., Yamins, D. L. K., Pinto, N., Ardila, D., Solomon, E. A., Majaj, N. J., & DiCarlo, J. J. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003963 · Cited by 824 · cited 2×
“First and foremost, DNNs can predict neural activation in primate visual cortex better than other preceding models (Cadieu et al., 2014; Khaligh-Razavi &Kriegeskorte, 2014; Yamins, Hong, Cadieu, Solomon, Seibert, & DiCarlo, 2014; Kubilius, Schrimpf, Nayebi, Bear, Yamins, & DiCarlo, 2018; but see Eberhardt, Cader, & Se…”
BackgroundRevisiting Self-Supervised Visual Representation Learning
Kolesnikov, A., Zhai, X., & Beyer, L. (2019) · 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2019.00202 · Cited by 675
Kar, K., Kubilius, J., Schmidt, K., Issa, E. B., & DiCarlo, J. J. (2019) · Nature Neuroscience · doi:10.1038/s41593-019-0392-5 · Cited by 496
Recurrence is required to capture the representational dynamics of the human visual system
Kietzmann, T. C., Spoerer, C. J., Sörensen, L. K. A., Cichy, R. M., Hauk, O., & Kriegeskorte, N. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1905544116 · Cited by 406
If deep learning is the answer, what is the question?
Saxe, A., Nelli, S., & Summerfield, C. (2021) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-00395-8 · Cited by 397
Integrative benchmarking to advance neurally mechanistic models of human intelligence
Schrimpf, M., Kubilius, J., Lee, M. J., Ratan Murty, N. A., Ajemian, R., & DiCarlo, J. J. (2020) · Neuron · doi:10.1016/j.neuron.2020.07.040 · Cited by 233
Deep neural networks in computational neuroscience
Kietzmann, T. C., McClure, P., & Kriegeskorte, N. (2019) · Oxford Research Encyclopedia of Neuroscience · doi:10.1093/acrefore/9780190264086.013.46 · Cited by 186
Performance vs. competence in human–machine comparisons
Firestone, C. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1905334117 · Cited by 150 · cited 4×
“This excellent performance and the analogy between DNNs and the primate visual cortex have caused a fierce discussion about the use of DNNs as neuroscientific models of the brain (e.g., Rosenholtz, 2017; VanRullen, 2017; Majaj & Pelli, 2018; Kubilius, 2018; Cichy & Kaiser, 2019; Kietzmann, McClure, & Kriegeskorte, 201…”
BackgroundInfluentialAn ecologically motivated image dataset for deep learning yields better models of human vision
Mehrer, J., Spoerer, C. J., Jones, E. C., Kriegeskorte, N., & Kietzmann, T. C. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2011417118 · Cited by 140
Partial success in closing the gap between human and machine vision
Geirhos, R., Narayanappa, K., Mitzkus, B., Thieringer, T., Bethge, M., Wichmann, F. A., & Brendel, W. (2021) · arXiv · doi:10.48550/arxiv.2106.07411 · Cited by 129 · cited 1×
“The trial-by-trial performance of DNNs in perceptual tasks is also consistently different than that of humans (Geirhos et al., 2020; Geirhos et al., 2021).”
BackgroundMulti-level crowding and the paradox of object recognition in clutter
Manassi, M., & Whitney, D. (2018) · Current Biology · doi:10.1016/j.cub.2017.12.051 · Cited by 102 · cited 1×
“Example 1: Crowding Crowding is the ubiquitous breakdown of object recognition in the presence of nearby flankers (Bouma, 1970; Pelli, Palomares, & Majaj, 2004; Levi, 2008; Sayim, Westheimer, & Herzog, 2008; Manassi & Whitney, 2018).”
BackgroundIllusory Motion Reproduced by Deep Neural Networks Trained for Prediction
Watanabe, E., Kitaoka, A., Sakamoto, K., Yasugi, M., & Tanaka, K. (2018) · Frontiers in Psychology · doi:10.3389/fpsyg.2018.00345 · Cited by 101
(2021) · doi:10.1162/jocn_a_01755 · Cited by 85
Perception science in the age of deep neural networks.
VanRullen, R. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.00142 · Cited by 82 · cited 2×
“This excellent performance and the analogy between DNNs and the primate visual cortex have caused a fierce discussion about the use of DNNs as neuroscientific models of the brain (e.g., Rosenholtz, 2017; VanRullen, 2017; Majaj & Pelli, 2018; Kubilius, 2018; Cichy & Kaiser, 2019; Kietzmann, McClure, & Kriegeskorte, 201…”
MethodStorrs, K. R., Khaligh-Razavi, S.-M., & Kriegeskorte, N. (2020) · doi:10.1101/2020.03.23.003046 · Cited by 17
Cited by
The neuroconnectionist research programme
Doerig, A., Sommers, R. P., Seeliger, K., Richards, B., Ismael, J., Lindsay, G. W., Kording, K. P., Konkle, T., van Gerven, M. A. J., Kriegeskorte, N., et al. (2023) · Nature Reviews Neuroscience · doi:10.1038/s41583-023-00705-w · Cited by 233
Self-citationQuilty-Dunn, J., Porot, N., & Mandelbaum, E. (2023) · Behavioral and Brain Sciences · doi:10.1017/s0140525x22002849 · Cited by 205
Deep problems with neural network models of human vision
Bowers, J. S., Malhotra, G., Dujmović, M., Llera Montero, M., Tsvetkov, C., Biscione, V., Puebla, G., Adolfi, F., Hummel, J. E., Heaton, R. F., et al. (2023) · Behavioral and Brain Sciences · doi:10.1017/s0140525x22002813 · Cited by 204
Huber, L. S., Geirhos, R., & Wichmann, F. A. (2023) · Journal of Vision · doi:10.1167/jov.23.7.4 · Cited by 42
Metadata
- Authors
- Lonnqvist, B., Bornet, A., Doerig, A., & Herzog, M. H.
- Venue
- Journal of Vision
- Published
- 2021-09-22
- Volume
- 21
- Issue
- 10
- Pages
- 17
- ISSN
- 1534-7362 (ISSN-L)
- OpenAlex
- W3200213953
- Cited by
- 21
- References
- 28
- FWCI
- 1.46
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.