Performance vs. competence in human–machine comparisons
Open accessAbstract
Does the human mind resemble the machines that can behave like it? Biologically inspired machine-learning systems approach “human-level” accuracy in an astounding variety of domains, and even predict human brain activity—raising the exciting possibility that such systems represent the world like we do. However, even seemingly intelligent machines fail in strange and “unhumanlike” ways, threatening their status as models of our minds. How can we know when human–machine behavioral differences reflect deep disparities in their underlying capacities, vs. when such failures are only superficial or peripheral? This article draws on a foundational insight from cognitive science—the distinction between performance and competence —to encourage “species-fair” comparisons between humans and machines. The performance/competence distinction urges us to consider whether the failure of a system to behave as ideally hypothesized, or the failure of one creature to behave like another, arises not because the system lacks the relevant knowledge or internal capacities (“competence”), but instead because of superficial constraints on demonstrating that knowledge (“performance”). I argue that this distinction has been neglected by research comparing human and machine behavior, and that it should be essential to any such comparison. Focusing on the domain of image classification, I identify three factors contributing to the species-fairness of human–machine comparisons, extracted from recent work that equates such constraints. Species-fair comparisons level the playing field between natural and artificial intelligence, so that we can separate more superficial differences from those that may be deep and enduring.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Without representational evidence, behavioral similarities between models and humans remain speculative.
“Without representational evidence, behavioral similarities between models and humans remain speculative”
From From Human Cognition to Neural Activations: Probing the Computational Primitives of Spatial Reasoning in LLMs (2026) · cites this paper as Firestone, 2020
In “What Species-Fair Comparisons Show” (page not supplied), Firestone says humanlike behavior alone is insufficient to establish humanlike underlying processes. But the paper does not require representational evidence: it treats species-fair behavioral comparisons as valuable evidence and says a mixed strategy is likely best. (LLM, conf=0.87)
Experimental — accuracy not yet measuredWithout a fair comparison that evaluates humans in the same settings, strong claims cannot be made about whether the properties listed in Table 1 make LLMs qualitatively unlike humans.
“Without a fair comparison (59–63) that evaluates humans in the same settings, we cannot make strong claims about whether the properties listed in Table 1 make LLMs qualitatively unlike humans.”
From Embers of autoregression show how large language models are shaped by the problem they are trained to solve (2024) · cites this paper as [60]
Similar performance-competence issues apply when assessing the capabilities of machine learning systems and particularly LLMs.
“More recent work has suggested that similar issues apply when assessing the capabilities of machine learning systems (Firestone, 2020), and particularly LLMs (Lampinen, 2022).”
From Machine Psychology (2023) · cites this paper as Firestone, 2020
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759 · see citing works
ASPECTS OF THE THEORY OF SYNTAX
Chomsky, N. (1964) · doi:10.21236/ad0616323 · Cited by 13,587 · see citing works
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653 · see citing works
I.—COMPUTING MACHINERY AND INTELLIGENCE
TURING, A. M. (1950) · Mind · doi:10.1093/mind/lix.236.433 · Cited by 10,290 · cited 1× · see citing works
“Both classical and contemporary discussions of such issues tend to focus on machine-learning “ successes ”— e.g., asking what amazing feat a machine must perform in order to be relevantly humanlike (31 – 33).”
BackgroundIntriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143 · see citing works
Computing Machinery and Intelligence (1950)
Turing, A. M. (2004) · doi:10.1093/oso/9780198250791.003.0017 · Cited by 6,120 · see citing works
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images
Nguyen, A., Yosinski, J., & Clune, J. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298640 · Cited by 3,258 · see citing works
Building machines that learn and think like people
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., & Gershman, S. J. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x16001837 · Cited by 2,550 · see citing works
Computer vision and pattern recognition
Zheng, N., Loizou, G. D., Jiang, X., Lan, X., & Li, X. (2007) · International Journal of Computer Mathematics · doi:10.1080/00207160701303912 · Cited by 2,463 · see citing works
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224 · see citing works
Performance-optimized hierarchical models predict neural responses in higher visual cortex
Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1403112111 · Cited by 1,924 · see citing works
Curiosity-Driven Exploration by Self-Supervised Prediction
Pathak, D., Agrawal, P., Efros, A. A., & Darrell, T. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw.2017.70 · Cited by 1,815 · see citing works
Lakhani, P., & Sundaram, B. (2017) · Radiology · doi:10.1148/radiol.2017162326 · Cited by 1,775 · see citing works
Understanding Neural Networks Through Deep VisualizationPreprint
Yosinski, J., Clune, J., Nguyen, A. M., Fuchs, T. J., & Lipson, H. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06579 · Cited by 1,687 · see citing works
Using goal-driven deep learning models to understand sensory cortex
Yamins, D. L. K., & DiCarlo, J. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4244 · Cited by 1,621 · see citing works
Meta-learning with memory-augmented neural networks
Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., & Lillicrap, T. (2016) · International Conference on Machine Learning · Cited by 1,504 · see citing works
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., & Brendel, W. (2018) · arXiv · doi:10.48550/arxiv.1811.12231 · Cited by 1,490 · cited 2× · see citing works
“ANNs may also diverge from humans on images with conflicting cues, as when a cat ’ s shape is rendered in the texture of elephant skin; many ANNs say “ elephant ” , whereas humans know not to (55, 56).”
BackgroundMethodRahwan, I., Cebrian, M., Obradovich, N., Bongard, J., Bonnefon, J.-F., Breazeal, C., Crandall, J. W., Christakis, N. A., Couzin, I. D., Jackson, M. O., et al. (2019) · Nature · doi:10.1038/s41586-019-1138-y · Cited by 1,364 · see citing works
Deep supervised, but not unsupervised, models may explain IT cortical representation
Khaligh-Razavi, S.-M., & Kriegeskorte, N. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003915 · Cited by 1,204 · see citing works
Kriegeskorte, N. (2015) · Annual Review of Vision Science · doi:10.1146/annurev-vision-082114-035447 · Cited by 1,093 · see citing works
Adversarial attacks on medical machine learning
Finlayson, S. G., Bowers, J. D., Ito, J., Zittrain, J. L., Beam, A. L., & Kohane, I. S. (2019) · Science · doi:10.1126/science.aaw4399 · Cited by 1,088 · see citing works
Traffic-Sign Detection and Classification in the Wild
Zhu, Z., Liang, D., Zhang, S., Huang, X., Li, B., & Hu, S. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.232 · Cited by 1,062 · see citing works
Synthesizing Robust Adversarial ExamplesPreprint
Athalye, A., Engstrom, L., Ilyas, A., & Kwok, K. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.07397 · Cited by 973 · see citing works
Adversarial Examples Are Not Bugs, They Are Features
Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., & Madry, A. (2019) · arXiv · doi:10.48550/arxiv.1905.02175 · Cited by 971 · see citing works
Guclu, U., & van Gerven, M. A. J. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.5023-14.2015 · Cited by 961 · see citing works
A feedforward architecture accounts for rapid categorization
Serre, T., Oliva, A., & Poggio, T. (2007) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0700622104 · Cited by 918 · see citing works
Deep Learning: A Critical Appraisal
Marcus, G. (2018) · arXiv · doi:10.48550/arxiv.1801.00631 · Cited by 853 · see citing works
Cadieu, C. F., Hong, H., Yamins, D. L. K., Pinto, N., Ardila, D., Solomon, E. A., Majaj, N. J., & DiCarlo, J. J. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003963 · Cited by 824 · see citing works
Cichy, R. M., Khosla, A., Pantazis, D., Torralba, A., & Oliva, A. (2016) · Scientific Reports · doi:10.1038/srep27755 · Cited by 716 · see citing works
Terrace, H. S., Petitto, L. A., Sanders, R. J., & Bever, T. G. (1979) · Science · doi:10.1126/science.504995 · Cited by 687 · see citing works
High-Frequency Component Helps Explain the Generalization of Convolutional Neural Networks
Wang, H., Wu, X., Huang, Z., & Xing, E. P. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00871 · Cited by 679 · see citing works
Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future
Lindsay, G. W. (2020) · Journal of Cognitive Neuroscience · doi:10.1162/jocn_a_01544 · Cited by 571 · see citing works
A critique of pure learning and what artificial neural networks can learn from animal brains
Zador, A. M. (2019) · Nature Communications · doi:10.1038/s41467-019-11786-6 · Cited by 553 · see citing works
ViZDoom: A Doom-based AI research platform for visual reinforcement learning
Kempka, M., Wydmuch, M., Runc, G., Toczek, J., & Jaśkowski, W. (2016) · IEEE Conference on Computational Intelligence and Games (CIG) · doi:10.1109/cig.2016.7860433 · Cited by 550 · see citing works
Kell, A. J. E., Yamins, D. L. K., Shook, E. N., Norman-Haignere, S. V., & McDermott, J. H. (2018) · Neuron · doi:10.1016/j.neuron.2018.03.044 · Cited by 544 · see citing works
Brown, T. B., Mané, D., Roy, A., Abadi, M., & Gilmer, J. (2017) · arXiv · doi:10.48550/arxiv.1712.09665 · Cited by 509 · see citing works
Brain-Score: Which Artificial Neural Network for Object Recognition is most Brain-Like?Preprint
Schrimpf, M., Kubilius, J., Hong, H., Majaj, N. J., Rajalingham, R., Issa, E. B., Kar, K., Bashivan, P., Prescott-Roy, J., Geiger, F., et al. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/407007 · Cited by 463 · see citing works
Synthesizing Robust Adversarial Examples
Athalye, A., Engstrom, L., Ilyas, A., & Kwok, K. (2018) · International Conference on Machine Learning · Cited by 432 · see citing works
Neural population control via deep image synthesis
Bashivan, P., Kar, K., & DiCarlo, J. J. (2019) · Science · doi:10.1126/science.aav9436 · Cited by 426 · see citing works
A Study and Comparison of Human and Deep Learning Recognition Performance under Visual Distortions
Dodge, S., & Karam, L. J. (2017) · International Conference on Computer Communications and Networks (ICCCN) · doi:10.1109/icccn.2017.8038465 · Cited by 408 · see citing works
Rajalingham, R., Issa, E. B., Bashivan, P., Kar, K., Schmidt, K., & DiCarlo, J. J. (2018) · Journal of Neuroscience · doi:10.1523/jneurosci.0388-18.2018 · Cited by 360 · see citing works
Deep convolutional networks do not classify based on global object shape
Baker, N., Lu, H., Erlikhman, G., & Kellman, P. J. (2018) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1006613 · Cited by 346 · see citing works
Visual Turing test for computer vision systems
Geman, D., Geman, S., Hallonquist, N., & Laurent Younes, B. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1422953112 · Cited by 323 · see citing works
Intuitions about support in 4.5-month-old infants
Needham, A. W., & Renée Baillargeon, B. (1993) · Cognition · doi:10.1016/0010-0277(93)90002-d · Cited by 301 · see citing works
Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNetPreprint
Kümmerer, M., Theis, L., & Bethge, M. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1411.1045 · Cited by 299 · see citing works
Deep Neural Networks as a Computational Model for Human Shape Sensitivity
Kubilius, J., Bracci, S., & de Beeck, H. O. (2016) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1004896 · Cited by 296 · see citing works
Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects
Alcorn, M. A., Li, Q., Gong, Z., Wang, C., Mai, L., Ku, W., & Nguyen, A. (2019) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2019.00498 · Cited by 265 · see citing works
Deep Learning: The Good, the Bad, and the Ugly
Serre, T. (2019) · Annual Review of Vision Science · doi:10.1146/annurev-vision-091718-014951 · Cited by 261 · see citing works
Ponce, C. R., Xiao, W., Schade, P. F., Hartmann, T. S., Kreiman, G., & Livingstone, M. S. (2019) · Cell · doi:10.1016/j.cell.2019.04.005 · Cited by 245 · see citing works
Analysing Mathematical Reasoning Abilities of Neural Models
Saxton, D., Grefenstette, E., Hill, F., & Kohli, P. (2019) · arXiv · doi:10.48550/arxiv.1904.01557 · Cited by 241 · see citing works
Allen, C., & Bekoff, M. (1997) · The MIT Press eBooks · doi:10.7551/mitpress/6395.001.0001 · Cited by 221 · see citing works
On the Measure of Intelligence
Chollet, F. (2019) · arXiv · doi:10.48550/arxiv.1911.01547 · Cited by 218 · see citing works
Adversarial Examples that Fool both Computer Vision and Time-Limited HumansPreprint
Elsayed, G. F., Shankar, S., Cheung, B., Papernot, N., Kurakin, A., Goodfellow, I., & Sohl‐Dickstein, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1802.08195 · Cited by 214 · see citing works
Striking individual differences in color perception uncovered by ‘the dress’ photograph
Lafer-Sousa, R., Hermann, K. L., & Conway, B. R. (2015) · Current Biology · doi:10.1016/j.cub.2015.04.053 · Cited by 208 · see citing works
LaVAN: Localized and Visible Adversarial NoisePreprint
Karmon, D., Zoran, D., & Goldberg, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.02608 · Cited by 203 · see citing works
Early Methods for Detecting Adversarial ImagesPreprint
Hendrycks, D., & Gimpel, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.00530 · Cited by 191 · see citing works
The Origins and Prevalence of Texture Bias in Convolutional Neural NetworksPreprint
Hermann, K. L., Chen, T., & Kornblith, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.09071 · Cited by 177 · see citing works
Ducklings imprint on the relational concept of “same or different”
Martinho, A., & Kacelnik, A. (2016) · Science · doi:10.1126/science.aaf4247 · Cited by 164 · see citing works
Adversarial Manipulation of Deep RepresentationsPreprint
Sabour, S., Cao, Y., Faghri, F., & Fleet, D. J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1511.05122 · Cited by 164 · see citing works
Atoms of recognition in human and computer vision
Ullman, S., Assif, L., Fetaya, E., & Harari, D. (2016) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1513198113 · Cited by 151 · see citing works
Empiricism without magic: transformational abstraction in deep convolutional neural networks
Buckner, C. (2018) · Synthese · doi:10.1007/s11229-018-01949-1 · Cited by 149 · see citing works
Comparing machines and humans on a visual categorization test
Fleuret, F., Li, T., Dubout, C., Wampler, E. K., Yantis, S., & Geman, D. (2011) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1109168108 · Cited by 142 · see citing works
Simulating a Primary Visual Cortex at the Front of CNNs Improves Robustness to Image PerturbationsPreprint
Dapello, J., Marques, T., Schrimpf, M., Geiger, F., Cox, D., & DiCarlo, J. J. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.06.16.154542 · Cited by 137 · see citing works
Jozwik, K. M., Kriegeskorte, N., Storrs, K. R., & Mur, M. (2017) · Frontiers in Psychology · doi:10.3389/fpsyg.2017.01726 · Cited by 133 · see citing works
Does Object Recognition Work for Everyone?Preprint
DeVries, T., Misra, I., Wang, C., & van der Maaten, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.02659 · Cited by 132 · see citing works
Morgan’s Canon, meet Hume’s Dictum: avoiding anthropofabulation in cross-species comparisons
Buckner, C. J. (2013) · Biology & Philosophy · doi:10.1007/s10539-013-9376-0 · Cited by 114 · see citing works
Cognitive Psychology for Deep Neural Networks: A Shape Bias Case StudyPreprint
Ritter, S., Barrett, D. G. T., Santoro, A., & Botvinick, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.08606 · Cited by 114 · see citing works
Humans can decipher adversarial images
Zhou, Z., & Firestone, C. (2019) · Nature Communications · doi:10.1038/s41467-019-08931-6 · Cited by 113 · see citing works
Self-citationNot-So-CLEVR: learning same–different relations strains feedforward neural networks
Kim, J., Ricci, M. R., & Serre, T. (2018) · Interface Focus · doi:10.1098/rsfs.2018.0011 · Cited by 111 · see citing works
Kuzovkin, I., Vicente, R., Petton, M., Lachaux, J.-P., Baciu, M., Kahane, P., Rheims, S., Vidal, J. R., & Aru, J. (2018) · Communications Biology · doi:10.1038/s42003-018-0110-y · Cited by 104 · see citing works
Computational mechanisms underlying cortical responses to the affordance properties of visual scenes
Bonner, M. F., & Epstein, R. A. (2018) · PLOS Computational Biology · doi:10.1371/journal.pcbi.1006111 · Cited by 94 · see citing works
Humans can decipher adversarial images
Zhou, Z., & Firestone, C. (2019) · RePEc: Research Papers in Economics · Cited by 87 · see citing works
Self-citation(2021) · doi:10.1162/jocn_a_01755 · Cited by 85 · see citing works
'Artiphysiology' reveals V4-like shape tuning in a deep network trained for image classification
Pospisil, D. A., Pasupathy, A., & Bair, W. (2018) · eLife · doi:10.7554/elife.38242 · Cited by 72 · see citing works
PsyPhy: A Psychophysics Driven Evaluation Framework for Visual Recognition
RichardWebster, B., Anthony, S. E., & Scheirer, W. J. (2018) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2849989 · Cited by 68 · see citing works
The hard lessons of DARPA's robotics challenge [News]
Guizzo, E., & Ackerman, E. (2015) · IEEE Spectrum · doi:10.1109/mspec.2015.7164385 · Cited by 58 · see citing works
Crosby, M., Beyret, B., & Halina, M. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0050-3 · Cited by 54 · see citing works
Canaan, R., Salge, C., Togelius, J., & Nealen, A. (2019) · International Conference on the Foundations of Digital Games (FDG) · doi:10.1145/3337722.3337750 · Cited by 54 · see citing works
Deep learning—Using machine learning to study biological vision
Majaj, N. J., & Pelli, D. G. (2018) · Journal of Vision · doi:10.1167/18.13.2 · Cited by 50 · see citing works
25 Years of CNNs: Can We Compare to Human Abstraction Capabilities?
Stabinger, S., Rodríguez-Sánchez, A., & Piater, J. (2016) · Lecture notes in computer science · doi:10.1007/978-3-319-44781-0_45 · Cited by 47 · see citing works
Brendel, W., Rauber, J., Kurakin, A., Papernot, N., Veliqi, B., Mohanty, S. P., Laurent, F., Salathé, M., Bethge, M., Yu, Y., et al. (2019) · The Springer series on challenges in machine learning · doi:10.1007/978-3-030-29135-8_5 · Cited by 46 · see citing works
Comparing the Visual Representations and Performance of Humans and Deep Neural Networks
Jacobs, R. A., & Bates, C. (2018) · Current Directions in Psychological Science · doi:10.1177/0963721418801342 · Cited by 45 · see citing works
What do adversarial images tell us about human vision?
Dujmović, M., Malhotra, G., & Bowers, J. S. (2020) · eLife · doi:10.7554/elife.55978 · Cited by 44 · see citing works
High Frequency Component Helps Explain the Generalization of Convolutional Neural NetworksPreprint
Wang, H., Wu, X., Huang, Z., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.13545 · Cited by 43 · see citing works
Predicting eye movement patterns from fMRI responses to natural scenes
O’Connell, T. P., & Chun, M. M. (2018) · Nature Communications · doi:10.1038/s41467-018-07471-9 · Cited by 39 · see citing works
Apply rich psychological terms in AI with care
Shevlin, H., & Halina, M. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0039-y · Cited by 38 · see citing works
Diverse deep neural networks all predict human IT well, after training and fittingPreprint
Storrs, K. R., Kietzmann, T. C., Walther, A., Mehrer, J., & Kriegeskorte, N. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.05.07.082743 · Cited by 37 · see citing works
A competence framework for artificial intelligence research
Miracchi, L. J. (2019) · Philosophical Psychology · doi:10.1080/09515089.2019.1607692 · Cited by 32 · see citing works
Emergent Properties of Foveated Perceptual SystemsPreprint
Deza, A., & Konkle, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.07991 · Cited by 29 · see citing works
2011 IEEE conference on computational intelligence and games
(2010) · IEEE Transactions on Computational Intelligence and AI in Games · doi:10.1109/tciaig.2011.2106610 · Cited by 27 · see citing works
Fantastic DNimals and where to find them
Scholte, H. S. (2017) · NeuroImage · doi:10.1016/j.neuroimage.2017.12.077 · Cited by 27 · see citing works
Leveling the Playing Field -- Fairness in AI Versus Human Game BenchmarksPreprint
Canaan, R., Salge, C., Togelius, J., & Nealen, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.07008 · Cited by 21 · see citing works
Naive Realism and the Science of (Some) Illusions
Phillips, I. B. (2016) · Philosophical Topics · doi:10.5840/philtopics201644227 · Cited by 20 · see citing works
Modeling Biological Immunity to Adversarial Examples
Kim, E., Rego, J., Watkins, Y., & Kenyon, G. T. (2020) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr42600.2020.00472 · Cited by 20 · see citing works
Deep Nets: What have they ever done for Vision?Preprint
Yuille, A., & Liu, C. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04025 · Cited by 19 · see citing works
Visible Progress on Adversarial Images and a New Saliency Map.
Hendrycks, D., & Gimpel, K. (2016) · Cited by 18 · see citing works
The Notorious Difficulty of Comparing Human and Machine Perception
Funke, C. M., Borowski, J., Stosio, K., Brendel, W., Wallis, T. S. A., & Bethge, M. (2020) · Cited by 15 · see citing works
Exploring Perceptual Illusions in Deep Neural NetworksPreprint
Ward, E. J. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/687905 · Cited by 11 · see citing works
Deep Neural Networks Carve the Brain at its JointsPreprint
Bertolero, M. A., & Bassett, D. S. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.02.20.958082 · Cited by 11 · see citing works
Can You Hear Me <i>Now</i>? Sensitive Comparisons of Human and Machine Perception
Lepori, M. A., & Firestone, C. (2022) · Cognitive Science · doi:10.1111/cogs.13191 · Cited by 7 · see citing works
Self-citationEvidence from machines that learn and think like people
Forbus, K. D., & Gentner, D. (2017) · Behavioral and Brain Sciences · doi:10.1017/s0140525x17000139 · Cited by 7 · see citing works
The Comparative Psychology of Artificial IntelligencesPreprint
Buckner, C. J. (2019) · PhilSci-Archive (University of Pittsburgh) · Cited by 6 · see citing works
Minimal Images in Deep Neural Networks: Fragile Object Recognition in Natural ImagesPreprint
Srivastava, S., Ben-Yosef, G., & Boix, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.03227 · Cited by 4 · see citing works
Can you hear me $\textit{now}$? Sensitive comparisons of human and machine perceptionPreprint
Lepori, M. A., & Firestone, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.12362 · Cited by 1 · see citing works
Self-citationAdversarial Vision ChallengePreprint
Brendel, W., Rauber, J., Kurakin, A., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.01976
Shortcut Learning in Deep Neural Networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., et al. (2020) · arXiv · doi:10.48550/arxiv.2004.07780
Five Points to Check when Comparing Visual Perception in Humans and Machines
Funke, C. M., Borowski, J., Stosio, K., Brendel, W., Wallis, T. S. A., & Bethge, M. (2020) · arXiv · doi:10.48550/arxiv.2004.09406
Deep Neural Networks Carve the Brain at its Joints
Bertolero, M. A., Moraczewski, D., Thomas, A., & Bassett, D. S. (2020) · arXiv · doi:10.48550/arxiv.2002.08891
Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects
Alcorn, M. A., Li, Q., Gong, Z., Wang, C., Mai, L., Ku, W.-S., & Nguyen, A. (2018) · arXiv · doi:10.48550/arxiv.1811.11553
Flagged first, then by citations
Research works printed but not linked: 3
- J. Buolamwini, T. Gebru, "Gender shades" in Proceedings of the First Conference on Fairness, Accountability and Transparency (Association for Computing Machinery, New York, NY, 2018), pp. 77-91.
- M. Crosby, B. Beyret, M. Halina, Animal-AI Olympics. Nat. Mach. Intell. 1, 257 (2019).
- S. Dodge, L. Karam, "Human and deep learning recognition performance under visual distortions" in International Conference on Computer Communication and Networks (Institute of Electrical and Electronics Engineers Inc., 2017).
Cited by
Sorted by citations
Testing theory of mind in large language models and humans
Strachan, J. W. A., Albergo, D., Borghini, G., Pansardi, O., Scaliti, E., Gupta, S., Saxena, K., Rufo, A., Panzeri, S., Manzi, G., et al. (2024) · Nature Human Behaviour · doi:10.1038/s41562-024-01882-z · Cited by 299 · see citing works
The neuroconnectionist research programme
Doerig, A., Sommers, R. P., Seeliger, K., Richards, B., Ismael, J., Lindsay, G. W., Kording, K. P., Konkle, T., van Gerven, M. A. J., Kriegeskorte, N., et al. (2023) · Nature Reviews Neuroscience · doi:10.1038/s41583-023-00705-w · Cited by 233 · see citing works
The Fallacy of AI Functionality
Raji, I. D., Kumar, I. E., Horowitz, A., & Selbst, A. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533158 · Cited by 251 · see citing works
Deepfake detection by human crowds, machines, and machine-informed crowds
Groh, M., Epstein, Z., Firestone, C., & Picard, R. (2022) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2110013119 · Cited by 248 · see citing works
Self-citationQuilty-Dunn, J., Porot, N., & Mandelbaum, E. (2023) · Behavioral and Brain Sciences · doi:10.1017/s0140525x22002849 · Cited by 205 · see citing works
McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., & Griffiths, T. L. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2322420121 · Cited by 96 · see citing works
Pellert, M., Lechner, C. M., Wagner, C., Rammstedt, B., & Strohmaier, M. (2024) · Perspectives on Psychological Science · doi:10.1177/17456916231214460 · Cited by 97 · see citing works
Hagendorff, T., Dasgupta, I., Binz, M., Chan, S. C. Y., Lampinen, A., Wang, J. X., Akata, Z., & Schulz, E. (2023) · arXiv · doi:10.48550/arxiv.2303.13988 · Cited by 68 · see citing works
Differences between human and machine perception in medical diagnosis
Makino, T., Jastrzębski, S., Oleszkiewicz, W., Chacko, C., Ehrenpreis, R., Samreen, N., Chhor, C., Kim, E., Lee, J., Pysarenko, K., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-10526-z · Cited by 30 · see citing works
Lampinen, A. (2024) · Computational Linguistics · doi:10.1162/coli_a_00525 · Cited by 32 · see citing works
Re-evaluating Theory of Mind evaluation in large language models
Hu, J., Sosa, F., & Ullman, T. (2025) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2023.0499 · Cited by 10 · see citing works
Disagreement & classification in comparative cognitive science
Boyle, A. (2024) · Noûs · doi:10.1111/nous.12480 · Cited by 23 · see citing works
Can You Hear Me <i>Now</i>? Sensitive Comparisons of Human and Machine Perception
Lepori, M. A., & Firestone, C. (2022) · Cognitive Science · doi:10.1111/cogs.13191 · Cited by 7 · see citing works
Self-citationAnthropocentric Bias in Language Model Evaluation
Millière, R., & Rathkopf, C. (2026) · Computational Linguistics · doi:10.1162/coli.a.582 · Cited by 6 · see citing works
“Competence if often defined as the system’s internal knowledge underlying a particular capacity, and performance as the observable behavior of a system exercising it (Firestone 2020).”
When Can AI Models Explain Learning? Validity Criteria for AI as Cognitive Models in Education
Wang, P., Viberg, O., Law, E. L.-C., & Greiff, S. (2026) · Educational Psychology Review · doi:10.1007/s10648-026-10202-y · Cited by 3 · see citing works
An, J., Yang, L., Wang, M., Lu, L., An, W., & Yang, E. (2026) · arXiv · doi:10.48550/arxiv.2603.26323
“Without representational evidence, behavioral similarities between models and humans remain speculative(Firestone, 2020).”
- Authors
- Firestone, C.
- Affiliations
- Johns Hopkins University (US)
- Corresponding author
- Firestone, Chaz
- ORCID
- Firestone, Chaz
- Venue
- Proceedings of the National Academy of Sciences
- Publisher
- National Academy of Sciences
- Published
- 2020-10-13
- Volume
- 117
- Issue
- 43
- Pages
- 26562-26571
- ISSN
- 0027-8424 (ISSN-L)
- 1091-6490
- PMC
- PMC7604508
- OpenAlex
- W3092323704
- Cited by
- 150
- References
- 112
- Citations by year
- 2026: 19 · 2025: 19 · 2024: 21 · 2023: 31 · 2022: 30 · 2021: 26 · 2020: 4
- FWCI
- 8.94
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Proceedings of the National Academy of Sciences (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- PhilPapers (PhilPapers Foundation) (submittedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Psychiatry, Mental Health, Neuroscience
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Philosophy and Theoretical Science
- Social Sciences › Arts and Humanities › History and Philosophy of Science › Cybernetics and Technology in Society
- Keywords
- human-machine comparison, image classification, machine learning, cognitive science, models of mind, knowledge representation
- MeSH terms
- Machine Learning, Animals, Artificial Intelligence, Brain, Humans, Image Processing, Computer-Assisted, Species Specificity, Visual Perception, Knowledge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
