Engineering a Less Artificial Intelligence
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
Despite enormous progress in machine learning, artificial neural networks still lag behind brains in their ability to generalize to new situations. Given identical training data, differences in generalization are caused by many defining features of a learning algorithm, such as network architecture and learning rule. Their joint effect, called "inductive bias," determines how well any learning algorithm-or brain-generalizes: robust generalization needs good inductive biases. Artificial networks use rather nonspecific biases and often latch onto patterns that are only informative about the statistics of the training data but may not generalize to different scenarios. Brains, on the other hand, generalize across comparatively drastic changes in the sensory input all the time. We highlight some shortcomings of state-of-the-art learning algorithms compared to biological brains and discuss several ideas about how neuroscience can guide the quest for better inductive biases by providing useful constraints on representations and network architecture.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
A logical calculus of the ideas immanent in nervous activity
McCulloch, W. S., & Pitts, W. (1943) · The Bulletin of Mathematical Biophysics · doi:10.1007/bf02478259 · Cited by 18,566
No free lunch theorems for optimization
Wolpert, D. H., & Macready, W. G. (1997) · IEEE Transactions on Evolutionary Computation · doi:10.1109/4235.585893 · Cited by 14,237
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
Intriguing properties of neural networks
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013) · arXiv · doi:10.48550/arxiv.1312.6199 · Cited by 8,143
Towards Deep Learning Models Resistant to Adversarial Attacks
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017) · arXiv · doi:10.48550/arxiv.1706.06083 · Cited by 6,602
DeepLabCut: markerless pose estimation of user-defined body parts with deep learning
Mathis, A., Mamidanna, P., Cury, K. M., Abe, T., Murthy, V. N., Mathis, M. W., & Bethge, M. (2018) · Nature Neuroscience · doi:10.1038/s41593-018-0209-y · Cited by 5,187
Self-citationDeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., & Darrell, T. (2013) · arXiv · doi:10.48550/arxiv.1310.1531 · Cited by 4,019
Hierarchical models of object recognition in cortex
Riesenhuber, M., & Poggio, T. (1999) · Nature Neuroscience · doi:10.1038/14819 · Cited by 3,309 · cited 2×
“Past attempts of including structure from neuroscience were focused on first-order properties, such as Gabor-shaped receptive fields in early layers, or other functional properties derived from parametric models of neuronal activity (Riesenhuber and Poggio, 1999).”
BackgroundMethodA Survey on Multi-Task Learning
Zhang, Y., & Yang, Q. (2017) · arXiv · doi:10.48550/arxiv.1707.08114 · Cited by 2,910
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838
Deep visual domain adaptation: A survey
Wang, M., & Deng, W. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.05.083 · Cited by 2,209
Fully integrated silicon probes for high-density recording of neural activity
Jun, J. J., Steinmetz, N. A., Siegle, J. H., Denman, D. J., Bauza, M., Barbarits, B., Lee, A. K., Anastassiou, C. A., Andrei, A., Aydın, Ç., et al. (2017) · Nature · doi:10.1038/nature24636 · Cited by 2,190
Performance-optimized hierarchical models predict neural responses in higher visual cortex
Yamins, D. L. K., Hong, H., Cadieu, C. F., Solomon, E. A., Seibert, D., & DiCarlo, J. J. (2014) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1403112111 · Cited by 1,924
How to grow a mind: Statistics, structure, and abstraction
Tenenbaum, J. B., Kemp, C., Griffiths, T. L., & Goodman, N. D. (2011) · Science · doi:10.1126/science.1192788 · Cited by 1,815
Using goal-driven deep learning models to understand sensory cortex
Yamins, D. L. K., & DiCarlo, J. J. (2016) · Nature Neuroscience · doi:10.1038/nn.4244 · Cited by 1,621
Deep supervised, but not unsupervised, models may explain IT cortical representation
Khaligh-Razavi, S.-M., & Kriegeskorte, N. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003915 · Cited by 1,204
London, M., & Häusser, M. (2005) · Annual Review of Neuroscience · doi:10.1146/annurev.neuro.28.061604.135703 · Cited by 1,149 · cited 1×
“Real neurons are vastly more complex machines, with nonlinear interactions within and between different branches of their dendrites (London and H € ausser, 2005).”
BackgroundTaskonomy: Disentangling Task Transfer Learning
Zamir, A. R., Sax, A., Shen, W., Guibas, L., Malik, J., & Savarese, S. (2018) · 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2018.00391 · Cited by 1,128
Graves, A., Wayne, G., & Danihelka, I. (2014) · arXiv · doi:10.48550/arxiv.1410.5401 · Cited by 1,058 · cited 1×
“…modules according to this network structure remains unknown, although some work has started to explore such motifs as memory modules (Weston et al., 2014; Graves et al., 2014) or elements akin to voluntary eye movements or attention (Jaderberg et al., 2015; Mnih et al., 2014; Vaswani et al., 2017).”
BackgroundGuclu, U., & van Gerven, M. A. J. (2015) · Journal of Neuroscience · doi:10.1523/jneurosci.5023-14.2015 · Cited by 961
Principles of connectivity among morphologically defined cell types in adult neocortex
Jiang, X., Shen, S., Cadwell, C. R., Berens, P., Sinz, F., Ecker, A. S., Patel, S., & Tolias, A. S. (2015) · Science · doi:10.1126/science.aac9462 · Cited by 908
Self-citationCadieu, C. F., Hong, H., Yamins, D. L. K., Pinto, N., Ardila, D., Solomon, E. A., Majaj, N. J., & DiCarlo, J. J. (2014) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003963 · Cited by 824 · cited 7×
“As far as we know, training on a single task—albeit using a large dataset such as ImageNet—is not sufficient for this purpose (see Section 2).”
MethodInfluentialA functional microcircuit for cat visual cortex
Douglas, R. J., & Martin, K. A. (1991) · The Journal of Physiology · doi:10.1113/jphysiol.1991.sp018733 · Cited by 736 · cited 1×
“Cortical microcircuits comprise many genetically and functionally distinct cell types (Douglas and Martin, 1991; Jiang et al., 2015) that may perform operations like gating, homeostatic regulation, divisive normalization, and more sophisticated operations.”
BackgroundLearning multiple visual domains with residual adapters
Rebuffi, S.-A., Bilen, H., & Vedaldi, A. (2017) · arXiv · doi:10.48550/arxiv.1705.08045 · Cited by 605
A model of inductive bias learning
Baxter, J. (2000) · Journal of Artificial Intelligence Research · doi:10.1613/jair.731 · Cited by 582 · cited 4×
“From empirical and theoretical results in machine learning, we also know that solving many tasks at once helps improve the inductive bias of a class of networks by constraining the space of solutions (Caruana, 1993; Baxter, 2000; Zhang and Yang, 2017; Ruder, 2017).”
BackgroundInfluentialApproximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
Brendel, W., & Bethge, M. (2019) · arXiv · doi:10.48550/arxiv.1904.00760 · Cited by 323
Self-citationPonce, C. R., Xiao, W., Schade, P. F., Hartmann, T. S., Kreiman, G., & Livingstone, M. S. (2019) · Cell · doi:10.1016/j.cell.2019.04.005 · Cited by 245
Bottger, J., Schafer, A., Lohmann, G., Villringer, A., & Margulies, D. S. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2013.114 · Cited by 59
Cited by
Shortcut learning in deep neural networks
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-00257-z · Cited by 2,224
Self-citationA massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence
Allen, E. J., St-Yves, G., Wu, Y., Breedlove, J. L., Prince, J. S., Dowdle, L. T., Nau, M., Caron, B., Pestilli, F., Charest, I., et al. (2022) · Nature Neuroscience · doi:10.1038/s41593-021-00962-x · Cited by 577
Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
Das, A., & Rad, P. (2020) · arXiv · doi:10.48550/arxiv.2006.11371 · Cited by 504
If deep learning is the answer, what is the question?
Saxe, A., Nelli, S., & Summerfield, C. (2021) · Nature Reviews Neuroscience · doi:10.1038/s41583-020-00395-8 · Cited by 397
“The recent literature is studded with comparisons of the behaviour and activity of biological and artificial systems 7–21 , summarized in a growing number of review articles 22–30 .”
BackgroundCatalyzing next-generation Artificial Intelligence through NeuroAI
Zador, A., Escola, S., Richards, B., Ölveczky, B., Bengio, Y., Boahen, K., Botvinick, M., Chklovskii, D., Churchland, A., Clopath, C., et al. (2023) · Nature Communications · doi:10.1038/s41467-023-37180-x · Cited by 300
Self-citation
Metadata
- Authors
- Sinz, F. H., Pitkow, X., Reimer, J., Bethge, M., & Tolias, A. S.
- Venue
- Neuron
- Publisher
- Elsevier BV
- Published
- 2019-09
- Volume
- 103
- Issue
- 6
- Pages
- 967-979
- ISSN
- 0896-6273 (ISSN-L)
- OpenAlex
- W2975512858
- Cited by
- 167
- References
- 31
- FWCI
- 10.25
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.