Finite Versus Infinite Neural Networks: an Empirical Study
Open accessThe DOI matches its OpenAlex record; 25 of the 25 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
Zagoruyko, S., & Komodakis, N. (2016) · Procedings of the British Machine Vision Conference 2016 · doi:10.5244/c.30.87 · Cited by 7,244
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
Xiao, H., Rasul, K., & Vollgraf, R. (2017) · arXiv · doi:10.48550/arxiv.1708.07747 · Cited by 6,108
mixup: Beyond Empirical Risk Minimization
Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017) · arXiv · doi:10.48550/arxiv.1710.09412 · Cited by 5,831
RandAugment: Practical automated data augmentation with a reduced search space
Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1909.13719 · Cited by 3,223 · cited 1×
“Extending this strategy to more augmentations such as crop or mixup [108], or to broader augmentations strategies like AutoAugment [109] and RandAugment [110], becomes rapidly infeasible.”
BackgroundRandaugment: Practical automated data augmentation with a reduced search space
Cubuk, E. D., Zoph, B., Shlens, J., & Le, Q. V. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) · doi:10.1109/cvprw50498.2020.00359 · Cited by 3,223
End to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048 · cited 1×
“Understanding biases in models is of critical importance if we hope to prevent them from perpetuating and exaggerating existing racial, gender, and other social biases [120–123].”
BackgroundPopular Ensemble Methods: An Empirical Study
Opitz, D., & Maclin, R. (1999) · Journal of Artificial Intelligence Research · doi:10.1613/jair.614 · Cited by 2,953
Understanding deep learning requires rethinking generalization
Zhang, C., Bengio, S., Hardt, M., Recht, B., & Vinyals, O. (2016) · arXiv · doi:10.48550/arxiv.1611.03530 · Cited by 2,838 · cited 1×
“This has led to a great deal of work on the interplay of overparameterization and generalization [93–102].”
BackgroundRokach, L. (2010) · Artificial Intelligence Review · doi:10.1007/s10462-009-9124-7 · Cited by 2,657
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 1×
“Understanding biases in models is of critical importance if we hope to prevent them from perpetuating and exaggerating existing racial, gender, and other social biases [120–123].”
BackgroundThe “independent components” of natural scenes are edge filters
Bell, A. J., & Sejnowski, T. J. (1997) · Vision Research · doi:10.1016/s0042-6989(97)00121-1 · Cited by 2,201
Reconciling modern machine-learning practice and the classical bias–variance trade-off
Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1903070116 · Cited by 1,595
Neural Tangent Kernel: Convergence and Generalization in Neural Networks
Jacot, A., Gabriel, F., & Hongler, C. (2018) · arXiv · doi:10.48550/arxiv.1806.07572 · Cited by 1,562
AutoAugment: Learning Augmentation Policies from Data
Cubuk, E. D., Zoph, B., Mane, D., Vasudevan, V., & Le, Q. V. (2018) · arXiv · doi:10.48550/arxiv.1805.09501 · Cited by 1,106
An Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969
Deep double descent: where bigger models and more data hurt*
Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., & Sutskever, I. (2021) · Journal of Statistical Mechanics: Theory and Experiment · doi:10.1088/1742-5468/ac3a74 · Cited by 565 · cited 1×
“Of particular interest has been the phenomenon of double descent, in which performance increases overall with parameter account, but drops dramatically when the neural network is roughly critically parameterized [103–105].”
BackgroundAn Empirical Model of Large-Batch Training
McCandlish, S., Kaplan, J., Amodei, D., & Team, O. D. (2018) · arXiv · doi:10.48550/arxiv.1812.06162 · Cited by 543
Measuring the Effects of Data Parallelism on Neural Network Training
Shallue, C. J., Lee, J., Antognini, J., Sohl-Dickstein, J., Frostig, R., & Dahl, G. E. (2018) · arXiv · doi:10.48550/arxiv.1811.03600 · Cited by 229
Self-citationDeep learning generalizes because the parameter-function map is biased towards simple functions
Valle-Pérez, G., Camargo, C. Q., & Louis, A. A. (2018) · arXiv · doi:10.48550/arxiv.1805.08522 · Cited by 105
Infinite attention: NNGP and NTK for deep attention networks
Hron, J., Bahri, Y., Sohl-Dickstein, J., & Novak, R. (2020) · arXiv · doi:10.48550/arxiv.2006.10540 · Cited by 40 · cited 2×
“It has also enabled practical advances in kernel methods [8, 9, 15, 16, 26, 41–43], Bayesian deep learning [44–46], active learning [47], and semi-supervised learning [17].”
BackgroundMethodSelf-citationOber, S. W., & Aitchison, L. (2020) · arXiv · doi:10.48550/arxiv.2005.08140 · Cited by 20
Scalable Uncertainty for Computer Vision With Functional Variational Inference
Carvalho, E. D. C., Clark, R., Nicastro, A., & Kelly, P. H. J. (2020) · 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr42600.2020.01202 · Cited by 14
A Gaussian Process perspective on Convolutional Neural Networks
Borovykh, A. (2018) · arXiv · doi:10.48550/arxiv.1810.10798 · Cited by 9
Cited by
Explaining Neural Scaling Laws
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., & Sharma, U. (2021) · arXiv · doi:10.48550/arxiv.2102.06701 · Cited by 182
“If we further assume, however, that f T on average has been trained such as to be sufficiently close to a local minimum of the loss, such that , then typical instances will also obey the O ( ϵ ) scaling.”
BackgroundSelf-citationExplaining neural scaling laws
Bahri, Y., Dyer, E., Kaplan, J., Lee, J., & Sharma, U. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2311878121 · Cited by 181
Self-citationCross-modal autoencoder framework learns holistic representations of cardiovascular state
Radhakrishnan, A., Friedman, S. F., Khurshid, S., Ng, K., Batra, P., Lubitz, S. A., Philippakis, A. A., & Uhler, C. (2023) · Nature Communications · doi:10.1038/s41467-023-38125-0 · Cited by 82
“We considered the NTK since it was shown to have superior performance on supervised learning problems 69,70 .”
Representations and generalization in artificial and brain neural networks
Li, Q., Sorscher, B., & Sompolinsky, H. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2311805121 · Cited by 27
Metadata
- Authors
- Lee, J., Schoenholz, S. S., Pennington, J., Adlam, B., Xiao, L., Novak, R., & Sohl‐Dickstein, J.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2020-07-31
- Volume
- 33
- Pages
- 15156-15172
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2007.15801
- OpenAlex
- W3101069636
- Cited by
- 70
- References
- 25
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.