Feature Learning in Infinite-Width Neural Networks
Open accessAbstract
As its width tends to infinity, a deep neural network's behavior under gradient descent can become simplified and predictable (e.g. given by the Neural Tangent Kernel (NTK)), if it is parametrized appropriately (e.g. the NTK parametrization). However, we show that the standard and NTK parametrizations of a neural network do not admit infinite-width limits that can learn features, which is crucial for pretraining and transfer learning such as with BERT. We propose simple modifications to the standard parametrization to allow for feature learning in the limit. Using the *Tensor Programs* technique, we derive explicit formulas for such limits. On Word2Vec and few-shot learning on Omniglot via MAML, two canonical tasks that rely crucially on feature learning, we compute these limits exactly. We find that they outperform both NTK baselines and finite-width networks, with the latter approaching the infinite-width feature learning performance as width increases. More generally, we classify a natural space of neural network parametrizations that generalizes standard, NTK, and Mean Field parametrizations. We show 1) any parametrization in this space either admits feature learning or has an infinite-width training dynamics given by kernel gradient descent, but not both; 2) any such infinite-width limit can be computed using the Tensor Programs technique. Code for our experiments can be found at github.com/edwardjhu/TP4.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Typically α is set as 16 or 32 and never tuned.
“Typically α is set as 16 or 32 and never tuned (Hu et al., 2022;Yang & Hu, 2020).”
From AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (2023) · cites this paper as Yang & Hu, 2020
The passages from the algorithm section (p. 37) and Table 4 (p. 38) concern α as a hyperparameter of the paper’s own network; the authors explicitly sweep it. Searches for 16, 32, and α tuning found no passage establishing the claimed typical practice, so the paper’s use of α does not verify the citing sentence. (LLM, conf=0.78)
Experimental — accuracy not yet measuredLoRA scales ∆x by α/r where α is a constant in r, so the magnitude of output can be consistent given different r.
“LoRA scales ∆x by α/r where α is a constant in r.As a result, the magnitude of output can be consistent given different r.”
From AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning (2023) · cites this paper as Yang & Hu, 2020
Section 3.2 (p. 9) defines r as an exponent governing how feature changes scale with network width, not as a LoRA rank. Searches for LoRA, α/r scaling, and output magnitude across ranks found no passage addressing the attributed claim. (LLM, conf=0.75)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
PyTorch: An Imperative Style, High-Performance Deep Learning Library
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019) · arXiv · doi:10.48550/arxiv.1912.01703 · Cited by 31,473 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
TensorFlow: A system for large-scale machine learningPreprint
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.5555/3026877.3026899 · Cited by 16,369 · see citing works
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512 · see citing works
Gaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · see citing works
Neural Tangent Kernel: Convergence and Generalization in Neural Networks
Jacot, A., Gabriel, F., & Hongler, C. (2018) · arXiv · doi:10.48550/arxiv.1806.07572 · Cited by 1,562 · see citing works
On First-Order Meta-Learning Algorithms
Nichol, A., Achiam, J., & Schulman, J. (2018) · arXiv · doi:10.48550/arxiv.1803.02999 · Cited by 1,438 · see citing works
A Convergence Theory for Deep Learning via Over-ParameterizationPreprint
Allen-Zhu, Z., Li, Y., & Song, Z. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.03962 · Cited by 847 · see citing works
The Dynamics of Message Passing on Dense Graphs, with Applications to Compressed Sensing
Bayati, M., & Andrea Montanari, A. (2011) · IEEE Transactions on Information Theory · doi:10.1109/tit.2010.2094817 · Cited by 779 · see citing works
A mean field view of the landscape of two-layer neural networks
Mei, S., Andrea Montanari, A., & Nguyen, P.-M. (2018) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1806579115 · Cited by 521 · see citing works
Gradient Descent Provably Optimizes Over-parameterized Neural NetworksPreprint
Du, S. S., Zhai, X., Póczos, B., & Singh, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.02054 · Cited by 490 · see citing works
Exponential expressivity in deep neural networks through transient chaosPreprint
Poole, B., Lahiri, S., Raghu, M., Sohl‐Dickstein, J., & Ganguli, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.05340 · Cited by 299 · see citing works
On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal TransportPreprint
Chizat, L., & Bach, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.09545 · Cited by 268 · see citing works
Oymak, S., & Soltanolkotabi, M. (2020) · IEEE Journal on Selected Areas in Information Theory · doi:10.1109/jsait.2020.2991332 · Cited by 225 · see citing works
Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU NetworksPreprint
Zou, D., Cao, Y. J., Zhou, D., & Gu, Q. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.08888 · Cited by 177 · see citing works
Yang, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.04760 · Cited by 157 · see citing works
Self-citationResurrecting the sigmoid in deep learning through dynamical isometry: theory and practicePreprint
Pennington, J., Schoenholz, S. S., & Ganguli, S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.04735 · Cited by 150 · see citing works
Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic LossPreprint
Chizat, L., & Bach, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.04486 · Cited by 130 · see citing works
Train Big, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers
Li, Z., Wallace, E., Shen, S., Lin, K., Keutzer, K., Klein, D., & Gonzalez, J. E. (2020) · International Conference on Machine Learning · Cited by 125 · see citing works
Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitPreprint
Song, M. N., Misiakiewicz, T., & Andrea Montanari, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.06015 · Cited by 123 · see citing works
Trainability and Accuracy of Neural Networks: An Interacting Particle System ApproachPreprint
Rotskoff, G. M., & Vanden‐Eijnden, E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.00915 · Cited by 93 · see citing works
Rotskoff, G. M., & Vanden‐Eijnden, E. (2018) · arXiv (Cornell University) · Cited by 89 · see citing works
A Mean Field Theory of Batch NormalizationPreprint
Yang, G., Pennington, J., Rao, V., Sohl‐Dickstein, J., & Schoenholz, S. S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.08129 · Cited by 84 · see citing works
Self-citationThe large learning rate phase of deep learning: the catapult mechanismPreprint
Lewkowycz, A., Bahri, Y., Dyer, E., Sohl‐Dickstein, J., & Gur-Ari, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02218 · Cited by 83 · see citing works
Kernel and Rich Regimes in Overparametrized ModelsPreprint
Woodworth, B., Gunasekar, S., Savarese, P., Moroshko, E., Golan, I., Lee, J. D., Soudry, D., & Srebro, N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.05827 · Cited by 82 · see citing works
Li, Z., Wallace, E., Shen, S., Lin, K., Keutzer, K., Klein, D., & Gonzalez, J. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.11794 · Cited by 76 · see citing works
Chen, M., Pennington, J., & Schoenholz, S. S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.05394 · Cited by 70 · see citing works
Dynamics of Deep Neural Networks and Neural Tangent HierarchyPreprint
Huang, J., & Yau, H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.08156 · Cited by 69 · see citing works
Tensor Programs II: Neural Tangent Kernel for Any ArchitecturePreprint
Yang, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.14548 · Cited by 48 · see citing works
Self-citationHow to Start Training: The Effect of Initialization and ArchitecturePreprint
Hanin, B., & Rolnick, D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.01719 · Cited by 47 · see citing works
Yang, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.12478 · Cited by 45 · see citing works
Self-citationOn the Selection of Initialization and Activation Function for Deep Neural NetworksPreprint
Hayou, S., Doucet, A., & Rousseau, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.08266 · Cited by 43 · see citing works
Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients?Preprint
Hanin, B. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.03744 · Cited by 36 · see citing works
A mean-field limit for certain deep neural networksPreprint
Araújo, D., Oliveira, R. I., & Yukimura, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.00193 · Cited by 35 · see citing works
Mean Field Limit of the Learning Dynamics of Multilayer Neural NetworksPreprint
Nguyen, P.-M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.02880 · Cited by 31 · see citing works
Learning Over-Parametrized Two-Layer ReLU Neural Networks beyond NTKPreprint
Li, Y., Ma, T., & Zhang, H. R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.04596 · Cited by 27 · see citing works
On the infinite width limit of neural networks with a standard parameterizationPreprint
Sohl‐Dickstein, J., Novak, R., Schoenholz, S. S., & Lee, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.07301 · Cited by 24 · see citing works
Dynamical Isometry and a Mean Field Theory of LSTMs and GRUsPreprint
Gilboa, D., Chang, B., Chen, M., Yang, G., Schoenholz, S. S., Ed, H., & Pennington, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1901.08987 · Cited by 23 · see citing works
Self-citationTensor Programs III: Neural Matrix LawsPreprint
Yang, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.10685 · Cited by 18 · see citing works
Self-citationYang, G., & Schoenholz, S. S. (2018) · International Conference on Learning Representations · Cited by 12 · see citing works
Self-citationA Rigorous Framework for the Mean Field Limit of Multilayer Neural NetworksPreprint
Nguyen, P.-M., & Pham, H. T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.11443 · Cited by 11 · see citing works
Why bigger is not always better: on finite and infinite neural networksPreprint
Aitchison, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1910.08013 · Cited by 11 · see citing works
The Nonlinearity Coefficient - Predicting Generalization in Deep Neural NetworksPreprint
Philipp, G., & Carbonell, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.00179 · Cited by 10 · see citing works
Deep kernel processesPreprint
Aitchison, L., Yang, A. X., & Ober, S. W. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01590 · Cited by 5 · see citing works
Wider Networks Learn Better FeaturesPreprint
Gilboa, D., & Gur-Ari, G. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.11572 · Cited by 3 · see citing works
Dynamically Stable Infinite-Width Limits of Neural ClassifiersPreprint
Golikov, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.06574 · Cited by 2 · see citing works
Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural NetworksPreprint
Fang, C., Lee, J. D., Yang, P., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.01452
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Mean Field Analysis of Neural Networks: A Law of Large Numbers
Sirignano, J., & Spiliopoulos, K. (2018) · arXiv · doi:10.48550/arxiv.1805.01053
Mean Field Analysis of Deep Neural Networks
Sirignano, J., & Spiliopoulos, K. (2019) · arXiv · doi:10.48550/arxiv.1903.04440
Kernel and Rich Regimes in Overparametrized Models
Woodworth, B., Gunasekar, S., Lee, J. D., Moroshko, E., Savarese, P., Golan, I., Soudry, D., & Srebro, N. (2020) · arXiv · doi:10.48550/arxiv.2002.09277
Flagged first, then by citations
Research works printed but not linked: 3
- [8][8] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. arXiv:2005.14165 [cs] , July 2020. URL http://arxiv.org/abs/2005.14165 . - · arXiv
- [44][44] Samuel S. Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep Information Propagation. 2017. URL https://openreview.net/pdf?id=H1W1UN9gg . -
- [55][55] Greg Yang and Samuel S. Schoenholz. Mean Field Residual Network: On the Edge of Chaos. In Advances in neural information processing systems , 2017. -
Cited by
Sorted by citations
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · see citing works
Priors in Bayesian Deep Learning: A Review
Fortuin, V. (2022) · International Statistical Review · doi:10.1111/insr.12502 · Cited by 126 · see citing works
Neural networks and quantum field theory
Halverson, J., Maiti, A., & Stoner, K. (2021) · Machine Learning Science and Technology · doi:10.1088/2632-2153/abeca3 · Cited by 85 · see citing works
Representation Learning via Quantum Neural Tangent Kernels
Liu, J., Tacchino, F., Glick, J. R., Jiang, L., & Mezzacapo, A. (2022) · PRX Quantum · doi:10.1103/prxquantum.3.030323 · Cited by 67 · see citing works
AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W., & Zhao, T. (2023) · arXiv · doi:10.48550/arxiv.2303.10512 · Cited by 47 · see citing works
Algorithms going wild – A review of machine learning techniques for terrestrial ecology
Cipriano, C., Noce, S., Mereu, S., & Santini, M. (2025) · Ecological Modelling · doi:10.1016/j.ecolmodel.2025.111164 · Cited by 44 · see citing works
The edge of chaos: quantum field theory and deep neural networks
Grosvenor, K. T., & Jefferson, R. (2022) · SciPost Physics · doi:10.21468/scipostphys.12.3.081 · Cited by 30 · see citing works
Asymptotics of representation learning in finite Bayesian neural networks*
Zavatone-Veth, J. A., Canatar, A., Ruben, B. S., & Pehlevan, C. (2022) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ac98a6 · Cited by 25 · see citing works
Exact marginal prior distributions of finite Bayesian neural networksPreprint
Zavatone-Veth, J. A., & Pehlevan, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.11734 · Cited by 19 · see citing works
Laziness, barren plateau, and noises in machine learning
Liu, J., Lin, Z., & Jiang, L. (2024) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ad35a3 · Cited by 21 · see citing works
Gradient descent on infinitely wide neural networks: global convergence and generalization
Bach, F., & Chizat, L. (2023) · EMS Press eBooks · doi:10.4171/icm2022/121 · Cited by 13 · see citing works
Suh, N., & Cheng, G. (2024) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-040522-013920 · Cited by 15 · see citing works
Mallik, M., Allaert, B., Egea-López, E., Gaillot, D. P., Wiart, J., & Clavier, L. (2024) · IEEE Access · doi:10.1109/access.2024.3380835 · Cited by 10 · see citing works
A duality connecting neural network and cosmological dynamics
Krippendorf, S., & Spannowsky, M. (2022) · Machine Learning Science and Technology · doi:10.1088/2632-2153/ac87e9 · Cited by 9 · see citing works
Neural Networks as Kernel Learners: The Silent Alignment EffectPreprint
Atanasov, A., Bordelon, B., & Pehlevan, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.00034 · Cited by 9 · see citing works
Nitanda, A., Wu, D. C.-Y., & Suzuki, T. (2022) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ac98a8 · Cited by 8 · see citing works
Aligned and oblique dynamics in recurrent neural networksPreprint
Schuessler, F., Mastrogiuseppe, F., Ostojic, S., & Barak, O. (2024) · eLife · doi:10.7554/elife.93060 · Cited by 8 · see citing works
Evolution of neural activity in circuits bridging sensory and abstract knowledge
Mastrogiuseppe, F., Hiratani, N., & Latham, P. E. (2023) · eLife · doi:10.7554/elife.79908 · Cited by 7 · see citing works
Six lectures on linearized neural networks
Misiakiewicz, T., & Andrea Montanari, A. (2024) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ad292a · Cited by 5 · see citing works
Precise learning curves and higher-order scaling limits for dot-product kernel regression *
Xiao, L., Hu, H., Misiakiewicz, T., Lu, Y. M., & Pennington, J. (2023) · Journal of Statistical Mechanics Theory and Experiment · doi:10.1088/1742-5468/ad01b7 · Cited by 5 · see citing works
Aligned and oblique dynamics in recurrent neural networks
Schuessler, F., Mastrogiuseppe, F., Ostojic, S., & Barak, O. (2024) · eLife · doi:10.7554/elife.93060.3 · Cited by 3 · see citing works
Dan, L., Zheng, K., Wu, C., Wang, X., & Wang, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128778 · Cited by 3 · see citing works
The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at InitializationPreprint
Li, M. B., Nica, M., & Roy, D. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.04013 · Cited by 3 · see citing works
Aligned and oblique dynamics in recurrent neural networksPreprint
Schuessler, F., Mastrogiuseppe, F., Ostojic, S., & Barak, O. (2024) · eLife · doi:10.7554/elife.93060.2 · Cited by 3 · see citing works
Dynamical transition in controllable quantum neural networks with large depth
Zhang, B., Liu, J., Wu, X.-C., Jiang, L., & Zhuang, Q. (2024) · Nature Communications · doi:10.1038/s41467-024-53769-2 · Cited by 3 · see citing works
Nitanda, A., Wu, D. C.-Y., & Suzuki, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.15477 · Cited by 2 · see citing works
A Mechanism for Producing Aligned Latent Spaces with AutoencodersPreprint
Jain, S., Radhakrishnan, A., & Uhler, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.15456 · Cited by 2 · see citing works
Collective variables of neural networks: empirical time evolution and scaling laws
Tovey, S., Krippendorf, S., Spannowsky, M., Nikolaou, K., & Holm, C. (2025) · Machine Learning Science and Technology · doi:10.1088/2632-2153/adee76 · Cited by 1 · see citing works
Properties of the After KernelPreprint
Long, P. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.10585 · Cited by 1 · see citing works
Gradient-trained Weights in Wide Neural Networks Align Layerwise to Error-scaled Input Correlations.Preprint
Boopathy, A., & Fiete, I. (2021) · arXiv (Cornell University) · Cited by 1 · see citing works
When does preconditioning help or hurt generalization
Амари, Ш.-И., Ba, J. L., Grosse, R., Li, X., Nitanda, A., Suzuki, T., Wu, D. C.-Y., & Xu, J. (2021) · Institutional Repositories DataBase (IRDB)
Report on scipost_202110_00024v1
(2021) · SciPost Physics · doi:10.21468/scipost.report.4037
Report on scipost_202110_00024v1
(2021) · SciPost Physics · doi:10.21468/scipost.report.3967
A study of field theories via neural networks
Maiti, A. (2023) · doi:10.17760/d20486931
A note on regularised NTK dynamics with an application to PAC-Bayesian trainingPreprint
Clerico, E., & Guedj, B. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.13259
Aligned and oblique dynamics in recurrent neural networksPreprint
Schuessler, F., Mastrogiuseppe, F., Ostojic, S., & Barak, O. (2024) · eLife · doi:10.7554/elife.93060.1
- Authors
- Yang, G., & Hu, E. J.
- Affiliations
- Microsoft (United States) (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2020
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2011.14522
- OpenAlex
- W3110116503
- Cited by
- 35
- References
- 34
- Citations by year
- 2025: 2 · 2024: 10 · 2023: 5 · 2022: 6 · 2021: 11 · 2020: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Keywords
- infinite-width neural networks, representation learning, neural tangent kernel, pre-training, transfer learning, BERT, Word2Vec, few-shot learning, model-agnostic meta-learning, Omniglot, neural network parametrization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
