The DOI matches its OpenAlex record; 42 of the 42 references it cites were found in a registry or an index.
Optimizing Neural Networks with Kronecker-factored Approximate Curvature
Open accessOverview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Reducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928
(2010) · Cited by 14,134
(2021) · A First Course in Statistical Programming with R · doi:10.1017/9781108993456.011 · Cited by 13,724
(2006) · Springer eBooks · Cited by 8,123
Improving neural networks by preventing co-adaptation of feature detectors
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012) · arXiv · doi:10.48550/arxiv.1207.0580 · Cited by 6,382
The Levenberg-Marquardt algorithm: Implementation and theory
Moré, J. J. (1978) · Lecture Notes in Mathematics · doi:10.1007/bfb0067700 · Cited by 5,773
ADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
(2013) · Cited by 3,740
Some methods of speeding up the convergence of iteration methods
Polyak, B. T. (1964) · USSR Computational Mathematics and Mathematical Physics · doi:10.1016/0041-5553(64)90137-5 · Cited by 3,069
Natural Gradient Works Efficiently in Learning
Amari, S.-I. (1998) · Neural Computation · doi:10.1162/089976698300017746 · Cited by 2,681
LeCun, Y., Bottou, L., Orr, G., & Müller, K. (1998) · neural information processing systems · Cited by 1,333
The ubiquitous Kronecker product
Loan, C. F. V. (2000) · Journal of Computational and Applied Mathematics · doi:10.1016/s0377-0427(00)00393-9 · Cited by 981
Deep learning via Hessian-free optimization
Martens, J. (2010) · International Conference on Machine Learning · Cited by 732
Self-citationJoint mean-covariance models with applications to longitudinal data: unconstrained parameterisation
Pourahmadi, M. (1999) · Biometrika · doi:10.1093/biomet/86.3.677 · Cited by 598
Computational Methods for Linear Matrix Equations
Simoncini, V. (2016) · SIAM Review · doi:10.1137/130912839 · Cited by 487
Sample size selection in optimization methods for machine learning
Byrd, R. H., Chin, G. M., Nocedal, J., & Wu, Y. (2012) · Mathematical Programming · doi:10.1007/s10107-012-0572-5 · Cited by 369
Improving the convergence of back-propagation learning with second-order methods
Becker, S., & LeCun, Y. (1989) · Cited by 331
A Stochastic Quasi-Newton Method for Online Convex Optimization
Schraudolph, N. N., Yu, J., & Günter, S. (2007) · ANU Open Research (Australian National University) · Cited by 324
Experiments on Learning by Back Propagation.
Plaut, D. C., Nowlan, S. J., & Hinton, G. E. (1986) · Cited by 323
Hybrid deterministic-stochastic methods for data fitting
Friedlander, M. P., & Schmidt, M. (2011) · Cited by 314
New insights and perspectives on the natural gradient method
Martens, J. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1412.1193 · Cited by 283
Self-citationSchaul, T., Zhang, S., & LeCun, Y. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1206.1106 · Cited by 266
Fast Curvature Matrix-Vector Products for Second-Order Gradient Descent
Schraudolph, N. N. (2002) · Neural Computation · doi:10.1162/08997660260028683 · Cited by 265
Deep Learning Made Easier by Linear Transformations in Perceptrons
Raiko, T., Harri Valpola, B., & LeCun, Y. (2012) · International Conference on Artificial Intelligence and Statistics · Cited by 229
Training Deep and Recurrent Networks with Hessian-Free Optimization
Martens, J., & Sutskever, I. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_27 · Cited by 202
Self-citationInformation-Geometric Optimization Algorithms: A Unifying Picture via Invariance Principles
Ollivier, Y., Arnold, L., Auger, A., & Nikolaus Hansen, C. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1106.3708 · Cited by 199
Revisiting Natural Gradient for Deep NetworksPreprint
Pascanu, R., & Bengio, Y. (2014) · arXiv (Cornell University) · Cited by 170
Adaptive natural gradient learning algorithms for various stochastic models
Park, H., Amari, S.-I., & Fukumizu, K. (2000) · Neural Networks · doi:10.1016/s0893-6080(00)00051-4 · Cited by 164
An Improved Newton Iteration for the Generalized Inverse of a Matrix, with Applications
Pan, V., & Schreiber, R. (1991) · SIAM Journal on Scientific and Statistical Computing · doi:10.1137/0912058 · Cited by 161
Parallel training of DNNs with Natural Gradient and Parameter Averaging
Povey, D., Zhang, X., & Khudanpur, S. (2014) · arXiv · doi:10.48550/arxiv.1410.7455 · Cited by 148
Covariance Estimation: The GLM and Regularization Perspectives
Pourahmadi, M. (2011) · Cited by 145
Note on Learning Rate Schedules for Stochastic Optimization
Darken, C. J., & Moody, J. E. (1990) · neural information processing systems · Cited by 134
Krylov Subspace Descent for Deep LearningPreprint
Vinyals, O., & Povey, D. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1111.4259 · Cited by 126
Riemannian metrics for neural networks I: feedforward networks
Ollivier, Y. (2015) · Information and Inference A Journal of the IMA · doi:10.1093/imaiai/iav006 · Cited by 94
Swersky, K., Chen, B., Marlin, B. M., & de Freitas, N. (2010) · Information Theory and Applications Workshop (ITA) · doi:10.1109/ita.2010.5454138 · Cited by 67
Mean-normalized stochastic gradient for large-scale deep learning
Wiesler, S., Richard, A., Schlüter, R., & Ney, H. (2014) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.2014.6853582 · Cited by 65
On “Natural” Learning and Pruning in Multilayered Perceptrons
Heskes, T. (2000) · Neural Computation · doi:10.1162/089976600300015637 · Cited by 60
Centering Neural Network Gradient Factors
Schraudolph, N. N. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35289-8_14 · Cited by 39
Training Neural Networks with Stochastic Hessian-Free OptimizationPreprint
Kiros, R. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.3641 · Cited by 32
Vatanen, T., Raiko, T., Harri Valpola, B., & LeCun, Y. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-42054-2_55 · Cited by 31
Sharpness in rates of convergence for the symmetric Lanczos method
Li, R. (2009) · Mathematics of Computation · doi:10.1090/s0025-5718-09-02258-3 · Cited by 24
Cited by
Learning to learn by gradient descent by gradient descent
Andrychowicz, M., Denil, M., Gomez, S., Hoffman, M. W., Pfau, D., Schaul, T., Shillingford, B., & de Freitas, N. (2016) · arXiv · doi:10.48550/arxiv.1606.04474 · Cited by 1,486
A survey of uncertainty in deep neural networks
Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10562-9 · Cited by 1,406
Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
Chaudhry, A., Dokania, P. K., Ajanthan, T., & Torr, P. H. S. (2018) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-01252-6_33 · Cited by 1,091
“By construction (outer product of gradients), Fθ is positive semi-definite (PSD) which makes it highly attractive for second-order optimization techniques [1,17,2,9,15].”
BackgroundMethodA Comprehensive Survey of Continual Learning: Theory, Method and Application
Wang, L., Zhang, X., Su, H., & Zhu, J. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3367329 · Cited by 1,113
“To overcome these issues, the Hessian can be approximated by the Fisher information matrix (FIM) ) For ease of computation, the FIM can be further simplified with a diagonal approximation [49], [53] or a Kronecker-factored approximation [50], [54].”
MethodWeight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
Salimans, T., & Kingma, D. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1602.07868 · Cited by 883
Metadata
- Authors
- Martens, J., & Grosse, R.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2015-03-19
- Pages
- 2408-2417
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1503.05671
- OpenAlex
- W2155894447
- Cited by
- 414
- References
- 42
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.