The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Fast Exact Multiplication by the Hessian
Abstract
Just storing the Hessian H (the matrix of second derivatives δ 2 E/δw i δw j of the error E with respect to each pair of weights) of a large neural network is difficult. Since a common use of a large matrix like H is to compute its product with various vectors, we derive a technique that directly calculates Hv, where v is an arbitrary vector. To calculate Hv, we first define a differential operator R v{f(w)} = (δ/δr)f(w + rv)|r=0, note that R v{▽w} = Hv and R v{w} = v, and then apply R v{·} to the equations used to compute ▽w. The result is an exact and numerically stable procedure for computing Hv, which takes about as much computation, and is about as local, as a gradient evaluation. We then apply the technique to a one pass gradient calculation algorithm (backpropagation), a relaxation gradient calculation algorithm (recurrent backpropagation), and two stochastic gradient calculation algorithms (Boltzmann machines and weight perturbation). Finally, we show that this technique can be used at the heart of many iterative techniques for computing various properties of H, obviating any need to calculate the full Hessian.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Press, W. H., Teukolsky, S. A., Vetterling, W. T., & Flannery, B. P. (1994) · Cited by 15,411
A scaled conjugate gradient algorithm for fast supervised learning
Møller, M. F. (1993) · Neural Networks · doi:10.1016/s0893-6080(05)80056-5 · Cited by 4,052
A learning algorithm for Boltzmann machines
Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985) · Cognitive Science · doi:10.1207/s15516709cog0901_7 · Cited by 3,402
(1989) · Cited by 3,214
A practical Bayesian framework for backpropagation networks
MacKay, D. J. C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.3.448 · Cited by 2,917
Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences
Werbos, P. J. (1974) · Medical Entomology and Zoology · Cited by 2,789
Beyond regression : new fools for prediction and analysis in the behavioral sciences
Werbos, P. J. (1974) · Medical Entomology and Zoology · Cited by 2,739
(1993) · CaltechAUTHORS (California Institute of Technology) · Cited by 1,498
Connectionist learning procedures
Hinton, G. E. (1989) · Artificial Intelligence · doi:10.1016/0004-3702(89)90049-0 · Cited by 1,430
Training with Noise is Equivalent to Tikhonov Regularization
Bishop, C. M. (1995) · Neural Computation · doi:10.1162/neco.1995.7.1.108 · Cited by 1,325
Stationary and nonstationary learning characteristics of the LMS adaptive filter
Widrow, B., McCool, J. M., Larimore, M. G., & Johnson, C. R. (1976) · Proceedings of the IEEE · doi:10.1109/proc.1976.10286 · Cited by 1,268
Stationary and Nonstationary Learning Characteristics of the LMS Adaptive Filter
Widrow, B., McCool, J., Larimore, M. G., & Johnson, C. R. (1977) · Aspects of Signal Processing · doi:10.1007/978-94-010-1223-2_23 · Cited by 1,195
Generalization of back-propagation to recurrent neural networks
Pineda, F. J. (1987) · Physical Review Letters · doi:10.1103/physrevlett.59.2229 · Cited by 931 · cited 2×
“The recurrent backpropagation algorithm (Almeida, 1987; Pineda, 1987) consists of a set of forward equations which relax to a solution for the gradient, xi = 9 j wjiyj (10)”
MethodInfluentialA Scaled Conjugate Gradient Algorithm for Fast Supervised Learning
Møller, M. F. (1990) · DAIMI Report Series · doi:10.7146/dpb.v19i339.6570 · Cited by 447
Moody, J. E. (1991) · Cited by 437
Bayesian Regularization and Pruning Using a Laplace Prior
Williams, P. M. (1995) · Neural Computation · doi:10.1162/neco.1995.7.1.117 · Cited by 377
Improving the convergence of back-propagation learning with second-order methods
Becker, S., & LeCun, Y. (1989) · Cited by 331
A learning rule for asynchronous perceptrons with feedback in a combinatorial environment
Almeida, L. B. (1990) · IEEE Press eBooks · Cited by 280
Learning Algorithms for Connectionist Networks: Applied Gradient Methods of Nonlinear Optimization
Watrous, R. L. (1988) · ScholarlyCommons (University of Pennsylvania) · Cited by 242
Backpropagation: past and future
Werbos (1988) · IEEE International Conference on Neural Networks · doi:10.1109/icnn.1988.23866 · Cited by 228
BACKPROPAGATION: PAST AND FUTURE
Werbos, P. J. (1988) · Cited by 224
A Fast Stochastic Error-Descent Algorithm for Supervised Learning and Optimization
Cauwenberghs, G. (1992) · CaltechAUTHORS (California Institute of Technology) · Cited by 179
Exact Calculation of the Hessian Matrix for the Multilayer Perceptron
Bishop, C. (1992) · Neural Computation · doi:10.1162/neco.1992.4.4.494 · Cited by 146
Computing second derivatives in feed-forward networks: a review
Buntine, W., & Weigend, A. S. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.286919 · Cited by 105
Automatic Hessians by reverse accumulation
Christianson, B. (1992) · IMA Journal of Numerical Analysis · doi:10.1093/imanum/12.2.135 · Cited by 102
A Parallel Gradient Descent Method for Learning in Analog VLSI Neural Networks
Alspector, J., Meir, R., Yuhas, B. P., Jayakumar, A. K., & Lippe, D. (1992) · neural information processing systems · Cited by 98
Automatic Learning Rate Maximization by On-Line Estimation of the Hessian's Eigenvectors
LeCun, Y., Simard, P. Y., & Pearlmutter, B. A. (1993) · Maynooth University ePrints and eTheses Archive (Maynooth University) · Cited by 79
Self-citationSecond Order Properties of Error Surfaces: Learning Time and Generalization
LeCun, Y., Kanter, I., & Solla, S. A. (1990) · neural information processing systems · Cited by 68
The Eigenvalues of Mega-dimensional Matrices
Skilling, J. (1989) · Maximum Entropy and Bayesian Methods · doi:10.1007/978-94-015-7860-8_48 · Cited by 66
Summed Weight Neuron Perturbation: An O(N) Improvement Over Weight Perturbation
Flower, B., & Jabri, M. A. (1992) · Cited by 43
Jabri, M. A., & Flower, B. (1991) · Neural Computation · doi:10.1162/neco.1991.3.4.546 · Cited by 37
Møller, M. F. (1993) · DAIMI Report Series · doi:10.7146/dpb.v22i432.6748 · Cited by 28
Analog VLSI Implementation of Gradient Descent
Kirch, D. B., Kerns, D., Fleischer, K., & Barr, A. H. (1992) · neural information processing systems · Cited by 16
Second-order training for recurrent neural networks without teacher-forcing
Zuben, F. J. V., & Netto, M. L. D. A. (2002) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/icnn.1995.487520 · Cited by 4
Cited by
Deep learning in neural networks: an overview
Schmidhuber, J. (2015) · Neural Networks · doi:10.1016/j.neunet.2014.09.003 · Cited by 18,570
Optimization Methods for Large-Scale Machine Learning
Bottou, L., Curtis, F. E., & Nocedal, J. (2018) · SIAM Review · doi:10.1137/16m1080173 · Cited by 3,128
Practical Recommendations for Gradient-Based Training of Deep Architectures
Bengio, Y. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_26 · Cited by 2,259
“Other potential beneficial applications of such symbolic manipulations include parallelization and additional differential operators (such as the Roperator, recently implemented in Theano, which is very useful to compute the product of a Jacobian matrix partialf(x) ∂x or Hessian matrix ∂ (2)L(x,θ) ∂θ(2) with a vector…”
BackgroundMethodStatic Analysis of Shape in TensorFlow Programs
The Theano Development Team, Al‐Rfou, R., Alain, G., Almahairi, A., Angermueller, C., Bahdanau, D., Ballas, N., Bastien, F., Bayer, J., Belikov, A., et al. (2016) · arXiv (Cornell University) · doi:10.4230/lipics.ecoop.2020.15 · Cited by 1,589
Understanding Black-box Predictions via Influence Functions
Koh, P. W., & Liang, P. (2017) · arXiv · doi:10.48550/arxiv.1703.04730 · Cited by 1,514
Metadata
- Authors
- Pearlmutter, B. A.
- Venue
- Neural Computation
- Publisher
- MIT Press - Journals
- Published
- 1994-01
- Volume
- 6
- Issue
- 1
- Pages
- 147-160
- ISSN
- 0899-7667 (ISSN-L)
- 1530-888X
- OpenAlex
- W2006903949
- Cited by
- 540
- References
- 34
- FWCI
- 8.15
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.