CoChat

Ill-Conditioning in Neural Network Training Problems

Saarinen, S., Bramley, R., Cybenko, G. · 1993 · SIAM Journal on Scientific Computing

The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.

Abstract

The training problem for feedforward neural networks is nonlinear parameter estimation that can be solved by a variety of optimization techniques. Much of the literature on neural networks has focused on variants of gradient descent. The training of neural networks using such techniques is known to be a slow process with more sophisticated techniques not always performing significantly better. This paper shows that feedforward neural networks can have ill-conditioned Hessians and that this ill-conditioning can be quite common. The analysis and experimental results in this paper lead to the conclusion that many network training problems are ill conditioned and may not be solved more efficiently by higher-order optimization methods. While the analyses used in this paper are for completely connected layered networks, they extend to networks with sparse connectivity as well. The results suggest that neural networks can have considerable redundancy in parameterizing the function space in a neighborhood of a local minimum, independently of whether or not the solution has a small residual.

Overview

Integrity signals

Registry

Crossref · 10.1137/0914044

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 12 references

Research works printed but not linked: 6

  1. M. D. Buhmann, Multivariate interpolation in odd dimensional Euclidean spaces using multi-quadratics, Tech. Rep., DAMTP 1988/NA6, Dept. of Applied Mathematics and Theoretical Physics, Univ. of Cambridge, Cambridge, U.K., 1988
  2. C. Fraley, Ph.D. Thesis, Solution of nonlinear least-squares problems, Stanford Univ., Stanford, CA, 1987, June, Tech. Rep. # STAN-CS-87-1165, Dept. of Computer Science, Stanford University, Stanford
  3. Golub G., 1989, Matrix computations
  4. Powell M., 1987, Algorithms for approximation (Shrivenham, 1985), 10, 143
  5. Saarinen S., 1991, Automatic differentiation of algorithms (Breckenridge, CO, 1991), 31
  6. Sejnowski T., 1987, Complex Systems, 1, 1134

Cited by

Items 101–145 of 145

Metadata

Authors
Saarinen, S., Bramley, R., & Cybenko, G.
Venue
SIAM Journal on Scientific Computing
Published
1993-05
Volume
14
Issue
3
Pages
693-714
ISSN
  • 1064-8275 (ISSN-L)
  • 1095-7197
OpenAlex
W1977923618
Cited by
141
References
11
Citations by year
2026: 1 · 2025: 4 · 2024: 2 · 2023: 4 · 2022: 5 · 2021: 7 · 2020: 6 · 2019: 6 · 2018: 4 · 2017: 2 · 2016: 6 · 2013: 5 · 2012: 5
FWCI
8.37
Type
journal article
Language
en
Open access
closed
Locations
SIAM Journal on Scientific Computing (publishedVersion)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
  • Physical Sciences › Mathematics › Numerical Analysis › Advanced Optimization Algorithms Research
Keywords
ill-conditioning, feedforward neural networks, neural network training, nonlinear parameter estimation, gradient descent, sparse connectivity, local minima

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.