Ill-Conditioning in Neural Network Training Problems
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
The training problem for feedforward neural networks is nonlinear parameter estimation that can be solved by a variety of optimization techniques. Much of the literature on neural networks has focused on variants of gradient descent. The training of neural networks using such techniques is known to be a slow process with more sophisticated techniques not always performing significantly better. This paper shows that feedforward neural networks can have ill-conditioned Hessians and that this ill-conditioning can be quite common. The analysis and experimental results in this paper lead to the conclusion that many network training problems are ill conditioned and may not be solved more efficiently by higher-order optimization methods. While the analyses used in this paper are for completely connected layered networks, they extend to networks with sparse connectivity as well. The results suggest that neural networks can have considerable redundancy in parameterizing the function space in a neighborhood of a local minimum, independently of whether or not the solution has a small residual.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Normalizing the data improves algorithm efficiency and accuracy.19”
From Artificial neural network classification of pharyngeal high‐resolution manometry with impedance data (2013) · cites this paper as [19]
Citations
Cites
Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations
Rumelhart, D. E., McClelland, J. L., & AU (1986) · doi:10.7551/mitpress/5236.001.0001 · Cited by 15,686
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Self-citationIntroduction to the theory of neural computation
(1994) · Neural Networks · doi:10.1016/0893-6080(94)90014-0 · Cited by 6,401
The Levenberg-Marquardt algorithm: Implementation and theory
Moré, J. J. (1978) · Lecture Notes in Mathematics · doi:10.1007/bfb0067700 · Cited by 5,773
Iterative Solution of Nonlinear Equations in Several Variables
I., E., Ortega, J. M., & Rheinboldt, W. C. (1971) · Mathematics of Computation · doi:10.2307/2004942 · Cited by 4,408
Introduction to linear and nonlinear programming
Polak, E. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100535 · Cited by 2,100
Gill, P. E., Murray, W., & Wright, M. H. (2019) · doi:10.1137/1.9781611975604 · Cited by 1,737
Restart procedures for the conjugate gradient method
Powell, M. J. D. (1977) · Mathematical Programming · doi:10.1007/bf01593790 · Cited by 1,717
Numerical methods for computing angles between linear subspaces
Björck, Ȧ., & Golub, G. H. (1973) · Mathematics of Computation · doi:10.1090/s0025-5718-1973-0348991-3 · Cited by 865
Approximation Theory and Methods
Powell, M. J. D. (1981) · Cambridge University Press eBooks · doi:10.1017/cbo9781139171502 · Cited by 771
Research works printed but not linked: 6
- M. D. Buhmann, Multivariate interpolation in odd dimensional Euclidean spaces using multi-quadratics, Tech. Rep., DAMTP 1988/NA6, Dept. of Applied Mathematics and Theoretical Physics, Univ. of Cambridge, Cambridge, U.K., 1988
- C. Fraley, Ph.D. Thesis, Solution of nonlinear least-squares problems, Stanford Univ., Stanford, CA, 1987, June, Tech. Rep. # STAN-CS-87-1165, Dept. of Computer Science, Stanford University, Stanford
- Golub G., 1989, Matrix computations
- Powell M., 1987, Algorithms for approximation (Shrivenham, 1985), 10, 143
- Saarinen S., 1991, Automatic differentiation of algorithms (Breckenridge, CO, 1991), 31
- Sejnowski T., 1987, Complex Systems, 1, 1134
Cited by
Nonlinear black-box modeling in system identification: a unified overview
Sjöberg, J., Zhang, Q., Ljung, L., Benveniste, A., Delyon, B., Glorennec, P.-Y., Hjalmarsson, H., & Juditsky, A. (1995) · Automatica · doi:10.1016/0005-1098(95)00120-8 · Cited by 1,942
Understanding Batch NormalizationPreprint
Bjorck, N., Gomes, C. P., Selman, B., & Weinberger, K. Q. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.02375 · Cited by 777
Statistical aspects of neural networks
Ripley, B. D. (1993) · Networks and Chaos — Statistical and Probabilistic Aspects · doi:10.1007/978-1-4899-3099-6_2 · Cited by 504
The 2017 IEEE International Conference on Fuzzy Systems
(2016) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2016.2545826 · Cited by 308
JETNET 3.0—A versatile artificial neural network package
Peterson, C., Rögnvaldsson, T., & Lönnblad, L. (1994) · Computer Physics Communications · doi:10.1016/0010-4655(94)90120-1 · Cited by 204
Metadata
- Authors
- Saarinen, S., Bramley, R., & Cybenko, G.
- ORCID
- Cybenko, G.
- Venue
- SIAM Journal on Scientific Computing
- Published
- 1993-05
- Volume
- 14
- Issue
- 3
- Pages
- 693-714
- ISSN
- 1064-8275 (ISSN-L)
- 1095-7197
- OpenAlex
- W1977923618
- Cited by
- 141
- References
- 11
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 2 · 2023: 4 · 2022: 5 · 2021: 7 · 2020: 6 · 2019: 6 · 2018: 4 · 2017: 2 · 2016: 6 · 2013: 5 · 2012: 5
- FWCI
- 8.37
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- SIAM Journal on Scientific Computing (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Mathematics › Numerical Analysis › Advanced Optimization Algorithms Research
- Keywords
- ill-conditioning, feedforward neural networks, neural network training, nonlinear parameter estimation, gradient descent, sparse connectivity, local minima
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Ill-Conditioning in Neural Network Training Problems. https://app.cochat.ai/papers/10.1137/0914044
- The paper
- Saarinen, S., Bramley, R., & Cybenko, G. (1993). Ill-Conditioning in Neural Network Training Problems. SIAM Journal on Scientific Computing. https://doi.org/10.1137/0914044
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.