Ill-Conditioning in Neural Network Training Problems
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
The training problem for feedforward neural networks is nonlinear parameter estimation that can be solved by a variety of optimization techniques. Much of the literature on neural networks has focused on variants of gradient descent. The training of neural networks using such techniques is known to be a slow process with more sophisticated techniques not always performing significantly better. This paper shows that feedforward neural networks can have ill-conditioned Hessians and that this ill-conditioning can be quite common. The analysis and experimental results in this paper lead to the conclusion that many network training problems are ill conditioned and may not be solved more efficiently by higher-order optimization methods. While the analyses used in this paper are for completely connected layered networks, they extend to networks with sparse connectivity as well. The results suggest that neural networks can have considerable redundancy in parameterizing the function space in a neighborhood of a local minimum, independently of whether or not the solution has a small residual.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Normalizing the data improves algorithm efficiency and accuracy.19”
From Artificial neural network classification of pharyngeal high‐resolution manometry with impedance data (2013) · cites this paper as [19]
Citations
Cites
Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations
Rumelhart, D. E., McClelland, J. L., & AU (1986) · doi:10.7551/mitpress/5236.001.0001 · Cited by 15,686
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Self-citationIntroduction to the theory of neural computation
(1994) · Neural Networks · doi:10.1016/0893-6080(94)90014-0 · Cited by 6,401
The Levenberg-Marquardt algorithm: Implementation and theory
Moré, J. J. (1978) · Lecture Notes in Mathematics · doi:10.1007/bfb0067700 · Cited by 5,773
Iterative Solution of Nonlinear Equations in Several Variables
I., E., Ortega, J. M., & Rheinboldt, W. C. (1971) · Mathematics of Computation · doi:10.2307/2004942 · Cited by 4,408
Introduction to linear and nonlinear programming
Polak, E. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100535 · Cited by 2,100
Gill, P. E., Murray, W., & Wright, M. H. (2019) · doi:10.1137/1.9781611975604 · Cited by 1,737
Restart procedures for the conjugate gradient method
Powell, M. J. D. (1977) · Mathematical Programming · doi:10.1007/bf01593790 · Cited by 1,717
Numerical methods for computing angles between linear subspaces
Björck, Ȧ., & Golub, G. H. (1973) · Mathematics of Computation · doi:10.1090/s0025-5718-1973-0348991-3 · Cited by 865
Approximation Theory and Methods
Powell, M. J. D. (1981) · Cambridge University Press eBooks · doi:10.1017/cbo9781139171502 · Cited by 771
Research works printed but not linked: 6
- M. D. Buhmann, Multivariate interpolation in odd dimensional Euclidean spaces using multi-quadratics, Tech. Rep., DAMTP 1988/NA6, Dept. of Applied Mathematics and Theoretical Physics, Univ. of Cambridge, Cambridge, U.K., 1988
- C. Fraley, Ph.D. Thesis, Solution of nonlinear least-squares problems, Stanford Univ., Stanford, CA, 1987, June, Tech. Rep. # STAN-CS-87-1165, Dept. of Computer Science, Stanford University, Stanford
- Golub G., 1989, Matrix computations
- Powell M., 1987, Algorithms for approximation (Shrivenham, 1985), 10, 143
- Saarinen S., 1991, Automatic differentiation of algorithms (Breckenridge, CO, 1991), 31
- Sejnowski T., 1987, Complex Systems, 1, 1134
Cited by
Items 101–145 of 145
Neurodynamics of learning and network performance
Wilson, C. L., Blue, J. L., & Omidvar, O. M. (1997) · Journal of Electronic Imaging · doi:10.1117/12.272656 · Cited by 4
Midilli, Y. E., & Parshutin, S. (2019) · Information Technology and Management Science · doi:10.7250/itms-2019-0003 · Cited by 4
Tang, B.-J., Shi, W., Li, R., & Cao, X.-L. (2025) · Sustainable Production and Consumption · doi:10.1016/j.spc.2025.12.004 · Cited by 3
Shenfield, A., Rodrigues, M., Nooreldeen, H., & Moreno-Cuesta, J. (2017) · IEEE Conference on Computational Intelligence in Bioinformatics and Computational Biology (CIBCB) · doi:10.1109/cibcb.2017.8058544 · Cited by 3
“Many authors [2], [30] have shown that standardising the input variables to a neural network significantly improves the performance of the training process by improving the numerical condition of the optimisation problem.”
BackgroundTwo Frameworks for Improving Gradient-Based Learning Algorithms
Ventresca, M., & Tizhoosh, H. R. (2008) · Studies in computational intelligence · doi:10.1007/978-3-540-70829-2_12 · Cited by 4
Iteratively reweighted least squares based learning
Warner, B., & Misra, M. (2002) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.1998.685967 · Cited by 3
“The issue of near multicollinearity in the multilayered feed-forward network has been discussed in [ 6 ], but it can also be seen by examining Xaug.”
BackgroundA VARIANT OF BACK-PROPAGATION ALGORITHM FOR MULTILAYER FEED-FORWARD NETWORK
Ahlawat, A., & Pandey, S. (2007) · Cited by 3
Complexity-optimized sparse Bayesian learning for scalable classification tasks
Luo, J., Chen, J., Xiang, J., Wong, C.-M., & Vong, C. (2025) · Information Sciences · doi:10.1016/j.ins.2025.122447 · Cited by 2
What can one expect when solving PDEs using shallow neural networks?
He, R. Y., Liang, Y., Zhao, H., & Zhong, Y. (2026) · Handbook of numerical analysis · doi:10.1016/bs.hna.2026.03.008 · Cited by 2
Deep transformer: A framework for 2D text image rectification from planar transformations
Yan, C., Hu, J., & Zhang, C. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.02.015 · Cited by 2
Variation modes of vehicle interior roominess
Lee, S.-D., & Kim, S.-L. (2019) · Proceedings of the Institution of Mechanical Engineers Part D Journal of Automobile Engineering · doi:10.1177/0954407019844394 · Cited by 2
“PCA is also termed as (1) EVD of the covariance or correlation matrix (XX), (2) SVD of X, (3) proper orthogonal decomposition, (4) empirical orthogonal functions in climate research, (5) Karhunen–Loeve transformation in electrical engineering, (6) Hotelling’s transformation in multivariate quality control, (7) spectra…”
MethodSun, F., Liu, C., Wu, L., Wang, L., Liu, S., & Yang, B. (2020) · IEEE Congress on Evolutionary Computation (CEC) · doi:10.1109/cec48606.2020.9185557 · Cited by 2
Parametrization and Conditioning of Hinging Hyperplane Models
Pucar, P., & Sjoberg, J. S. (1996) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)58332-9 · Cited by 2
“NN models are often ill-conditioned, see (Sj(cid:127)oberg and Ljung, 1992; Moody, 1992; Saarinen et al., 1993), which means that some of the directions in the parameter space have little in(cid:13)uence on the model.”
BackgroundMeade, A. J., & Zeldin, B. A. (1998) · Mathematical and Computer Modelling · doi:10.1016/s0895-7177(98)00018-1 · Cited by 2
Numeric Data Augmentation using Structural Constraint Wasserstein Generative Adversarial Networks
Wang, W., Wang, C., Cui, T., Gong, R., Tang, Z., Zhou, X., & Li, Y. (2020) · IEEE International Symposium on Circuits and Systems proceedings · doi:10.1109/iscas45731.2020.9181232 · Cited by 2
An Inverse-Free and Scalable Sparse Bayesian Extreme Learning Machine for Classification Problems
Luo, J., Vong, C., Liu, Z., & Chen, C. (2021) · IEEE Access · doi:10.1109/access.2021.3089539 · Cited by 2
“In some particular cases, the Hessian matrix (cid:56) might be ill-conditioned, which results in the failed training [18], [19], i.e., the optimal output weight w cannot be found.”
BackgroundMultilayer Perceptrons: Other Learing Techniques
Du, K.-L., & Swamy, M. N. S. (2019) · Springer eBooks · doi:10.1007/978-1-4471-7452-3_6 · Cited by 2
Deterministic and Stochastic Logarithmic Barrier Function Methods for Neural Network Training
Trafalis, T. B., & Tutunji, T. A. (1997) · Applied optimization · doi:10.1007/978-1-4613-3400-2_13 · Cited by 3
Artificial Neural Network Modeling of Forest Tree GrowthPreprint
Gordon, C. J. (1999) · arXiv (Cornell University) · doi:10.48550/arxiv.physics/9906012 · Cited by 2
“However,Saarinen et al.(1993) show that the Jacobian m atrix ofE with respectto theNN weightsisgenerally rank de cientm aking the NN training num erically ill-conditioned.”
BackgroundOn practical use of stagewise second-order backpropagation for multi-stage neural-network learning
Mizutani, E., & Dreyfus, S. E. (2007) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/ijcnn.2007.4371146 · Cited by 2
“In all the simulations that follow, we employ a secondorder trust-region method [17] that exploits negative curvature (unlike the methods in [16], [19]), and MLP's weights are randomly-generated per trial uniformly in [-0.”
BackgroundMethodGeneralized Factor Neural Network Model for High-dimensional Regression
Guo, Z., Cucuringu, M., & Shestopaloff, A. Y. (2026) · International Journal of Forecasting · doi:10.1016/j.ijforecast.2026.07.009 · Cited by 1
Firework Swarm Learning (FSL): A Universal Machine Learning Framework
Tan, Y., Liu, Y., & Chen, J. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-7175-5_1 · Cited by 1
Automatic Regularization of Multilayered-Perceptron Training by Weight Orthogonalization
Amavasai, B. P., & Rockett, P. I. (2008) · Journal of Intelligent Systems · doi:10.1515/jisys.2008.17.s1.57 · Cited by 1
“Saarinen et al. (1993) showed experimentally that the Hessian matrix in a neural network is rank-deficient and ill-conditioned which implies that some weights in the network are highly”
BackgroundMethodA Reduced Memory Marquardt Algorlthm for Training Feedforward Neural Networks
Si, J., & Zhou, G. (1996) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)58333-0 · Cited by 1
Biomimetic control methods for nonholonomic mobile robots
Syam, R. (2005) · Cited by 1
The Virtuous Cycle of Quantum-Classical Machine LearningPreprint
Tomašev, N., McClean, J. R., & Bausch, J. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.10563 · Cited by 1
Sensitivity of nonlinear network training to affine transformed inputs
Yu, C., MANRY, M. T., & Narasimha, P. L. (2005)
Generalization of raster images containing patterns featuring stochastic repetitiveness
Rataj, A. (2007) · Machine Graphics & Vision International Journal archive · doi:10.5555/1375845.1375857
Pei, J., & Mai, E. C. (2007) · Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE · doi:10.1117/12.715956
An Efficient Algorithm for Feed-Forward Neural Network Regression Analysis
Mayekawa, S.-I. (1995) · Behaviormetrika · doi:10.2333/bhmk.22.67
Scattered Data Approximation of Weapons Bay Flow Classification
Yoo, J., & Meade, A. (2005) · AIAA Infotech@Aerospace Conference · doi:10.2514/6.2005-7169
Multistep Second Order Training For The Multilayer Perceptron
Robinson, M. (2013) · UTA ResearchCommons (University of Texas Arlington)
NEO: NEuro-Inspired Optimization—A Fractional Time Series Approach
Chatterjee, S., Das, S., & Pequito, S. (2021) · Frontiers in Physiology · doi:10.3389/fphys.2021.724044 · Cited by 2
“Furthermore, our simulations suggest that the major advantages of NEO lie in cases where the Hessian is ill-conditioned, which is particularly important in the context of several real-world problems, for instance, neural networks (Saarinen et al., 1993).”
BackgroundIF: Iterative Fractional Optimization
Chatterjee, S., Das, S., & Pequito, S. (2021) · European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN) · doi:10.14428/esann/2021.es2021-133
“3: Convergence profile showing the evolution of functional values with iteration number when the IF algorithm is used to find the minimizer of neural networks often possess ill-conditioned Hessians, as demonstrated in [9].”
BackgroundMethodDéveloppement de la technique de scattérométrie neuronale dynamique
Kalioubi, I. E. (2015) · theses.fr (ABES) · doi:10.70675/bbaecec0z8104z4ee8zbf40z13c509a1fa8e
Frank, N. B., Pughe-Sanford, J. L., & Grauer, S. J. (2026) · Journal of Computational Physics · doi:10.1016/j.jcp.2026.115114 · Cited by 1
“To see this, consider a quadratic loss with ordered Hessian eigenvalues Using the optimal step size, given by η = 2/ ( λ 1 + λ n ) , progress near a critical point θ ∗ satisfies where ∥·∥ 2 is the Euclidean norm [65].”
From radiation pattern straight to antenna geometry
Burtsev, V. D., Vosheva, T. S., Zarenin, A. A., & Filonov, D. (2025) · Photonics and Nanostructures - Fundamentals and Applications · doi:10.1016/j.photonics.2025.101477
Classification par réseaux de neurones dans le cadre de la scattérométrie ellipsométrique
Fawzi, Z. S. (2016) · theses.fr (ABES) · doi:10.70675/71256d40z33c0z4b05zb9bezd7969815fef8
Unsupervised pretraining of state representations in a rewardless environment
Merckling, A. (2021) · theses.fr (ABES) · doi:10.70675/e5589029zb8c4z4364zb923zbb4c70deaaf9
Détection et inspection par réseaux de neurones dans le cadre de la scattérométrie ellipsométrique
Tchéré, M. G. (2023) · theses.fr (ABES) · doi:10.70675/228d59ccz81cdz4bfcz8deaz774d7c8b4a43
Neuronale Regressionsmodelle in der Devisenkursprognose
Hann, T. H. (2000) · Wirtschaftswissenschaftliche Beiträge · doi:10.1007/978-3-642-57656-0_1
Metadata
- Authors
- Saarinen, S., Bramley, R., & Cybenko, G.
- ORCID
- Cybenko, G.
- Venue
- SIAM Journal on Scientific Computing
- Published
- 1993-05
- Volume
- 14
- Issue
- 3
- Pages
- 693-714
- ISSN
- 1064-8275 (ISSN-L)
- 1095-7197
- OpenAlex
- W1977923618
- Cited by
- 141
- References
- 11
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 2 · 2023: 4 · 2022: 5 · 2021: 7 · 2020: 6 · 2019: 6 · 2018: 4 · 2017: 2 · 2016: 6 · 2013: 5 · 2012: 5
- FWCI
- 8.37
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- SIAM Journal on Scientific Computing (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Mathematics › Numerical Analysis › Advanced Optimization Algorithms Research
- Keywords
- ill-conditioning, feedforward neural networks, neural network training, nonlinear parameter estimation, gradient descent, sparse connectivity, local minima
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.