Ill-Conditioning in Neural Network Training Problems
The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
The training problem for feedforward neural networks is nonlinear parameter estimation that can be solved by a variety of optimization techniques. Much of the literature on neural networks has focused on variants of gradient descent. The training of neural networks using such techniques is known to be a slow process with more sophisticated techniques not always performing significantly better. This paper shows that feedforward neural networks can have ill-conditioned Hessians and that this ill-conditioning can be quite common. The analysis and experimental results in this paper lead to the conclusion that many network training problems are ill conditioned and may not be solved more efficiently by higher-order optimization methods. While the analyses used in this paper are for completely connected layered networks, they extend to networks with sparse connectivity as well. The results suggest that neural networks can have considerable redundancy in parameterizing the function space in a neighborhood of a local minimum, independently of whether or not the solution has a small residual.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Normalizing the data improves algorithm efficiency and accuracy.19”
From Artificial neural network classification of pharyngeal high‐resolution manometry with impedance data (2013) · cites this paper as [19]
Citations
Cites
Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations
Rumelhart, D. E., McClelland, J. L., & AU (1986) · doi:10.7551/mitpress/5236.001.0001 · Cited by 15,686
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883
Self-citationIntroduction to the theory of neural computation
(1994) · Neural Networks · doi:10.1016/0893-6080(94)90014-0 · Cited by 6,401
The Levenberg-Marquardt algorithm: Implementation and theory
Moré, J. J. (1978) · Lecture Notes in Mathematics · doi:10.1007/bfb0067700 · Cited by 5,773
Iterative Solution of Nonlinear Equations in Several Variables
I., E., Ortega, J. M., & Rheinboldt, W. C. (1971) · Mathematics of Computation · doi:10.2307/2004942 · Cited by 4,408
Introduction to linear and nonlinear programming
Polak, E. (1974) · IEEE Transactions on Automatic Control · doi:10.1109/tac.1974.1100535 · Cited by 2,100
Gill, P. E., Murray, W., & Wright, M. H. (2019) · doi:10.1137/1.9781611975604 · Cited by 1,737
Restart procedures for the conjugate gradient method
Powell, M. J. D. (1977) · Mathematical Programming · doi:10.1007/bf01593790 · Cited by 1,717
Numerical methods for computing angles between linear subspaces
Björck, Ȧ., & Golub, G. H. (1973) · Mathematics of Computation · doi:10.1090/s0025-5718-1973-0348991-3 · Cited by 865
Approximation Theory and Methods
Powell, M. J. D. (1981) · Cambridge University Press eBooks · doi:10.1017/cbo9781139171502 · Cited by 771
Research works printed but not linked: 6
- M. D. Buhmann, Multivariate interpolation in odd dimensional Euclidean spaces using multi-quadratics, Tech. Rep., DAMTP 1988/NA6, Dept. of Applied Mathematics and Theoretical Physics, Univ. of Cambridge, Cambridge, U.K., 1988
- C. Fraley, Ph.D. Thesis, Solution of nonlinear least-squares problems, Stanford Univ., Stanford, CA, 1987, June, Tech. Rep. # STAN-CS-87-1165, Dept. of Computer Science, Stanford University, Stanford
- Golub G., 1989, Matrix computations
- Powell M., 1987, Algorithms for approximation (Shrivenham, 1985), 10, 143
- Saarinen S., 1991, Automatic differentiation of algorithms (Breckenridge, CO, 1991), 31
- Sejnowski T., 1987, Complex Systems, 1, 1134
Cited by
Nonlinear black-box modeling in system identification: a unified overview
Sjöberg, J., Zhang, Q., Ljung, L., Benveniste, A., Delyon, B., Glorennec, P.-Y., Hjalmarsson, H., & Juditsky, A. (1995) · Automatica · doi:10.1016/0005-1098(95)00120-8 · Cited by 1,942
Understanding Batch NormalizationPreprint
Bjorck, N., Gomes, C. P., Selman, B., & Weinberger, K. Q. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1806.02375 · Cited by 777
Statistical aspects of neural networks
Ripley, B. D. (1993) · Networks and Chaos — Statistical and Probabilistic Aspects · doi:10.1007/978-1-4899-3099-6_2 · Cited by 504
The 2017 IEEE International Conference on Fuzzy Systems
(2016) · IEEE Computational Intelligence Magazine · doi:10.1109/mci.2016.2545826 · Cited by 308
JETNET 3.0—A versatile artificial neural network package
Peterson, C., Rögnvaldsson, T., & Lönnblad, L. (1994) · Computer Physics Communications · doi:10.1016/0010-4655(94)90120-1 · Cited by 204
Ngia, L. S. H., & Sjoberg, J. (2000) · IEEE Transactions on Signal Processing · doi:10.1109/78.847778 · Cited by 189
“The reason is that the neural net minimization problems are often ill-conditioned as viewed in [25] and [26].”
BackgroundNeural Networks in System Identification
Sjöberg, J., Hjalmarsson, H., & Ljung, L. (1994) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)47737-8 · Cited by 181
Two highly efficient second-order algorithms for training feedforward networks
Ampazis, N., & Perantonis, S. J. (2002) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2002.1031939 · Cited by 161
“The ill-conditioning of the Jacobian becomes even more prominent in the case of multilayer feedforward networks due to possible redundancy of the synaptic weights and also due to the saturation of the sigmoid activation functions [ 13 ].”
BackgroundOvertraining, regularization and searching for a minimum, with application to neural networks
SJÖBERG, J., & LJUNG, L. (1995) · International Journal of Control · doi:10.1080/00207179508921605 · Cited by 159
“The results in this paper apply to all ill-conditioned, parametrized models, and we will especially address adaptive Neural Networks (NN) that typically belong to this group (Saarinen et al. 1993).”
BackgroundA deep learning solution approach for high-dimensional random differential equations
Nabian, M. A., & Meidani, H. (2019) · Probabilistic Engineering Mechanics · doi:10.1016/j.probengmech.2019.05.001 · Cited by 164
“…particular, we address the numerical difficulties that arise when the operator L in Equation 7 includes second or higher-order derivatives, which are more expensive to compute when the response is represented by a deep neural networks [21], and could potentially lead to ill-conditioned Hessians [40].”
BackgroundTraining neural nets with the reactive tabu search
Battiti, R., & Tecchiolli, G. (1995) · IEEE Transactions on Neural Networks · doi:10.1109/72.410361 · Cited by 126
Du, K.-L., Leung, C.-S., Mow, W. H., & Swamy, M. N. S. (2022) · Mathematics · doi:10.3390/math10244730 · Cited by 135
A generalized learning paradigm exploiting the structure of feedforward neural networks
Parisi, R., Claudio, E. D. D., Orlandi, G., & Rao, B. D. (1996) · IEEE Transactions on Neural Networks · doi:10.1109/72.548172 · Cited by 101
Ajemian, R., D’Ausilio, A., Moorman, H., & Bizzi, E. (2013) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1320116110 · Cited by 94
“This problem of network ill-conditioning (19) arises when the error function is characterized by sharply different slopes for different directions in weight space (numerically assessed from the eigenvalues of the Hessian matrix).”
BackgroundNeural-network construction and selection in nonlinear modeling
Rivals, I., & Personnaz, L. (2003) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2003.811356 · Cited by 90
“…studies of the ill-conditioning of neural networks deal with their training rather than with their approval, like in [Zhou & Si 1998] where an algorithm avoiding the Jacobian rank deficiency is presented, or in [Saarinen et al. 1993] where the Hessian rank deficiency is studied during training.”
MethodFast adaptive digital equalization by recurrent neural networks
Parisi, R., Claudio, E. D. D., Orlandi, G., & Rao, B. D. (1997) · IEEE Transactions on Signal Processing · doi:10.1109/78.650099 · Cited by 94
“Moreover, a recent analysis [23] has demonstrated that learning in NN’s is very often an ill-conditioned problem since the Hessian matrix [6] is badly ill conditioned; this implies that a maximum likelihood (ML) weight identification problem may suffer from lack of information (e.”
BackgroundConstruction of confidence intervals for neural networks based on least squares estimation
Rivals, I., & Personnaz, L. (2000) · Neural Networks · doi:10.1016/s0893-6080(99)00080-5 · Cited by 86
Badra, J., Khaled, F., Tang, M., Pei, Y., Kodavasal, J., Pal, P., Owoyele, O., Fütterer, C., Brenner, M., & Farooq, A. (2020) · Journal of Energy Resources Technology · doi:10.1115/1.4047978 · Cited by 84
Improved Adaptive–Reinforcement Learning Control for Morphing Unmanned Air Vehicles
Valasek, J., Doebbler, J., Tandale, M. D., & Meade, A. J. (2008) · IEEE Transactions on Systems Man and Cybernetics Part B (Cybernetics) · doi:10.1109/tsmcb.2008.922018 · Cited by 79
Separable non-linear least-squares minimization-possible improvements for neural net fitting
Sjoberg, J. S., & Viberg, M. (2002) · Machine learning for signal processing ... · doi:10.1109/nnsp.1997.622415 · Cited by 77
AghaKouchak, A., Pan, B., Mazdiyasni, O., Sadegh, M., Jiwa, S., Zhang, W., Love, C. A., Madadgar, S., Papalexiou, S. M., Davis, S. J., et al. (2022) · Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences · doi:10.1098/rsta.2021.0288 · Cited by 67
“modelling complex phenomenon like drought; however, they can be prone to overfitting, and the computations involved can easily become a black box if proper attention is not paid to model construction [167].”
BackgroundNeural network-based systems for handprint OCR applications
Ganis, M. D., Wilson, C. L., & Blue, J. L. (1998) · IEEE Transactions on Image Processing · doi:10.1109/83.704304 · Cited by 73
“(Though since is suboptimal, the global minimum is unlikely to be the best one.) 2) If the nodes of the network are sigmoidal, the Jacobian of with respect to weights tends to be ill-conditioned [ 43 ], causing problems with the convergence of the optimization process.”
BackgroundForecasting tourism demand by extracting fuzzy Takagi–Sugeno rules from trained SVMs
Xu, X., Law, R., Chen, W., & Tang, L. (2016) · CAAI Transactions on Intelligence Technology · doi:10.1016/j.trit.2016.03.004 · Cited by 66
Learning improvement of neural networks used in structural optimization
Lagaros, N. D., & Papadrakakis, M. (2003) · Advances in Engineering Software · doi:10.1016/s0965-9978(03)00112-1 · Cited by 54
“[14] the most widely used architecture, that of feed-forward NN, is likely to produce illconditioned Jacobian matrices due to the bad properties of the activation function used and that this type of illconditioning is encountered in many applications.”
MethodMulti-layer Perceptron Error Surfaces: Visualization, Structure and Modelling
Gallagher, M. R. (2000) · The University of Queensland · doi:10.14264/157842 · Cited by 67
Application of Classification Models to Pharyngeal High-Resolution Manometry
Mielens, J. D., Hoffman, M. R., Ciucci, M. R., McCulloch, T. M., & Jiang, J. J. (2012) · Journal of Speech, Language, and Hearing Research · doi:10.1044/1092-4388(2011/11-0088 · Cited by 46
Three‐Dimensional Analysis of Pharyngeal High‐Resolution Manometry Data
Geng, Z., Hoffman, M. R., Jones, C. A., McCulloch, T. M., & Jiang, J. J. (2013) · The Laryngoscope · doi:10.1002/lary.23987 · Cited by 41
“Normalizing the data improves algorithm efficiency and accuracy.(17) Additionally, principal component analysis was used to reduce dimensionality and improve generalization.”
MethodLearning with first, second, and no derivatives: A case study in high energy physics
Battiti, R., & Tecchiolli, G. (1994) · Neurocomputing · doi:10.1016/0925-2312(94)90054-x · Cited by 46
A new sparse model for traffic sign classification using soft histogram of oriented gradients
Kassani, P. H., & Teoh, A. B. J. (2016) · Applied Soft Computing · doi:10.1016/j.asoc.2016.12.037 · Cited by 38
“The culprit of overfitting on MP and RP models is ill-conditioning [44] that often occurs in large r.”
BackgroundCai, X., Tyagi, K., & MANRY, M. T. (2011) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2011.6033634 · Cited by 36
“… ൌ leads to ۱ ࢇ ൌ ܅ ܀ ் (23) where ە ۖ ۖ ۔ ۖ ۖ ۓ ۱ ࢇ ൌ ͳ ܰ ௩ ܡ ܠ ത ் ே ೡ ୀଵ ܀ ࢇ ൌ ͳ ܰ ௩ ܠ ത ܠ ത ் ே ೡ ୀଵ (24) Since the equations in (23) are often ill conditioned [25][26], we solve them using OLS[27] which is equivalent to using the QR decomposition [28].”
Lower dimensional kernels for video discriminators
Kahembwe, E., & Ramamoorthy, S. (2020) · Neural Networks · doi:10.1016/j.neunet.2020.09.016 · Cited by 51
Fuzzy C-means clustering based construction and training for second order RBF network
Tyagi, K., Cai, X., & MANRY, M. T. (2011) · IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) · doi:10.1109/fuzzy.2011.6007713 · Cited by 38
“Since the equations in (25) are often ill conditioned [28][29], we solve them using OLS[30] which is equivalent to using the QR decomposition [31].”
MethodA modular framework for the automatic classification of chromosomes in Q-band images
Poletti, E., Grisan, E., & Ruggeri, A. (2011) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2011.07.013 · Cited by 35
Artificial neural network classification of pharyngeal high‐resolution manometry with impedance data
Hoffman, M. R., Mielens, J. D., Omari, T. I., Rommel, N., Jiang, J. J., & McCulloch, T. M. (2013) · The Laryngoscope · doi:10.1002/lary.23655 · Cited by 34
“Normalizing the data improves algorithm efficiency and accuracy.(19) Additionally, principal component analysis was used to reduce dimensionality and improve generalization.”
MethodMahapatra, A. K., Panda, N., Mahapatra, M., Jena, T., & Mohanty, A. K. (2024) · Cluster Computing · doi:10.1007/s10586-024-04750-7 · Cited by 33
“For instance, COPs like engineering design with constraint tightness, objective function landscape, etc.[9], feature selection (FS) with high feature co-relationships and neural network training can exhibit such behaviour [10, 11].”
Chen, M., Fan, M.-H., Chen, Y.-L., & Wei, H.-M. (2013) · Neural Network World · doi:10.14311/nnw.2013.23.023 · Cited by 33
“(1993) [53], Sarle (1999) [54], and Hsieh et al.”
MethodInfluentialAn Alternative Approach Based on Artificial Neural Networks to Study Controlled Drug Release
Reis, M. A. A., Sinisterra, R. D., & Belchior, J. C. (2003) · Journal of Pharmaceutical Sciences · doi:10.1002/jps.10569 · Cited by 33
Fuzzy parameter adaptation in optimization: some neural net training examples
Arabshahi, P., Choi, J. J., Marks, R. J., & Caudell, T. P. (1996) · IEEE Computational Science and Engineering · doi:10.1109/99.486761 · Cited by 33
A statistical procedure for determining the optimal number of hidden neurons of a neural model
Rivals, I., & Personnaz, L. (2000) · Cited by 32
Engine Combustion System Optimization Using CFD and Machine Learning: A Methodological Approach
Badra, J., Khaled, F., Tang, M., Pei, Y., Kodavasal, J., Pal, P., Owoyele, O., Fütterer, C., Brenner, M., & Farooq, A. (2019) · ASME Internal Combustion Engine Division Fall Technical Conference (ICEF) · doi:10.1115/icef2019-7238 · Cited by 29
“Ill-conditioning of neural networks and the importance of data normalization have been discussed in detail by many authors [46, 47].”
BackgroundA system identification perspective on neural nets
Lennart Ljung, A., & Sjoberg, J. S. (2003) · Machine learning for signal processing ... · doi:10.1109/nnsp.1992.253670 · Cited by 27
ADAPTIVE ALGORITHMS FOR NEURAL NETWORK SUPERVISED LEARNING: A DETERMINISTIC OPTIMIZATION APPROACH
Magoulas, G. D., & Vrahatis, M. N. (2006) · International Journal of Bifurcation and Chaos · doi:10.1142/s0218127406015805 · Cited by 28
“Another important consideration for the convergence success of an iterative scheme is its susceptibility to ill-conditioning: the minimization of the network’s learning error is often ill-conditioned, especially when there are many hidden neurons [Saarinen et al., 1993].”
BackgroundGlobally convergent algorithms with local learning rates
Magoulas, G. D., Plagianakos, V. P., & Vrahatis, M. N. (2002) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2002.1000148 · Cited by 27
“Training of multilayer feedforward neural networks can be considered as a highly nonlinear minimization problem, involving sigmoid functions that have infinitely broad regions with arbitrary small derivative [3], [26].”
BackgroundA new algorithm for learning in piecewise-linear neural networks
Gad, E., Atiya, A. F., Shaheen, S. I., & El-Dessouki, A. (2000) · Neural Networks · doi:10.1016/s0893-6080(00)00024-1 · Cited by 28
“The analysis in Saarinen, Bramley and Cybenko (1993) shows why this ill conditioning is very common with NN learning.”
BackgroundSparse pseudoinverse incremental extreme learning machine
Kassani, P. H., Teoh, A. B. J., & Kim, E. (2018) · Neurocomputing · doi:10.1016/j.neucom.2018.01.087 · Cited by 23
Synthesis of Optimal Strategies for Differential Games by Neural Networks
Pesch, H. J., Gabler, I., Miesbach, S., & Breitner, M. H. (1995) · Birkhäuser Boston eBooks · doi:10.1007/978-1-4612-4274-1_6 · Cited by 22
“[34] Saarinen, S., Bramley, R., and Cybenko, G., Ill-Conditioning inNeural Network Training Problems , SIAM Journal of Scienti c and Com-puting 14, pp. 693{714, 1993.”
BackgroundSome numerical aspects of the training problem for feed-forward neural nets
McKeown, J., Stella, F. A., & Hall, G. (1997) · Neural Networks · doi:10.1016/s0893-6080(97)00015-4 · Cited by 21
Mizutani, E., & Dreyfus, S. E. (2008) · Neural Networks · doi:10.1016/j.neunet.2007.12.038 · Cited by 21
Variable projections neural network training
Pereyra, V. L., Scherer, G., & Wong, F. S. (2006) · Mathematics and Computers in Simulation · doi:10.1016/j.matcom.2006.06.017 · Cited by 20
Solving the Ill-Conditioning in Neural Network Learning
van der Smagt, P., & Hirzinger, G. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-35289-8_13 · Cited by 22
Wilson, C. L., Blue, J. L., & Omidvar, O. M. (1995) · doi:10.6028/nist.ir.5695 · Cited by 21
NEURAL NETWORK AS AN OPHTHALMOLOGIC DISEASE CLASSIFIER
Treigys, P., & Šaltenis, V. (2007) · Information Technology And Control · doi:10.5755/j01.itc.36.4.11895 · Cited by 18
Training Dynamics and Neural Network Performance
Wilson, C. L., Blue, J. L., & Omidvar, O. M. (1997) · Neural Networks · doi:10.1016/s0893-6080(96)00119-0 · Cited by 20
O’Leary-Roseberry, T., Alger, N., & Ghattas, O. N. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.06738 · Cited by 20
“Gradient based methods converge very slowly for ill-conditioned problems [3, 20, 21, 34].”
A MLP Classifier for Both Printed and Handwritten Bangla Numeral Recognition
Majumdar, A. K., & Chaudhuri, B. B. (2006) · Lecture notes in computer science · doi:10.1007/11949619_71 · Cited by 17
“In such a case the Neural Network remains ill-conditioned initially [21].”
BackgroundHow to make sigma-pi neural networks perform perfectly on regular training sets
Lenze, B. (1994) · Neural Networks · doi:10.1016/0893-6080(94)90009-4 · Cited by 17
Prediction of Direct Methanol Fuel Cell Stack Performance Using Artificial Neural Network
Biswas, M. A. R., & Robinson, M. (2017) · Journal of Electrochemical Energy Conversion and Storage · doi:10.1115/1.4036811 · Cited by 17
Ellenius, J. (2000) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(00)00065-4 · Cited by 16
“Nevertheless, a problem with the MLP is that it is prone to give rise to ill-conditioned optimization problems that must be addressed [33].”
BackgroundSolving the Ill-Conditioning in Neural Network Learning
van der Smagt, P., & Hirzinger, G. (1998) · Lecture notes in computer science · doi:10.1007/3-540-49430-8_10 · Cited by 22
“It has long been noted [10,16] that the derivatives of the error function are usually ill-conditioned.”
BackgroundTOTEM: a digital processor for neural networks and Reactive Tabu Search
Battiti, R., Lee, P. W. L., Sartori, A., & Tecchiolli, G. (2002) · International Conference on Microelectronics for Neural Networks and Fuzzy Systems (MicroNeuro) · doi:10.1109/icmnn.1994.593147 · Cited by 16
Goraya, S. A., Sobh, N., & Masud, A. (2023) · Computational Mechanics · doi:10.1007/s00466-023-02334-7 · Cited by 14
“Since deeper networks have larger number of parameters, these oscillations might be due to redundancy in the parameterisation of the function space in a neighborhood of a local minimum [43].”
Improving gradient-based learning algorithms for large scale feedforward networks
Ventresca, M., & Tizhoosh, H. R. (2009) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2009.5178798 · Cited by 14
“Rank deficiency of the Jacobian is related to the concept of ill-conditioning [10], [18].”
BackgroundMethodShape-dependent characteristics of full-scale wind profiles
Clobes, M., Willecke, A., & Peil, U. (2011) · Journal of Wind Engineering and Industrial Aerodynamics · doi:10.1016/j.jweia.2011.05.005 · Cited by 14
Numerical condition of feedforward networks with opposite transfer functions
Ventresca, M., & Tizhoosh, H. R. (2008) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn.2008.4634257 · Cited by 14
“Additionally, the condition will change during training if nonlinear hidden or output neurons are used [5].”
BackgroundMethodApproximation properties of local bases assembled from neural network transfer functions
Meade, A. J., & Zeldin, B. A. (1998) · Mathematical and Computer Modelling · doi:10.1016/s0895-7177(98)00144-7 · Cited by 14
“[7].”
Pei, J., Mai, E. C., Wright, J. P., & Smyth, A. W. (2007) · IEEE International Conference on Neural Networks/IEEE ... International Conference on Neural Networks · doi:10.1109/ijcnn.2007.4371284 · Cited by 13
“, [24]).”
ResultA machine learning-genetic algorithm approach for rapid optimization of internal combustion engines
Badra, J., Owoyele, O., Pal, P., & Som, S. (2022) · Elsevier eBooks · doi:10.1016/b978-0-323-88457-0.00003-5 · Cited by 12
Neural Nets and Related Model Structures for Nonlinear System Identification
Sjoberg, J. S., & Ngia, L. S. H. (1998) · Springer eBooks · doi:10.1007/978-1-4615-5703-6_1 · Cited by 11
Classification of glottic insufficiency and tension asymmetry using a multilayer perceptron
Hoffman, M. R., Surender, K., Devine, E. E., & Jiang, J. J. (2012) · The Laryngoscope · doi:10.1002/lary.23549 · Cited by 10
Pedersen, M. W. (2002) · Machine learning for signal processing ... · doi:10.1109/nnsp.1997.622416 · Cited by 10
AN INITIALIZATION METHOD FOR FEEDFORWARD ARTIFICIAL NEURAL NETWORKS USING POLYNOMIAL BASES
VARNAVA, T. M., & MEADE, A. J. (2011) · Advances in Adaptive Data Analysis · doi:10.1142/s1793536911000684 · Cited by 9
“The conjugate gradient method, the Levenberg–Marquardt method and other nonlinear optimization methods can be used for training [Saarinen et al. (1993)] each with its own merits and limitations.”
BackgroundMethodOptimization of Deep Learning Hyperparameters with Experimental Design in Exchange Rate Prediction
Midilli, Y. E., & Parshutin, S. (2020) · International Scientific Conference on Information Technology and Management Science of Riga Technical University (ITMS) · doi:10.1109/itms51158.2020.9259300 · Cited by 9
A Novel Graphic Bending Transformation on Benchmark
Liu, C., Sun, F., Ni, Q., Wang, L., & Yang, B. (2020) · Conference proceedings/Conference proceedings - IEEE International Conference on Systems, Man, and Cybernetics · doi:10.1109/smc42975.2020.9283291 · Cited by 9
“In other word, an ill conditional function is difficult to be optimized because it has such properties: the change in some variables or directions does slight influence on function value, but a minute change in other variables or directions can instill a drastically substantial change in function value [2], [3].”
BackgroundEgan, W. J., Angel, S. M., & Morgan, S. L. (2000) · Journal of Chemometrics · doi:10.1002/1099-128x(200101)15:1<29::aid-cem600>3.0.co;2-a · Cited by 10
Application of Scattered Data Approximation to a Rotorcraft Health Monitoring Problem
Meade, A. J. (2004) · Journal of Aerospace Computing Information and Communication · doi:10.2514/1.1978 · Cited by 8
Wang, B., Bai, F., & Zhang, K. (2023) · Complex & Intelligent Systems · doi:10.1007/s40747-023-01238-6 · Cited by 8
“A similar issue arises for DNN training [37–39], where each iterative update of the network parameters is essentially a step of gradient descent.”
Myles, A. J., Murray, A. F., Wallace, A. R., Barnard, J., & Smith, G. (1997) · Neural Computing and Applications · doi:10.1007/bf01413859 · Cited by 10
“[65] discuss the problem of ill-conditioning in MLP training in more depth.”
BackgroundMahapatra, A. K., Panda, N., & Pattanayak, B. K. (2023) · Progress in Artificial Intelligence · doi:10.1007/s13748-023-00306-9 · Cited by 7
“That is, even a minor alteration may outcome a candidate solution with a transformed structural behavior than the parent solution from which it is derived [40, 41].”
BackgroundEdwards, P. J., Peacock, A. M., Renshaw, D., Hannah, J. M., & Murray, A. F. (2002) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2002.1000137 · Cited by 7
“RAINING neural networks, and particularly multilayer perceptrons (MLPs), is a nontrivial task [ 1 ].”
BackgroundModeling with Neural Networks: Principles and Model Design Methodology
Dreyfus, G. (2005) · Springer eBooks · doi:10.1007/3-540-28847-3_2 · Cited by 7
Dash, G. H., & Kajiji, N. (2008) · Springer optimization and its applications · doi:10.1007/978-0-387-76682-9_7 · Cited by 7
Feedforward neural nets as discretization schemes for ODEs and DAEs
Gerstberger, R., & Rentrop, P. (1997) · Journal of Computational and Applied Mathematics · doi:10.1016/s0377-0427(97)00085-x · Cited by 7
Wavelets and Regression Analysis
Zhang, Q. (1995) · Lecture notes in statistics · doi:10.1007/978-1-4612-2544-7_24 · Cited by 7
Building neural networks through linear programming
Zhang, Q., & Fuchs, J. (1999) · IFAC Proceedings Volumes · doi:10.1016/s1474-6670(17)56901-3 · Cited by 7
Sequential function approximation for the solution of differential equations
Meade, A. J., Kokkolaras, M., & Zeldin, B. A. (1997) · Communications in Numerical Methods in Engineering · doi:10.1002/(sici)1099-0887(199712)13:12 · Cited by 7
Classification algorithms on the cell processor
Wyganowski, M. (2008) · RIT Scholar Works (Rochester Institute of Technology) · Cited by 7
Why feed-forward networks are in a bad shape
van der Smagt, P., & Hirzinger, G. (1998) · Perspectives in neural computing · doi:10.1007/978-1-4471-1599-1_20 · Cited by 6
“[3] list a few cases in which the Hessian may become (nearly) singular.”
BackgroundGrafting constructive algorithm in feedforward neural network learning
Zhang, S., & Xie, L. (2022) · Applied Intelligence · doi:10.1007/s10489-022-04082-2 · Cited by 6
“3, there are two datasets with different sample numbers, it is easy to find that there is an ill-condition when the number of the hidden nodes is close to the number of the samples [23].”
BackgroundNeural networks and nonlinear optimization i the representation of continuous functions
Dixon, L. C. W., & Mills, D. J. (1992) · Optimization methods & software · doi:10.1080/10556789208805513 · Cited by 6
Luo, J., Gan, Y., Vong, C., Wong, C.-M., & Chen, C. (2021) · Neurocomputing · doi:10.1016/j.neucom.2021.06.025 · Cited by 6
Influence of Digital Fluctuations on Behavior of Neural Networks
Netay, I. V. (2022) · Indian Journal of Artificial Intelligence and Neural Networking · doi:10.54105/ijainn.a1061.123122 · Cited by 5
Uduagbomen, J., Leeson, M. S., Liu, Z., Lakshminarayana, S., & Xu, T. (2024) · Applied Optics · doi:10.1364/ao.524426 · Cited by 5
Toda, Y., Kodai, Y., Hiwada, E., & Kubota, N. (2011) · IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) · doi:10.1109/fuzzy.2011.6007712 · Cited by 5
Multistage Newton’s Approach for Training Radial Basis Function Neural Networks
Tyagi, K., Rane, C., Irie, B., & MANRY, M. T. (2021) · SN Computer Science · doi:10.1007/s42979-021-00757-8 · Cited by 7
“Equation (9) is often ill-conditioned [43], meaning that the auto-correlation matrix is nearly singular.”
BackgroundImproved Learning of Neural Nets Through Global Search
Plagianakos, V. P., Magoulas, G. D., & Vrahatis, M. N. (2006) · Springer eBooks · doi:10.1007/0-387-30927-6_15 · Cited by 6
“At this point it is worth mentioning an important consideration for adopting an iterative scheme in practical learning tasks is its susceptibility to ill-conditioning: the minimization of the network's learning error is often ill-conditioned, especially when there are many hidden units [51].”
BackgroundDEVELOPMENT OF AN AUTOMATIC DETECTION SYSTEM FOR MEASURING PAVEMENT CRACK DEPTH ON FLORIDA ROADWAYS
Lu, J. J., Mei, X., & Gunaratne, M. (2002) · Cited by 4
Overview of data store management for sliding-window learning using MLP networks
Izzeldin, H., Asirvadam, V. S., & Saad, N. B. (2012) · International Conference on Intelligent and Advanced Systems · doi:10.1109/icias.2012.6306158 · Cited by 4
“Also ill-conditioning is reflected in terms of error curves containing several flat areas and saddle points [3].”
BackgroundDevelopment of a Low-Cost, Noninvasive, Portable Visual Speech Recognition Program
Kohlberg, G. D., Gal, K., & Lalwani, A. K. (2016) · Annals of Otology Rhinology & Laryngology · doi:10.1177/0003489416650689 · Cited by 4
“The input vectors were then scaled so that each variable in the data set ranged from −1 to +1 with a mean of 0 and standard deviation of 1.(23) The ANN classifier was composed of a multilayer perceptron with 1 hidden layer of nodes.”
MethodMultilayer Perceptrons: Other Learning Techniques
Du, K.-L., & Swamy, M. N. S. (2013) · Springer eBooks · doi:10.1007/978-1-4471-5571-3_5 · Cited by 4
Improved Adaptive-Reinforcement Learning Control for Morphing Unmanned Air Vehicles
Doebbler, J., Tandale, M. D., Valasek, J., & Meade, A. (2005) · AIAA Infotech@Aerospace Conference · doi:10.2514/6.2005-7159 · Cited by 4
Metadata
- Authors
- Saarinen, S., Bramley, R., & Cybenko, G.
- ORCID
- Cybenko, G.
- Venue
- SIAM Journal on Scientific Computing
- Published
- 1993-05
- Volume
- 14
- Issue
- 3
- Pages
- 693-714
- ISSN
- 1064-8275 (ISSN-L)
- 1095-7197
- OpenAlex
- W1977923618
- Cited by
- 141
- References
- 11
- Citations by year
- 2026: 1 · 2025: 4 · 2024: 2 · 2023: 4 · 2022: 5 · 2021: 7 · 2020: 6 · 2019: 6 · 2018: 4 · 2017: 2 · 2016: 6 · 2013: 5 · 2012: 5
- FWCI
- 8.37
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- SIAM Journal on Scientific Computing (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Mathematics › Numerical Analysis › Advanced Optimization Algorithms Research
- Keywords
- ill-conditioning, feedforward neural networks, neural network training, nonlinear parameter estimation, gradient descent, sparse connectivity, local minima
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.