Parameter Conjugate Gradient with Secant Equation Based Elman Neural Network and its Convergence Analysis

Abstract

Elman neural network (ENN) is one of the local recursive networks with a feedback mechanism. The parameter conjugate gradient method is a promising alternative to the gradient descent method, due to its faster convergence speed that results from searching for the conjugate descent direction with an adaptive step size (obtained by Wolfe conditions). However, there are still some challenges such as how to avoid the sawtooth phenomenon in gradient algorithms to improve the learning accuracy of the second‐order curvature of an objective function. As such, this paper presents a novel parametric conjugate gradient method that is based on the secant equation for training ENN in an effective way. Strict proof of the theoretical convergence of the proposed algorithm is provided in detail. In particular, the weak convergence and strong convergence of the algorithm, as well as the monotonicity of the error function are proved. Except for the theoretical analysis, the three numerical experiments have been conducted by applying the algorithm to three problems of classification, regression, and function approximation on nine real‐world datasets. The experimental results have demonstrated the feasibility of the proposed algorithm and the correctness of this theoretical analysis.