aSGD: Stochastic Gradient Descent with Adaptive Batch Size for Every Parameter
Open accessAbstract
In recent years, deep neural networks (DNN) have been widely used in many fields. Lots of effort has been put into training due to their numerous parameters in a deep network. Some complex optimizers with many hyperparameters have been utilized to accelerate the process of network training and improve its generalization ability. It often is a trial-and-error process to tune these hyperparameters in a complex optimizer. In this paper, we analyze the different roles of training samples on a parameter update, visually, and find that a training sample contributes differently to the parameter update. Furthermore, we present a variant of the batch stochastic gradient decedent for a neural network using the ReLU as the activation function in the hidden layers, which is called adaptive stochastic gradient descent (aSGD). Different from the existing methods, it calculates the adaptive batch size for each parameter in the model and uses the mean effective gradient as the actual gradient for parameter updates. Experimental results over MNIST show that aSGD can speed up the optimization process of DNN and achieve higher accuracy without extra hyperparameters. Experimental results over synthetic datasets show that it can find redundant nodes effectively, which is helpful for model compression.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Combining the suggested ECDSA with the enhanced ECC-based three-factor multiserver authentication approach proposed by Bao et al. gives an even stronger and better security mechanism for IIoT systems.
“We can even combine the suggested ECDSA, which the blockchain utilizes to generate new public keys, with the enhanced ECC-based three-factor multiserver authentication approach proposed by Bao et al. [185] to give an even stronger and better security mechanism for IIoT systems.”
From A Critical Overview of Industrial Internet of Things Security and Privacy Issues Using a Layer-Based Hacking Scenario (2023) · cites this paper as [185]
The Experiments and Conclusions sections discuss adaptive stochastic gradient descent, not authentication for IIoT systems. Searches for Bao’s ECC-based three-factor approach and for ECDSA, blockchain, and IIoT found no passage addressing the cited claim. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · see citing works
(2014) · Cited by 33,419 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · see citing works
Learning Internal Representations by Error Propagation
Rumelhart, D. E., Hinton, G. E., & Williams, R. J. (1985) · doi:10.21236/ada164453 · Cited by 16,501 · see citing works
(2010) · Cited by 14,134 · see citing works
A Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643 · see citing works
(2010) · Cited by 8,261 · see citing works
ADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322 · see citing works
An overview of gradient descent optimization algorithms
Ruder, S. (2016) · arXiv · doi:10.48550/arxiv.1609.04747 · Cited by 4,962 · see citing works
(2011) · Cited by 3,939 · see citing works
Deep Learning using Rectified Linear Units (ReLU)Preprint
Agarap, A. F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1803.08375 · Cited by 3,792 · see citing works
Multi-column deep neural networks for image classification
Ciresan, D. C., Meier, U., & Schmidhuber, J. (2012) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr.2012.6248110 · Cited by 3,760 · see citing works
(2013) · Cited by 3,740 · see citing works
End to End Learning for Self-Driving Cars
Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016) · arXiv · doi:10.48550/arxiv.1604.07316 · Cited by 3,198 · see citing works
Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition
Dahl, G. E., Yu, D., Deng, L., & Acero, A. (2012) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/tasl.2011.2134090 · Cited by 3,011 · see citing works
An Overview of Evolutionary Algorithms for Parameter Optimization
Bäck, T., & Schwefel, H. (1993) · Evolutionary Computation · doi:10.1162/evco.1993.1.1.1 · Cited by 2,111 · see citing works
Practical markov chain monte carlo
Geyer, C. J. (1992) · Statistical Science · doi:10.1214/ss/1177011137 · Cited by 2,045 · see citing works
Activation Functions: Comparison of trends in Practice and Research for Deep LearningPreprint
Nwankpa, C. E., Ijomah, W., Gachagan, A., & Marshall, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.03378 · Cited by 1,304 · see citing works
On Early Stopping in Gradient Descent Learning
Yao, Y., Rosasco, L., & Caponnetto, A. (2007) · Constructive Approximation · doi:10.1007/s00365-006-0663-2 · Cited by 990 · see citing works
The influence of the sigmoid function parameters on the speed of backpropagation learning
Han, J., & Moraga, C. (1995) · Lecture notes in computer science · doi:10.1007/3-540-59497-3_175 · Cited by 986 · see citing works
An Analysis of Deep Neural Network Models for Practical Applications
Canziani, A., Paszke, A., & Culurciello, E. (2016) · arXiv · doi:10.48550/arxiv.1605.07678 · Cited by 969 · see citing works
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2572 · Cited by 827 · see citing works
Don't Decay the Learning Rate, Increase the Batch SizePreprint
Smith, S. L., Kindermans, P.-J., Ying, C., & Le, Q. V. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1711.00489 · Cited by 791 · see citing works
Cancer Diagnosis Using Deep Learning: A Bibliographic Review
Munir, K., Elahi, H., Ayub, A., Frezza, F., & Rizzi, A. (2019) · Cancers · doi:10.3390/cancers11091235 · Cited by 430 · see citing works
Radiuk, P. (2017) · Information Technology and Management Science · doi:10.1515/itms-2017-0003 · Cited by 243 · see citing works
Deep learning for self-driving cars
Rao, Q., & Frtunikj, J. (2018) · doi:10.1145/3194085.3194087 · Cited by 183 · see citing works
Kumar, R. (2017) · Apress eBooks · doi:10.1007/978-1-4842-3069-5_4 · Cited by 155 · see citing works
Accelerating Minibatch Stochastic Gradient Descent using Stratified SamplingPreprint
Zhao, P., & Zhang, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.3080 · Cited by 82 · see citing works
Performance comparison of artificial neural network models for daily rainfall prediction
Devi, S. R., Arulmozhivarman, P., Venkatesh, C., & Agarwal, P. (2016) · Machine Intelligence Research · doi:10.1007/s11633-016-0986-2 · Cited by 49 · see citing works
Chattopadhyay, S., & Chattopadhyay, G. (2018) · Journal of Atmospheric and Solar-Terrestrial Physics · doi:10.1016/j.jastp.2018.07.015 · Cited by 36 · see citing works
Prediction of rainfall based on weather parameter using artificial neural network
Hashim, F. R., Daud, N. G. N., Ahmad, K. A., Adnan, J., & Rizman, Z. I. (2018) · Journal of Fundamental and Applied Sciences · doi:10.4314/jfas.v9i3s.38 · Cited by 22 · see citing works
Yang, N., Tang, H., Yue, J., Yang, X., & Xu, Z. (2020) · IEEE Access · doi:10.1109/access.2020.3013621 · Cited by 13 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 7
- Fakoor, R.; Ladhak, F.; Nazi, A.; Huber, M. Using deep learning to enhance cancer diagnosis and classification. In Proceedings of the International Conference on Machine Learning; ACM: New York, NY, USA, 2013; Volume 28, pp. 3937-3949.
- Sutton, R.S.; Barto, A.G. Reinforcement Learning: An Introduction; MIT Press: Cambridge, MA, USA, 2018.
- Tieleman, T.; Hinton, G. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. Coursera Neural Netw. Mach. Learn. 2012, 4, 26-31.
- Bengio, Y.; Senécal, J.S. Quick Training of Probabilistic Neural Nets by Importance Sampling; AISTATS: Key West, FL, USA, 2003; pp. 1-9.
- Nair, V.; Hinton, G.E. Rectified Linear Units Improve Restricted Boltzmann Machines; ICML: Haifa, Israel, 2010.
- Goodfellow, I.; Bengio, Y.; Courville, A.; Bengio, Y. Deep Learning; MIT Press: Cambridge, UK, 2016; Volume 1.
- Ji Chao, T.H. Visualization and Coding of Original Feature Space Partitioning Process Based on Fully Connected Neural Network. J. Signal Process. 2020, 36, 486-494. [CrossRef] · DOI
Cited by
Sorted by citations
Deep Autoencoder Neural Networks: A Comprehensive Review and New Perspectives
Mienye, I. D., & Swart, T. G. (2025) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-025-10260-5 · Cited by 115 · see citing works
“Stochastic gradient descent (SGD), a widely used optimization algorithm in neural network training, updates the model weights iteratively based on a subset of the entire data set, known as a minibatch [23].”
Eyeleko, A. H., & Feng, T. (2023) · IEEE Internet of Things Journal · doi:10.1109/jiot.2023.3308195 · Cited by 38 · see citing works
A Deep Learning Optimizer Based on Grünwald–Letnikov Fractional Order Definition
Zhou, X., Zhao, C., & Huang, Y. (2023) · Mathematics · doi:10.3390/math11020316 · Cited by 25 · see citing works
“As can be seen from Figure 5, when the hyperparameter α = 0.3 of FCAdam_G-L DJIA has the highest precision of train loss convergence, and the , _G-L is faster than the SGD and the SGDM, and when α 0.1, FCSGD_G-L It can be seen from Figure 4, on DJIA, that the train loss convergence speed of FCSGD G-L to train DJIA, du…”
BackgroundMethodInfluentialYu, H., Cui, Y., Li, S., Kang, S., Yao, Z., & Wei, Z. (2023) · Remote Sensing of Environment · doi:10.1016/j.rse.2023.113819 · Cited by 13 · see citing works
Evaluation of Optimization Algorithms for Measurement of Suspended Solids
López-Betancur, D., González, E., Guerrero-Méndez, C., Saucedo-Anaya, T., Rivera, M. M., Olmos-Trujillo, E., & Gómez-Jiménez, S. (2024) · Water · doi:10.3390/w16131761 · Cited by 9 · see citing works
CRCS: An automatic image processing pipeline for hormone level analysis of Cushing’s disease
Li, H., Xie, J., Song, J., Jin, C., Xin, H., Pan, X., Ke, J., Yuan, Y., Shen, H., & Ning, G. (2023) · Methods · doi:10.1016/j.ymeth.2023.12.003 · Cited by 2 · see citing works
Li, J. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3590151
“Work [11] looked at how supervised machine learning may be applied to expect student exam performance.”
BackgroundPronosumarto, M. H. A., Sambhuwara, J., Usman, K., & Fuadah, Y. N. (2024) · Lecture notes in electrical engineering · doi:10.1007/978-981-97-1463-6_21
Dehghani, M., Aly, M., Said, S. M., Abdelhameed, D., & Rodríguez, J. (2025) · IEEE Conference on Power Electronics and Renewable Energy (CPERE) · doi:10.1109/cpere65146.2025.11240027
“To construct the output 𝑌 𝑎𝑡𝑡𝑎𝑐𝑘 ∈ (0, 1) for DNN, information about the attack must be revealed in the DNN: According to the 𝑅 training data set {𝑊 𝑣 (𝑟), 𝑊 𝐼 (𝑟),𝑌 𝑎𝑡𝑡𝑎𝑐𝑘 (𝑟)} 𝑟=1𝑅 , the learning parameters in DNN are trained using numerous gradient descent algorithms [15].”
VivoOptiNet: A Gradient Update Adam Optimizer for ECG Arrhythmia Classification
Patel, K., Pandey, J. P., & Dutta, M. K. (2025) · International Journal of Pattern Recognition and Artificial Intelligence · doi:10.1142/s0218001425400063
Cross-Validated Regularization for Robust Mahalanobis Metric Learning
Mones, M. M. (2025) · Jurnal Pendidikan Matematika · doi:10.47134/ppm.v3i1.2078
- Authors
- Shi, H., Yang, N., Tang, H., & Yang, X.
- Affiliations
- Beijing Normal University (CN)
- Shandong University (CN)
- Corresponding author
- Tang, Hong
- Venue
- Mathematics
- Publisher
- MDPI AG
- Published
- 2022-03-09
- Volume
- 10
- Issue
- 6
- Pages
- 863
- ISSN
- 2227-7390 (ISSN-L)
- OpenAlex
- W4221055587
- Cited by
- 11
- References
- 22
- Citations by year
- 2025: 4 · 2024: 2 · 2023: 5
- FWCI
- 0.90
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Mathematics (publishedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Natural Science Foundation of China (41971280)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Data Classification
- Keywords
- adaptive batch size, stochastic gradient descent, deep neural network training, hyperparameter optimization, ReLU activation, model compression, optimization acceleration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
