The DOI matches its Crossref record; 3 of the 3 references it cites were found in a registry or an index.
Efficient mini-batch training for stochastic optimization
Abstract
Stochastic gradient descent (SGD) is a popular technique for large-scale optimization problems in machine learning. In order to parallelize SGD, minibatch training needs to be employed to reduce the communication cost. However, an increase in minibatch size typically decreases the rate of convergence. This paper introduces a technique based on approximate optimization of a conservatively regularized objective function within each minibatch. We prove that the convergence rate does not decrease with increasing minibatch size. Experiments demonstrate that with suitable implementations of approximate optimization, the resulting algorithm can outperform standard SGD in many scenarios.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162
Factor graphs and the sum-product algorithm
Kschischang, F. R., Frey, B. J., & Loeliger, H.-A. (2001) · IEEE Transactions on Information Theory · doi:10.1109/18.910572 · Cited by 6,486 · cited 2×
“Graphical Model Inference [15, 14]: In undirected graphical models (and factor graphs) the relationship between random variables can be encoded by clique potentials ψC(w), as given by the Hammersley-Clifford theorem [1].”
MethodBetter Mini-Batch Algorithms via Accelerated Gradient Methods
Cotter, A., Shamir, O., Srebro, N., & Sridharan, K. (2011) · arXiv · doi:10.48550/arxiv.1106.4574 · Cited by 173
Cited by
Federated Optimization in Heterogeneous Networks
Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., & Smith, V. (2018) · arXiv · doi:10.48550/arxiv.1812.06127 · Cited by 4,023
Opportunities and obstacles for deep learning in biology and medicine
Ching, T., Himmelstein, D. S., Beaulieu-Jones, B. K., Kalinin, A. A., Do, B. T., Way, G. P., Ferrero, E., Agapow, P.-M., Zietz, M., Hoffman, M. M., et al. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0387 · Cited by 2,310
Hardware implementation of memristor-based artificial neural networks
Aguirre, F., Sebastian, A., Le Gallo, M., Song, W., Wang, T., Yang, J. J., Lu, W., Chang, M.-F., Ielmini, D., Yang, Y., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-45670-9 · Cited by 564
“For stochastic evaluation, the most common optimization algorithms are: the i) Stochastic Gradient Descent 179 (the stochastic equivalent of the Gradient Descent 165 method previously mentioned, assuming that one epoch consists of only 1 training input vector) and Mini-batch Gradient Descent 180 (which is a generaliza…”
(2017) · doi:10.1109/intellisys.2017.8324237 · Cited by 298
Cryptocurrency Portfolio Management with Deep Reinforcement Learning
Jiang, Z., & Liang, J. (2016) · arXiv · doi:10.48550/arxiv.1612.01277 · Cited by 298
Metadata
- Authors
- Li, M., Zhang, T., Chen, Y., & Smola, A. J.
- Venue
- Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining
- Publisher
- ACM
- Published
- 2014-08-24
- Pages
- 661-670
- OpenAlex
- W2072566913
- Cited by
- 742
- References
- 3
- FWCI
- 17.00
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
