The DOI matches its Crossref record; 140 of the 140 references it cites were found in a registry or an index.
A Survey of Optimization Methods From a Machine Learning Perspective
Open accessAbstract
Machine learning develops rapidly, which has made many theoretical breakthroughs and is widely applied in various fields. Optimization, as an important part of machine learning, has attracted much attention of researchers. With the exponential growth of data amount and the increase of model complexity, optimization methods in machine learning face more and more challenges. A lot of work on solving optimization problems or improving optimization methods in machine learning has been proposed successively. The systematic retrospect and summary of the optimization methods from the perspective of machine learning are of great significance, which can offer guidance for both developments of optimization and machine learning research. In this article, we first describe the optimization problems in machine learning. Then, we introduce the principles and progresses of commonly used optimization methods. Finally, we explore and give some challenges and open problems for the optimization in machine learning.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805
Boyd, S., & Vandenberghe, L. (2004) · doi:10.1017/cbo9780511804441 · Cited by 56,601
(2014) · Cited by 33,419
Human-level control through deep reinforcement learning
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. (2015) · Nature · doi:10.1038/nature14236 · Cited by 30,357
Reinforcement Learning: An Introduction
Sutton, R. S., & Barto, A. G. (1998) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.1998.712192 · Cited by 27,225
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545
Reinforcement Learning: An Introduction
(2005) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2004.842673 · Cited by 24,496
Pan, S. J., & Yang, Q. (2010) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2009.191 · Cited by 23,387
Book Review: Pattern Recognition and Machine Learning
Nasrabadi, N. M. (2007) · Journal of Electronic Imaging · doi:10.1117/1.2819119 · Cited by 22,066
Boyd, S., Parikh, N., Chu, E., Peleato, B., & Eckstein, J. (2011) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000016 · Cited by 17,075
Breiman, L. (1996) · Machine Learning · doi:10.1007/bf00058655 · Cited by 16,325
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Algorithm AS 136: A K-Means Clustering Algorithm
Hartigan, J. A., & Wong, M. A. (1979) · Applied Statistics · doi:10.2307/2346830 · Cited by 14,523
Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network
Ledig, C., Theis, L., Huszar, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., et al. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.19 · Cited by 12,696
Boyd, S. (2010) · now publishers, Inc. eBooks · doi:10.1561/9781601984616 · Cited by 12,677
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235
TensorFlow: A system for large-scale machine learning
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.08695 · Cited by 11,822
Wold, S., Esbensen, K., & Geladi, P. (1987) · Chemometrics and Intelligent Laboratory Systems · doi:10.1016/0169-7439(87)80084-9 · Cited by 11,606
An Introduction to Genetic Algorithms
Mitchell, M. (1996) · doi:10.7551/mitpress/3927.001.0001 · Cited by 11,340
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008
A Stochastic Approximation Method
Robbins, H., & Monro, S. (1951) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177729586 · Cited by 9,643
Playing Atari with Deep Reinforcement Learning
Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013) · arXiv · doi:10.48550/arxiv.1312.5602 · Cited by 9,471
(2006) · Springer Series in Operations Research and Financial Engineering · doi:10.1007/978-0-387-40065-5 · Cited by 9,008
Reinforcement Learning: A Survey
Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.301 · Cited by 8,972
Speech recognition with deep recurrent neural networks
Graves, A., Mohamed, A.-R., & Hinton, G. (2013) · 2013 IEEE International Conference on Acoustics, Speech and Signal Processing · doi:10.1109/icassp.2013.6638947 · Cited by 8,893
Goodall, C. (1988) · Technometrics · doi:10.1080/00401706.1988.10488412 · Cited by 8,675
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Finn, C., Abbeel, P., & Levine, S. (2017) · arXiv · doi:10.48550/arxiv.1703.03400 · Cited by 8,512
Reinforcement Learning: An Introduction
Johnson, J. D., Li, J., & Chen, Z. (2000) · Neurocomputing · doi:10.1016/s0925-2312(00)00324-6 · Cited by 8,414
(2010) · Cited by 8,261
Asynchronous Methods for Deep Reinforcement Learning
Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T. P., Harley, T., Silver, D., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1602.01783 · Cited by 7,132
Learning Deep Architectures for AI
Bengio, Y. (2009) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000006 · Cited by 6,938
Large-Scale Video Classification with Convolutional Neural Networks
Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014) · 2014 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2014.223 · Cited by 6,354 · cited 1×
“ious fields of machine learning. CNNs are feedforward neural networks with convolution calculation. CNNs have been successfully used in many fields such as image processing [14], [15], video processing [16] and natural language processing (NLP) [17], [18]. RNNs are a kind of sequential model and very active in NLP [19],…”
Background3D Convolutional Neural Networks for Human Action Recognition
Ji, S., Xu, W., Yang, M., & Yu, K. (2013) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2012.59 · Cited by 6,235
ADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
An overview of gradient descent optimization algorithms
Ruder, S. (2016) · arXiv · doi:10.48550/arxiv.1609.04747 · Cited by 4,962
Learning Deep Architectures for AI
Bengio, Y. (2009) · now publishers, Inc. eBooks · doi:10.1561/9781601982957 · Cited by 4,941
Prototypical Networks for Few-shot Learning
Snell, J., Swersky, K., & Zemel, R. S. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.05175 · Cited by 4,755
Conditioning of quasi-Newton methods for function minimization
Shanno, D. F. (1970) · Mathematics of Computation · doi:10.1090/s0025-5718-1970-0274029-x · Cited by 3,872
An Introduction to Variational Methods for Graphical Models
Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., & Saul, L. K. (1999) · Machine Learning · doi:10.1023/a:1007665907178 · Cited by 3,791
(2013) · Cited by 3,740
Siamese Neural Networks for One-shot Image Recognition
Koch, G. E., Zemel, R. S., & Salakhutdinov, R. (2015) · Cited by 3,740
Multi-column deep neural networks for image classification
Ciresan, D., Meier, U., & Schmidhuber, J. (2012) · 2012 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2012.6248110 · Cited by 3,720
Probabilistic Principal Component Analysis
Tipping, M. E., & Bishop, C. M. (1999) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/1467-9868.00196 · Cited by 3,679
Some methods of speeding up the convergence of iteration methods
Polyak, B. T. (1964) · USSR Computational Mathematics and Mathematical Physics · doi:10.1016/0041-5553(64)90137-5 · Cited by 3,069
An algorithm for quadratic programming
Frank, M., & Wolfe, P. (1956) · Naval Research Logistics Quarterly · doi:10.1002/nav.3800030109 · Cited by 3,030
Linear spatial pyramid matching using sparse coding for image classification
Yang, J., Yu, K., Gong, Y., & Huang, T. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206757 · Cited by 2,828
Optimization as a Model for Few-Shot Learning
Ravi, S., & Larochelle, H. (2017) · International Conference on Learning Representations · Cited by 2,766
Pattern recognition and machine learning
(2007) · Choice Reviews Online · doi:10.5860/choice.44-5091 · Cited by 2,650
SIGNATURE VERIFICATION USING A “SIAMESE” TIME DELAY NEURAL NETWORK
BROMLEY, J., BENTZ, J. W., BOTTOU, L., GUYON, I., LECUN, Y., MOORE, C., SÄCKINGER, E., & SHAH, R. (1994) · Series in Machine Perception and Artificial Intelligence · doi:10.1142/9789812797926_0003 · Cited by 2,350
Recurrent Convolutional Neural Networks for Text Classification
Lai, S., Xu, L., Liu, K., & Zhao, J. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9513 · Cited by 2,280
Self-citationRecurrent Neural Network Regularization
Zaremba, W., Sutskever, I., & Vinyals, O. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1409.2329 · Cited by 2,123
Robust Stochastic Approximation Approach to Stochastic Programming
Nemirovski, A., Juditsky, A., Lan, G., & Shapiro, A. (2009) · SIAM Journal on Optimization · doi:10.1137/070704277 · Cited by 1,945
Accelerating Stochastic Gradient Descent using Predictive Variance Reduction
Johnson, R., & Zhang, T. (2013) · Cited by 1,889
On the Convergence of Adam and Beyond
Reddi, S. J., Kale, S., & Kumar, S. (2019) · arXiv · doi:10.48550/arxiv.1904.09237 · Cited by 1,771
Quasi-Newton Methods, Motivation and Theory
Dennis, Jr., & Moré, J. J. (1977) · SIAM Review · doi:10.1137/1019005 · Cited by 1,553
Meta-learning with memory-augmented neural networks
Santoro, A., Bartunov, S., Botvinick, M., Wierstra, D., & Lillicrap, T. (2016) · International Conference on Machine Learning · Cited by 1,504
Rock: A robust clustering algorithm for categorical attributes
Guha, S., Rastogi, R., & Shim, K. (2000) · Information Systems · doi:10.1016/s0306-4379(00)00022-3 · Cited by 1,472
Pixel Recurrent Neural Networks
van den Oord, A., Kalchbrenner, N., & Kavukcuoglu, K. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.06759 · Cited by 1,437
Stochastic variational inference
Hoffman, M. D., Blei, D. M., Wang, C., & Paisley, J. W. (2013) · Journal of Machine Learning Research · doi:10.5555/2567709.2502622 · Cited by 1,333
Problem complexity and method efficiency in optimization
(1984) · Mathematics and Computers in Simulation · doi:10.1016/0378-4754(84)90101-0 · Cited by 1,329
Derivative-free optimization: a review of algorithms and comparison of software implementations
Rios, L. M., & Sahinidis, N. V. (2013) · Journal of Global Optimization · doi:10.1007/s10898-012-9951-y · Cited by 1,276
Preconditioning Techniques for Large Linear Systems: A Survey
Benzi, M. (2002) · Journal of Computational Physics · doi:10.1006/jcph.2002.7176 · Cited by 1,221
Jolliffe, I. (2011) · International Encyclopedia of Statistical Science · doi:10.1007/978-3-642-04898-2_455 · Cited by 1,206
A Survey of Recent Advances in Hierarchical Clustering Algorithms
Murtagh, F. (1983) · The Computer Journal · doi:10.1093/comjnl/26.4.354 · Cited by 1,058
The Traffic Assignment Problem: Models and Methods
Patriksson, M. (1994) · Medical Entomology and Zoology · Cited by 1,047
Recurrent Neural Network for Text Classification with Multi-Task Learning
Liu, P., Qiu, X., & Huang, X. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1605.05101 · Cited by 972
Minimizing finite sums with the stochastic average gradient
Schmidt, M., Le Roux, N., & Bach, F. (2017) · Mathematical Programming · doi:10.1007/s10107-016-1030-6 · Cited by 922
A clustering technique for summarizing multivariate data
Ball, G. H., & Hall, D. J. (1967) · Behavioral Science · doi:10.1002/bs.3830120210 · Cited by 912
Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization
Jaggi, M. (2013) · Infoscience (Ecole Polytechnique Fédérale de Lausanne) · Cited by 911
A method for unconstrained convex minimization problem with the rate of convergence o(1/k^2)
Nesterov, Y. E. (1983) · Medical Entomology and Zoology · Cited by 865
SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
Defazio, A., Bach, F., & Lacoste-Julien, S. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.0202 · Cited by 834
Semi-Supervised Support Vector Machines
Bennett, K. P., & Demiriz, A. (1998) · Cited by 826
Low-rank matrix completion using alternating minimization
Jain, P., Netrapalli, P., & Sanghavi, S. (2013) · Proceedings of the forty-fifth annual ACM symposium on Theory of Computing · doi:10.1145/2488608.2488693 · Cited by 817
DRAW: A Recurrent Neural Network For Image Generation
Gregor, K., Danihelka, I., Graves, A., Rezende, D. J., & Wierstra, D. (2015) · Cited by 785
Semi-Supervised Classification by Low Density Separation
Chapelle, O., Zien, A., & Ghahramani, R. C. Z. (2005) · Cited by 765
Deep Reinforcement Learning: An Overview
Li, Y. (2017) · arXiv · doi:10.48550/arxiv.1701.07274 · Cited by 762
Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features
Ullah, A., Ahmad, J., Muhammad, K., Sajjad, M., & Baik, S. W. (2018) · IEEE Access · doi:10.1109/access.2017.2778011 · Cited by 756
Deep learning via Hessian-free optimization
Martens, J. (2010) · Cited by 722
Problem Complexity and Method Efficiency in Optimization
Darzentas, J., Nemirovsky, A., & Yudin, D. B. (1984) · Journal of the Operational Research Society · doi:10.2307/2581380 · Cited by 650
Adjiman, C. S., Dallwig, S., Floudas, C. A., & Neumaier, A. (1998) · Computers & Chemical Engineering · doi:10.1016/s0098-1354(98)00027-1 · Cited by 526
Linear spatial pyramid matching using sparse coding for image classification
Yang, J., Yu, K., Gong, Y., & Huang, T. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvprw.2009.5206757 · Cited by 496
A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
Roux, N. L., Schmidt, M., & Bach, F. (2012) · arXiv (Cornell University) · doi:10.48550/arxiv.1202.6258 · Cited by 481
He, B. S., Yang, H., & Wang, S. L. (2000) · Journal of Optimization Theory and Applications · doi:10.1023/a:1004603514434 · Cited by 465
CVXPY: A Python-Embedded Modeling Language for Convex Optimization
Diamond, S., & Boyd, S. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.00943 · Cited by 462
A Stochastic Quasi-Newton Method for Large-Scale Optimization
Byrd, R. H., Hansen, S. L., Nocedal, J., & Singer, Y. (2016) · SIAM Journal on Optimization · doi:10.1137/140954362 · Cited by 459
Stochastic Gradient Hamiltonian Monte Carlo
Chen, T., Fox, E. B., & Guestrin, C. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1402.4102 · Cited by 434
Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
Wu, Y., Mansimov, E., Liao, S., Grosse, R., & Ba, J. L. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.05144 · Cited by 432
Multimodal semi-supervised learning for image classification
Guillaumin, M., Verbeek, J., & Schmid, C. (2010) · 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2010.5540120 · Cited by 418
Optimization Techniques for Semi-Supervised Support Vector Machines
Chapelle, O., Sindhwani, V., & Keerthi, S. S. (2008) · MPG.PuRe (Max Planck Society) · doi:10.5555/1390681.1390688 · Cited by 401
Statistical guarantees for the EM algorithm: From population to sample-based analysis
Balakrishnan, S., Wainwright, M. J., & Yu, B. (2017) · The Annals of Statistics · doi:10.1214/16-aos1435 · Cited by 384
Training recurrent neural networks
Hinton, G. E., & Sutskever, I. (2013) · Cited by 380
Coresets, sparse greedy approximation, and the Frank-Wolfe algorithm
Clarkson, K. L. (2010) · ACM Transactions on Algorithms · doi:10.1145/1824777.1824783 · Cited by 379
Improving Generalization Performance by Switching from Adam to SGD
Keskar, N. S., & Socher, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.07628 · Cited by 373
Non-convex Optimization for Machine Learning
Jain, P., & Kar, P. (2017) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000058 · Cited by 364
SGD-QN: Careful Quasi-Newton Stochastic Gradient Descent
Bordes, A., Bottou, L., & Gallinari, P. (2009) · HAL (Le Centre pour la Communication Scientifique Directe) · Cited by 334
A Stochastic Quasi-Newton Method for Online Convex Optimization
Schraudolph, N. N., Yu, J., & Günter, S. (2007) · ANU Open Research (Australian National University) · Cited by 324
Semi-Supervised Dimensionality Reduction
Zhang, D., Zhou, Z.-H., & Chen, S. (2007) · Proceedings of the 2007 SIAM International Conference on Data Mining · doi:10.1137/1.9781611972771.73 · Cited by 313
Semi-supervised regression with co-training
Zhou, Z., & Li, M. (2005) · International Joint Conference on Artificial Intelligence · Cited by 288
An ADMM Algorithm for a Class of Total Variation Regularized Estimation Problems
Wahlberg, B., Boyd, S., Annergren, M., & Wang, Y. (2012) · IFAC Proceedings Volumes · doi:10.3182/20120711-3-be-2027.00310 · Cited by 256
A convex relaxation approach for computing minimal partitions
Pock, T., Chambolle, A., Cremers, D., & Bischof, H. (2009) · 2009 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2009.5206604 · Cited by 253
Gradient Descent for General Reinforcement Learning
Baird, L. C., & Moore, A. (1998) · Cited by 232
Learning rate schedules for faster stochastic gradient search
Darken, C., Chang, J. Y. T., & Moody, J. E. (2003) · Machine learning for signal processing ... · doi:10.1109/nnsp.1992.253713 · Cited by 219
Nonlinear modeling, estimation and predictive control in APMonitor
Hedengren, J. D., Shishavan, R. A., Powell, K. M., & Edgar, T. F. (2014) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2014.04.013 · Cited by 218
Understanding Alternating Minimization for Matrix Completion
Hardt, M. (2014) · 2014 IEEE 55th Annual Symposium on Foundations of Computer Science · doi:10.1109/focs.2014.75 · Cited by 216
Stochastic Variance Reduction for Nonconvex Optimization
Reddi, S. J., Hefny, A., Sra, S., Póczos, B., & Smola, A. J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06160 · Cited by 215
Chen, Y., & Wainwright, M. J. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.03025 · Cited by 187
Non-convex Optimization for Machine Learning
Jain, P., & Kar, P. (2017) · now publishers, Inc. eBooks · doi:10.1561/9781680833690 · Cited by 185
A modified Frank-Wolfe algorithm for solving the traffic assignment problem
Fukushima, M. (1984) · Transportation Research Part B Methodological · doi:10.1016/0191-2615(84)90029-8 · Cited by 184
Adaptive dimension reduction for clustering high dimensional data
Ding, C., He, X., Hongyuan Zha, B., & Simon, H. D. (2003) · Proceedings · doi:10.1109/icdm.2002.1183897 · Cited by 164
Multichannel Variable-Size Convolution for Sentence Classification
Yin, W., & Schütze, H. (2015) · Proceedings of the Nineteenth Conference on Computational Natural Language Learning · doi:10.18653/v1/k15-1021 · Cited by 162
Natasha 2: Faster Non-Convex Optimization Than SGDPreprint
Allen-Zhu, Z. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.08694 · Cited by 159
Unsupervised and semi-supervised multi-class support vector machines
Xu, L., & Schuurmans, D. (2005) · Cited by 151
Gaussian Process Approach to Remote Sensing Image Classification
Bazi, Y., & Melgani, F. (2009) · IEEE Transactions on Geoscience and Remote Sensing · doi:10.1109/tgrs.2009.2023983 · Cited by 135
Note on Learning Rate Schedules for Stochastic Optimization
Darken, C. J., & Moody, J. E. (1990) · neural information processing systems · Cited by 134
Branch and Bound for Semi-Supervised Support Vector Machines
Chapelle, O., Sindhwani, V., & Keerthi, S. S. (2007) · The MIT Press eBooks · doi:10.7551/mitpress/7503.003.0032 · Cited by 124
A Bi-Projection Neural Network for Solving Constrained Quadratic Optimization Problems
Xia, Y., & Wang, J. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2500618 · Cited by 121
Variance Reduction for Faster Non-Convex OptimizationPreprint
Allen-Zhu, Z., & Hazan, E. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.05643 · Cited by 114
Adaptive dimension reduction for clustering high dimensional data
Ding, C. H. Q., He, X., Hongyuan Zha, B., & Simon, H. D. (2002) · OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) · doi:10.2172/807420 · Cited by 108
Jain, P., Kakade, S. M., Kidambi, R., Netrapalli, P., & Sidford, A. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1610.03774 · Cited by 104
Zhang, S., Xia, Y., & Wang, J. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2441697 · Cited by 98
A Parallel Gradient Descent Method for Learning in Analog VLSI Neural Networks
Alspector, J., Meir, R., Yuhas, B. P., Jayakumar, A. K., & Lippe, D. (1992) · neural information processing systems · Cited by 98
Fast matrix completion without the condition numberPreprint
Hardt, M., & Wootters, M. K. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1407.4070 · Cited by 84
Convex and Scalable Weakly Labeled SVMsPreprint
Li, Y.-F., Tsang, I. W.-H., Kwok, J. T.-Y., et al. (2013) · arXiv (Cornell University) · doi:10.5555/2567709.2567730 · Cited by 84
Training Recurrent Neural Networks
Sutskever, I. (2013) · Cited by 79
Information-theoretic lower bounds on the oracle complexity of convex optimizationPreprint
Agarwal, A., Wainwright, M. J., Bartlett, P. L., & Ravikumar, P. (2009) · arXiv (Cornell University) · Cited by 77
Sub-sampled Newton Methods with Non-uniform SamplingPreprint
Xu, P., Yang, J., Roosta, F., Ré, C., & Mahoney, M. W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.00559 · Cited by 75
Sub-Sampled Newton Methods II: Local Convergence RatesPreprint
Roosta, F., & Mahoney, M. W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1601.04738 · Cited by 74
Schaul, T., & Schmidhuber, J. (2010) · Scholarpedia · doi:10.4249/scholarpedia.4650 · Cited by 73
Fast and Robust General Purpose Clustering Algorithms
Estivill‐Castro, V., & Yang, J. (2004) · Data Mining and Knowledge Discovery · doi:10.1023/b:dami.0000015869.08323.b3 · Cited by 72
Stochastic Conjugate Gradient Algorithm With Variance Reduction
Jin, X., Zhang, X.-Y., Huang, K.-Z., & Geng, G.-G. (2017) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2868835 · Cited by 72
Fast and Robust General Purpose Clustering Algorithms
Estivill‐Castro, V., & Yang, J. (2000) · Lecture notes in computer science · doi:10.1007/3-540-44533-1_24 · Cited by 64
Two Projection Neural Networks With Reduced Model Complexity for Nonlinear Programming
Xia, Y., Wang, J., & Guo, W. (2019) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2019.2927639 · Cited by 56
Second-order Optimization for Neural Networks
Martens, J. (2016) · TSpace (University of Toronto) · Cited by 47
Robust Regression Estimation Based on Low-Dimensional Recurrent Neural Networks
Xia, Y., & Wang, J. (2018) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2018.2814824 · Cited by 35
Compatible natural gradient policy search
Pajarinen, J., Thai, H. L., Akrour, R., Peters, J., & Neumann, G. M. (2019) · Machine Learning · doi:10.1007/s10994-019-05807-0 · Cited by 21
Liu, G. N. (2014) · Cited by 9
Global convergence of online limited memory BFGS
(2015) · Journal of Machine Learning Research · doi:10.5555/2789272.2912100 · Cited by 2
Cited by
Domingos, L. C. F., Santos, P. E., Skelton, P. S. M., Brinkworth, R. S. A., & Sammut, K. (2022) · Sensors · doi:10.3390/s22062181 · Cited by 93
“This procedure is computed by gradient descent, for which various optimisation methods exist (such as Stochastic Gradient Descent (SGD), minibatch gradient descent method (MSGD), and Adaptive moment estimation (ADAM), amongst others [55]).”
MethodA comprehensive survey and taxonomy on privacy-preserving deep learning
Tran, A.-T., Luong, T.-D., & Huynh, V.-N. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127345 · Cited by 21
Metadata
- Authors
- Sun, S., Cao, Z., Zhu, H., & Zhao, J.
- Venue
- IEEE Transactions on Cybernetics
- Published
- 2020-08
- Volume
- 50
- Issue
- 8
- Pages
- 3668-3681
- ISSN
- 2168-2267 (ISSN-L)
- 2168-2275
- OpenAlex
- W2990346675
- Cited by
- 789
- References
- 140
- FWCI
- 29.70
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
