Review of Deep Learning Algorithms and Architectures
Open accessAbstract
Deep learning (DL) is playing an increasingly important role in our lives. It has already made a huge impact in areas, such as cancer diagnosis, precision medicine, self-driving cars, predictive forecasting, and speech recognition. The painstakingly handcrafted feature extractors used in traditional learning, classification, and pattern recognition systems are not scalable for large-sized data sets. In many cases, depending on the problem complexity, DL can also overcome the limitations of earlier shallow networks that prevented efficient training and abstractions of hierarchical representations of multi-dimensional training data. Deep neural network (DNN) uses multiple (deep) layers of units with highly optimized algorithms and architectures. This paper reviews several optimization methods to improve the accuracy of the training and to reduce training time. We delve into the math behind training algorithms used in recent deep networks. We describe current shortcomings, enhancements, and implementations. The review also covers different types of deep architectures, such as deep convolution networks, deep residual networks, recurrent neural networks, reinforcement learning, variational autoencoders, and others.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
ANN, Deep Neural Network (DNN), SVM, GA, LR, and DT are among the prominent AI techniques that gained strong traction in recent years, as are ML, DL, and cognitive AI models in relation to the planning and development of smart cities.
“As consistent with the findings for the second and third periods, ANN, Deep Neural Network (DNN), SVM, GA, LR, and DT are among the prominent AI techniques that gained strong traction in recent years (Şerban and Lytras 2020; Shrestha and Mahmood 2019), so are ML, DL, and cognitive AI models in relation to the planning and development of smart cities (Kamrowska-Załuska 2021; Şerban and Lytras 2020”
From Environmentally sustainable smart cities and their converging AI, IoT, and big data technologies and solutions: an integrated approach to an extensive literature review (2023) · cites this paper as Shrestha and Mahmood 2019
Deep learning is gaining traction as a type of ANN with more layers of units.
“Deep learning is gaining traction as a type of ANN with more layers of units (Shrestha and Mahmood, 2019).”
From Artificial intelligence for sustainability: Challenges, opportunities, and a research agenda (2020) · cites this paper as Shrestha and Mahmood, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474 · cited 5× · see citing works
“tion with a 152-layer DNN on the ImageNet dataset [62].”
BackgroundMethodInfluentialHochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 2× · see citing works
“The input, forget and output gates dictates new data going into the cell, what remains in the cell and the cell values used in the calculation of the output of the LSTM block respectively [41], [43].”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 4× · see citing works
“Multilayer network training cost on MNIST dataset using different adaptive learning algorithms [58].”
BackgroundMethodInfluentialLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126 · see citing works
Gradient-based learning applied to document recognition
Lecun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998) · Proceedings of the IEEE · doi:10.1109/5.726791 · Cited by 58,805 · cited 2× · see citing works
“7-layer architecture of CNN for character recognition [28].”
BackgroundMethodAdaptation in Natural and Artificial Systems
Holland, J. H. (1992) · The MIT Press eBooks · doi:10.7551/mitpress/1090.001.0001 · Cited by 47,340 · see citing works
(2014) · Cited by 33,419 · see citing works
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Ioffe, S., & Szegedy, C. (2015) · arXiv · doi:10.48550/arxiv.1502.03167 · Cited by 27,818 · cited 2× · see citing works
“[59] S. Ioffe and C. Szegedy, ‘‘Batch normalization: Accelerating deep network training by reducing internal covariate shift,’’ CoRR, Feb. 2015.”
BackgroundLearning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1179 · Cited by 25,545 · cited 1× · see citing works
“GRUs are smaller in size than LSTM as they don’t include the output gate, and can perform better than LSTM on only some simpler datasets [44], [45].”
BackgroundReducing the dimensionality of data with neural networks
Hinton, G. E., & Salakhutdinov, R. R. (2006) · Science · doi:10.1126/science.1127647 · Cited by 20,928 · see citing works
Auto-Encoding Variational Bayes
Kingma, D. P., & Welling, M. (2013) · arXiv · doi:10.48550/arxiv.1312.6114 · Cited by 20,067 · see citing works
Mastering the game of Go with deep neural networks and tree search
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al. (2016) · Nature · doi:10.1038/nature16961 · Cited by 15,612 · cited 1× · see citing works
“AlphaGo, Google’s program that beat the human Go champion also uses reinforcement learning [91].”
BackgroundDermatologist-level classification of skin cancer with deep neural networks
Esteva, A., Kuprel, B., Novoa, R. A., Ko, J., Swetter, S. M., Blau, H. M., & Thrun, S. (2017) · Nature · doi:10.1038/nature21056 · Cited by 14,362 · cited 1× · see citing works
“, [12] reported that an deep learning network was able to learn from 129,450 images of 2,032 diseases and was able to diagnose at the same level as 21 board certified dermatologists.”
Background(2010) · Cited by 14,134 · see citing works
Approximation by superpositions of a sigmoidal function
Cybenko, G. (1989) · Mathematics of Control, Signals, and Systems · doi:10.1007/bf02551274 · Cited by 13,883 · cited 1× · see citing works
“The universal approximation theorem showed that a single hidden layer was able to solve any continuous problem [3].”
BackgroundExtreme learning machine: Theory and applications
Huang, G.-B., Zhu, Q.-Y., & Siew, C.-K. (2006) · Neurocomputing · doi:10.1016/j.neucom.2005.12.126 · Cited by 13,285 · see citing works
The perceptron: A probabilistic model for information storage and organization in the brain.
Rosenblatt, F. (1958) · Psychological Review · doi:10.1037/h0042519 · Cited by 12,280 · cited 1× · see citing works
“In 1957, Frank Rosenblatt created the perceptron, the first prototype of what we now know as a neural network [1].”
BackgroundUnsupervised representation learning with deep convolutional generative adversarial networks
Radford, A., Metz, L., & Chintala, S. (2015) · arXiv · doi:10.48550/arxiv.1511.06434 · Cited by 12,235 · cited 1× · see citing works
“Deep convolutional generative adversarial networks proposed by [33].”
Background(2016) · Cited by 11,679 · see citing works
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
Chung, J., Gulcehre, C., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.3555 · Cited by 11,008 · cited 1× · see citing works
“GRUs are smaller in size than LSTM as they don’t include the output gate, and can perform better than LSTM on only some simpler datasets [44], [45].”
BackgroundMachine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955 · cited 1× · see citing works
“ML will continue to impact and disrupt all areas of our lives from education, finance, governance, healthcare, manufacturing, marketing and others [7].”
Background(2010) · Cited by 8,261 · see citing works
Chiang, L. H., Russell, E. L., & Braatz, R. D. (2001) · Advanced textbooks in control and signal processing · doi:10.1007/978-1-4471-0347-9_4 · Cited by 6,684 · see citing works
Approximation capabilities of multilayer feedforward networks
Hornik, K. (1991) · Neural Networks · doi:10.1016/0893-6080(91)90009-t · Cited by 6,091 · cited 1× · see citing works
“It was mathematically proven as well [4], which further questioned the validity of DNN.”
BackgroundLearning to Forget: Continual Prediction with LSTM
Gers, F. A., Schmidhuber, J., & Cummins, F. (2000) · Neural Computation · doi:10.1162/089976600300015015 · Cited by 5,869 · cited 2× · see citing works
“The input, forget and output gates dictates new data going into the cell, what remains in the cell and the cell values used in the calculation of the output of the LSTM block respectively [41], [43].”
MethodDimensionality Reduction by Learning an Invariant Mapping
Hadsell, R., Chopra, S., & LeCun, Y. (2006) · 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2 (CVPR'06) · doi:10.1109/cvpr.2006.100 · Cited by 5,336 · see citing works
Deep Reinforcement Learning: A Brief Survey
Arulkumaran, K., Deisenroth, M. P., Brundage, M., & Bharath, A. A. (2017) · IEEE Signal Processing Magazine · doi:10.1109/msp.2017.2743240 · Cited by 4,733 · cited 2× · see citing works
“With DRL, problems that were intractable with reinforcement learning can now be solved with higher number of hidden layers of deep networks and reinforcement learning based Q-learning algorithm that maximizes the reward for actions taken by the agent [13].”
BackgroundConvolutional networks for images, speech, and time series
LeCun, Y., & Bengio, Y. (1998) · HAL · doi:10.5555/303568.303704 · Cited by 4,637 · see citing works
Whitley, D. (1994) · Statistics and Computing · doi:10.1007/bf00175354 · Cited by 4,523 · cited 1× · see citing works
“of GA can be executed independently in islands and few members exchanged between the island every so often [81].”
BackgroundGenerative Adversarial Networks
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.2661 · Cited by 4,477 · see citing works
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1406.1078 · Cited by 3,317 · cited 1× · see citing works
“GRUs are smaller in size than LSTM as they don’t include the output gate, and can perform better than LSTM on only some simpler datasets [44], [45].”
BackgroundUnsupervised Learning of Depth and Ego-Motion from Video
Zhou, T., Brown, M., Snavely, N., & Lowe, D. G. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.700 · Cited by 2,910 · cited 1× · see citing works
“Reference [22] demonstrated enhance results with unsupervised learning using unstructured video sequences for camera motion estimation and monocular depth.”
BackgroundA Practical Guide to Training Restricted Boltzmann Machines
Hinton, G. E. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-35289-8_32 · Cited by 2,852 · cited 6× · see citing works
“The probability of every pair of visible and hidden vectors is given by the following [40].”
BackgroundInfluentialDeep learning applications and challenges in big data analytics
Najafabadi, M. M., Villanustre, F., Khoshgoftaar, T. M., Seliya, N., Wald, R., & Muharemagic, E. (2015) · Journal of Big Data · doi:10.1186/s40537-014-0007-7 · Cited by 2,594 · cited 3× · see citing works
“[47] raises similar challenges learning from real-time streaming data such as credit cards”
BackgroundInfluentialGenerative Adversarial Text to Image Synthesis
Reed, S., Akata, Z., Yan, X., Logeswaran, L., Schiele, B., & Lee, H. (2016) · arXiv · doi:10.48550/arxiv.1605.05396 · Cited by 2,494 · cited 1× · see citing works
“, [94] were able to successfully synthesize realistic images from text.”
BackgroundConvolutional networks and applications in vision
LeCun, Y., Kavukcuoglu, K., & Farabet, C. (2010) · Proceedings of 2010 IEEE International Symposium on Circuits and Systems · doi:10.1109/iscas.2010.5537907 · Cited by 2,213 · cited 1× · see citing works
“asserts that one way we could achieve this would be to utilize and integrate both unsupervised and supervised learning [24].”
BackgroundDo we need hundreds of classifiers to solve real world classification problems
Fernández-Delgado, M., Cernadas, E., Barro, S., & AmorimDinani (2014) · Journal of Machine Learning Research · doi:10.5555/2627435.2697065 · Cited by 2,131 · cited 1× · see citing works
“Nonetheless [27] evaluated 179 classifiers and concluded that parallel random forest or parRF_t, which is essentially parallel implementation of variation of decision tree, performed the best.”
Background(2001) · Cited by 1,932 · see citing works
Salakhutdinov, R., & Hinton, G. E. (2009) · Cited by 1,890 · see citing works
Buhmann, M. (2003) · Springer eBooks · doi:10.1007/978-3-540-70529-1_306 · Cited by 1,797 · see citing works
Extreme Learning Machine for Multilayer Perceptron
Tang, J., Deng, C., & Huang, G.-B. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2424995 · Cited by 1,456 · cited 1× · see citing works
“came up with a multilayer version of ELM in 2016 [70] to take on more complex problems.”
BackgroundPouyanfar, S., Sadiq, S. B., Yan, Y., Tian, H., Tao, Y., Reyes, M. P., Shyu, M., Chen, S., & Iyengar, S. S. (2018) · ACM Computing Surveys · doi:10.1145/3234150 · Cited by 1,435 · see citing works
An Introduction to Deep Reinforcement Learning
François-Lavet, V., Henderson, P., Islam, R., Bellemare, M. G., & Pineau, J. (2018) · Foundations and Trends® in Machine Learning · doi:10.1561/2200000071 · Cited by 1,353 · see citing works
Big data deep learning: Challenges and perspectives
Chen, X.-W., & Lin, X. (2014) · IEEE Access · doi:10.1109/access.2014.2325029 · Cited by 1,255 · cited 9× · see citing works
“Below is the equation [23] for change in weights, where c is the momentum factor and α is the learning rate, and v and h are visible and hidden units respectively.”
BackgroundMethodInfluentialHuang, L., Joseph, A. D., Nelson, B., Rubinstein, B. I. P., & Tygar, J. D. (2011) · Proceedings of the 4th ACM workshop on Security and artificial intelligence · doi:10.1145/2046684.2046692 · Cited by 1,194 · cited 1× · see citing works
“Reference [86] demonstrates these and other vulnerabilities and discusses how application domain, features and data distribution can be used to reduce the risk and impact of such adversarial attacks.”
BackgroundVery Deep Convolutional Networks for Text Classification
Conneau, A., Schwenk, H., Barrault, L., & Lecun, Y. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1104 · Cited by 1,046 · cited 2× · see citing works
“Reference [62] addressed this issue with shortcut connections between the convolution blocks to let the gradients to propagate more freely and they, along with [74] were able to validate the benefits of the shortcuts with 10/101/152-layers and 49 layers respectively.”
BackgroundMethodLearning multiple layers of representation
Hinton, G. E. (2007) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2007.09.004 · Cited by 1,013 · cited 3× · see citing works
“Here is the equation [66] for adjusting the recognition weights rij.”
BackgroundMethodInfluentialThe Design of Innovation: Lessons from and for Competent Genetic Algorithms
Goldberg, D. E. (2002) · Cited by 912 · see citing works
Miikkulainen, R. P., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., et al. (2018) · Elsevier eBooks · doi:10.1016/b978-0-12-815480-9.00015-3 · Cited by 854 · see citing works
Genetic Algorithms for the Travelling Salesman Problem: A Review of Representations and Operators
Larrañaga, P., Kuijpers, C. M. H., Murga, R. H., Inza, I., & Dizdarevic, S. (1999) · Artificial Intelligence Review · doi:10.1023/a:1006529012972 · Cited by 849 · see citing works
Deep learning via Hessian-free optimization
Martens, J. (2010) · Cited by 722 · see citing works
Convolutional Learning of Spatio-temporal Features
Taylor, G. W., Fergus, R., LeCun, Y., & Bregler, C. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-15567-3_11 · Cited by 645 · cited 1× · see citing works
“The deep nature of the CNN results in high quality representations while maintaining locality, reduced parameters and invariance to minor variations in the input image [30].”
BackgroundAdvances in Instance Selection for Instance-Based Learning Algorithms
Brighton, H., & Mellish, C. (2002) · Data Mining and Knowledge Discovery · doi:10.1023/a:1014043630878 · Cited by 534 · cited 1× · see citing works
“Instead of training the network with all the data set, we can preselect a smaller but representative data set from the full training distribution set using instance selection methods [95] or Monte Carlo sampling [48].”
MethodNagpal, K., Foote, D., Liu, Y., Chen, P.-H. C., Wulczyn, E., Tan, F., Olson, N., Smith, J. L., Mohtashamian, A., Wren, J. H., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0112-2 · Cited by 495 · cited 2× · see citing works
“Google AI [10] was able to beat the average accuracy of US board certified general pathologists in grading prostate cancer by 70% to 61%.”
BackgroundAutomatic Speech Recognition: A Deep Learning Approach
Yu, D., & Li Juan, D. E. N. G. (2014) · Signals and communication technology · doi:10.1007/978-1-4471-5779-3 · Cited by 475 · cited 2× · see citing works
“GMM are very effective in extracting speech features and modeling the probability density function to a desired level of accuracy as long as we have sufficient components, and the estimation maximization makes it easy to fit the model [87].”
BackgroundMethodMulti-instance multi-label learning
Zhou, Z., Zhang, M.-L., Huang, S.-J., & Li, Y.-F. (2011) · Artificial Intelligence · doi:10.1016/j.artint.2011.10.002 · Cited by 424 · see citing works
Genetic Algorithm Performance with Different Selection Strategies in Solving TSP
Razali, N. M., & Geraghty, J. (2011) · Cited by 359 · see citing works
A Machine Learning Approach for Non-blind Image Deconvolution
Schuler, C. J., Burger, H. C., Harmeling, S., & Scholkopf, B. (2013) · 2013 IEEE Conference on Computer Vision and Pattern Recognition · doi:10.1109/cvpr.2013.142 · Cited by 319 · cited 1× · see citing works
“Reference [32] achieves this by leveraging Fourier transformation to regularize inversion of the blurred images and denoising.”
MethodA Framework for Designing the Architectures of Deep Convolutional Neural Networks
Albelwi, S. A., & Mahmood, A. (2017) · Entropy · doi:10.3390/e19060242 · Cited by 283 · see citing works
Self-citationUnconstrained Face Recognition: Identifying a Person of Interest From a Media Collection
Best-Rowden, L., Han, H., Otto, C., Klare, B. F., & Jain, A. K. (2014) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2014.2359577 · Cited by 205 · cited 1× · see citing works
“Reference [64] proposes a method to utilize multiple images of the unconstrained source to enhance the recognition process.”
MethodA Multiobjective Sparse Feature Learning Model for Deep Neural Networks
Gong, M., Liu, J., Li, H., Cai, Q., & Su, L. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2015.2469673 · Cited by 198 · cited 3× · see citing works
“Their learning procedure continuously iterates between evolutionary optimization step and the stochastic gradient descent to optimize the reconstruction error [71].”
BackgroundMethodInfluentialGenerative or Discriminative? Getting the Best of Both Worlds
Bishop, C., & Lasserre, J. (2007) · doi:10.1093/oso/9780199214655.003.0001 · Cited by 191 · see citing works
Large-scale information retrieval with latent semantic indexing
Letsche, T. A., & Berry, M. W. (1997) · Information Sciences · doi:10.1016/s0020-0255(97)00044-3 · Cited by 187 · cited 1× · see citing works
“Deep learning can help with semantic indexing to enable information to be more readily accessible in search engines [14], [65].”
BackgroundA Supervised Learning and Control Method to Improve Particle Swarm Optimization Algorithms
Dong, W., & Zhou, M. (2016) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2016.2560128 · Cited by 180 · cited 6× · see citing works
“The supervised learning module provided real-time feedback with back diffusion (BD) to retain diversity and social attractor renewal to overcome stagnation [76].”
BackgroundInfluentialDeepBox: Learning Objectness with Convolutional Networks
Kuo, W., Hariharan, B., & Malik, J. (2015) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.285 · Cited by 178 · cited 1× · see citing works
“Reference [68] proposed a four-layered CNN termed DeepBox that outperformed larger networks in speed and accuracy.”
BackgroundLean GHTorrent: GitHub data on demand
Gousios, G., Vasilescu, B., Serebrenik, A., & Zaidman, A. (2014) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/2597073.2597126 · Cited by 173 · cited 1× · see citing works
“Github is the largest hosting service provider of source code in the world [25].”
BackgroundParticle Swarm Model Selection
Escalante, H. J., Montes, M., & Sucar, L. E. (2009) · doi:10.5555/1577069.1577084 · Cited by 172 · cited 2× · see citing works
“used PSO for hyperparameter optimization to build a competitive model that ranked among the top relative to other comparable methods [52].”
MethodA Survey on Deep Learning in Big Data
Gheisari, M., Wang, G., & Bhuiyan, M. Z. A. (2017) · IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC) · doi:10.1109/cse-euc.2017.215 · Cited by 154 · cited 1× · see citing works
“Deep learning can help with semantic indexing to enable information to be more readily accessible in search engines [14], [65].”
BackgroundRate-coded Restricted Boltzmann Machines for Face Recognition
Teh, Y. W., & Hinton, G. E. (2000) · Cited by 134 · see citing works
A recurrent neural network for classification of unevenly sampled variable stars
Naul, B., Bloom, J. S., Pérez, F., & van der Walt, S. (2017) · Nature Astronomy · doi:10.1038/s41550-017-0321-z · Cited by 131 · cited 1× · see citing works
“demonstrated LSTM andGRU based autoencoders for automatic feature extractions [46].”
BackgroundA recurrent neural network for classification of unevenly sampled variable stars
Naul, B., Bloom, J. S., Pérez, F., & van der Walt, S. (2018) · eScholarship (California Digital Library) · Cited by 101 · see citing works
Deep Learning-Based Model Reduction for Distributed Parameter Systems
Wang, M., Li, H., Chen, X., & Chen, Y. (2016) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2016.2605159 · Cited by 98 · cited 1× · see citing works
“[37] successfully implemented a deep autoencoder with stacks of RBM blocks similar to Figure 6 to achieve better modeling accuracy and efficiency than the proper orthogonal decomposition (POD) method for dimensionality reduction of distributed parameter systems (DPSs).”
MethodDeep Learning: A ReviewPreprint
Vargas, R. H., Mosavi, A. H., & Ruiz, R. (2018) · Preprints.org · doi:10.20944/preprints201810.0218.v1 · Cited by 93 · cited 1× · see citing works
“Other review papers [13]–[16] focus on specific areas and implementations without encompass the full scope of the field.”
BackgroundKhan, S. H., Rahmani, H., Shah, S. A. A., & Bennamoun, M. (2018) · Synthesis lectures on computer vision · doi:10.1007/978-3-031-01821-3_4 · Cited by 79 · see citing works
An Improved Polynomial Neural Network Classifier Using Real-Coded Genetic Algorithm
Lin, C., Prasad, M., & Saxena, A. K. (2015) · IEEE Transactions on Systems Man and Cybernetics Systems · doi:10.1109/tsmc.2015.2406855 · Cited by 78 · cited 2× · see citing works
“Standard GA operations including selection, crossover and mutation were used on parameters that included partial descriptions (PDs) of inputs in the first layer, bias and all input features [82].”
MethodSOM: Stochastic initialization versus principal components
Akinduko, A. A., Mirkes, E. M., & Gorban, A. N. (2015) · Information Sciences · doi:10.1016/j.ins.2015.10.013 · Cited by 73 · cited 1× · see citing works
“The weights of the closest node and its neighbors from the input data are updated to bring them closer to the input data with the formula below [18].”
BackgroundGuo, Y. P., Cai, Q., Samuels, D. C., Ye, F., Long, J., Li, C.-I., Winther, J. F., Tawn, E. J., Stovall, M. A., Lähteenmäki, P. M., et al. (2012) · Mutation Research/Genetic Toxicology and Environmental Mutagenesis · doi:10.1016/j.mrgentox.2012.02.006 · Cited by 57 · see citing works
Multiclass Semisupervised Learning Based Upon Kernel Spectral Clustering
Mehrkanoon, S., Alzate, C., Mall, R., Langone, R., & Suykens, J. A. K. (2015) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2014.2322377 · Cited by 50 · cited 2× · see citing works
“[72] proposed a multiclass learning algorithm based on Kernel Spectral Clustering (KSC) using both labeled and unlabeled data.”
BackgroundMethodImproving Genetic Algorithm with Fine-Tuned Crossover and Scaled Architecture
Shrestha, A., & Mahmood, A. (2016) · Journal of Mathematics · doi:10.1155/2016/4015845 · Cited by 38 · cited 10× · see citing works
“[89] A. Shrestha and A.Mahmood, ‘‘Enhancing siamese networks training with importance sampling,’’ in Proc.”
BackgroundMethodInfluentialSelf-citationKernel Spectral Clustering and Applications
Langone, R., Mall, R., Alzate, C., & Suykens, J. A. K. (2016) · Springer eBooks · doi:10.1007/978-3-319-24211-8_6 · Cited by 30 · see citing works
ML for Flood Forecasting at Scale
Nevo, S., Anisimov, V., Elidan, G., El-Yaniv, R., Giencke, P., Gigi, Y., Hassidim, A., Moshe, Z., Schlesinger, M., Shalev, G., et al. (2019) · arXiv · doi:10.48550/arxiv.1901.09583 · Cited by 23 · cited 1× · see citing works
“Deep learning has a potential to improve human lives with more accurate diagnosis of diseases like cancer [10], discovery of new drugs, prediction of natural disasters [11].”
BackgroundDeep and Modular Neural Networks
Chen, K. (2015) · Springer eBooks · doi:10.1007/978-3-662-43505-2_28 · Cited by 22 · cited 1× · see citing works
“The smaller networks perform specific task which are later combined as part of a single output of the entire network [19].”
BackgroundMultimodal integration learning of object manipulation behaviors using deep neural networks
Noda, K., Arie, H., Suga, Y., & Ogata, T. (2013) · IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros.2013.6696582 · Cited by 20 · cited 1× · see citing works
“It uses greedy layer by layer unsupervised pretraining and fin-tuning with backpropagation [35].”
MethodEnhancing Siamese Networks Training with Importance Sampling
Shrestha, A., & Mahmood, A. (2019) · International Conference on Agents and Artificial Intelligence (ICAART) · doi:10.5220/0007371706100615 · Cited by 6 · cited 1× · see citing works
“This issue doesn’t apply to siamese network as the number of outputs of the softmax in the twin networks doesn’t have the requirement to match the number of classes [89].”
BackgroundSelf-citationParallel neural network training with OpenCL
Krpan, N., & Jakobović, D. (2012) · International Convention on Information and Communication Technology, Electronics and Microelectronics · Cited by 3 · see citing works
Flagged first, then by citations
3 of the 88 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- M. Minsky and S. A. Papert, Perceptrons: An Introduction to Computa- tional Geometry, Expanded Edition. Cambridge, MA, USA: MIT Press, 1969, p. 258.
- P. J. Werbos, '''Beyond Regression:' New tools for prediction and analysis in the behavioral sciences,'' Ph.D. dissertation, Harvard Univ., Cambridge, MA, USA, 1975.
- A. J. R. Simpson, ''Uniform learning in a deep neural network via 'oddball' stochastic gradient descent,'' CoRR, Oct. 2015.
- V. Vapnik and R. Izmailov, ''Learning using privileged information: Simi- larity control and knowledge transfer,'' J. Mach. Learn. Res., vol. 16, no. 1, pp. 2023-2049, Jan. 2015.
- Y. Wu, ''Google's neural machine translation system: Bridging the gap between human and machine translation,'' CoRR, Sep. 2016.
Cited by
Sorted by citations
Review of deep learning: concepts, CNN architectures, challenges, applications, future directions
Alzubaidi, L., Zhang, J., Humaidi, A. J., Al-Dujaili, A., Duan, Y., Al-Shamma, O., Santamaría, J., Fadhel, M. A., Al-Amidie, M., & Farhan, L. (2021) · Journal of Big Data · doi:10.1186/s40537-021-00444-8 · Cited by 7,906 · see citing works
Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next
Cuomo, S., Cola, V. S. D., Giampaolo, F., Rozza, G., Raissi, M., & Piccialli, F. (2022) · Journal of Scientific Computing · doi:10.1007/s10915-022-01939-z · Cited by 2,882 · see citing works
“The main principle of reinforcement learning is to have an agent learn from its surroundings through exploration and by defining a reward (Shrestha and Mahmood, 2019).”
BackgroundArtificial intelligence for sustainability: Challenges, opportunities, and a research agenda
Nishant, R., Kennedy, M., & Corbett, J. (2020) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2020.102104 · Cited by 1,203 · see citing works
Taye, M. M. (2023) · Computers · doi:10.3390/computers12050091 · Cited by 1,073 · see citing works
“…rules to function; rather, it leverages a massive amount of data to map the provided input to particular labels. is created utilising multiple layers of algorithms (arti fi cial neural networks, or ANNs), each of which delivers a unique interpretation of the data provided to it [25,26].”
BackgroundDeep learning modelling techniques: Current progress, applications, advantages, and challenges
Ahmed, S. F., Alam, M. S. B., Hassan, M., Rozbu, M. R., Ishtiak, T., Rafa, N., Mofijur, M., Shawkat Ali, A. B. M., & Gandomi, A. H. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10466-8 · Cited by 1,079 · see citing works
Alzubaidi, L., Bai, J., Al-Sabaawi, A., Santamaría, J., Albahri, A. S., Al-dabbagh, B. S. N., Fadhel, M. A., Manoufali, M., Zhang, J., Al-Timemy, A. H., et al. (2023) · Journal Of Big Data · doi:10.1186/s40537-023-00727-2 · Cited by 992 · see citing works
AI revolutionizing industries worldwide: A comprehensive overview of its diverse applications
Rashid, A. B., & Kausik, M. A. K. (2024) · Hybrid Advances · doi:10.1016/j.hybadv.2024.100277 · Cited by 764 · see citing works
AI applications to medical images: From machine learning to deep learning
Castiglioni, I., Rundo, L., Codari, M., Leo, G. D., Salvatore, C., Interlenghi, M., Gallivanone, F., Cozzi, A., D’Amico, N. C., & Sardanelli, F. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.02.006 · Cited by 737 · see citing works
Taye, M. M. (2023) · Computation · doi:10.3390/computation11030052 · Cited by 732 · see citing works
Machine Learning for Electrocatalyst and Photocatalyst Design and Discovery
Mai, H., Le, T. C., Chen, D., Winkler, D. A., & Caruso, R. A. (2022) · Chemical Reviews · doi:10.1021/acs.chemrev.2c00061 · Cited by 574 · see citing works
Application of deep learning algorithms in geotechnical engineering: a short critical review
Zhang, W., Li, H., Li, Y., Liu, H., Chen, Y., & Ding, X. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-09967-1 · Cited by 545 · see citing works
Loey, M., Smarandache, F., & M. Khalifa, N. E. (2020) · Symmetry · doi:10.3390/sym12040651 · Cited by 538 · see citing works
“Table 2 shows a series of major contributions in the field of the neural network to deep learning [21].”
BackgroundVidal, C., Malysz, P., Kollmeyer, P., & Emadi, A. (2020) · IEEE Access · doi:10.1109/access.2020.2980961 · Cited by 512 · see citing works
“For a more detailed explanation about the methods and its basic math, interested readers are referred to [9], [10].”
BackgroundMethodVali, A., Comai, S., & Matteucci, M. (2020) · Remote Sensing · doi:10.3390/rs12152495 · Cited by 479 · see citing works
“The authors of [136] provide a review discussing the optimisation methods vs.”
BackgroundDeep Learning for Change Detection in Remote Sensing Images: Comprehensive Review and Meta-Analysis
Khelifi, L., & Mignotte, M. (2020) · IEEE Access · doi:10.1109/access.2020.3008036 · Cited by 459 · see citing works
“Hence, this function D(s, a) is represented as the summation of current reward R and future reward α · max[D(s, a)] in the following way [120]:”
BackgroundA review of machine learning in processing remote sensing data for mineral exploration
Shirmard, H., Farahbakhsh, E., Müller, R. D., & Chandra, R. (2021) · Remote Sensing of Environment · doi:10.1016/j.rse.2021.112750 · Cited by 434 · see citing works
“Deep learning methods have the ability to model complex and large datasets and are loosely grouped into feedforward and recurrent neural network architectures (Shrestha and Mahmood, 2019).”
BackgroundHoeser, T., & Kuenzer, C. (2020) · Remote Sensing · doi:10.3390/rs12101667 · Cited by 375 · see citing works
“Such a network then transports values from the input layer via the weights over all hidden layers to the output layer and can be seen as a non-linear function of higher order, mapping input values to outputs [5,25].”
BackgroundApplication of machine learning in groundwater quality modeling - A comprehensive review
Haggerty, R., Sun, J., Yu, H., & Li, Y. (2023) · Water Research · doi:10.1016/j.watres.2023.119745 · Cited by 344 · see citing works
“…Sze et al., 2017; Zhao et al., 2017) optimized for machine learning algorithms (MLAs), especially for deep learning (Emmert-Streib et al., 2020; LeCun et al., 2015; Shrestha and Mahmood, 2019), data-centric ML methods started to emerge in recent years with a remarkable leap forward in performance.”
BackgroundEbayyeh, A. A. R. M. A., & Mousavi, A. (2020) · IEEE Access · doi:10.1109/access.2020.3029127 · Cited by 314 · see citing works
“A special case of DNNs is when the matrix multiplications include convolutional filter operations, which is common in DNNs that are designed for image and video analysis.”
BackgroundInfluentialBibri, S. E., Alexandre, A., Sharifi, A., & Krogstie, J. (2023) · Energy Informatics · doi:10.1186/s42162-023-00259-2 · Cited by 299 · see citing works
“…Neural Network (DNN), SVM, GA, LR, and DT are among the prominent AI techniques that gained strong traction in recent years (Şerban and Lytras 2020; Shrestha and Mahmood 2019), so are ML, DL, and cognitive AI models in relation to the planning and development of smart cities (Kamrowska-Załuska…”
BackgroundNovel Transfer Learning Approach for Medical Imaging with Limited Labeled Data
Alzubaidi, L., Al‐Amidie, M., Al‐Asadi, A., Humaidi, A. J., Al-Shamma, O., Fadhel, M. A., Zhang, J., Santamaría, J., & Duan, Y. (2021) · Cancers · doi:10.3390/cancers13071590 · Cited by 301 · see citing works
“Furthermore, these layers efficiently decrease the network sensitivity to image distortions, scales, and shifts along with shared weights and local receptive fields (inside the same filter) [38].”
BackgroundMethodManoharan, A., Begam, M., Aparow, V. R., & Sooriamoorthy, D. (2022) · Journal of Energy Storage · doi:10.1016/j.est.2022.105384 · Cited by 297 · see citing works
Ren, Z. G., & Du, C. (2023) · Energy Reports · doi:10.1016/j.egyr.2023.01.108 · Cited by 292 · see citing works
The Role of Machine Learning Algorithms for Diagnosing Diseases
Ibrahim, I., & Abdulazeez, A. M. (2021) · Journal of Applied Science and Technology Trends · doi:10.38094/jastt20179 · Cited by 281 · see citing works
“The application of these methods is used for a number of medical functions, such as diagnosis of diseases [25,26].”
Blockchain and AI-Based Solutions to Combat Coronavirus (COVID-19)-Like Epidemics: A Survey
Nguyen, D. C., Ding, M., Pathirana, P. N., & Seneviratne, A. (2021) · IEEE Access · doi:10.1109/access.2021.3093633 · Cited by 286 · see citing works
“adjustment of weight values among perceptrons [62].”
Deploying artificial intelligence for climate change adaptation
Leal Filho, W., Wall, T., Rui Mucova, S. A., Nagy, G. J., Balogun, A.-L., Luetz, J. M., Ng, A. W., Kovaleva, M., Safiul Azam, F. M., Alves, F., et al. (2022) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.121662 · Cited by 266 · see citing works
Internet of Things: Evolution, Concerns and Security Challenges
Malhotra, P., Singh, Y., Anand, P., Bangotra, D. K., Singh, P. K., & Hong, W.-C. (2021) · Sensors · doi:10.3390/s21051809 · Cited by 260 · see citing works
“Nowadays, the technique is finding usage in the sector of anomaly detection [93,94], the approach is fused with other methods for anomaly detection, thus providing a profitable proposal in this sector.”
MethodRecent Progresses in Machine Learning Assisted Raman Spectroscopy
Qi, Y., Hu, D., Jiang, Y., Wu, Z., Zheng, M., Chen, E. X., Liang, Y., Sadi, M. A., Zhang, K., & Chen, Y. P. (2023) · Advanced Optical Materials · doi:10.1002/adom.202203104 · Cited by 256 · see citing works
Deep learning in mammography images segmentation and classification: Automated CNN approach
Salama, W. M., & Aly, M. H. (2021) · Alexandria Engineering Journal · doi:10.1016/j.aej.2021.03.048 · Cited by 255 · see citing works
“A. Shrestha et al. developed a new algorithm to describe deep learning [6].”
BackgroundA Deep Learning-Based Phishing Detection System Using CNN, LSTM, and LSTM-CNN
Alshingiti, Z., Alaqel, R., Al-Muhtadi, J. F., Haq, Q. E. U., Saleem, K., & Faheem, M. H. (2023) · Electronics · doi:10.3390/electronics12010232 · Cited by 244 · see citing works
“For training deep learning models, a large amount of labeled data is required, substantial computing power, and neural network architectures that contain numerous layers [15,16].”
BackgroundTransformer Architecture and Attention Mechanisms in Genome Data Analysis: A Comprehensive Review
Choi, S. R., & Lee, M. (2023) · Biology · doi:10.3390/biology12071033 · Cited by 265 · see citing works
“Deep learning, or deep neural networks (DNNs), employs artificial neural networks with multiple layers, a feature that makes it remarkably capable of learning complex patterns from large datasets [23].”
BackgroundMulti-Disease Prediction Based on Deep Learning: A Survey
Xie, S., Yu, Z., & Lv, Z. (2021) · Computer Modeling in Engineering & Sciences · doi:10.32604/cmes.2021.016728 · Cited by 240 · see citing works
Karataş, F., Abedi, F. Y., Ozek Gunyel, F., Karadeniz, D., & Kuzgun, Y. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-12574-6 · Cited by 252 · see citing works
“…architectures and algorithms to efficiently train on large datasets, enabling complex hierarchical representations and transformations that power its capabilities across voice recognition, chatbots, search, and computer vision (Moshayedi et al., 2022; Rizk et al., 2019; Shrestha & Mahmood, 2019).”
A review on social spam detection: Challenges, open issues, and future directions
Rao, S., Verma, A. K., & Bhatia, T. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115742 · Cited by 225 · see citing works
An Efficient Deep Learning-Based Skin Cancer Classifier for an Imbalanced Dataset
Alam, T. M., Shaukat, K., Khan, W. A., Hameed, I. A., Almuqren, L., Raza, M. A., Aslam, M., & Luo, S. (2022) · Diagnostics · doi:10.3390/diagnostics12092115 · Cited by 207 · see citing works
Jahangir, H., Tayarani, H., Gougheri, S. S., Golkar, M. A., Ahmadian, A., & Elkamel, A. (2020) · IEEE Transactions on Industrial Electronics · doi:10.1109/tie.2020.3009604 · Cited by 203 · see citing works
Deep learning ensemble 2D CNN approach towards the detection of lung cancer
Shah, A. A., Malik, H. A. M., Muhammad, A., Alourani, A., & Butt, Z. A. (2023) · Scientific Reports · doi:10.1038/s41598-023-29656-z · Cited by 211 · see citing works
“A final solution Deep Ensemble 2D CNN is developed with the help of the Deep Learning Algorithm 22 to detect Lung Nodules from CT Scan images.”
MethodKumar, V., Azamathulla, H. M., Sharma, K. V., Mehta, D., & Tota‐Maharaj, K. (2023) · Sustainability · doi:10.3390/su151310543 · Cited by 195 · see citing works
“The parameters of the model are changed during training to reduce the discrepancy between the model’s predictions and the actual labels in the training data [34].”
BackgroundEmotion recognition from unimodal to multimodal analysis: A review
Ezzameli, K., & Mahersia, H. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101847 · Cited by 195 · see citing works
Pathways and challenges of the application of artificial intelligence to geohazards modelling
Dikshit, A., Pradhan, B., & Alamri, A. M. (2020) · Gondwana Research · doi:10.1016/j.gr.2020.08.007 · Cited by 191 · see citing works
Deep learning in analytical chemistry
Debus, B., Parastar, H., Harrington, P. D. B., & Kirsanov, D. O. (2021) · TrAC Trends in Analytical Chemistry · doi:10.1016/j.trac.2021.116459 · Cited by 191 · see citing works
Deep Neuro-Fuzzy System application trends, challenges, and future perspectives: a systematic survey
Talpur, N., Abdulkadir, S. J., Alhussian, H., Hasan, M. H., Aziz, N., & Bamhdi, A. M. (2022) · Artificial Intelligence Review · doi:10.1007/s10462-022-10188-3 · Cited by 188 · see citing works
“The advancement from shallow to deep learning models has increased the possibility of dealing with complex and nonlinear functions (Shrestha and Mahmood 2019).”
BackgroundA Survey on Kolmogorov-Arnold Network
Somvanshi, S., Javed, S. A., Islam, M. M., Pandit, D., & Das, S. (2025) · ACM Computing Surveys · doi:10.1145/3743128 · Cited by 214 · see citing works
“This structural shift differentiates KANs from Multi-Layer Perceptrons (MLPs), which use fixed activation functions at the nodes, offering KANs the advantage of adaptability and greater alignment with the decomposition of multivariate functions [3].”
Taking the leap between analytical chemistry and artificial intelligence: A tutorial review
Ayres, L. B., Gómez, F. J. V., Linton, J. R., Silva, M. F., & García, C. D. (2021) · Analytica Chimica Acta · doi:10.1016/j.aca.2021.338403 · Cited by 178 · see citing works
An analysis of process fault diagnosis methods from safety perspectives
Arunthavanathan, R., Khan, F. I., Ahmed, S., & Imtiaz, S. A. (2020) · Computers & Chemical Engineering · doi:10.1016/j.compchemeng.2020.107197 · Cited by 173 · see citing works
Molecular design in drug discovery: a comprehensive review of deep generative models
Cheng, Y., Gong, Y., Liu, Y., Song, B., & Zou, Q. (2021) · Briefings in Bioinformatics · doi:10.1093/bib/bbab344 · Cited by 169 · see citing works
A comprehensive survey on optimizing deep learning models by metaheuristics
Akay, B., Karaboğa, D., & Akay, R. (2021) · Artificial Intelligence Review · doi:10.1007/s10462-021-09992-0 · Cited by 177 · see citing works
Machine Learning for Perovskite Solar Cells and Component Materials: Key Technologies and Prospects
Liu, Y., Tan, X., Liang, J., Han, H., Xiang, P., & Yan, W. (2023) · Advanced Functional Materials · doi:10.1002/adfm.202214271 · Cited by 174 · see citing works
AI-Enabled Secure Microservices in Edge Computing: Opportunities and Challenges
Al-Doghman, F., Moustafa, N., Khalil, I., Sohrabi, N., Tari, Z., & Zomaya, A. Y. (2022) · IEEE Transactions on Services Computing · doi:10.1109/tsc.2022.3155447 · Cited by 179 · see citing works
“The usefulness of this operation is that the features do not require to be defined at the beginning and the model would gradually learn to point out the best results by itself [64].”
BackgroundElallid, B. B., Benamar, N., Hafid, A. S., Rachidi, T., & Mrani, N. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2022.03.013 · Cited by 179 · see citing works
“(Shrestha and Mahmood, 2019).”
Progress and Challenges for Memtransistors in Neuromorphic Circuits and Systems
Yan, X., Qian, J. H., Sangwan, V. K., & Hersam, M. C. (2021) · Advanced Materials · doi:10.1002/adma.202108025 · Cited by 164 · see citing works
Khallaf, R., & Khallaf, M. (2021) · Automation in Construction · doi:10.1016/j.autcon.2021.103760 · Cited by 170 · see citing works
A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU
Shiri, F. M., Perumal, T., Mustapha, N., & Mohamed, R. (2023) · Journal of Artificial Intelligence · doi:10.32604/jai.2024.054314 · Cited by 537 · see citing works
“The units sum the inputs after multiplying them by their corresponding weights [4].”
BackgroundEnabling AI in Future Wireless Networks: A Data Life Cycle Perspective
Nguyen, D. C., Cheng, P., Ding, M., Lopez-Perez, D., Pathirana, P. N., Li, J., Seneviratne, A., Li, Y., & Poor, H. V. (2021) · IEEE Communications Surveys & Tutorials · doi:10.1109/comst.2020.3024783 · Cited by 160 · see citing works
“1) Convolutional Neural Networks: CNN is a type of neural network used mainly for image processing and recognition with large pixel datasets [53].”
BackgroundConstruction of deep learning-based disease detection model in plants
Jung, M., Song, J. S., Shin, A.-Y., Choi, B., Go, S., Kwon, S., Park, J., Park, S. G., & Kim, Y. (2023) · Scientific Reports · doi:10.1038/s41598-023-34549-2 · Cited by 157 · see citing works
Hybrid CNN-LSTM approaches for identification of type and locations of transmission line faults
Moradzadeh, A., Teimourzadeh, H., Mohammadi‐Ivatloo, B., & Pourhossein, K. (2021) · International Journal of Electrical Power & Energy Systems · doi:10.1016/j.ijepes.2021.107563 · Cited by 155 · see citing works
Deep Transfer Learning Models for Medical Diabetic Retinopathy Detection
Khalifa, N. E. M., Loey, M., Taha, M. H. N., & Mohamed, H. (2019) · Acta Informatica Medica · doi:10.5455/aim.2019.27.327-332 · Cited by 148 · see citing works
“Table 1 shows a series of major contributions in the field of DL (12).”
BackgroundXi, Z., Hopkinson, C., Rood, S. B., & Peddle, D. R. (2020) · ISPRS Journal of Photogrammetry and Remote Sensing · doi:10.1016/j.isprsjprs.2020.08.001 · Cited by 146 · see citing works
Arwa, E. O., & Folly, K. A. (2020) · IEEE Access · doi:10.1109/access.2020.3038735 · Cited by 147 · see citing works
“Many training algorithms have been developed to get the neural network to arrive at the optimal θ , namely, momentum, back propagation, Levenberg–Marquardt algorithm, etc., [75].”
BackgroundApplication of Internet of Things and artificial intelligence for smart fitness: A survey
Farrokhi, A., Farahbakhsh, R., Rezazadeh, J., & Minerva, R. (2021) · Computer Networks · doi:10.1016/j.comnet.2021.107859 · Cited by 153 · see citing works
Yaghoubi, E., Yaghoubi, E., Khamees, A., & Vakili, A. (2024) · Neural computing & applications (Print) · doi:10.1007/s00521-024-09893-7 · Cited by 139 · see citing works
“They can automatically acquire hierarchical features [369, 370].”
Artificial Intelligence Revolutionises Weather Forecast, Climate Monitoring and Decadal Prediction
Dewitte, S., Cornelis, J. P., Müller, R. W., & Munteanu, A. (2021) · Remote Sensing · doi:10.3390/rs13163209 · Cited by 142 · see citing works
“backpropagation algorithm has been the most popular one for many years [25].”
BackgroundMathematical Analysis and Performance Evaluation of the GELU Activation Function in Deep Learning
Lee, M. (2023) · Journal of Mathematics · doi:10.1155/2023/4229924 · Cited by 142 · see citing works
“To minimize the loss function, we utilize optimization algorithms [24] that update the weights and biases of the network iteratively.”
MethodZhu, H., Lin, C., Liu, G., Wang, D., Qin, S., Li, A., Xu, J., & He, Y. (2024) · Frontiers in Plant Science · doi:10.3389/fpls.2024.1435016 · Cited by 143 · see citing works
Deep learning-based approaches for robust classification of cervical cancer
Paçal, İ., & Kılıçarslan, S. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08757-w · Cited by 146 · see citing works
“…decision tree, Bayesian and logistic regression, which are classical machine methods, offer a good solution to problems that are simple and require little data, but as the problem complexity or data increases, these structures experience a bottleneck and lack the ability to generalize [10, 46, 47].”
BackgroundJahangir, R., Teh, Y. W., Nweke, H. F., Mujtaba, G., Al-Garadi, M. A., & Ali, I. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.114591 · Cited by 146 · see citing works
Artificial intelligence in cardiology: Hope for the future and power for the present
Karatzia, L., Aung, N., & Aksentijević, D. (2022) · Frontiers in Cardiovascular Medicine · doi:10.3389/fcvm.2022.945726 · Cited by 143 · see citing works
“Abbreviations: EHRs, electronic health records; CMR, cardiac magnetic resonance; CT, computed tomography; IoT, internet of things; SPECT, single photon-emission computerised tomography. recognition, natural language processing, computer vision, and predictive analysis (13).”
A bird's-eye view of deep learning in bioimage analysis
Meijering, E. (2020) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2020.08.003 · Cited by 126 · see citing works
“Notwithstanding the great arsenal of software toolkits for deep learning available today [54–57], there is still much room for the development of higher-level, user-friendly platforms that make it easier also for non-experts to adopt and use existing DNNs or to design and deploy their own solutions.”
BackgroundThe challenges of modern computing and new opportunities for optics
Li, C., Zhang, X., Li, J., Fang, T., & Dong, X. (2021) · PhotoniX · doi:10.1186/s43074-021-00042-0 · Cited by 136 · see citing works
“Recently, Sorger et al. [52] demonstrated that the optical-electrical hybrid CNN still works well if the phase information in the Fourier filter plane is abandoned.”
MethodInfluentialTrends and gaps in photovoltaic power forecasting with machine learning
Alcañiz, A., Grzebyk, D., Ziar, H., & Isabella, O. (2022) · Energy Reports · doi:10.1016/j.egyr.2022.11.208 · Cited by 134 · see citing works
Sajitha, P., Andrushia, A. D., Anand, N., & Naser, M. Z. (2024) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2024.100572 · Cited by 134 · see citing works
Deep learning technologies for shield tunneling: Challenges and opportunities
Zhou, C., Gao, Y., Chen, E. J., Ding, L., & Qin, W. (2023) · Automation in Construction · doi:10.1016/j.autcon.2023.104982 · Cited by 132 · see citing works
Empowering vertical farming through IoT and AI-Driven technologies: A comprehensive review
Rathor, A. S., Choudhury, S., Sharma, A., Nautiyal, P., & Shah, G. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e34998 · Cited by 130 · see citing works
“Overcoming these challenges will unlock break-throughs across all facets of ML and AI in the field of VF [145].”
BackgroundMachine learning and optimization for production rescheduling in Industry 4.0
Li, Y., Carabelli, S., Fadda, E., Manerba, D., Tadei, R., & Terzo, O. (2020) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-020-05850-5 · Cited by 127 · see citing works
Survey of Deep Learning Paradigms for Speech Processing
Bhangale, K. B., & Kothandaraman, M. (2022) · Wireless Personal Communications · doi:10.1007/s11277-022-09640-y · Cited by 131 · see citing works
“The learning algorithms have several drawbacks: vanishing gradient problem, local minima, larger training time, over-fitting, etc. [32].”
BackgroundDiabetes detection based on machine learning and deep learning approaches
Wee, B. F., Sivakumar, S., Lim, K. H., Wong, W. K., & Juwono, F. H. (2023) · Multimedia Tools and Applications · doi:10.1007/s11042-023-16407-5 · Cited by 127 · see citing works
“One of the benefits is that deep learning algorithms possess various built in functions such as feature selection and feature extraction [54].”
Sharma, S., Singh, G., & Sharma, M. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104450 · Cited by 131 · see citing works
Android Mobile Malware Detection Using Machine Learning: A Systematic Review
Senanayake, J., Kalutarage, H., & Al-Kadri, M. O. (2021) · Electronics · doi:10.3390/electronics10131606 · Cited by 126 · see citing works
“It works with models such as artificial neural networks (ANN) and consists of a higher or deeper number of processing layers [31].”
A review on Deep Learning approaches for low-dose Computed Tomography restoration
Kulathilake, K. A. S. H., Abdullah, N. A., Sabri, A. Q. M., & Lai, K. W. (2021) · Complex & Intelligent Systems · doi:10.1007/s40747-021-00405-x · Cited by 124 · see citing works
“The network models based on the discriminative approach represent bottom–up execution to separate learned data based on a decision boundary [61].”
MethodNutakki, M., & Mandava, S. (2023) · Engineering applications of artificial intelligence · doi:10.1016/j.engappai.2022.105721 · Cited by 104 · see citing works
Machine learning for condensed matter physics
Bedolla-Montiel, E. A., Padierna, L. C., & Castañeda-Priego, R. (2020) · Journal of Physics Condensed Matter · doi:10.1088/1361-648x/abb895 · Cited by 123 · see citing works
Deep learning for accelerated and robust MRI reconstruction
Heckel, R., Jacob, M., Chaudhari, A. S., Perlman, O., & Shimron, E. (2024) · Magnetic Resonance Materials in Physics Biology and Medicine · doi:10.1007/s10334-024-01173-8 · Cited by 144 · see citing works
Applications of machine learning in routine laboratory medicine: Current state and future directions
Rabbani, N., Kim, G. Y. E., Suarez, C. J., & Chen, J. H. (2022) · Clinical Biochemistry · doi:10.1016/j.clinbiochem.2022.02.011 · Cited by 118 · see citing works
Yang, X., Guan, J., Ding, L., You, Z., Lee, V. C. S., Hasan, M. R. M., & Cheng, X. (2021) · Journal of Traffic and Transportation Engineering (English Edition) · doi:10.1016/j.jtte.2021.03.005 · Cited by 117 · see citing works
Digital Twin in the Maritime Domain: A Review and Emerging Trends
Madusanka, N. S., Fan, Y., Yang, S., & Xiang, X. (2023) · Journal of Marine Science and Engineering · doi:10.3390/jmse11051021 · Cited by 125 · see citing works
“Diagnostics and predictive forecasting can be incorporated into the DT with its neural network with a deep network from the acquisition of data to the training of the final algorithm [115].”
BackgroundHuang, C., He, R., Ai, B., Molisch, A. F., Lau, B. K., Haneda, K., Liu, B., Wang, C., Yang, M., Oestges, C., et al. (2022) · IEEE Transactions on Antennas and Propagation · doi:10.1109/tap.2022.3149663 · Cited by 115 · see citing works
“Widely used deep neural network (DNN) structures include Convolution Neural Network (CNN), Restricted Boltzmann Machine, Long Short-Term Memory (LSTM), etc [22].”
MethodLifelo, Z., Ding, J., Ning, H., Ain, Q. U., & Dhelim, S. (2024) · Electronics · doi:10.3390/electronics13244874 · Cited by 125 · see citing works
Er, M. J., Chen, J., Zhang, Y., & Gao, W. (2023) · Sensors · doi:10.3390/s23041990 · Cited by 119 · see citing works
Improving the efficiency of RMSProp optimizer by utilizing Nestrove in deep learning
Elshamy, R., Abu-Elnasr, O., Elhoseny, M., & Elmougy, S. (2023) · Scientific Reports · doi:10.1038/s41598-023-35663-x · Cited by 112 · see citing works
“RMSProp can be thought of as an extension of AdaGrad which uses a moving (1) θ = θ − η · ∇F (θ)”
State-of-the-Art of Stress Prediction from Heart Rate Variability Using Artificial Intelligence
Haque, Y., Zawad, R. S., Rony, C. S. A., Banna, H. A., Ghosh, T., Kaiser, M., & Mahmud, M. (2023) · Cognitive Computation · doi:10.1007/s12559-023-10200-0 · Cited by 105 · see citing works
“T h e a r c h it ec t u r e o f C NN i s b a s e d on t h e v i s u a l c o r t e x o f hu m a n b e i ng s [ 82 ]”
Naghib, A., Gharehchopogh, F. S., & Zamanifar, A. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-024-11101-w · Cited by 112 · see citing works
Deep Learning for Biospectroscopy and Biospectral Imaging: State-of-the-Art and Perspectives
He, H., Yan, S., Lyu, D., Xu, M., Ye, R., Zheng, P., Lu, X., Wang, L., & Ren, B. W. (2021) · Analytical Chemistry · doi:10.1021/acs.analchem.0c04671 · Cited by 111 · see citing works
A Review on the emerging technology of TinyML
Tsoukas, V., Gkogkidis, A., Boumpa, E., & Kakarountas, A. P. (2024) · ACM Computing Surveys · doi:10.1145/3661820 · Cited by 122 · see citing works
“The most widely used DL network types are Recurrent Neural Networks (RNNs) , Convolutional Neural Networks (CNNs) , and Long Short-Term Memory Networks (LSTMs) [190].”
MethodNair, A. K., Sahoo, J., & Raj, E. D. (2023) · Computer Standards & Interfaces · doi:10.1016/j.csi.2023.103720 · Cited by 116 · see citing works
Improved Automatic Summarization of Subroutines via Attention to File Context
Haque, S., LeClair, A., Wu, L., & McMillan, C. (2020) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3379597.3387449 · Cited by 108 · see citing works
“ds what can be listed in one paper, with thousands of papers discussing overall improvementsaswellasadjustmentsforspecificdomains[10].Nevertheless, this history has been chronicled in several surveys [36, 37, 45]. We include enough details in Section 4 to understand the basic encoder-decoderarchitectureandhowourapproa…”
BackgroundDigital Twin-Driven Remaining Useful Life Prediction for Gear Performance Degradation: A Review
He, B., Liu, L., & Zhang, D. (2021) · Journal of Computing and Information Science in Engineering · doi:10.1115/1.4049537 · Cited by 108 · see citing works
Digital hair removal by deep learning for skin lesion segmentation
Li, W., Raj, A. N. J., Tjahjadi, T., & Zhuang, Z. (2021) · Pattern Recognition · doi:10.1016/j.patcog.2021.107994 · Cited by 108 · see citing works
Hyperspectral Band Selection Using Attention-Based Convolutional Neural Networks
Lorenzo, P. R., Tulczyjew, Ł., Marcinkiewicz, M., & Nalepa, J. (2020) · IEEE Access · doi:10.1109/access.2020.2977454 · Cited by 104 · see citing works
Anomaly detection based on Artificial Intelligence of Things: A Systematic Literature Mapping
Trilles, S., Hammad, S. S., & Iskandaryan, D. (2024) · Internet of Things · doi:10.1016/j.iot.2024.101063 · Cited by 118 · see citing works
Abnormal Driving Detection With Normalized Driving Behavior Data: A Deep Learning Approach
Hu, J., Zhang, X., & Maybank, S. J. (2020) · IEEE Transactions on Vehicular Technology · doi:10.1109/tvt.2020.2993247 · Cited by 104 · see citing works
“The literature shows that deep-layered neural networks can learn more powerful models than shallow ones with a large amount of labeled data [19].”
Background
- Authors
- Shrestha, A., & Mahmood, A.
- Affiliations
- University of Bridgeport (US)
- Venue
- IEEE Access
- Published
- 2019
- Volume
- 7
- Pages
- 53040-53065
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W2942231644
- Cited by
- 1,883
- References
- 88
- Citations by year
- 2026: 116 · 2025: 317 · 2024: 360 · 2023: 375 · 2022: 297 · 2021: 259 · 2020: 139 · 2019: 20
- FWCI
- 81.55
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion)
- University of Bridgeport ScholarWorks (University of Bridgeport) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Stochastic Gradient Optimization Techniques
- Keywords
- deep learning, deep learning architectures, deep convolutional neural networks, ResNet, recurrent neural networks, reinforcement learning, variational autoencoder, feature extraction, hierarchical representation, training time reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
