Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 253 of the 253 references it cites were found in a registry or an index.

Deep learning modelling techniques: current progress, applications, advantages, and challenges

Open access

Abstract

Deep learning (DL) is revolutionizing evidence-based decision-making techniques that can be applied across various sectors. Specifically, it possesses the ability to utilize two or more levels of non-linear feature transformation of the given data via representation learning in order to overcome limitations posed by large datasets. As a multidisciplinary field that is still in its nascent phase, articles that survey DL architectures encompassing the full scope of the field are rather limited. Thus, this paper comprehensively reviews the state-of-art DL modelling techniques and provides insights into their advantages and challenges. It was found that many of the models exhibit a highly domain-specific efficiency and could be trained by two or more methods. However, training DL models can be very time-consuming, expensive, and requires huge samples for better accuracy. Since DL is also susceptible to deception and misclassification and tends to get stuck on local minima, improved optimization of parameters is required to create more robust models. Regardless, DL has already been leading to groundbreaking results in the healthcare, education, security, commercial, industrial, as well as government sectors. Some models, like the convolutional neural network (CNN), generative adversarial networks (GAN), recurrent neural network (RNN), recursive neural networks, and autoencoders, are frequently used, while the potential of other models remains widely unexplored. Pertinently, hybrid conventional DL architectures have the capacity to overcome the challenges experienced by conventional models. Considering that capsule architectures may dominate future DL models, this work aimed to compile information for stakeholders involved in the development and use of DL models in the contemporary world.

Overview

Integrity signals

Registry

Crossref · 10.1007/s10462-023-10466-8

Cites

253 of 253 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 252 references

Research works printed but not linked: 54

  1. Adhikari A, Ram A, Tang R, Lin J (2019) DocBERT: BERT for document classification. arXiv:1904.08398 · arXiv
  2. Alain G, Bengio Y, Courville A, Fergus R, Manning C (2014) What regularized auto-encoders learn from the data-generating distribution. J Mach Learn Res 15(1):3563–3593
  3. Alpaydin E (2020) Introduction to machine learning. MIT Press, Cambridge
  4. Arora S, Ma T, Moitra A (2015) Simple, efficient, and neural algorithms for sparse coding. PMLR, pp 113–149
  5. Ba J, Hinton G, Mnih V, Leibo JZ, Ionescu C (2016) Using fast weights to attend to the recent past. Adv Neural Inf Process Syst 29:4338–4346
  6. Bartunov S, Rae JW, Osindero S, Lillicrap TP (2019) Meta-learning deep energy-based memory models. https://arXiv.org/1910.02720
  7. Bau D, Zhu JY, Strobelt H, Zhou B, Tenenbaum JB, Freeman WT, Torralba A (2019) GaN dissection: visualizing and understanding generative adversarial networks. In: 7th international conference on learning representations, ICLR 2019
  8. Bousmalis K, Trigeorgis G, Silberman N, Krishnan D, Erhan D (2016) Domain separation networks. In: Advances in neural information processing systems
  9. Brahma S (2018) Improved sentence modeling using suffix bidirectional LSTM. https://arXiv.org/1805.07340
  10. Brock A, Donahue J, Simonyan K (2019) Large scale GaN training for high fidelity natural image synthesis. In: 7th international conference on learning representations, ICLR 2019
  11. Case C, Casper J, Catanzaro B, Diamos G, Elsen E (2014) Deep speech: scaling up end-to-end speech recognition. https://arXiv.org/1412.5567
  12. Collobert R, Weston J, Bottou L, Karlen M, Kavukcuoglu K, Kuksa P (2011) Natural language processing (Almost) from scratch. J Mach Learn Res 12:2493–2537
  13. Dahl GE, Ranzato M, Mohamed AR, Hinton G (2010) Phone recognition with the mean-covariance restricted Boltzmann machine. Adv Neural Inf Process Syst 23:469–477
  14. Demeester T, Sutskever I, Chen K, Dean J, Corado G (2016) Distributed representations of words and phrases and their compositionality. EMNLP 2016 - Conference empirical methods natural language process processing
  15. Deoras A, Povey D, Mikolov T, Burget L, Černocký J (2011) Strategies for training large scale neural network language models. In: IEEE workshop on automatic speech recognition and understanding pp 196–201
  16. Gallicchio C Micheli A Pedrelli L (2018a) Deep echo state networks for diagnosis of Parkinson’s disease. In: ESANN 2018a - Proceedings, European symposium on artificial neural networks, computational intelligence and machine learning
  17. Gehring J, Auli M, Grangier D, Yarats D, Dauphin YN (2017) Convolutional sequence to sequence learning. In: 34th International conference on machine learning, ICML 2017
  18. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, Cambridge
  19. Goodfellow IJ, Warde-Farley D, Mirza M, Courville A, Bengio Y (2013) Maxout networks. In: 30th International conference on machine learning, ICML 2013.
  20. Hamilton WL, Ying R, Leskovec J (2017) Representation learning on graphs: methods and applications. https://arXiv.org/1709.05584
  21. Irsoy O, Cardie C (2014) Deep recursive neural networks for compositionality in language. In: Advances in neural information processing systems
  22. Kaiser Ł, Sutskever I (2016) Neural GPUs learn algorithms. In: 4th International conference on learning representations, ICLR 2016 - conference track proceedings
  23. Karras T, Aila T, Laine S, Lehtinen J (2018) Progressive growing of GANs for improved quality, stability, and variation. In: 6th international conference on learning representations, ICLR 2018 - conference track proceedings
  24. Kim JH, On KW, Lim W, Kim J, Ha JW, Zhang BT (2017) Hadamard product for low-rank bilinear pooling. In: 5th international conference on learning representations, ICLR 2017 - conference track proceedings
  25. Kiros R, Zhu Y, Salakhutdinov R, Zemel RS, Torralba A, Urtasun R, Fidler S (2015) Skip-thought vectors. Advances in neural information processing systems
  26. Li JB, Schmidt FR, Kolter JZ (2019a) Adversarial camera stickers: a physical camera-based attack on deep learning systems. In: International conference on machine learning. pp 3896–3904
  27. Lin Z, Feng M, Dos Santos CN, Yu M, Xiang B, Zhou B, Bengio Y (2017) A structured self-attentive sentence embedding. In: 5th international conference on learning representations, ICLR 2017 - conference track proceedings
  28. Liu P, Qiu X, Xuanjing H (2016) Recurrent neural network for text classification with multi-task learning. In: IJCAI international joint conference on artificial intelligence
  29. Mesnil G, Dauphin Y, Yao K, Bengio Y, Deng L, Hakkani-tur D, He X (2015) Using recurrent neural networks for slot filling in spoken language understanding. IEEE/ACM trans audio, speech Lang Process 23:530–539
  30. Mikolov T, Chen K, Corrado G Dean J (2013) Efficient estimation of word representations in vector space. In: 1st international conference on learning representations, ICLR 2013 - workshop track proceedings
  31. Misra D (2019) Mish: a self regularized non-monotonic neural activation function. https://arXiv.org/1908.08681
  32. Miyato T, Kataoka T, Koyama M, Yoshida Y (2018) Spectral normalization for generative adversarial networks. In: 6th international conference on learning representations, ICLR 2018 - conference track proceedings
  33. Ngiam J, Chen Z, Wei Koh P, Ng AY (2011) Learning deep energy models. In: Proceedings of the 28th international conference on machine learning (ICML-11) pp 1105–1112
  34. Nwankpa CE, Ijomah W, Gachagan A, Marshall S (2018) Activation functions: comparison of trends in practice and research for deep learning. https://arXiv.org/1811.03378
  35. Odena A, Olah C, Shlens J (2017) Conditional image synthesis with auxiliary classifier gans. In: 34th International conference on machine learning, ICML 2017
  36. Oka A, Ishimura N, Ishihara S (2021) A new dawn for the use of artificial intelligence in gastroenterology. Hepatol Pancreatol Diagn 11:1719
  37. Oord VD, Dieleman S, Schrauwen B (2013) Deep content-based music recommendation. Neural Inform Process Syst 26:1–9
  38. Papineni K, Roukos S, Ward T, Zhu W-J (2001) BLEU: a method for automatic evaluation of machine translation. Assoc Comput Linguist. https://doi.org/10.3115/10730831073135 · doi:10.3115/10730831073135
  39. Paula EL, Ladeira M, Carvalho RN, Marzag T (2016) Deep learning anomaly detection as support fraud investigation in Brazilian exports and anti-money laundering. In: 2016 15th IEEE International conference on machine learning and applications (ICMLA) pp 954–960. https://doi.org/10.1109/ICMLA.2016.73 · doi:10.1109/icmla.2016.73
  40. Paulus R, Xiong C, Socher R (2018) A deep reinforced model for abstractive summarization. In: 6th international conference on learning representations, ICLR 2018 - conference track proceedings
  41. Saremi S, Mehrjou A, Schölkopf B, Hyvärinen A (2018) Deep energy estimator networks. https://arXiv.1805.08306
  42. Shoeibi A, Ghassemi N, Khodatars M, Jafari M, Hussain S, Alizadehsani R (2020) Application of deep learning techniques for automated detection of epileptic seizures: a Review. https://arXiv.org/2007.01276
  43. Socher R, Chen D, Manning CD, Ng AY (2013) Reasoning with neural tensor networks for knowledge base completion. Adv Neural Inf Proc Syst 1:e2
  44. Sønderby CK, Caballero J, Theis L, Shi W, Huszár F (2017) Amortised map inference for image super-resolution. In: 5th international conference on learning representations, ICLR 2017 - conference track proceedings
  45. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Advances in neural information processing systems
  46. Sutskever I, Hinton G (2007) Learning multilevel distributed representations for high-dimensional sequences. J Machine Learn Res. 2:548–555
  47. Taigman Y, Polyak A, Wolf L (2017) Unsupervised cross-domain image generation. In: 5th international conference on learning representations, ICLR 2017 - conference track proceedings
  48. Tang Y (2013) Deep learning using linear support vector machines. https://arXiv.org/1306.0239
  49. Tkachenko Y (2015) Autonomous CRM control via CLV approximation with deep reinforcement learning in discrete and continuous action space. arXiv:1504.01840. https://arXiv.org/1504.01840 · arXiv
  50. Tompson J, Jain A, Lecun Y, Bregler C (2014) Joint training of a convolutional network and a graphical model for human pose estimation. 27:1–9 https://arXiv.org/1406.2984
  51. Trabelsi C, Bilaniuk O, Zhang Y, Serdyuk D, Subramanian S, Santos JF, Mehri S, Rostamzadeh N, Bengio, Y, Pal CJ (2018) Deep complex networks. In: 6th international conference on learning representations, ICLR 2018 - conference track proceedings
  52. Vaswani A (2017) Attention is all you need . Adv Neural Inf Process Syst 2017 pp 5999–6009 arXiv:1706.03762v5 · arXiv
  53. Wang X, Jiang, W, Luo Z (2016c) Combination of convolutional and recurrent neural network for sentiment analysis of short texts. In: COLING 2016 - 26th international conference on computational linguistics, proceedings of COLING 2016: technical papers
  54. Yang B, Yih W tau, He X, Gao J, Deng L (2015) Embedding entities and relations for learning and inference in knowledge bases. In: 3rd international conference on learning representations, ICLR 2015 - conference track proceedings

Cited by

See all 1,079

Metadata

Authors
Ahmed, S. F., Alam, M. S. B., Hassan, M., Rozbu, M. R., Ishtiak, T., Rafa, N., Mofijur, M., Shawkat Ali, A. B. M., & Gandomi, A. H.
Affiliations
  • Asian University for Women (BD)
  • Asian Institute of Technology (TH)
  • Carnegie Mellon University (US)
  • Carleton University (CA)
  • University of Cambridge (GB)
  • University of Technology Sydney (AU)
  • Obuda University (HU)
Corresponding author
  • Ahmed, Shams Forruque
  • Gandomi, Amir H.
Venue
Artificial Intelligence Review
Published
2023-04-17
Volume
56
Issue
11
Pages
13521-13617
ISSN
  • 0269-2821 (ISSN-L)
  • 1573-7462
OpenAlex
W4366091323
Cited by
1,079
References
252
Citations by year
2026: 337 · 2025: 500 · 2024: 222 · 2023: 20
FWCI
101.71
Type
journal article
Language
en
Open access
Licence
Locations
  • Artificial Intelligence Review (publishedVersion, cc-by)
  • Apollo (University of Cambridge) (submittedVersion, cc-by)
Funders
University of Technology Sydney
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Neural Networks and Applications
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Advanced Neural Network Applications
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
deep learning architectures, representation learning, convolutional neural networks, generative adversarial networks, recurrent neural networks, autoencoder, capsule networks, training problems, model robustness, local minima, deep learning applications

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Deep learning modelling techniques: current progress, applications, advantages, and challenges. https://app.cochat.ai/papers/10.1007/s10462-023-10466-8
The paper
Ahmed, S. F., Alam, M. S. B., Hassan, M., Rozbu, M. R., Ishtiak, T., Rafa, N., Mofijur, M., Shawkat Ali, A. B. M., & Gandomi, A. H. (2023). Deep learning modelling techniques: current progress, applications, advantages, and challenges. Artificial Intelligence Review. https://doi.org/10.1007/s10462-023-10466-8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.