CoChat

Artificial Intelligence in Malware and Network Intrusion Detection: A Comprehensive Survey of Techniques, Datasets, Challenges, and Future Directions

Open access
Moamin, Saif A. H., Abdulhameed, Muhannad Kaml, Al-Amri, Rusul Mansoor, Radhi, Ahmed Dheyaa, Naser, Rusul Kadhim, Pheng, Liaw Geok · 2025 · Babylonian Journal of Artificial Intelligence

The DOI matches its Crossref record; 21 of the 22 references it cites were found in a registry or an index, and 1 are not checked yet.

Abstract

As cyber threats evolve in complexity and scale, traditional detection mechanisms are increasingly inadequate. This paper presents a comprehensive survey of artificial intelligence (AI) applications in malware and network intrusion detection, emphasizing the integration of intelligent techniques for real-time threat mitigation. We categorize state-of-the-art methods across supervised, unsupervised, and reinforcement learning, including deep learning architectures such as CNNs, RNNs, LSTMs, and ensemble models. Static, dynamic, and hybrid analysis techniques are compared, with a focus on feature engineering, behavioral modeling, and real-world deployment constraints. A novel AI-based Malware Detection and Prevention Framework is proposed, combining machine learning classifiers with Network Intrusion Prevention Systems (NIPS) to enhance proactive defense capabilities. The study evaluates publicly available and synthetic datasets, addressing challenges such as class imbalance, adversarial evasion, and data scarcity. We also highlight ethical considerations including bias, privacy, and accountability in AI-enabled cybersecurity systems. Case studies from mobile and IoT ecosystems demonstrate the practicality and limitations of AI-based defenses in dynamic threat landscapes. Finally, the paper outlines future research directions in explainable AI, automated model generation (AutoML), and adaptive, context-aware intrusion prevention systems. This work serves as a critical resource for developing resilient, scalable, and intelligent cybersecurity infrastructures aligned with modern digital ecosystems

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.58496/bjai/2025/008

Cites

0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 69

  1. R. C. Pivetta de Araujo et al., "Serial mass screening for tuberculosis among incarcerated persons in Brazil," Clinical Infectious Diseases, vol. 78, no. 6, pp. 1669-1676, 2024.
  2. Y. Li and Q. Liu, "A comprehensive review study of cyber-attacks and cyber security; Emerging trends and recent developments," Energy Reports, vol. 7, pp. 8176-8186, 2021.
  3. M. Fahad, H. Airf, A. Kumar, and H. K. Hussain, "Securing against apts: Advancements in detection and mitigation," BIN: Bulletin Of Informatics, vol. 1, no. 2, 2023.
  4. L. Wang, C. Sun, C. Zhang, W. Nie, and K. Huang, "Application of knowledge graph in software engineering field: A systematic literature review," Information and Software Technology, vol. 164, p. 107327, 2023.
  5. I. P. Saputra, E. Utami, and A. H. Muhammad, "Comparison of anomaly based and signature based methods in detection of scanning vulnerability," in 2022 9th International Conference on Electrical Engineering, Computer Science and Informatics (EECSI), 2022: IEEE, pp. 221-225.
  6. A. Aboalassaad and T. Malik, "The Impact of IPS and Firewall Placement on Network Security and Performance," in 2024 Cyber Research Conference-Ireland (Cyber-RCI), 2024: IEEE, pp. 1-7.
  7. R. Al-Amri, A. Hadi, A. H. Mousa, H. Hasan, and M. Kadhim, "The development of a deep learning model for predicting stock prices," Journal of Advanced Research in Applied Sciences and Engineering Technology, vol. 31, no. 3, pp. 208-219, 2023.
  8. V. Hassija et al., "Interpreting black-box models: a review on explainable artificial intelligence," Cognitive Computation, vol. 16, no. 1, pp. 45-74, 2024.
  9. R. M. Al-Amri, A. A. Hadi, M. S. Kadhim, A. H. Mousa, A. Z. Matloob, and H. F. Hasan, "Enhancement of The Performance of Machine Learning Algorithms to Rival Deep Learning Algorithms in Predicting Stock Prices," Babylonian Journal of Artificial Intelligence, vol. 2024, pp. 118-127, 2024.
  10. H. Alqahtani and G. Kumar, "Advances in artificial intelligence for detecting algorithmically generated domains: Current trends and future prospects," Engineering Applications of Artificial Intelligence, vol. 138, p. 109410, 2024.
  11. P. Bhat, S. Behal, and K. Dutta, "A system call-based android malware detection approach with homogeneous & eterogeneous ensemble machine learning," Computers & Security, vol. 130, p. 103277, 2023.
  12. T. Bilot, N. El Madhoun, K. Al Agha, and A. Zouaoui, "A survey on malware detection with graph representation learning," ACM Computing Surveys, vol. 56, no. 11, pp. 1-36, 2024.
  13. M. A. Talukder et al., "Machine learning-based network intrusion detection for big and imbalanced data using oversampling, stacking feature embedding and feature extraction," Journal of big data, vol. 11, no. 1, p. 33, 2024.
  14. M. Schmitt, "Securing the digital world: Protecting smart infrastructures and digital industries with artificial intelligence (AI)-enabled malware and intrusion detection," Journal of Industrial Information Integration, vol. 36, p. 100520, 2023.
  15. R. M. Al-Amri, D. N. Hamood, and A. K. Farhan, "Theoretical background of cryptography," Mesopotamian Journal of CyberSecurity, vol. 2023, pp. 7-15, 2023.
  16. A. A. Hashim and A. H. Mousa, "An evaluation framework for diabetes prediction techniques using machine learning," in BIO Web of Conferences, 2024, vol. 97: EDP Sciences, p. 00125.
  17. M. E. Manaa, F. J. Abd Al-Razaq, and H. A. Al-Khamees, "Enhancing IoT Security: An Optimization Algorithm for Fog Layer-Based DDoS Attack Mitigation Framework," Iraqi Journal of Science, 2025.
  18. A. Gaurav, B. B. Gupta, and P. K. Panigrahi, "A comprehensive survey on machine learning approaches for malware detection in IoT-based enterprise information system," Enterprise Information Systems, vol. 17, no. 3, p. 2023764, 2023.
  19. S. M. Ali, R. M. Al-Amri, and A. K. Farhan, "Generation of Dynamic Substitution Boxes Using HSM Chaos System for Application in Color Images Encrypting," International Journal of Intelligent Engineering & Systems, vol. 16, no. 6, 2023.
  20. D. Trizna, L. Demetrio, B. Biggio, and F. Roli, "Nebula: Self-attention for dynamic malware analysis," IEEE Transactions on Information Forensics and Security, 2024.
  21. G. Karat, J. M. Kannimoola, N. Nair, A. Vazhayil, S. VG, and P. Poornachandran, "CNN-LSTM hybrid model for enhanced malware analysis and detection," Procedia Computer Science, vol. 233, pp. 492-503, 2024.
  22. H. Jmila and M. I. Khedher, "Adversarial machine learning for network intrusion detection: A comparative study," Computer Networks, vol. 214, p. 109073, 2022.
  23. C. P. Chenet, A. Savino, and S. Di Carlo, "A survey on hardware-based malware detection approaches," IEEE Access, 2024.
  24. Y. Li, W. Zhao, Y. Su, W. Li, and C. Yuan, "Overview of cloud computing deployment mode and technology development trend," in 2023 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting (BMSB), 2023: IEEE, pp. 1-5.
  25. M. Ali, S. Shiaeles, M. Papadaki, and B. V. Ghita, "Agent-based vs agent-less sandbox for dynamic behavioral analysis," in 2018 Global Information Infrastructure and Networking Symposium (GIIS), 2018: IEEE, pp. 1-5.
  26. C. Li et al., "DMalNet: Dynamic malware analysis based on API feature engineering and graph learning," Computers & Security, vol. 122, p. 102872, 2022.
  27. Z. Dai et al., "An intrusion detection model to detect zero-day attacks in unseen data using machine learning," PloS one, vol. 19, no. 9, p. e0308469, 2024.
  28. S. Kumar, P. Ahlawat, and J. Sahni, "IOT malware detection using static and dynamic analysis techniques: A systematic literature review," Security and Privacy, vol. 7, no. 6, p. e444, 2024.
  29. A. Q. Raheema, M. Kamalrudin, and N. R. Dzakiyullah, "Multi-Level Fusion for Enhanced Host-based Malware Detection in ICT-Enabled Smart Cities," Fusion: Practice & Applications, vol. 15, no. 2, 2024.
  30. A. K. Ghazi-Tehrani and H. N. Pontell, "Phishing evolves: Analyzing the enduring cybercrime," in The New Technology of Financial Crime: Routledge, 2022, pp. 35-61.
  31. H. A. Noman, Q. Al-Maatouk, and S. A. Noman, "A static analysis tool for malware detection," in 2021 International Conference on Data Analytics for Business and Industry (ICDABI), 2021: IEEE, pp. 661-665.
  32. M. V. S. Babu and K. Banana, "A study on narrow artificial intelligence-An overview," Int. J. Eng. Sci. Adv. Technol, vol. 24, pp. 210-219, 2024.
  33. M. Ashik et al., "Detection of malicious software by analyzing distinct artifacts using machine learning and deep learning algorithms," Electronics, vol. 10, no. 14, p. 1694, 2021.
  34. E. E. Abdallah and A. F. Otoom, "Intrusion detection systems using supervised machine learning techniques: a survey," Procedia Computer Science, vol. 201, pp. 205-212, 2022.
  35. A. Alshammari and A. Aldribi, "Apply machine learning techniques to detect malicious network traffic in cloud computing," Journal of Big Data, vol. 8, no. 1, p. 90, 2021.
  36. Ü. Atila, M. Uçar, K. Akyol, and E. Uçar, "Plant leaf disease classification using EfficientNet deep learning model," Ecological Informatics, vol. 61, p. 101182, 2021.
  37. M. Ozkan-Okay, R. Samet, Ö. Aslan, and D. Gupta, "A comprehensive systematic literature review on intrusion detection systems," IEEE Access, vol. 9, pp. 157727-157760, 2021.
  38. O. H. Abdulganiyu, T. Ait Tchakoucht, and Y. K. Saheed, "A systematic literature review for network intrusion detection system (IDS)," International journal of information security, vol. 22, no. 5, pp. 1125-1162, 2023.
  39. H. Satilmiş, S. Akleylek, and Z. Y. Tok, "A systematic literature review on host-based intrusion detection systems," Ieee Access, vol. 12, pp. 27237-27266, 2024.
  40. H. Satilmiş, S. Akleylek, and Z. Y. Tok, "A systematic literature review on host-based intrusion detection systems," Ieee Access, vol. 12, pp. 27237-27266, 2024.
  41. B. Nawaal, U. Haider, I. U. Khan, and M. Fayaz, "Signature-based intrusion detection system for IoT," in Cyber Security for Next-Generation Computing Technologies: CRC Press, 2024, pp. 141-158.
  42. S. Einy, C. Oz, and Y. D. Navaei, "The anomaly-and signature-based IDS for network security using hybrid inference systems," Mathematical Problems in Engineering, vol. 2021, no. 1, p. 6639714, 2021.
  43. M. A. Alsoufi et al., "Anomaly-based intrusion detection systems in iot using deep learning: A systematic literature review," Applied sciences, vol. 11, no. 18, p. 8383, 2021.
  44. M. Bhavsar, K. Roy, J. Kelly, and O. Olusola, "Anomaly-based intrusion detection system for IoT application," Discover Internet of things, vol. 3, no. 1, p. 5, 2023.
  45. Z. K. Maseer, R. Yusof, N. Bahaman, S. A. Mostafa, and C. F. M. Foozy, "Benchmarking of machine learning for anomaly based intrusion detection systems in the CICIDS2017 dataset," IEEE access, vol. 9, pp. 22351-22370, 2021.
  46. G. Karat, J. M. Kannimoola, N. Nair, A. Vazhayil, S. VG, and P. Poornachandran, "CNN-LSTM hybrid model for enhanced malware analysis and detection," Procedia Computer Science, vol. 233, pp. 492-503, 2024.
  47. J. S. Bailey and M. R. Burch, Research methods in applied behavior analysis. Routledge, 2024.
  48. F. A. Vadhil, M. L. Salihi, and M. F. Nanne, "Machine learning-based intrusion detection system for detecting web attacks," IAES International Journal of Artificial Intelligence, vol. 13, no. 1, pp. 711-721, 2024.
  49. G. Singh and N. Khare, "A survey of intrusion detection from the perspective of intrusion datasets and machine learning techniques," International Journal of Computers and Applications, vol. 44, no. 7, pp. 659-669, 2022.
  50. A. Khanan, Y. A. Mohamed, A. H. H. Mohamed, and M. Bashir, "From bytes to insights: a systematic literature review on unraveling IDS datasets for enhanced cybersecurity understanding," IEEE Access, vol. 12, pp. 59289- 59317, 2024.
  51. M. S. Iqbal, W. Ahmad, R. Alizadehsani, S. Hussain, and R. Rehman, "Breast cancer dataset, classification and detection using deep learning," in Healthcare, 2022, vol. 10, no. 12: MDPI, p. 2395.
  52. E. Alomari, I. Katib, A. Albeshri, T. Yigitcanlar, and R. Mehmood, "Iktishaf+: a big data tool with automatic labeling for road traffic social sensing and event detection using distributed machine learning," Sensors, vol. 21, no. 9, p. 2993, 2021.
  53. H. Yukhymenko, R. Staab, M. Vero, and M. Vechev, "A synthetic dataset for personal attribute inference," Advances in Neural Information Processing Systems, vol. 37, pp. 120735-120779, 2024.
  54. Ammara, D. A., Ding, J., & Tutschku, K. (2024). "Synthetic Data Generation in Cybersecurity: A Comparative Analysis." arXiv preprint arXiv:2410.16326. · arXiv
  55. P. Bedi, N. Gupta, and V. Jindal, "I-SiamIDS: An Improved Siam-IDS for Handling Class Imbalance in Network- Based Intrusion Detection Systems," arXiv preprint arXiv:2009.10940, Sep. 2020. · arXiv
  56. H. J. Ahn, S. Sharma, R. Matam, and J. Park, "A comprehensive survey on Android malware detection using machine learning," IEEE Access, vol. 11, pp. 39193-39226, 2023. doi: 10.1109/ACCESS.2023.3260207. · doi:10.1109/access.2023.3260207
  57. A. Kumar, S. Sharma, and R. Singh, "Static analysis framework for permission-based dataset generation and android malware detection using machine learning," EURASIP Journal on Information Security, vol. 2024, no. 33, pp. 1-17, Oct. 2024
  58. M. Nasir, A. R. Javed, M. A. Tariq, M. Asim, and T. Baker, "Feature engineering and deep learning-based intrusion detection framework for securing edge IoT," Journal of Supercomputing, vol. 78, pp. 1-15, 2022.
  59. B. Lim, S. O. Arik, N. Loeff, and T. Pfister, "Temporal Fusion Transformers for Interpretable Multi-horizon Time Series Forecasting," arXiv preprint arXiv:1912.09363, Dec. 2019. · arXiv
  60. Y. Ren, D. Zhao, D. Luo, H. Ma, and P. Duan, "Global-local temporal convolutional network for traffic flow prediction," IEEE Transactions on Intelligent Transportation Systems, vol. 21, no. 5, pp. 1935-1944, May 2020. doi: 10.1109/TITS.2019.2919620. · doi:10.1109/tits.2019.2919620
  61. L. Zhao et al., "T-GCN: A Temporal Graph Convolutional Network for Traffic Prediction," arXiv preprint arXiv:1811.05320, Nov. 2018. · arXiv
  62. O. A. Madamidola, F. Ngobigha, and A. Ez-zizi, "Detecting new obfuscated malware variants: A lightweight and interpretable machine learning approach," Intelligent Systems with Applications, vol. 25, Mar. 2025.
  63. M. A. Hossain and M. S. Islam, "Enhanced detection of obfuscated malware in memory dumps: a machine learning approach for advanced cybersecurity," Cybersecurity, vol. 7, Art. no. 16, Jan. 2024.
  64. K. A. Dhanya, D. O. K., G. K. T., and P. Vinod, "Detection of Obfuscated Mobile Malware with Machine Learning and Deep Learning Models," in Machine Learning and Metaheuristics Algorithms, and Applications, Springer, Singapore, 2021, pp. 245-258.
  65. J. Zhu, J. Jang-Jaccard, A. Singh, P. A. Watters, and S. Camtepe, "Task-Aware Meta Learning-based Siamese Neural Network for Classifying Obfuscated Malware," arXiv preprint arXiv:2110.13409, Oct. 2021. · arXiv
  66. E. Alhajjar, P. Maxwell, and N. D. Bastian, "Adversarial Machine Learning in Network Intrusion Detection Systems," arXiv preprint arXiv:2004.11898, 2020. · arXiv
  67. X. Zhao, K. W. Fok, and V. L. L. Thing, "Enhancing Network Intrusion Detection Performance using Generative Adversarial Networks," arXiv preprint arXiv:2404.07464, 2024. · arXiv
  68. H. A. Alatwi and C. Morisset, "Adversarial Machine Learning in Network Intrusion Detection Domain: A Systematic Review," arXiv preprint arXiv:2112.03315, 2021. · arXiv
  69. M. Roesch, "Snort -Lightweight Intrusion Detection for Networks," in Proceedings of the 13th USENIX Conference on System Administration (LISA '99), Seattle, WA, USA, 1999, pp. 229-238.

Cited by

See all 23

Metadata

Authors
Moamin, S. A. H., Abdulhameed, M. K., Al-Amri, R. M., Radhi, A. D., Naser, R. K., & Pheng, L. G.
Affiliations
  • University of Kerbala (IQ)
  • University of Al-Ameed (IQ)
  • Universiti Teknikal Malaysia Melaka (MY)
Venue
Babylonian Journal of Artificial Intelligence
Published
2025-06-13
Volume
2025
Pages
77-98
ISSN
3006-5437 (ISSN-L)
OpenAlex
W4411279063
Cited by
23
References
0
Citations by year
2026: 11 · 2025: 11
FWCI
10.30
Type
journal article
Language
en
Open access
Locations
Babylonian Journal of Artificial Intelligence (publishedVersion)
Funders
University of Kerbala
Topics
  • Physical Sciences › Computer Science › Software › Advanced Malware Detection Techniques
  • Physical Sciences › Computer Science › Computer Networks and Communications › Network Security and Intrusion Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Adversarial Robustness in Machine Learning
Keywords
malware detection, network intrusion detection, intrusion prevention systems, deep learning for cybersecurity, static and dynamic analysis, class imbalance, evasion attacks, data scarcity, mobile security, Internet of Things security, explainable AI, automated machine learning

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Artificial Intelligence in Malware and Network Intrusion Detection: A Comprehensive Survey of Techniques, Datasets, Challenges, and Future Directions. https://app.cochat.ai/papers/10.58496/bjai/2025/008
The paper
Moamin, S. A. H., Abdulhameed, M. K., Al-Amri, R. M., Radhi, A. D., Naser, R. K., & Pheng, L. G. (2025). Artificial Intelligence in Malware and Network Intrusion Detection: A Comprehensive Survey of Techniques, Datasets, Challenges, and Future Directions. Babylonian Journal of Artificial Intelligence. https://doi.org/10.58496/bjai/2025/008

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.