Student or educator? Sign up with your school email and get 6 months of Pro free.
Cited by 5

The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.

Characterizing the Impact of Data-Damaged Models on Generalization Strength in Intrusion Detection

D’hooge, Laurens, Verkerken, Miel, Wauters, Tim, De Turck, Filip, Volckaert, Bruno · 2023 · Journal of Cybersecurity and Privacy

Abstract

Generalization is a longstanding assumption in articles concerning network intrusion detection through machine learning. Novel techniques are frequently proposed and validated based on the improvement they attain when classifying one or more of the existing datasets. The necessary follow-up question of whether this increased performance in classification is meaningful outside of the dataset(s) is almost never investigated. This lacuna is in part due to the sparse dataset landscape in network intrusion detection and the complexity of creating new data. The introduction of two recent datasets, namely CIC-IDS2017 and CSE-CIC-IDS2018, opened up the possibility of testing generalization capability within similar academic datasets. This work investigates how well models from different algorithmic families, pretrained on CICIDS2017, are able to classify the samples in CSE-CIC-IDS2018 without retraining. Earlier work has shown how robust these models are to data reduction when classifying state-of-the-art datasets. This work experimentally demonstrates that the implicit assumption that strong generalized performance naturally follows from strong performance on a specific dataset is largely erroneous. The supervised machine learning algorithms suffered flat losses in classification performance ranging from 0 to 50% (depending on the attack class under test). For non-network-centric attack classes, this performance regression is most pronounced, but even the less affected models that classify the network-centric attack classes still show defects. Current implementations of intrusion detection systems (IDSs) with supervised machine learning (ML) as a core building block are thus very likely flawed if they have been validated on the academic datasets, without the consideration for their general performance on other academic or real-world datasets.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3390/jcp3020008

Cites

1 of 1 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 1 references

Research works printed but not linked: 33

  1. Denning, 1987, An intrusion-detection model, IEEE Trans. Softw. Eng., 13, 222, 10.1109/TSE.1987.232894 · DOI · Paper page
  2. Duessel, 2017, Detecting zero-day attacks using context-aware anomaly detection at the application-layer, Int. J. Inf. Secur., 16, 475, 10.1007/s10207-016-0344-y · DOI
  3. Kolias, 2017, TermID: A distributed swarm intelligence-based approach for wireless intrusion detection, Int. J. Inf. Secur., 16, 401, 10.1007/s10207-016-0335-z · DOI
  4. Shone, 2018, A deep learning approach to network intrusion detection, IEEE Trans. Emerg. Top. Comput. Intell., 2, 41, 10.1109/TETCI.2017.2772792 · DOI · Paper page
  5. Sethi, 2020, A context-aware robust intrusion detection system: A reinforcement learning-based approach, Int. J. Inf. Secur., 19, 657, 10.1007/s10207-019-00482-7 · DOI
  6. Quadir, 2020, An efficient algorithm to detect DDoS amplification attacks, J. Intell. Fuzzy Syst., 39, 8565, 10.3233/JIFS-189173 · doi:10.3233/jifs-189173
  7. Kannari, 2021, Network intrusion detection using sparse autoencoder with swish-PReLU activation model, J. Ambient. Intell. Humaniz. Comput., 12, 1
  8. Badji, J.C.J., and Diallo, C. (2022, January 20–22). A CNN-based Attack Classification versus an AE-based Unsupervised Anomaly Detection for Intrusion Detection Systems. Proceedings of the 2022 International Conference on Electrical, Computer and Energy Technologies (ICECET), Prague, Czech Republic. · DOI
  9. Wauters, 2020, Inter-dataset generalization strength of supervised machine learning methods for intrusion detection, J. Inf. Secur. Appl., 54, 102564
  10. Wauters, 2019, Classification hardness for supervised learners on 20 years of intrusion detection data, IEEE Access, 7, 167455, 10.1109/ACCESS.2019.2953451 · DOI
  11. Sharafaldin, I., and CIC (2022, November 15). CIC-IDS2017. Available online: https://www.unb.ca/cic/datasets/ids-2017.html. · DOI · Paper page
  12. Govindarajan, M., and Chandrasekaran, R. (2012, January 24–26). Intrusion detection using an ensemble of classification methods. Proceedings of the World Congress on Engineering and Computer Science (WCECS), San Francisco, CA, USA.
  13. Lu, L., Teng, S., Zhang, W., Zhang, Z., Fei, L., and Fang, X. (2019, January 16–18). Two-Layer Intrusion Detection Model Based on Ensemble Classifier. Proceedings of the CCF Conference on Computer Supported Cooperative Work and Social Computing, Kunming, China. · doi:10.1007/978-981-15-1377-0_8
  14. Kuang, 2012, A novel approach of KPCA and SVM for intrusion detection, J. Comput. Inf. Syst., 8, 3237
  15. Wickramasinghe, C.S., Marino, D.L., Amarasinghe, K., and Manic, M. (2018, January 21–23). Generalization of deep learning for cyber-physical system security: A survey. Proceedings of the IECON 2018-44th Annual Conference of the IEEE Industrial Electronics Society, Washington, DC, USA. · DOI
  16. Sommer, R., and Paxson, V. (2010, January 16–19). Outside the closed world: On using machine learning for network intrusion detection. Proceedings of the 2010 IEEE Symposium on Security and Privacy, Oakland, CA, USA. · DOI · Paper page
  17. Gates, C., and Taylor, C. (2006, January 19–22). Challenging the anomaly detection paradigm: A provocative discussion. Proceedings of the 2006 Workshop on NEW Security Paradigms, Schloss Dagstuhl, Germany. · DOI
  18. Małowidzki, M., Berezinski, P., and Mazur, M. (2015, January 1). Network intrusion detection: Half a kingdom for a good dataset. Proceedings of the NATO STO SAS-139 Workshop, Lisbon, Portugal.
  19. Vasilomanolakis, E., Cordero, C.G., Milanov, N., and Mühlhäuser, M. (2016, January 25–29). Towards the creation of synthetic, yet realistic, intrusion detection datasets. Proceedings of the NOMS 2016-2016 IEEE/IFIP Network Operations and Management Symposium, Istanbul, Turkey. · DOI
  20. Ring, 2019, A survey of network-based intrusion detection data sets, Comput. Secur., 86, 147, 10.1016/j.cose.2019.06.005 · DOI · Paper page
  21. Li, Z., Das, A., and Zhou, J. (2005, January 7–10). Model generalization and its implications on intrusion detection. Proceedings of the International Conference on Applied Cryptography and Network Security, New York, NY, USA. · DOI
  22. Lin, Z., Shi, Y., and Xue, Z. (2022, January 16–19). Idsgan: Generative adversarial networks for attack generation against intrusion detection. Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining, Chengdu, China. · DOI · Paper page
  23. Newlin, M., Reith, M., and DeYoung, M. (July, January 4–). Synthetic Data Generation with Machine Learning for Network Intrusion Detection Systems. Proceedings of the European Conference on Cyber Warfare and Security, Coimbra, Portugal.
  24. Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. (2019, January 9–15). Do imagenet classifiers generalize to imagenet?. Proceedings of the International Conference on Machine Learning, PMLR, Long Beach, CA, USA.
  25. Sharafaldin, 2018, Toward generating a new intrusion detection dataset and intrusion traffic characterization, ICISSp, 1, 108
  26. Sharafaldin, 2018, Towards a reliable intrusion detection benchmark dataset, Softw. Netw., 2018, 177
  27. Hastie, T., Tibshirani, R., Friedman, J.H., and Friedman, J.H. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, Springer. · DOI · Paper page
  28. Chen, T., and Guestrin, C. (2016, January 13–17). Xgboost: A scalable tree boosting system. Proceedings of the 22nd Acm Sigkdd International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA. · DOI · Paper page
  29. Geurts, 2006, Extremely randomized trees, Mach. Learn., 63, 3, 10.1007/s10994-006-6226-1 · DOI · Paper page
  30. D’hooge, L., Wauters, T., Volckaert, B., and De Turck, F. (2019, January 2–4). In-depth comparative evaluation of supervised machine learning approaches for detection of cybersecurity threats. Proceedings of the 4th International Conference on Internet of Things, Big Data and Security (IoTBDS), Crete, Greece. · DOI
  31. Young, 2018, Recent trends in deep learning based natural language processing, IEEE Comput. IntelligenCe Mag., 13, 55, 10.1109/MCI.2018.2840738 · DOI · Paper page
  32. Zhao, 2019, Object detection with deep learning: A review, IEEE Trans. Neural Networks Learn. Syst., 30, 3212, 10.1109/TNNLS.2018.2876865 · DOI · Paper page
  33. Mighan, 2021, A novel scalable intrusion detection system based on deep learning, Int. J. Inf. Secur., 20, 387, 10.1007/s10207-020-00508-5 · doi:10.1007/s10207-020-00508-5

Metadata

Authors
D’hooge, L., Verkerken, M., Wauters, T., De Turck, F., & Volckaert, B.
Venue
Journal of Cybersecurity and Privacy
Publisher
MDPI AG
Published
2023-04-03
Volume
3
Issue
2
Pages
118-144
ISSN
2624-800X
Cited by
5
References
1
Language
en

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Characterizing the Impact of Data-Damaged Models on Generalization Strength in Intrusion Detection. https://app.cochat.ai/papers/10.3390/jcp3020008
The paper
D’hooge, L., Verkerken, M., Wauters, T., De Turck, F., & Volckaert, B. (2023). Characterizing the Impact of Data-Damaged Models on Generalization Strength in Intrusion Detection. Journal of Cybersecurity and Privacy. https://doi.org/10.3390/jcp3020008

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.