The DOI matches its Crossref record; 15 of the 16 references it cites were found in a registry or an index; none of the 17 sources its claims cite is checked for support yet.
Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study
Open accessAbstract
Numerous studies have demonstrated the effectiveness of machine learning techniques in application to network intrusion detection. And yet, the adoption of machine learning for securing large-scale network environments remains challenging. The community acknowledges that network security presents unique challenges for machine learning, and the lack of training data representative of modern traffic remains one of the most intractable issues. New attempts are continuously made to develop high quality benchmark datasets and proper data collection methodologies. The CICIDS2017 dataset is one of the recent results, created to meet the demanding criterion of representativeness for network intrusion detection. In this paper we revisit CICIDS2017 and its data collection pipeline and analyze correctness, validity and overall utility of the dataset for the learning task. During this in-depth analysis, we uncover a series of problems with traffic generation, flow construction, feature extraction and labelling that severely affect the aforementioned properties. We investigate the causes of these shortcomings and address most of them by applying an improved data processing methodology. As a result, more than 20 percent of original traffic traces are reconstructed or relabelled. Machine learning benchmarks on the final dataset demonstrate significant improvements. Our study exemplifies how data collection issues may have enormous impact on model evaluation and provides recommendations for their anticipation and prevention.
Overview
CoChat Summary
TL;DR
The paper audits CICIDS2017 and its CICFlowMeter pipeline, finding errors in TCP flow construction, labelling, attack simulation, and feature extraction that make many flows meaningless or mislabelled. After fixing the extractor and regenerating/relabelling the dataset, Random Forest benchmarks improve on several attack classes despite unchanged aggregate metrics.
Key findings
- More than 25% of all flows are meaningless artefacts; TCP appendices alone make up 25.9% of the dataset, and for several attack categories their fraction reaches 50%. More than 20% of original traffic traces are reconstructed or relabelled.
- CICFlowMeter treats the first FIN packet as connection close, violating TCP; it ignores RST for termination; timeout can swap flow direction, causing attack flows to risk being mislabelled as benign. The DoS Hulk tool is misimplemented (`Connection: Close` instead of `Keep-Alive`) and ineffective.
- Labelling relies on source/destination IP and time window rather than flow content; many attack flows have no forward data transfer and are relabelled as “X-Attempted” (447,362 attempted flows in the final dataset). The final payload filter sharply reduces Web Brute Force (1507→151), Web Attack XSS (652→27), Bot (1956→738), DoS Slowhttptest (5499→1742), and DoS slowloris (5791→4001).
- Random Forest per-class F1 improves on Web Brute Force (0.77→0.95), XSS (0.18→0.79), Bot (0.60→0.99), Heartbleed (0.77→0.93), and Infiltration (0.81→0.91), while aggregate accuracy and weighted average remain 0.99. Shortcut learning is evident: appendices are differentiated by `Fwd Header Len`, `Total Fwd Pkt`, `Min Seg Size Fwd`, and `Init Win bytes Fwd`; RST presence was a learned shortcut for Bot in the Original dataset.
Methods
- Manual methodological analysis of raw CICIDS2017 and the CICFlowMeter source code across attack simulation, flow construction, labelling, feature extraction, and ML benchmarking, using the original paper, dataset website, and GitHub tool documentation.
- Regenerated the dataset in three stages: Original, Intermediate (fix TCP appendices and RST termination), and Final (payload filter except Benign and PortScan); evaluated a Random Forest classifier on each with a 75–25 train-test split, assigning Attempted flows to BENIGN, and reported per-class precision/recall/F1 plus feature importance.
- Dataset described as over 2.8 million traffic traces in 15 classes, with 80 extracted high-level statistical features; CICFlowMeter outputs 83 features excluding the label.
Limitations
- Stated: the detailed TCP appendix timeout/direction-swap phenomenon is out of scope; results may differ with other ML models; some attack classes have extremely small test sets (Heartbleed, XSS, SQL, Infiltration); CSE-CIC-IDS2018 is only preliminarily investigated and needs further analysis; documentation on flow construction and attack parameters is lacking; diversity within attack categories is not openly discussed.
- Evident: evaluation uses only a Random Forest classifier, so per-class benchmark improvements are model-specific; manual inspection may not exhaustively cover all flows/classes; Web Attack SQL Injection ends with 12 traces total and all 3 test traces mispredicted as Benign, indicating poor representation and distinguishability.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 5 Not valid · 12 Not checked · 59 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“During this in-depth analysis, we uncover a series of problems with traffic generation, flow construction, feature extraction and labelling that severely affect the aforementioned properties.”
- Also stated in: § V. CONCLUSION · plain
“Through analyzing traffic generation, attack simulation, flow construction, feature extraction, labeling and benchmarking stages of data collection, we discovered a series of problems that violate some of the established properties of a high quality NIDS dataset.”
§ Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study > Abstract
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § V. CONCLUSION · plain
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We observe, however, that the CICFlowMeter tool does not consider the RST packet as a valid way to terminate a TCP flow.”
§ Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study > 1) TCP "appendices".
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In Section IV we show that the TCP appendices are alarmingly present, making up 25.9 percent of the entire dataset and study their impact.”
§ Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study > 1) TCP "appendices".
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we observe that Web Attack -Brute Force and Web Attack -XSS largely consist of flows that have no data transfer in the forward direction, which means that the attacker never actually executes an attack within those flows.”
§ Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study > B. Labelling
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“We found that the time frames reported on the dataset information page [15] are not precise, therefore we had to tune them to reproduce the labelling.”
§ Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study > B. Labelling
What the sources say
- Not matchedcites [15]
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
34 more claims
Create a free account to see all 76 claims and every source verdict.
- It may state two separate claims as one. 25
- Probably not a substantial claim. 16
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study > 1) Original.
“BENIGN 2271326 1823964 1823964 ATTEMPTED* 0 0 447362 FTP-Patator 7934 3984 3973 SSH-Patator 5898 2988 2980 DoS GoldenEye 10293 7647 7567 DoS Hulk 230124 159048 158469 DoS Slowhttptest 5499 5109 1742 DoS slowloris 5791 5707 4001 Heartbleed 11 11 11 Web Brute Force 1507 1365 151 Web Attack XSS 652 679 27 Web Attack SQL 21 12 12 Infiltration 36 48 32 Bot 1956 2208 738”
§ Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study > 1) Original.
“3) Final: payload filter.Following our conclusion in III-B2, we apply the payload filter to all classes except for Benign and PortScan.This filter ensures that attack classes predominantly contain flows that actually exhibit malicious activity.”
Citations
Cites
Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization
Sharafaldin, I., Habibi Lashkari, A., & Ghorbani, A. A. (2018) · Proceedings of the 4th International Conference on Information Systems Security and Privacy · doi:10.5220/0006639801080116 · Cited by 5,318
(2011) · SpringerReference · doi:10.1007/springerreference_27387 · Cited by 3,053
Postel, J. (1981) · doi:10.17487/rfc0793 · Cited by 2,122
A survey of network-based intrusion detection data sets
Ring, M., Wunderlich, S., Scheuring, D., Landes, D., & Hotho, A. (2019) · Computers & Security · doi:10.1016/j.cose.2019.06.005 · Cited by 947
CICIDS-2017 Dataset Feature Analysis With Information Gain for Anomaly Detection
Kurniabudi, Stiawan, D., Darmawijoyo, Bin Idris, M. Y., Bamhdi, A. M., & Budiarto, R. (2020) · IEEE Access · doi:10.1109/access.2020.3009843 · Cited by 453
Towards a Reliable Intrusion Detection Benchmark Dataset
Sharafaldin, I., Canadian Institute for Cybersecurity (CIC), U., Gharib, A., Canadian Institute for Cybersecurity (CIC), U., Lashkari, A. H., Canadian Institute for Cybersecurity (CIC), U., Ghorbani, A. A., & Canadian Institute for Cybersecurity (CIC), U. (2017) · Software Networking · doi:10.13052/jsn2445-9739.2017.009 · Cited by 302
A Taxonomy of Network Threats and the Effect of Current Datasets on Intrusion Detection Systems
Hindy, H., Brosset, D., Bayne, E., Seeam, A. K., Tachtatzis, C., Atkinson, R., & Bellekens, X. (2020) · IEEE Access · doi:10.1109/access.2020.3000179 · Cited by 273
A survey and analysis of intrusion detection models based on CSE-CIC-IDS2018 Big Data
Leevy, J. L., & Khoshgoftaar, T. M. (2020) · Journal of Big Data · doi:10.1186/s40537-020-00382-x · Cited by 264
An Evaluation Framework for Intrusion Detection Dataset
Gharib, A., Sharafaldin, I., Lashkari, A. H., & Ghorbani, A. A. (2016) · 2016 International Conference on Information Science and Security (ICISS) · doi:10.1109/icissec.2016.7885840 · Cited by 217
A Detailed Analysis of the CICIDS2017 Data Set
Sharafaldin, I., Habibi Lashkari, A., & Ghorbani, A. A. (2019) · Communications in Computer and Information Science · doi:10.1007/978-3-030-25109-3_9 · Cited by 164
Research works printed but not linked: 3
- 0, Extended documentation of the corrected CICFlowMeter generated CICIDS dataset and the source code of the paper
- 2017, Intrusion detection evaluation dataset (cicids2017)
- 2021, CICFlowMeter tool
Cited by
Pitfalls in Machine Learning for Computer Security
Arp, D., Quiring, E., Pendlebury, F., Warnecke, A., Pierazzi, F., Wressnegger, C., Cavallaro, L., & Rieck, K. (2024) · Communications of the ACM · doi:10.1145/3643456 · Cited by 295
The Role of Machine Learning in Cybersecurity
Apruzzese, G., Laskov, P., Montes de Oca, E., Mallouli, W., Brdalo Rapa, L., Grammatopoulos, A. V., & Di Franco, F. (2023) · Digital Threats: Research and Practice · doi:10.1145/3545574 · Cited by 279
INSOMNIA: Towards Concept-Drift Robustness in Network Intrusion Detection
Andresini, G., Pendlebury, F., Pierazzi, F., Loglisci, C. A. A. M. D., Appice, A., & Cavallaro, L. (2021) · ACM Workshop on Artificial Intelligence and Security (AISec) · doi:10.1145/3474369.3486864 · Cited by 227
Error Prevalence in NIDS datasets: A Case Study on CIC-IDS-2017 and CSE-CIC-IDS-2018
Liu, L., Engelen, G., Lynar, T., Essam, D., & Joosen, W. (2022) · IEEE Conference on Communications and Network Security · doi:10.1109/cns56114.2022.9947235 · Cited by 225
Self-citationFed-ANIDS: Federated learning for anomaly-based network intrusion detection systems
Idrissi, M. J., Alami, H., El Mahdaouy, A., El Mekki, A., Oualil, S., Yartaoui, Z., & Berrada, I. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121000 · Cited by 190
Metadata
- Authors
- Engelen, G., Rimmer, V., & Joosen, W.
- Affiliations
- IMEC (BE)
- Venue
- 2021 IEEE Security and Privacy Workshops (SPW)
- Publisher
- IEEE
- Published
- 2021-05
- Pages
- 7-12
- OpenAlex
- W3180062783
- Semantic Scholar
- f2391bf235ef5342703b642c1a071228f24accbe
- Cited by
- 398
- References
- 16
- Citations by year
- 2026: 191 · 2025: 85 · 2024: 58 · 2023: 41 · 2022: 20 · 2021: 3
- FWCI
- 19.03
- Type
- conference paper
- Language
- en
- Open access
- green
- lirias.kuleuven.be
- Licence
- Locations
- IEEE Security and Privacy Workshops (SPW) (publishedVersion)
- Lirias (KU Leuven) (submittedVersion, other-oa)
- Funders
- KU Leuven
- Topics
- Physical Sciences › Computer Science › Computer Networks and Communications › Network Security and Intrusion Detection
- Physical Sciences › Computer Science › Artificial Intelligence › Internet Traffic Analysis and Secure E-voting
- Physical Sciences › Computer Science › Hardware and Architecture › Network Packet Processing and Optimization
- Keywords
- CICIDS2017, network intrusion detection, data collection pipeline, traffic generation, feature extraction, dataset representativeness, data processing, model evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-16, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-16 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study. https://app.cochat.ai/papers/10.1109/spw53761.2021.00009
- The paper
- Engelen, G., Rimmer, V., & Joosen, W. (2021). Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study. 2021 IEEE Security and Privacy Workshops (SPW). https://doi.org/10.1109/spw53761.2021.00009
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.