Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 15 of the 16 references it cites were found in a registry or an index; none of the 17 sources its claims cite is checked for support yet.

Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study

Open access
Engelen, Gints, Rimmer, Vera, Joosen, Wouter · 2021 · 2021 IEEE Security and Privacy Workshops (SPW)

Abstract

Numerous studies have demonstrated the effectiveness of machine learning techniques in application to network intrusion detection. And yet, the adoption of machine learning for securing large-scale network environments remains challenging. The community acknowledges that network security presents unique challenges for machine learning, and the lack of training data representative of modern traffic remains one of the most intractable issues. New attempts are continuously made to develop high quality benchmark datasets and proper data collection methodologies. The CICIDS2017 dataset is one of the recent results, created to meet the demanding criterion of representativeness for network intrusion detection. In this paper we revisit CICIDS2017 and its data collection pipeline and analyze correctness, validity and overall utility of the dataset for the learning task. During this in-depth analysis, we uncover a series of problems with traffic generation, flow construction, feature extraction and labelling that severely affect the aforementioned properties. We investigate the causes of these shortcomings and address most of them by applying an improved data processing methodology. As a result, more than 20 percent of original traffic traces are reconstructed or relabelled. Machine learning benchmarks on the final dataset demonstrate significant improvements. Our study exemplifies how data collection issues may have enormous impact on model evaluation and provides recommendations for their anticipation and prevention.

Overview

CoChat Summary

TL;DR

The paper audits CICIDS2017 and its CICFlowMeter pipeline, finding errors in TCP flow construction, labelling, attack simulation, and feature extraction that make many flows meaningless or mislabelled. After fixing the extractor and regenerating/relabelling the dataset, Random Forest benchmarks improve on several attack classes despite unchanged aggregate metrics.

Key findings

  • More than 25% of all flows are meaningless artefacts; TCP appendices alone make up 25.9% of the dataset, and for several attack categories their fraction reaches 50%. More than 20% of original traffic traces are reconstructed or relabelled.
  • CICFlowMeter treats the first FIN packet as connection close, violating TCP; it ignores RST for termination; timeout can swap flow direction, causing attack flows to risk being mislabelled as benign. The DoS Hulk tool is misimplemented (`Connection: Close` instead of `Keep-Alive`) and ineffective.
  • Labelling relies on source/destination IP and time window rather than flow content; many attack flows have no forward data transfer and are relabelled as “X-Attempted” (447,362 attempted flows in the final dataset). The final payload filter sharply reduces Web Brute Force (1507→151), Web Attack XSS (652→27), Bot (1956→738), DoS Slowhttptest (5499→1742), and DoS slowloris (5791→4001).
  • Random Forest per-class F1 improves on Web Brute Force (0.77→0.95), XSS (0.18→0.79), Bot (0.60→0.99), Heartbleed (0.77→0.93), and Infiltration (0.81→0.91), while aggregate accuracy and weighted average remain 0.99. Shortcut learning is evident: appendices are differentiated by `Fwd Header Len`, `Total Fwd Pkt`, `Min Seg Size Fwd`, and `Init Win bytes Fwd`; RST presence was a learned shortcut for Bot in the Original dataset.

Methods

  • Manual methodological analysis of raw CICIDS2017 and the CICFlowMeter source code across attack simulation, flow construction, labelling, feature extraction, and ML benchmarking, using the original paper, dataset website, and GitHub tool documentation.
  • Regenerated the dataset in three stages: Original, Intermediate (fix TCP appendices and RST termination), and Final (payload filter except Benign and PortScan); evaluated a Random Forest classifier on each with a 75–25 train-test split, assigning Attempted flows to BENIGN, and reported per-class precision/recall/F1 plus feature importance.
  • Dataset described as over 2.8 million traffic traces in 15 classes, with 80 extracted high-level statistical features; CICFlowMeter outputs 83 features excluding the label.

Limitations

  • Stated: the detailed TCP appendix timeout/direction-swap phenomenon is out of scope; results may differ with other ML models; some attack classes have extremely small test sets (Heartbleed, XSS, SQL, Infiltration); CSE-CIC-IDS2018 is only preliminarily investigated and needs further analysis; documentation on flow construction and attack parameters is lacking; diversity within attack categories is not openly discussed.
  • Evident: evaluation uses only a Random Forest classifier, so per-class benchmark improvements are model-specific; manual inspection may not exhaustively cover all flows/classes; Web Attack SQL Injection ends with 12 traces total and all 3 test traces mispredicted as Benign, indicating poor representation and distinguishability.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/spw53761.2021.00009

Cites

15 of 15 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

0 Valid · 5 Not valid · 12 Not checked · 59 Own results and interpretations

34 more claims

Create a free account to see all 76 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 25
  • Probably not a substantial claim. 16

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 16 references

Research works printed but not linked: 3

  1. 0, Extended documentation of the corrected CICFlowMeter generated CICIDS dataset and the source code of the paper
  2. 2017, Intrusion detection evaluation dataset (cicids2017)
  3. 2021, CICFlowMeter tool

Cited by

See all 398

Metadata

Authors
Engelen, G., Rimmer, V., & Joosen, W.
Affiliations
IMEC (BE)
Venue
2021 IEEE Security and Privacy Workshops (SPW)
Publisher
IEEE
Published
2021-05
Pages
7-12
OpenAlex
W3180062783
Cited by
398
References
16
Citations by year
2026: 191 · 2025: 85 · 2024: 58 · 2023: 41 · 2022: 20 · 2021: 3
FWCI
19.03
Type
conference paper
Language
en
Open access
Locations
  • IEEE Security and Privacy Workshops (SPW) (publishedVersion)
  • Lirias (KU Leuven) (submittedVersion, other-oa)
Funders
KU Leuven
Topics
  • Physical Sciences › Computer Science › Computer Networks and Communications › Network Security and Intrusion Detection
  • Physical Sciences › Computer Science › Artificial Intelligence › Internet Traffic Analysis and Secure E-voting
  • Physical Sciences › Computer Science › Hardware and Architecture › Network Packet Processing and Optimization
Keywords
CICIDS2017, network intrusion detection, data collection pipeline, traffic generation, feature extraction, dataset representativeness, data processing, model evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-11-16, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-11-16 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study. https://app.cochat.ai/papers/10.1109/spw53761.2021.00009
The paper
Engelen, G., Rimmer, V., & Joosen, W. (2021). Troubleshooting an Intrusion Detection Dataset: the CICIDS2017 Case Study. 2021 IEEE Security and Privacy Workshops (SPW). https://doi.org/10.1109/spw53761.2021.00009

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.