Data Quality Matters: A Case Study on Data Label Correctness for Security Bug Report Prediction
The DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
In the research of mining software repositories, we need to label a large amount of data to construct a predictive model. The correctness of the labels will affect the performance of a model substantially. However, limited studies have been performed to investigate the impact of mislabeled instances on a predictive model. To bridge the gap, in this article, we perform a case study on the security bug report (SBR) prediction. We found five publicly available datasets for SBR prediction contains many mislabeled instances, which lead to the poor performance of SBR prediction models of recent studies (e.g., the work of Peterset al.and Shuet al.). Furthermore, it might mislead the research direction of SBR prediction. In this article, we first improve the label correctness of these five datasets by manually analyzing each bug report, and we find 749 SBRs, which are originally mislabeled as Non-SBRs (NSBRs). We then evaluate the impacts of datasets label correctness by comparing the performance of the classification models on both the noisy (i.e., before our correction) and the clean (i.e., after our correction) datasets. The results show that the cleaned datasets result in improvement in the performance of classification models. The performance of the approaches proposed by Peterset al.and Shuet al.on the clean datasets is much better than on the noisy datasets. Furthermore, with the clean datasets, the simple text classification models could significantly outperform the security keywords-matrix-based approaches applied by Peterset al.and Shuet al.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
SMOTE synthetic minority over-sampling technique
Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002) · Journal of Artificial Intelligence Research · doi:10.1613/jair.953 · Cited by 32,865
Measuring nominal scale agreement among many raters.
Fleiss, J. L. (1971) · Psychological Bulletin · doi:10.1037/h0031619 · Cited by 8,733 · cited 1×
“We compute the Fleiss’s Kappa Coefficient [52], [55] to measure the agreement among the three annotators of each dataset.”
MethodDifferential Evolution: A Survey of the State-of-the-Art
Das, S., & Suganthan, P. N. (2011) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2010.2059031 · Cited by 5,307
Feurer, M., & Hutter, F. (2019) · The Springer Series on Challenges in Machine Learning · doi:10.1007/978-3-030-05318-5_1 · Cited by 1,414
Gan, Y., Zhang, Y., Cheng, D., Shetty, A., Rathi, P., Katarki, N., Bruno, A., Hu, J., Ritchken, B., Jackson, B., et al. (2019) · Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems · doi:10.1145/3297858.3304013 · Cited by 665 · cited 1×
“Data annotation is an extremely time-consuming work [24], [40], [41].”
BackgroundZhou, X., Peng, X., Xie, T., Sun, J., Ji, C., Li, W., & Ding, D. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2887384 · Cited by 410 · cited 1×
“Data annotation is an extremely time-consuming work [24], [40], [41].”
BackgroundRevisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models
Ghotra, B., McIntosh, S., & Hassan, A. E. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.91 · Cited by 364 · cited 1×
“Furthermore, the incorrectly labeled data may translate to inaccurate results that may in turn mislead the research direction of the target problem [5], [6], [7], [8], [9], [10].”
BackgroundTowards a Big Data Curated Benchmark of Inter-project Code Clones
Svajlenko, J., Islam, J. F., Keivanloo, I., Roy, C. K., & Mia, M. M. (2014) · 2014 IEEE International Conference on Software Maintenance and Evolution · doi:10.1109/icsme.2014.77 · Cited by 339
Data scientists in software teams: State of the art and challenges
Kim, M., Zimmermann, T., DeLine, R., & Begel, A. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2754374 · Cited by 199
Evaluating clone detection tools with BigCloneBench
Svajlenko, J., & Roy, C. K. (2015) · 2015 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsm.2015.7332459 · Cited by 172 · cited 1×
“[69], [70] built a large-scale benchmark for clone detection by mining and then manually checking clones of ten common functionalities.”
Background
Cited by
INFO: An efficient optimization algorithm based on weighted mean of vectors
Ahmadianfar, I., Heidari, A. A., Noshadian, S., Chen, H., & Gandomi, A. H. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.116516 · Cited by 822
Deepfake detection using deep learning methods: A systematic and comprehensive review
Heidari, A., Jafari Navimipour, N., Dag, H., & Unal, M. (2024) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1520 · Cited by 349
“For instance, security agencies could use this strategy to sway influential people's judgments, including politicians, posing national and international security risks (Wu et al., 2021; Z. Zhang, Luo, et al., 2020; Wei Zheng, Xun, Wu, et al., 2021).”
BackgroundA Comparative Study of Class Rebalancing Methods for Security Bug Report Classification
Zheng, W., Xun, Y., Wu, X., Deng, Z., Chen, X., & Sui, Y. (2021) · IEEE Transactions on Reliability · doi:10.1109/tr.2021.3118026 · Cited by 85
Self-citationInsights for variant clinical interpretation based on a benchmark of 65 variant effect predictors
Radjasandirane, R., Diharce, J., Gelly, J.-C., & de Brevern, A. G. (2025) · Genomics · doi:10.1016/j.ygeno.2025.111036 · Cited by 15
Metadata
- Authors
- Wu, X., Zheng, W., Xia, X., & Lo, D.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2022-07-01
- Volume
- 48
- Issue
- 7
- Pages
- 2541-2556
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W3134770171
- Cited by
- 132
- References
- 10
- FWCI
- 23.64
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Data Quality Matters: A Case Study on Data Label Correctness for Security Bug Report Prediction. https://app.cochat.ai/papers/10.1109/tse.2021.3063727
- The paper
- Wu, X., Zheng, W., Xia, X., & Lo, D. (2022). Data Quality Matters: A Case Study on Data Label Correctness for Security Bug Report Prediction. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2021.3063727
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.