The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Data Quality: Some Comments on the NASA Software Defect Datasets
Abstract
Background--Self-evidently empirical analyses rely upon the quality of their data. Likewise, replications rely upon accurate reporting and using the same rather than similar versions of datasets. In recent years, there has been much interest in using machine learners to classify software modules into defect-prone and not defect-prone categories. The publicly available NASA datasets have been extensively used as part of this research. Objective--This short note investigates the extent to which published analyses based on the NASA defect datasets are meaningful and comparable. Method--We analyze the five studies published in the IEEE Transactions on Software Engineering since 2007 that have utilized these datasets and compare the two versions of the datasets currently in use. Results--We find important differences between the two versions of the datasets, implausible values in one dataset and generally insufficient detail documented on dataset preprocessing. Conclusions--It is recommended that researchers 1) indicate the provenance of the datasets they use, 2) report any preprocessing in sufficient detail to enable meaningful replication, and 3) invest effort in understanding the data prior to applying machine learners.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Lessmann, S., Baesens, B., Mues, C., & Pietsch, S. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2008.35 · Cited by 1,226
Data Mining Static Code Attributes to Learn Defect Predictors
Menzies, T., Greenwald, J., & Frank, A. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.256941 · Cited by 1,215
A Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154 · cited 1×
“[7] found more than a quarter of relevant defect prediction studies, that is, 58 out of 208, made use of the NASA datasets).”
MethodThe case for open computer programs
Ince, D. C., Hatton, L., & Graham-Cumming, J. (2012) · Nature · doi:10.1038/nature10836 · Cited by 535
A systematic review of software fault prediction studies
Catal, C., & Diri, B. (2009) · Expert Systems with Applications · doi:10.1016/j.eswa.2008.10.027 · Cited by 530
A General Software Defect-Proneness Prediction Framework
Song, Q., Jia, Z., Shepperd, M., Ying, S., & Liu, J. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.90 · Cited by 374
Self-citationThe misuse of the NASA Metrics Data Program data sets for automated software defect prediction
Gray, D., Bowes, D., Davey, N., Sun, Y., & Christianson, B. (2011) · 15th Annual Conference on Evaluation & Assessment in Software Engineering (EASE 2011) · doi:10.1049/ic.2011.0012 · Cited by 167
Evolutionary Optimization of Software Quality Modeling with Multiple Repositories
Liu, Y., Khoshgoftaar, T. M., & Seliya, N. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.51 · Cited by 152
Pramesti Cahyadewi, A. (2025) · Universitas Pasundan institutional repositories & scientific journals (Universitas Pasundan) · Cited by 145
Comments on "Data Mining Static Code Attributes to Learn Defect Predictors"
Zhang, H., & Zhang, X. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.70706 · Cited by 140
Cited by
An Empirical Comparison of Model Validation Techniques for Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2584050 · Cited by 574
Gai, K., Qiu, M., & Sun, X. (2018) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2017.10.011 · Cited by 566
The Impact of Automated Parameter Optimization on Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2794977 · Cited by 415
Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models
Ghotra, B., McIntosh, S., & Hassan, A. E. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.91 · Cited by 364
Researcher Bias: The Use of Machine Learning in Software Defect Prediction
Shepperd, M., Bowes, D., & Hall, T. (2014) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2322358 · Cited by 359
Self-citationAutomated parameter optimization of classification techniques for defect prediction models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884857 · Cited by 331
Tantithamthavorn, C., Hassan, A. E., & Matsumoto, K. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2876537 · Cited by 327
A Systematic Literature Review and Meta-Analysis on Cross Project Defect Prediction
Hosseini, S., Turhan, B., & Gunarathna, D. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2770124 · Cited by 317
Generative Adversarial Networks for face generation: A survey
Kammoun, A., Slama, R., Tabia, H., Ouni, T., & Abid, M. (2022) · ACM Computing Surveys · doi:10.1145/1122445.1122456 · Cited by 311
A Comprehensive Investigation of the Role of Imbalanced Learning for Software Defect Prediction
Song, Q., Guo, Y., & Shepperd, M. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2836442 · Cited by 295
Self-citationHeterogeneous Defect Prediction
Nam, J., Fu, W., Kim, S., Menzies, T., & Tan, L. (2018) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2017.2720603 · Cited by 277
Experience with using the Parallel Workloads Archive
Feitelson, D. G., Tsafrir, D., & Krakov, D. (2014) · Journal of Parallel and Distributed Computing · doi:10.1016/j.jpdc.2014.06.013 · Cited by 276
Progress on approaches to software defect prediction
Li, Z., Jing, X.-Y., & Zhu, X. (2018) · IET Software · doi:10.1049/iet-sen.2017.0148 · Cited by 264
SEGRESS: Software Engineering Guidelines for REporting Secondary Studies
Kitchenham, B., Madeyski, L., & Budgen, D. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3174092 · Cited by 219
Metadata
- Authors
- Shepperd, M., Song, Q., Sun, Z., & Mair, C.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2013-09
- Volume
- 39
- Issue
- 9
- Pages
- 1208-1215
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2097883090
- Cited by
- 570
- References
- 21
- FWCI
- 84.80
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.