The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Data Quality: Some Comments on the NASA Software Defect Datasets
Abstract
Background--Self-evidently empirical analyses rely upon the quality of their data. Likewise, replications rely upon accurate reporting and using the same rather than similar versions of datasets. In recent years, there has been much interest in using machine learners to classify software modules into defect-prone and not defect-prone categories. The publicly available NASA datasets have been extensively used as part of this research. Objective--This short note investigates the extent to which published analyses based on the NASA defect datasets are meaningful and comparable. Method--We analyze the five studies published in the IEEE Transactions on Software Engineering since 2007 that have utilized these datasets and compare the two versions of the datasets currently in use. Results--We find important differences between the two versions of the datasets, implausible values in one dataset and generally insufficient detail documented on dataset preprocessing. Conclusions--It is recommended that researchers 1) indicate the provenance of the datasets they use, 2) report any preprocessing in sufficient detail to enable meaningful replication, and 3) invest effort in understanding the data prior to applying machine learners.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Lessmann, S., Baesens, B., Mues, C., & Pietsch, S. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2008.35 · Cited by 1,226
Data Mining Static Code Attributes to Learn Defect Predictors
Menzies, T., Greenwald, J., & Frank, A. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.256941 · Cited by 1,215
A Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154 · cited 1×
“[7] found more than a quarter of relevant defect prediction studies, that is, 58 out of 208, made use of the NASA datasets).”
MethodThe case for open computer programs
Ince, D. C., Hatton, L., & Graham-Cumming, J. (2012) · Nature · doi:10.1038/nature10836 · Cited by 535
A systematic review of software fault prediction studies
Catal, C., & Diri, B. (2009) · Expert Systems with Applications · doi:10.1016/j.eswa.2008.10.027 · Cited by 530
A General Software Defect-Proneness Prediction Framework
Song, Q., Jia, Z., Shepperd, M., Ying, S., & Liu, J. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.90 · Cited by 374
Self-citationThe misuse of the NASA Metrics Data Program data sets for automated software defect prediction
Gray, D., Bowes, D., Davey, N., Sun, Y., & Christianson, B. (2011) · 15th Annual Conference on Evaluation & Assessment in Software Engineering (EASE 2011) · doi:10.1049/ic.2011.0012 · Cited by 167
Evolutionary Optimization of Software Quality Modeling with Multiple Repositories
Liu, Y., Khoshgoftaar, T. M., & Seliya, N. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.51 · Cited by 152
Pramesti Cahyadewi, A. (2025) · Universitas Pasundan institutional repositories & scientific journals (Universitas Pasundan) · Cited by 145
Comments on "Data Mining Static Code Attributes to Learn Defect Predictors"
Zhang, H., & Zhang, X. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.70706 · Cited by 140
Fault Prediction using Early Lifecycle Data
Jiang, Y., Čukić, B., & Menzies, T. (2007) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2007.24 · Cited by 121
Data sets and data quality in software engineering
Liebchen, G., & Shepperd, M. (2008) · Predictive Models in Software Engineering · doi:10.1145/1370788.1370799 · Cited by 112
Self-citationThe Scientific Method in Practice: Reproducibility in the Computational SciencesPreprint
Stodden, V. (2010) · SSRN Electronic Journal · doi:10.2139/ssrn.1550193 · Cited by 111
Special issue on repeatable results in software engineering prediction
Menzies, T., & Shepperd, M. (2012) · Empirical Software Engineering · doi:10.1007/s10664-011-9193-5 · Cited by 107
Self-citationFinding conclusion stability for selecting the best effort predictor in software effort estimation
Keung, J., Kocagüneli, E., & Menzies, T. (2012) · Automated Software Engineering · doi:10.1007/s10515-012-0108-5 · Cited by 76
The Scientific Method in Practice: Reproducibility in the Computational SciencesPreprint
Stodden, V. (2010) · SSRN Electronic Journal · doi:10.7916/d8t15bnc · Cited by 74
A revival of integrity constraints for data cleaning
Fan, W., Geerts, F., & Jia, X. (2008) · Proceedings of the VLDB Endowment · doi:10.14778/1454159.1454220 · Cited by 46
Improving Credibility of Machine Learner Models in Software Engineering
Boetticher, G. D. (2007) · IGI Global eBooks · doi:10.4018/978-1-59140-941-1.ch003 · Cited by 27
Kaminsky, K., & Boetticher, G. D. (2004) · Annual Meeting of the North American Fuzzy Information Processing Society (NAFIPS) · doi:10.1109/nafips.2004.1336240 · Cited by 16
Data cleaning techniques for software engineering data sets
Liebchen, G. (2010) · OpenGrey (Institut de l'Information Scientifique et Technique) · Cited by 15
Data quality: cinderella at the software metrics ball?
Shepperd, M. (2011) · Workshop on Emerging Trends in Software Metrics · doi:10.1145/1985374.1985376 · Cited by 13
Self-citation
Cited by
An Empirical Comparison of Model Validation Techniques for Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2584050 · Cited by 574
Gai, K., Qiu, M., & Sun, X. (2018) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2017.10.011 · Cited by 566
The Impact of Automated Parameter Optimization on Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2794977 · Cited by 415
Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models
Ghotra, B., McIntosh, S., & Hassan, A. E. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.91 · Cited by 364
Researcher Bias: The Use of Machine Learning in Software Defect Prediction
Shepperd, M., Bowes, D., & Hall, T. (2014) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2322358 · Cited by 359
Self-citation
Metadata
- Authors
- Shepperd, M., Song, Q., Sun, Z., & Mair, C.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2013-09
- Volume
- 39
- Issue
- 9
- Pages
- 1208-1215
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2097883090
- Cited by
- 570
- References
- 21
- FWCI
- 84.80
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.