The DOI matches its Crossref record; 44 of the 44 references it cites were found in a registry or an index.
A critique of software defect prediction models
Abstract
Many organizations want to predict the number of defects (faults) in software systems, before they are deployed, to gauge the likely delivered quality and maintenance effort. To help in this numerous software metrics and statistical models have been developed, with a correspondingly large literature. We provide a critical review of this literature and the state-of-the-art. Most of the wide range of prediction models use size and complexity metrics to predict defects. Others are based on testing data, the "quality" of the development process, or take a multivariate approach. The authors of the models have often made heroic contributions to a subject otherwise bereft of empirical studies. However, there are a number of serious theoretical and practical problems in many studies. The models are weak because of their inability to cope with the, as yet, unknown relationship between defects and failures. There are fundamental statistical and data quality problems that undermine model validity. More significantly many prediction models tend to model only part of the underlying problem and seriously misspecify it. To illustrate these points the Goldilock's Conjecture, that there is an optimum module size, is used to show the considerable problems inherent in current defect prediction approaches. Careful and considered analysis of past and new results shows that the conjecture lacks support and that some models are misleading. We recommend holistic models for software defect prediction, using Bayesian belief networks, as alternative approaches to the single-issue models used at present. We also argue for research into a theory of "software decomposition" in order to test hypotheses about defect introduction and help construct a better science of software engineering.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
McCabe, T. J. (1976) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1976.233837 · Cited by 6,059
A metrics suite for object oriented design
Chidamber, S. R., & Kemerer, C. F. (1994) · IEEE Transactions on Software Engineering · doi:10.1109/32.295895 · Cited by 5,669
Multivariate Statistical Methods
(2008) · Encyclopedia of Public Health · doi:10.1007/978-1-4020-5614-7_2263 · Cited by 5,580
Lauritzen, S. L., & Spiegelhalter, D. J. (1988) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1988.tb01721.x · Cited by 3,902
Halstead, M. H. (1977) · CERN Document Server (European Organization for Nuclear Research) · Cited by 3,639
Software Metrics: A Rigorous and Practical Approach
Fenton, N., & Pfleeger, S. L. (2013) · Cited by 2,509
Self-citationMultivariate Statistical Methods: A Primer.
RAJ, & Manly, B. F. J. (1996) · Journal of the American Statistical Association · doi:10.2307/2291702 · Cited by 2,451
Halstead, M. H. (1977) · CERN Document Server (European Organization for Nuclear Research) · Cited by 2,339
Humphrey, W. S. (1989) · CERN Document Server (European Organization for Nuclear Research) · Cited by 2,005
A validation of object-oriented design metrics as quality indicators
Basili, V. R., Briand, L. C., & Melo, W. L. (1996) · IEEE Transactions on Software Engineering · doi:10.1109/32.544352 · Cited by 1,726
W, L. (1989) · Journal of Information Technology · doi:10.1057/jit.1989.28 · Cited by 1,378
Software Structure Metrics Based on Information Flow
Henry, S., & Kafura, D. (1981) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1981.231113 · Cited by 782
Quantitative analysis of faults and failures in a complex software system
Fenton, N. E., & Ohlsson, N. (2000) · IEEE Transactions on Software Engineering · doi:10.1109/32.879815 · Cited by 699
Self-citationSoftware errors and complexity: an empirical investigation0
Basili, V. R., & Perricone, B. T. (1984) · Communications of the ACM · doi:10.1145/69605.2085 · Cited by 634
Software measurement: a necessary scientific basis
Fenton, N. (1994) · IEEE Transactions on Software Engineering · doi:10.1109/32.268921 · Cited by 501
Self-citationPractical Software Metrics for Project Management and Process Improvement
Grady, R. B. (1992) · Cited by 442
The detection of fault-prone programs
Munson, J. C., & Khoshgoftaar, T. M. (1992) · IEEE Transactions on Software Engineering · doi:10.1109/32.135775 · Cited by 442
Optimizing Preventive Service of Software Products
Adams, E. N. (1984) · IBM Journal of Research and Development · doi:10.1147/rd.281.0002 · Cited by 373
Practical software metrics for project management and process improvement
Shepperd, M. (1993) · Information and Software Technology · doi:10.1016/0950-5849(93)90091-g · Cited by 354
Predicting fault-prone software modules in telephone switches
Ohlsson, N., & Alberg, H. (1996) · IEEE Transactions on Software Engineering · doi:10.1109/32.553637 · Cited by 346
Software testability: the new verification
Voas, J. M., & Miller, K. W. (1995) · IEEE Software · doi:10.1109/52.382180 · Cited by 327
Science and substance: a challenge to software engineers
Fenton, N., Pfleeger, S. L., & Glass, R. L. (1994) · IEEE Software · doi:10.1109/52.300094 · Cited by 265
Self-citationMaintenance metrics for the object oriented paradigm
Li, W., & Henry, S. (2002) · [1993] Proceedings First International Software Metrics Symposium · doi:10.1109/metric.1993.263801 · Cited by 264
How software process improvement helped Motorola
Diaz, M., & Sligo, J. (1997) · IEEE Software · doi:10.1109/52.605934 · Cited by 255
A critique of cyclomatic complexity as a software metric
Shepperd, M. (1988) · Software Engineering Journal · doi:10.1049/sej.1988.0003 · Cited by 240
Predicting software development errors using software complexity metrics
Khoshgoftaar, T. M., & Munson, J. C. (1990) · IEEE Journal on Selected Areas in Communications · doi:10.1109/49.46879 · Cited by 239
Identifying Error-Prone Software—An Empirical Study
Shen, V. Y., Yu, T.-J., Thebaut, S. M., & Paulsen, L. R. (1985) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1985.232222 · Cited by 209
An Example of Software System Debugging.
Akiyama, F. (1971) · IFIP Congress · Cited by 204
Reexamining the fault density component size connection
Hatton, L. (1997) · IEEE Software · doi:10.1109/52.582978 · Cited by 162
Some misconceptions about lines of code
Rosenberg, J. (2002) · IEEE International Software Metrics Symposium (METRICS) · doi:10.1109/metric.1997.637174 · Cited by 159
Cusumano, M. A. (1991) · doi:10.1093/oso/9780195062168.001.0001 · Cited by 157
Number of Faults per Line of Code
Lipow, M. (1982) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1982.235579 · Cited by 151
Japan's Software Factories: A Challenge to U.S. Management
Anchordoguy, M., & Cusumano, M. A. (1992) · Journal of Japanese Studies · doi:10.2307/132850 · Cited by 147
Remoortere, P. V. (1979) · Mathematics and Computers in Simulation · doi:10.1016/0378-4754(79)90027-2 · Cited by 142
Estimating the Number of Faults in Code
Gaffney, J. E. (1984) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1984.5010260 · Cited by 138
Software Science Revisited: A Critical Analysis of the Theory and Its Empirical Support
Shen, V. Y., Conte, S. D., & Dunsmore, H. E. (1983) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1983.236460 · Cited by 128
An Experiment in Software Error Data Collection and Analysis
Schneidewind, N. F., & Hoffmann, H.-M. (1979) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1979.234188 · Cited by 126
Prediction and control of ADA software defects
Compton, B. T., & Withrow, C. (1990) · Journal of Systems and Software · doi:10.1016/0164-1212(90)90040-s · Cited by 120
A case history analysis of software error cause-effect relationships
Nakajo, T., & Kume, H. (1991) · IEEE Transactions on Software Engineering · doi:10.1109/32.83917 · Cited by 106
Regression modelling of software quality: empirical investigation
Munson, J. C., & Khoshgoftaar, Y. M. (1990) · Information and Software Technology · doi:10.1016/0950-5849(90)90109-5 · Cited by 90
The evaluation of software systems' structure using quantitative software metrics
Henry, S. M., & Kafura, D. G. (1984) · Software Practice and Experience · doi:10.1002/spe.4380140606 · Cited by 89
New ways to get accurate reliability measures (software)
Brocklehurst, S., & Bev Littlewood, A. (1992) · IEEE Software · doi:10.1109/52.143100 · Cited by 84
M. H. Halstead's software science: a critical examination
Hamer, P. G., & Frewin, G. D. (1982) · doi:10.5555/800254.807762 · Cited by 76
Quantitative Estimates of Debugging Requirements
Ottenstein, L. M. (1979) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1979.234211 · Cited by 51
Cited by
Lessmann, S., Baesens, B., Mues, C., & Pietsch, S. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2008.35 · Cited by 1,226
A Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154
“Fenton and Neil conducted a critical review of software fault prediction research up to 1999 [1].”
Mining metrics to predict component failures
Nagappan, N., Ball, T., & Zeller, A. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134349 · Cited by 782
Moser, R., Pedrycz, W., & Succi, G. (2008) · Proceedings of the 13th international conference on Software engineering - ICSE '08 · doi:10.1145/1368088.1368114 · Cited by 701
Quantitative analysis of faults and failures in a complex software system
Fenton, N. E., & Ohlsson, N. (2000) · IEEE Transactions on Software Engineering · doi:10.1109/32.879815 · Cited by 699
Self-citation
Metadata
- Authors
- Fenton, N. E., & Neil, M.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 1999
- Volume
- 25
- Issue
- 5
- Pages
- 675-689
- ISSN
- 0098-5589 (ISSN-L)
- OpenAlex
- W2101728371
- Cited by
- 1,040
- References
- 44
- FWCI
- 46.58
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.