CoChat
Sign Up
Cited by 364Software Engineering ResearchSoftware Reliability and Analysis ResearchSoftware Engineering Techniques and Practices

The DOI matches its Crossref record; its reference list is not available.

Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models

Ghotra, Baljinder, McIntosh, Shane, Hassan, Ahmed E.·2015·2015 IEEE/ACM 37th IEEE International Conference on Software Engineering
DOI 10.1109/icse.2015.91
Open this paper in CoChat — free account

Abstract

Defect prediction models help software quality assurance teams to effectively allocate their limited resources to the most defect-prone software modules. A variety of classification techniques have been used to build defect prediction models ranging from simple (e.g., Logistic regression) to advanced techniques (e.g., Multivariate Adaptive Regression Splines (MARS)). Surprisingly, recent research on the NASA dataset suggests that the performance of a defect prediction model is not significantly impacted by the classification technique that is used to train it. However, the dataset that is used in the prior study is both: (a) noisy, i.e., Contains erroneous entries and (b) biased, i.e., Only contains software developed in one setting. Hence, we set out to replicate this prior study in two experimental settings. First, we apply the replicated procedure to the same (known-to-be noisy) NASA dataset, where we derive similar results to the prior study, i.e., The impact that classification techniques have appear to be minimal. Next, we apply the replicated procedure to two new datasets: (a) the cleaned version of the NASA dataset and (b) the PROMISE dataset, which contains open source software developed in a variety of settings (e.g., Apache, GNU). The results in these new datasets show a clear, statistically distinct separation of groups of techniques, i.e., The choice of classification technique has an impact on the performance of defect prediction models. Indeed, contrary to earl…

Overview

Integrity signals

Registry

Crossref · 10.1109/icse.2015.91

Cites

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 64 · Cited by 364

Research works printed but not linked: 62

  1. 10.1109/TSE.2012.45 · DOI
  2. 10.1007/s10515-010-0069-5 · DOI
  3. lemon, 2010, The effect of locality based learning on software defect prediction, 10.33915/etd.3018 · DOI
  4. 10.1007/s10994-005-0466-3 · DOI
  5. kotsiantis, 2007, Supervised machine learning: A review of classification techniques, Informatica, 31, 249
  6. 10.1109/TSE.2008.90 · DOI
  7. 10.1109/TSE.2007.256941 · DOI
  8. menzies, 2002, Metrics that matter, Proceedings of the 27th Annual NASA Goddard/IEEE Software Engineering Workshop (SEW- 27 '02), 51
  9. 10.1109/ASE.2011.6100072 · DOI
  10. 10.1109/TSE.2008.35 · DOI
  11. 10.1109/TSE.2003.1191795 · DOI
  12. witten, 2005, Data Mining Practical Machine Learning Tools and Techniques
  13. wang, 2011, Software defect prediction based on classifiers ensemble, Information and Computer Science, 8, 4241
  14. kaur, 2009, Static and dynamic complexity analysis of software metrics, World Academy of Science Engineering and Technology, 56, 2009
  15. zhou, 2009, Ensemble learning, Encyclopedia of Biometrics, 270, 10.1007/978-0-387-73003-5_293 · DOI
  16. 10.1109/ASE.2013.6693087 · DOI
  17. 10.1109/PROMISE.2007.10 · DOI
  18. 10.1145/2635868.2635909 · DOI
  19. 10.1109/32.544352 · DOI
  20. 10.1145/1159733.1159738 · DOI
  21. 10.1109/TSE.2005.112 · DOI
  22. hamerly, 2004, Learning the k in k-means, Advances in neural information processing systems, 281
  23. 10.1109/TSE.2011.103 · DOI · Paper page
  24. 10.1109/ICSE.2009.5070510 · DOI
  25. hammouda, 2000, A comparative study of data clustering techniques
  26. jelihovschi, 2014, Scottknott: A package for performing the scott-knott clustering algorithm in r, Trends in Applied and Computational Mathematics, 15, 3
  27. ho, 1998, The random subspace method for constructing decision forests, IEEE Transactions on Pattern Analysis and Machine Intelligence, 20, 832, 10.1109/34.709601 · DOI
  28. quinlan, 1993, C4 5 Programs for Machine Learning, 1
  29. 10.1145/1370750.1370759 · DOI
  30. singh, 2009, Neural networks in data mining, Journal of Theoretical and Applied Information Technology, 5, 36
  31. 10.1109/TSE.2013.11 · DOI
  32. 10.1109/TSE.2014.2322358 · DOI
  33. shaik, 2010, Metrics for object oriented design software systems: A survey, Journal of Emerging Trends in Engineering and Applied Sciences, 1, 190
  34. sehgal, 2012, Quality prediction of function based software using decision tree approach, Proceedings of the International Conference on Computer Engineering and Multimedia Technologies (ICCEMT), 43
  35. 10.3844/jcssp.2007.281.288 · DOI
  36. 10.1109/TPAMI.2006.211 · DOI
  37. 10.1016/B978-1-55860-377-6.50023-2 · DOI
  38. 10.1109/TIT.1967.1053964 · DOI · Paper page
  39. 10.1145/1368088.1368114 · DOI
  40. 10.1007/s10664-011-9173-9 · DOI
  41. demšar, 2006, Statistical comparisons of classifiers over multiple data sets, The Journal of Machine Learning Research, 7, 1
  42. 10.1023/A:1007607513941 · DOI
  43. 10.1109/6294.846201 · DOI
  44. 10.1007/s00357-007-0004-5 · DOI
  45. freund, 1996, Experiments with a New Boosting Algorithm, Proceedings of the International Conference on Machine Learning, 148
  46. 10.1007/BF00962234 · DOI
  47. 10.1007/978-3-540-79187-4_22 · DOI
  48. berson, 2004, An overview of data mining techniques, Building Data Mining Application for CRM
  49. 10.1145/1294948.1294953 · DOI
  50. 10.1109/MSR.2012.6224300 · DOI
  51. 10.1017/S0962492900000015 · DOI
  52. 10.1023/A:1009815306478 · DOI
  53. platt, 1999, Fast training of support vector machines using sequential minimal optimization, Advances in Kernel Methods, 185
  54. 10.1109/32.295895 · DOI
  55. 10.1109/TSE.2005.49 · DOI
  56. 10.1109/MOMPES.2008.11 · DOI
  57. 10.1109/MSR.2013.6624057 · DOI
  58. 10.1109/CSMR-WCRE.2014.6747166 · DOI
  59. 2005, Use of relative code churn measures to predict system defect density, Proceedings of the 27th International Conference on Software Engineering, 284
  60. nagappan, 2005, Static analysis tools as early indicators of pre-release defect density, Proc of International Conference on Software Engi-neering, 580
  61. 10.1109/32.553637 · DOI
  62. 10.1145/1134285.1134349 · DOI

Metadata

Authors
Ghotra, B., McIntosh, S., & Hassan, A. E.
Affiliations
Queen's University (CA)
ORCID
Venue
2015 IEEE/ACM 37th IEEE International Conference on Software Engineering
Publisher
IEEE
Published
2015-05
Pages
789-800
ISSN
0270-5257 (ISSN-L)
DOI
10.1109/icse.2015.91
OpenAlex
W4250023757
Cited by
364
References
64
Citations by year
2026: 13 · 2025: 20 · 2024: 24 · 2023: 38 · 2022: 45 · 2021: 55 · 2020: 36 · 2019: 46 · 2018: 39 · 2017: 25 · 2016: 16 · 2015: 7
FWCI
25.20
Type
conference paper
Language
en
Open access
closed
Locations
ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion)
Funders
Natural Sciences and Engineering Research Council of Canada
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
Keywords
software defect prediction, classification methods, model performance comparison, NASA dataset, PROMISE dataset, noisy data, data bias, open source software, software quality assurance, replication

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review