The DOI matches its Crossref record; 64 of the 64 references it cites were found in a registry or an index.
Automated parameter optimization of classification techniques for defect prediction models
Abstract
Defect prediction models are classifiers that are trained to identify defect-prone software modules. Such classifiers have configurable parameters that control their characteristics (e.g., the number of trees in a random forest classifier). Recent studies show that these classifiers may underperform due to the use of suboptimal default parameter settings. However, it is impractical to assess all of the possible settings in the parameter spaces. In this paper, we investigate the performance of defect prediction models where Caret --- an automated parameter optimization technique --- has been applied. Through a case study of 18 datasets from systems that span both proprietary and open source domains, we find that (1) Caret improves the AUC performance of defect prediction models by as much as 40 percentage points; (2) Caret-optimized classifiers are at least as stable as (with 35% of them being more stable than) classifiers that are trained using the default settings; and (3) Caret increases the likelihood of producing a top-performing classifier by as much as 83%. Hence, we conclude that parameter settings can indeed have a large impact on the performance of defect prediction models, suggesting that researchers should experiment with the parameters of the classification techniques. Since automated parameter optimization techniques like Caret yield substantially benefits in terms of performance improvement and stability, while incurring a manageable additional computational cost, they should be included in future defect prediction studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2025) · doi:10.32614/r.manuals · Cited by 352,181
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1994) · doi:10.1201/9780429246593 · Cited by 40,500
An Introduction to the Bootstrap
Efron, B., & Tibshirani, R. J. (1993) · doi:10.1007/978-1-4899-4541-9 · Cited by 21,230
The WEKA data mining software: an update
Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., & Witten, I. H. (2009) · ACM SIGKDD Explorations Newsletter · doi:10.1145/1656274.1656278 · Cited by 17,750 · cited 4×
“For example, for the number of decision trees of the random forest technique, the default settings vary from 10 for the bigrf R package [34], 50 for MATLAB [35], 100 for Weka [14], to 500 for the randomForest R package [33].”
BackgroundMethodInfluentialHe, H., & Garcia, E. A. (2009) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2008.239 · Cited by 10,359
A simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655
Harrell, F. E. (2001) · Springer Series in Statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,195
Building Predictive Models in R Using the caret Package
Kuhn, M. (2008) · Journal of Statistical Software · doi:10.18637/jss.v028.i05 · Cited by 8,863
Cohen, J. (1992) · Psychological Bulletin · doi:10.1037//0033-2909.112.1.155 · Cited by 8,771
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
(2012) · Cited by 8,009
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878
Estimating the error rate of a prediction rule: improvement on cross-validation
Efron, B. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477973 · Cited by 2,300
caret: Classification and Regression Training
Kuhn, M. (2007) · CRAN: Contributed Packages · doi:10.32614/cran.package.caret · Cited by 1,786
Graph-based approach for airborne light detection and ranging segmentation
Vilariño, D. L., Cabaleiro, J. C., Martínez, J., Rivera, F. F., & Pena, T. F. (2017) · Journal of Applied Remote Sensing · doi:10.1117/1.jrs.11.015020 · Cited by 1,721
Lessmann, S., Baesens, B., Mues, C., & Pietsch, S. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2008.35 · Cited by 1,226
A Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154 · cited 2×
“Nevertheless, we find that the optimal #boosting iterations parameter is 40, suggesting that the default parameter settings of the research toolkits are suboptimal for defect prediction datasets, which agrees with the suspicion of prior studies [15, 22, 51, 58].”
BackgroundResultneuralnet: Training of Neural Networks
Günther, F., & Fritsch, S. (2010) · The R Journal · doi:10.32614/rj-2010-006 · Cited by 757
Predicting Defects for Eclipse
Zimmermann, T., Premraj, R., & Zeller, A. (2007) · Third International Workshop on Predictor Models in Software Engineering (PROMISE'07: ICSE Workshops 2007) · doi:10.1109/promise.2007.10 · Cited by 753
A systematic review of software development cost estimation studies
Jørgensen, M., & Shepperd, M. (2007) · Brunel University Research Archive (BURA) (Brunel University London) · doi:10.1109/tse.2007.3 · Cited by 742
Cross-project defect prediction: a large scale experiment on data vs. domain vs. process
Zimmermann, T., Nagappan, N., Gall, H. C., Giger, E., & Murphy, B. (2009) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-25785 · Cited by 694
Random search for hyper-parameter optimization
BergstraJames, & BengioYoshua (2012) · Journal of Machine Learning Research · doi:10.5555/2188385.2188395 · Cited by 690
The PROMISE Repository of Empirical Software Engineering Data
Boetticher, G. D. (2007) · Medical Entomology and Zoology · Cited by 682
An extensive comparison of bug prediction approaches
D'Ambros, M., Lanza, M., & Robbes, R. (2010) · 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010) · doi:10.1109/msr.2010.5463279 · Cited by 637
Evaluating defect prediction approaches: a benchmark and an extensive comparison
D’Ambros, M., Lanza, M., & Robbes, R. (2012) · Empirical Software Engineering · doi:10.1007/s10664-011-9173-9 · Cited by 626
Is cross-validation valid for small-sample microarray classification?
Braga-Neto, U. M., & Dougherty, E. R. (2004) · Bioinformatics · doi:10.1093/bioinformatics/btg419 · Cited by 597
An Empirical Comparison of Model Validation Techniques for Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2584050 · Cited by 574
Self-citationData Quality: Some Comments on the NASA Software Defect Datasets
Shepperd, M., Song, Q., Sun, Z., & Mair, C. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2013.11 · Cited by 570
Towards identifying software project clusters with regard to defect prediction
Jureczko, M., & Madeyski, L. (2010) · Proceedings of the 6th International Conference on Predictive Models in Software Engineering · doi:10.1145/1868328.1868342 · Cited by 546
ReLink: recovering links between bugs and changes
Wu, R., Zhang, H., Kim, S., & Cheung, S.-C. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025120 · Cited by 412
Software defect prediction using ensemble learning on selected features
Laradji, I. H., Alshayeb, M., & Ghouti, L. (2015) · Information and Software Technology · doi:10.1016/j.infsof.2014.07.005 · Cited by 380
A General Software Defect-Proneness Prediction Framework
Song, Q., Jia, Z., Shepperd, M., Ying, S., & Liu, J. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.90 · Cited by 374
Researcher Bias: The Use of Machine Learning in Software Defect Prediction
Shepperd, M., Bowes, D., & Hall, T. (2014) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2322358 · Cited by 359
Dealing with noise in defect prediction
Kim, S., Zhang, H., Wu, R., & Gong, L. (2011) · Proceedings of the 33rd International Conference on Software Engineering · doi:10.1145/1985793.1985859 · Cited by 330
Techniques for evaluating fault prediction models
Jiang, Y., Cukic, B., & Ma, Y. (2008) · Empirical Software Engineering · doi:10.1007/s10664-008-9079-3 · Cited by 267
Reliability and validity in comparative studies of software prediction models
Myrtveit, I., Stensrud, E., & Shepperd, M. (2005) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2005.58 · Cited by 259
Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models
Ghotra, B., McIntosh, S., & Hassan, A. E. (2015) · Cited by 256
Self-citationSearch Based Software Engineering: Techniques, Taxonomy, Tutorial
Harman, M., McMinn, P., de Souza, J. T., & Yoo, S. (2012) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-25231-0_1 · Cited by 245
Ranking and Clustering Software Cost Estimation Models through a Multiple Comparisons Algorithm
Mittas, N., & Angelis, L. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.45 · Cited by 234
Exploiting the Essential Assumptions of Analogy-Based Effort Estimation
Kocaguneli, E., Menzies, T., Bener, A., & Keung, J. W. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.27 · Cited by 200
How, and Why, Process Metrics are better
Rahman, F., & Dévanbu, P. (2013) · Cited by 184
Revisiting the evaluation of defect prediction models
Mende, T., & Koschke, R. (2009) · Proceedings of the 5th International Conference on Predictor Models in Software Engineering · doi:10.1145/1540438.1540448 · Cited by 179
The Impact of Mislabelling on the Performance and Interpretation of Defect Prediction Models
Tantithamthavorn, C. K., McIntosh, S., Hassan, A. E., Ihara, A., & Matsumoto, K. (2015) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2015.93 · Cited by 169
Self-citationOn the dataset shift problem in software engineering prediction models
Turhan, B. (2011) · Empirical Software Engineering · doi:10.1007/s10664-011-9182-8 · Cited by 132
Special issue on repeatable results in software engineering prediction
Menzies, T., & Shepperd, M. (2012) · Empirical Software Engineering · doi:10.1007/s10664-011-9193-5 · Cited by 122
Can data transformation help in the detection of fault-prone modules?
Jiang, Y., Čukić, B., & Menzies, T. (2008) · doi:10.1145/1390817.1390822 · Cited by 104
Feed-Forward Neural Networks and Multinomial Log-Linear Models
Ripley, B. D. (2015) · Cited by 98
An investigation of the effect of module size on defect prediction using static measures
Koru, A. G., & Liu, H. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083172 · Cited by 92
The impact of parameter tuning on software effort estimation using learning machines
Song, L., Minku, L. L., & Yao, X. (2013) · doi:10.1145/2499393.2499394 · Cited by 86
Finding conclusion stability for selecting the best effort predictor in software effort estimation
Keung, J., Kocagüneli, E., & Menzies, T. (2012) · Automated Software Engineering · doi:10.1007/s10515-012-0108-5 · Cited by 85
Predicting Subsystem Failures using Dependency Graph Complexities
Zimmermann, T., & Nagappan, N. (2007) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2007.19 · Cited by 78
Comments on “Researcher Bias: The Use of Machine Learning in Software Defect Prediction”
Tantithamthavorn, C. K., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2553030 · Cited by 76
Self-citationReducing false alarms in software defect prediction by decision threshold optimization
Tosun, A., & Bener, A. (2009) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2009.5316006 · Cited by 67
Software Defect Prediction Using Regression via Classification
Bibi, S., Tsoumakas, G., Stamelos, I. G., & Vlahvas, I. (2006) · IEEE/ACS International Conference on Computer Systems and Applications (AICCSA) · doi:10.1109/aiccsa.2006.205110 · Cited by 61
Replication of defect prediction studies
Mende, T. (2010) · doi:10.1145/1868328.1868336 · Cited by 58
Cross-project defect prediction: a large scale experiment on data vs. domain vs. process
Zimmermann, T., Nagappan, N., Gall, H. C., Giger, E., & Murphy, T. B. (2009) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1595696.1595713 · Cited by 57
Tantithamthavorn, C. K. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2889160.2889256 · Cited by 28
Self-citationEvaluating Defect Prediction Models for a Large Evolving Software System
Mende, T., Koschke, R., & Leszak, M. (2009) · Euromicro Conference on Software Maintenance and Reengineering (CSMR) · doi:10.1109/csmr.2009.55 · Cited by 26
Comparing negative binomial and recursive partitioning models for fault prediction
Weyuker, E. J., Ostrand, T. J., & Bell, R. M. (2008) · Predictive Models in Software Engineering · doi:10.1145/1370788.1370792 · Cited by 19
Search-Based Software Engineering
Goues, C. L., & Yoo, S. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-09940-8 · Cited by 7
(2014) · doi:10.5555/2627435.2697065
Cited by
Flood Prediction Using Machine Learning Models: Literature Review
Mosavi, A., Ozturk, P., & Chau, K.-W. (2018) · Water · doi:10.3390/w10111536 · Cited by 1,672
An Empirical Comparison of Model Validation Techniques for Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2584050 · Cited by 574
Self-citationSapienz: multi-objective automated testing for Android applications
Mao, K., Harman, M., & Jia, Y. (2016) · Proceedings of the 25th International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931054 · Cited by 541
The Impact of Automated Parameter Optimization on Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2794977 · Cited by 415
Self-citationLineVul: a transformer-based line-level vulnerability prediction
Fu, M., & Tantithamthavorn, C. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3528452 · Cited by 384
Self-citation
Metadata
- Authors
- Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K.
- Venue
- Proceedings of the 38th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2016-05-14
- Pages
- 321-332
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W2367798545
- Cited by
- 331
- References
- 64
- FWCI
- 41.88
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.