Revisiting common bug prediction findings using effort-aware models
The DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
Bug prediction models are often used to help allocate software quality assurance efforts (e.g. testing and code reviews). Mende and Koschke have recently proposed bug prediction models that are effort-aware. These models factor in the effort needed to review or test code when evaluating the effectiveness of prediction models, leading to more realistic performance evaluations. In this paper, we revisit two common findings in the bug prediction literature: 1) Process metrics (e.g., change history) outperform product metrics (e.g., LOC), 2) Package-level predictions outperform file-level predictions. Through a case study on three projects from the Eclipse Foundation, we find that the first finding holds when effort is considered, while the second finding does not hold. These findings validate the practical significance of prior findings in the bug prediction literature and encourage their adoption in practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · cited 1×
“We use three well-known modeling techniques; regression model [25], regression tree [26] and random forest [27].”
MethodClassification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,522
Cockburn, A. (2001) · UCLA - Biblioteca de Ciencias y Tecnologia Felix Morales Bueno · Cited by 1,858
A validation of object-oriented design metrics as quality indicators
Basili, V. R., Briand, L. C., & Melo, W. L. (1996) · IEEE Transactions on Software Engineering · doi:10.1109/32.544352 · Cited by 1,726 · cited 2×
“However, as pointed out by Mende and Koschke [4], traditional prediction models [ 5 ]‐[10] typically ignore the effort needed to fix bugs, i.e., they do not distinguish between a predicted bug in a small module and a predicted bug in a large module.”
BackgroundAgile Software Development, Principles, Patterns, and Practices
Martin, R. (2002) · Cited by 1,466
Lessmann, S., Baesens, B., Mues, C., & Pietsch, S. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2008.35 · Cited by 1,226 · cited 2×
“There are also several studies on evaluation methods for prediction models [1], [4], [21], [22].”
BackgroundŚliwerski, J., Zimmermann, T., & Zeller, A. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083147 · Cited by 805 · cited 2×
“To obtain the number of bugs in source code files, we implemented the SZZ algorithm [29].”
BackgroundMethodMining metrics to predict component failures
Nagappan, N., Ball, T., & Zeller, A. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134349 · Cited by 782
Use of relative code churn measures to predict system defect density
Nagappan, N., & Ball, T. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062514 · Cited by 775
Predicting Defects for Eclipse
Zimmermann, T., Premraj, R., & Zeller, A. (2007) · Third International Workshop on Predictor Models in Software Engineering (PROMISE'07: ICSE Workshops 2007) · doi:10.1109/promise.2007.10 · Cited by 753
Cited by
A Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154
“These papers are: [[8]], [[9]], [[10]], [[11]], [[18]], [[21]], [[29]], [[31]], [[32]], [[37]], [[51]], [[56]], [[69]], [[73]], [[74]], [[76]], [[83]], [[86]], [[92]], [[98]], [[109]], [[110]], [[116]], [[117]], [[118]], [[120]], [[122]], [[127]], [[133]], [[135]], [[154]], [[160]], [[163]],…”
A large-scale empirical study of just-in-time quality assurance
Kamei, Y., Shihab, E., Adams, B., Hassan, A. E., Mockus, A., Sinha, A., & Ubayashi, N. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.70 · Cited by 715
Self-citationEvaluating defect prediction approaches: a benchmark and an extensive comparison
D’Ambros, M., Lanza, M., & Robbes, R. (2012) · Empirical Software Engineering · doi:10.1007/s10664-011-9173-9 · Cited by 626
Nam, J., Pan, S. J., & Kim, S. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606584 · Cited by 449
The Impact of Automated Parameter Optimization on Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2794977 · Cited by 415
Self-citationDeep Learning for Just-in-Time Defect Prediction
Yang, X., Lo, D., Xia, X., Zhang, Y., & Sun, J. (2015) · 2015 IEEE International Conference on Software Quality, Reliability and Security · doi:10.1109/qrs.2015.14 · Cited by 352
McIntosh, S., Kamei, Y., Adams, B., & Hassan, A. E. (2014) · Proceedings of the 11th Working Conference on Mining Software Repositories · doi:10.1145/2597073.2597076 · Cited by 348
Self-citationAn empirical study of the impact of modern code review practices on software quality
McIntosh, S., Kamei, Y., Adams, B., & Hassan, A. E. (2016) · Empirical Software Engineering · doi:10.1007/s10664-015-9381-9 · Cited by 332
Self-citationTantithamthavorn, C., Hassan, A. E., & Matsumoto, K. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2876537 · Cited by 327
Self-citationProgress on approaches to software defect prediction
Li, Z., Jing, X.-Y., & Zhu, X. (2018) · IET Software · doi:10.1049/iet-sen.2017.0148 · Cited by 264
TLEL: A two-layer ensemble learning approach for just-in-time defect prediction
Yang, X., Lo, D., Xia, X., & Sun, J. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.03.007 · Cited by 216
Predicting Defective Lines Using a Model-Agnostic Technique
Wattanakriengkrai, S., Thongtanunam, P., Tantithamthavorn, C., Hata, H., & Matsumoto, K. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.3023177 · Cited by 137
“One of the key benefits of defect prediction is to help developers perform a cost-effective SQA activity by priortizing defect-prone files in order to uncover maximal defects with minimal effort [28, 43, 53, 61].”
BackgroundMethodInfluentialSelf-citation
Metadata
- Authors
- Kamei, Y., Matsumoto, S., Monden, A., Matsumoto, K.-I., Adams, B., & Hassan, A. E.
- Venue
- 2010 IEEE International Conference on Software Maintenance
- Publisher
- IEEE
- Published
- 2010-09
- Pages
- 1-10
- ISSN
- 1063-6773 (ISSN-L)
- OpenAlex
- W2051978688
- Cited by
- 207
- References
- 27
- FWCI
- 7.42
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.