Predicting Defective Lines Using a Model-Agnostic Technique
Open accessThe DOI matches its Crossref record; 107 of the 109 references it cites were found in a registry or an index.
Abstract
Defect prediction models are proposed to help a team prioritize the areas of source code files that need Software Quality Assurance (SQA) based on the likelihood of having defects. However, developers may waste their unnecessary effort on the whole file while only a small fraction of its source code lines are defective. Indeed, we find that as little as 1-3 percent of lines of a file are defective. Hence, in this work, we propose a novel framework (calledLine-DP) to identify defective lines using a model-agnostic technique, i.e., an Explainable AI technique that provides information why the model makes such a prediction. Broadly speaking, ourLine-DPfirst builds a file-level defect model using code token features. Then, ourLine-DPuses a state-of-the-art model-agnostic technique (i.e., LIME) to identify risky tokens, i.e., code tokens that lead the file-level defect model to predict that the file will be defective. Then, the lines that contain risky tokens are predicted as defective lines. Through a case study of 32 releases of nine Java open source systems, our evaluation results show that ourLine-DPachieves an average recall of 0.61, a false alarm rate of 0.47, a top 20%LOC recall of 0.27, and an initial false alarm of 16, which are statistically better than six baseline approaches. Our evaluation shows that ourLine-DPrequires an average computation time of 10 seconds including model construction and defective line identification time. In addition, we find that 63 percent of defective lines that can be identified by ourLine-DPare related to common defects (e.g., argument change, condition change). These results suggest that ourLine-DPcan effectively identify defective lines that contain common defects while requiring a smaller amount of inspection effort and a manageable computation cost. The contribution of this paper builds an important step towards line-level defect prediction by leveraging a model-agnostic technique.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Local approaches, like just-in-time defect prediction [158], line-level defect prediction to improve debugging [215], analysis of the quality of code [200], prediction of the retention of syntax knowledge [143, 199], and counterfactual explanations [43, 75] explain individual predictions.”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [215]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538 · cited 1×
“Unlike the traditional model interpretation techniques (TMI) like variable importance for random forest [9] or the coefficients analysis for logistic regression models [27], the model-agnostic techniques can identify important features for a given file by estimating the contribution of each token…”
MethodStatistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
(2017) · Cited by 16,319
Harrell, F. E. (2001) · Springer Series in Statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,195
(2005) · PubMed · Cited by 7,011
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Extended static checking for Java
Flanagan, C., Leino, K. R. M., Lillibridge, M., Nelson, G., Saxe, J. B., & Stata, R. (2002) · Proceedings of the ACM SIGPLAN 2002 conference on Programming language design and implementation · doi:10.1145/512529.512558 · Cited by 1,340 · cited 1×
“ch as null pointer dereferencing and buffer overflows) at the line level. Various static analysis approaches are proposed including heuristic rule-based techniques (e.g., PMD [12]), complex algorithms [20], and hybrid approaches like FindBugs2 which incorporates the static data-flow analysis and the patternmatching anal…”
MethodData Mining Static Code Attributes to Learn Defect Predictors
Menzies, T., Greenwald, J., & Frank, A. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.256941 · Cited by 1,215 · cited 4×
“Defect prediction models have been proposed to predict the most risky areas of source code that are likely to have post-release defects [16, 55, 59, 77, 80, 89, 90].”
BackgroundMethodInfluentialA Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154 · cited 2×
“Defect prediction models are proposed to help SQA teams prioritize their effort by analyzing post-release software defects that occur in the previous release [16, 26, 55, 59, 77, 80].”
Backgroundcode2vec: Learning distributed representations of code
Alon, U., Zilberstein, M., Levy, O., & Yahav, E. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3290353 · Cited by 962 · cited 1×
“Instead of using hand-crafted software metrics, prior work directly uses semantic features of source code to build defect prediction models [7, 14, 35, 89, 90].”
MethodUse of relative code churn measures to predict system defect density
Nagappan, N., & Ball, T. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062514 · Cited by 775
A large-scale empirical study of just-in-time quality assurance
Kamei, Y., Shihab, E., Adams, B., Hassan, A. E., Mockus, A., Sinha, A., & Ubayashi, N. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.70 · Cited by 715 · cited 2×
“To achieve effective SQA prioritization, defect prediction models have been long investigated at different granularity levels, for example, packages [43], components [82], modules [44], files [43, 53], methods [28], and commits [45].”
BackgroundAutomatically learning semantic features for defect prediction
Wang, S., Liu, T., & Tan, L. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884804 · Cited by 689 · cited 3×
“Defect prediction models have been proposed to predict the most risky areas of source code that are likely to have post-release defects [16, 55, 59, 77, 80, 89, 90].”
BackgroundMethodA Novel Neural Source Code Representation Based on Abstract Syntax Tree
Zhang, J., Wang, X., Zhang, H., Sun, H., Wang, K., & Liu, X. (2019) · 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2019.00086 · Cited by 655 · cited 1×
“…optimizing the parameters [80] or using the advanced techniques (e.g., embedding techniques [62]), would improve the performance of our L INE -DP. Recent studies have shown that deep learning and embedding techniques can improve the predictive accuracy of file-level defect models [6, 15, 62, 92].”
MethodAn extensive comparison of bug prediction approaches
D'Ambros, M., Lanza, M., & Robbes, R. (2010) · 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010) · doi:10.1109/msr.2010.5463279 · Cited by 637 · cited 3×
“Defect prediction models have been proposed to predict the most risky areas of source code that are likely to have post-release defects [16, 55, 59, 77, 80, 89, 90].”
BackgroundMethodAn exploratory study of the pull-based software development model
Gousios, G., Pinzger, M., & Deursen, A. V. (2014) · Proceedings of the 36th International Conference on Software Engineering · doi:10.1145/2568225.2568260 · Cited by 628 · cited 1×
“Such code changes (or commits) must undergo rigorous SQA activities (e.g., Continuous Integration tests and code review) prior to merge into the main branch (e.g., a master branch) [24].”
BackgroundAre automated debugging techniques actually helping programmers?
Parnin, C., & Orso, A. (2011) · Proceedings of the 2011 International Symposium on Software Testing and Analysis · doi:10.1145/2001420.2001445 · Cited by 576
An Empirical Comparison of Model Validation Techniques for Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2584050 · Cited by 574
Self-citationWhy don't software developers use static analysis tools to find bugs?
Johnson, B., Song, Y., Murphy-Hill, E., & Bowdidge, R. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606613 · Cited by 568
Code completion with statistical language models
Raychev, V., Vechev, M., & Yahav, E. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594321 · Cited by 555 · cited 1×
“Prior work found that statistical language models can be leveraged to help developers in many software engineering tasks such as code completion [68, 84], code convention [5], and method names suggestion [6].”
BackgroundOn the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Dévanbu, P. (2012) · doi:10.5555/2337223.2337322 · Cited by 541
Cliff´s Delta Calculator: A non-parametric effect size program for two groups of observations
Macbeth, G., Razumiejczyk, E., & Ledesma, R. D. (2010) · Universitas Psychologica · doi:10.11144/javeriana.upsy10-2.cdcp · Cited by 442 · cited 1×
“We did not use the commonly-used Cohen’s D [11] and Cliff’s δ [51] to measure the effect size because both methods are not based on the assumption that the data is pairwise.”
MethodOn the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425 · cited 1×
“With a concept of software naturalness, statistical language models from Natural Language Processing (NLP) have been 2. http://findbugs.sourceforge.net/ used to measure the repetitiveness of source code in a software repository [33].”
MethodThe Impact of Automated Parameter Optimization on Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2794977 · Cited by 415
Self-citationSEQUENCER: Sequence-to-Sequence Learning for End-to-End Program Repair
Chen, Z., Kommrusch, S. J., Tufano, M., Pouchet, L.-N., Poshyvanyk, D., & Monperrus, M. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2940179 · Cited by 396 · cited 1×
“We use a taxonomy of Chen et al. [10] which is summarized in Table 4.”
MethodLearning natural coding conventions
Allamanis, M., Barr, E. T., Bird, C., & Sutton, C. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635883 · Cited by 376 · cited 1×
“Prior work found that statistical language models can be leveraged to help developers in many software engineering tasks such as code completion [68, 84], code convention [5], and method names suggestion [6].”
BackgroundSuggesting accurate method and class names
Allamanis, M., Barr, E. T., Bird, C., & Sutton, C. (2015) · Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering · doi:10.1145/2786805.2786849 · Cited by 369
Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models
Ghotra, B., McIntosh, S., & Hassan, A. E. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.91 · Cited by 364 · cited 4×
“For example, feature selection techniques [23, 39, 40], collinearity analysis [37– 39], class rebalancing techniques [75], classification techniques [22], parameter optimization [4, 21, 77, 80], model validation [79], and model interpretation [36, 37].”
MethodInfluentialInteractions with big data analytics
Fisher, D., DeLine, R., Czerwinski, M., & Drucker, S. (2012) · Interactions · doi:10.1145/2168931.2168943 · Cited by 361 · cited 2×
“Fisher et al. [19] raise a concern that the increased complexity of data analytics may incur additional computation cost of building defect prediction models.”
BackgroundResearcher Bias: The Use of Machine Learning in Software Defect Prediction
Shepperd, M., Bowes, D., & Hall, T. (2014) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2014.2322358 · Cited by 359 · cited 2×
“4 We consider that the accuracy of our file-level defect models is sufficient since prior study reported that a file-level prediction model typically has an MCC value of 0.3 [71].”
MethodInfluentialDeepBugs: a learning approach to name-based bug detection
Pradel, M., & Sen, K. (2018) · Proceedings of the ACM on Programming Languages · doi:10.1145/3276517 · Cited by 342 · cited 2×
“…optimizing the parameters [80] or using the advanced techniques (e.g., embedding techniques [62]), would improve the performance of our L INE -DP. Recent studies have shown that deep learning and embedding techniques can improve the predictive accuracy of file-level defect models [6, 15, 62, 92].”
MethodAutomated parameter optimization of classification techniques for defect prediction models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884857 · Cited by 331 · cited 5×
“Hence, in this work, we consider five well-known classification techniques [22, 77, 80], i.”
BackgroundMethodInfluentialSelf-citationTantithamthavorn, C., Hassan, A. E., & Matsumoto, K. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2876537 · Cited by 327 · cited 1×
“For example, feature selection techniques [23, 39, 40], collinearity analysis [37– 39], class rebalancing techniques [75], classification techniques [22], parameter optimization [4, 21, 77, 80], model validation [79], and model interpretation [36, 37].”
MethodSelf-citationDeep Semantic Feature Learning for Software Defect Prediction
Wang, S., Liu, T., Nam, J., & Tan, L. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2877612 · Cited by 320 · cited 4×
“Defect prediction models have been proposed to predict the most risky areas of source code that are likely to have post-release defects [16, 55, 59, 77, 80, 89, 90].”
BackgroundMethodInfluentialPyDriller: Python framework for mining software repositories
Spadini, D., Aniche, M., & Bacchelli, A. (2018) · Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3236024.3264598 · Cited by 303 · cited 1×
“We use the implementation of PyDriller package to extract such information from Git repositories [72].”
MethodAre deep neural networks the best choice for modeling source code?
Hellendoorn, V. J., & Devanbu, P. (2017) · Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering · doi:10.1145/3106237.3106290 · Cited by 292 · cited 2×
“We use a standard n-gram order of 6 with the Jelinek-Mercer smoothing function as prior work demonstrates that this configuration works well for source code [31].”
MethodMenzies, T., Dekhtyar, A., Distefano, J., & Greenwald, J. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.70721 · Cited by 277
Static analysis tools as early indicators of pre-release defect density
Nagappan, N., & Ball, T. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062558 · Cited by 272 · cited 1×
“Prior work also argued that it is beneficial to obtain early estimates of defect-proneness for areas of source code to help software development teams develop the most effective SQA resource management [57, 58].”
Revisiting the Impact of Classification Techniques on the Performance of Defect Prediction Models
Ghotra, B., McIntosh, S., & Hassan, A. E. (2015) · Cited by 256
On the "naturalness" of buggy code
Ray, B., Hellendoorn, V., Godhane, S., Tu, Z., Bacchelli, A., & Devanbu, P. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884848 · Cited by 252 · cited 10×
“Similar to prior work [64, 67], the lines that were modified by bug-fixing commits are identified as defective lines .”
BackgroundMethodResultInfluentialHow, and why, process metrics are better
Rahman, F., & Devanbu, P. (2013) · 2013 35th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2013.6606589 · Cited by 248 · cited 2×
“The state-of-the-art software metrics (e.g., code and process metrics) are often calculated at the class, file, and method levels [28, 63].”
BackgroundMethodPersonalized defect prediction
Jiang, T., Tan, L., & Kim, S. (2013) · 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2013.6693087 · Cited by 240 · cited 1×
“Instead of using hand-crafted software metrics, prior work directly uses semantic features of source code to build defect prediction models [7, 14, 35, 89, 90].”
MethodTu, Z., Su, Z., & Devanbu, P. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635875 · Cited by 238 · cited 1×
“Prior work found that statistical language models can be leveraged to help developers in many software engineering tasks such as code completion [68, 84], code convention [5], and method names suggestion [6].”
BackgroundTuning for software analytics: Is it really necessary?
Fu, W., Menzies, T., & Shen, X. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.04.017 · Cited by 210
Effort-Aware Defect Prediction Models
Mende, T., & Koschke, R. (2010) · 2010 14th European Conference on Software Maintenance and Reengineering · doi:10.1109/csmr.2010.18 · Cited by 209 · cited 4×
“Similar to prior work [43, 53, 64, 67], we use 20% of LOC as a fixed cutoff for an effort.”
BackgroundMethodInfluentialRevisiting common bug prediction findings using effort-aware models
Kamei, Y., Matsumoto, S., Monden, A., Matsumoto, K.-I., Adams, B., & Hassan, A. E. (2010) · 2010 IEEE International Conference on Software Maintenance · doi:10.1109/icsm.2010.5609530 · Cited by 207 · cited 6×
“One of the key benefits of defect prediction is to help developers perform a cost-effective SQA activity by priortizing defect-prone files in order to uncover maximal defects with minimal effort [28, 43, 53, 61].”
BackgroundMethodInfluentialSelf-citationUse of relative code churn measures to predict system defect density
Nagappan, N., & Ball, T. (2005) · Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005. · doi:10.1109/icse.2005.1553571 · Cited by 201
Which warnings should I fix first?
Kim, S., & Ernst, M. D. (2007) · Proceedings of the the 6th joint meeting of the European software engineering conference and the ACM SIGSOFT symposium on The foundations of software engineering · doi:10.1145/1287624.1287633 · Cited by 199 · cited 2×
“Prior work shows that a static analysis tool can be used to identify defect-prone lines [25, 42, 47, 64, 66, 67].”
BackgroundMethodA Framework for Evaluating the Results of the SZZ Approach for Identifying Bug-Introducing Changes
da Costa, D. A., McIntosh, S., Shang, W., Kulesza, U., Coelho, R., & Hassan, A. E. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2616306 · Cited by 197 · cited 1×
“To ensure the quality of the dataset, we followed an approach suggested by prior work [13, 91], i.e., focusing on the issues that are reported after a studied release; labelled as bugs; affected only the studied release; and already closed or fixed.”
MethodChange Bursts as Defect Predictors
Nagappan, N., Zeller, A., Zimmermann, T., Herzig, K., & Murphy, B. (2010) · 2010 IEEE 21st International Symposium on Software Reliability Engineering · doi:10.1109/issre.2010.25 · Cited by 195 · cited 3×
“Defect prediction models have been proposed to predict the most risky areas of source code that are likely to have post-release defects [16, 55, 59, 77, 80, 89, 90].”
BackgroundMethodCode completion with statistical language models
Raychev, V., Vechev, M., & Yahav, E. (2014) · ACM SIGPLAN Notices · doi:10.1145/2666356.2594321 · Cited by 189
How, and Why, Process Metrics are better
Rahman, F., & Dévanbu, P. (2013) · Cited by 184
Automatic Feature Learning for Predicting Vulnerable Software Components
Dam, H. K., Tran, T., Pham, T., Ng, S. W., Grundy, J., & Ghose, A. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2881961 · Cited by 184 · cited 1×
“…optimizing the parameters [80] or using the advanced techniques (e.g., embedding techniques [62]), would improve the performance of our L INE -DP. Recent studies have shown that deep learning and embedding techniques can improve the predictive accuracy of file-level defect models [6, 15, 62, 92].”
MethodLessons Learned from Using a Deep Tree-Based Model for Software Defect Prediction in Practice
Dam, H. K., Pham, T., Ng, S. W., Tran, T., Grundy, J., Ghose, A., Kim, T., & Kim, C.-J. (2019) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2019.00017 · Cited by 172 · cited 1×
“Instead of using hand-crafted software metrics, prior work directly uses semantic features of source code to build defect prediction models [7, 14, 35, 89, 90].”
MethodPerceptions, Expectations, and Challenges in Defect Prediction
Wan, Z., Xia, X., Hassan, A. E., Lo, D., Yin, J., & Yang, X. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2877678 · Cited by 169 · cited 3×
“However, developers could still waste an SQA effort on manually identifying the most risky lines, since the current prediction granularity is still perceived as coarse-grained [87].”
BackgroundMethodResultInfluentialZ-Ranking: Using Statistical Analysis to Counter the Impact of Static Analysis Approximations
Kremenek, T., & Engler, D. (2003) · Cited by 168
Fine-grained just-in-time defect prediction
Pascarella, L., Palomba, F., & Bacchelli, A. (2018) · Journal of Systems and Software · doi:10.1016/j.jss.2018.12.001 · Cited by 161 · cited 2×
“One of the key benefits of defect prediction is to help developers perform a cost-effective SQA activity by priortizing defect-prone files in order to uncover maximal defects with minimal effort [28, 43, 53, 61].”
BackgroundHow developers engage with static analysis tools in different contexts
Vassallo, C., Panichella, S., Palomba, F., Proksch, S., Gall, H. C., & Zaidman, A. (2020) · Empirical Software Engineering · doi:10.1007/s10664-019-09750-5 · Cited by 161 · cited 1×
“A static analysis tool could potentially be used to predict and rank defect-prone lines [85].”
MethodAgrawal, A., & Menzies, T. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180197 · Cited by 159 · cited 4×
“For example, feature selection techniques [23, 39, 40], collinearity analysis [37– 39], class rebalancing techniques [75], classification techniques [22], parameter optimization [4, 21, 77, 80], model validation [79], and model interpretation [36, 37].”
MethodInfluentialBug prediction based on fine-grained module histories
Hata, H., Mizuno, O., & Kikuno, T. (2012) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2012.6227193 · Cited by 159
Self-citationMining Software Defects: Should We Consider Affected Releases?
Yatish, S., Jiarpakdee, J., Thongtanunam, P., & Tantithamthavorn, C. K. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00075 · Cited by 156 · cited 4×
“To ensure the quality of the dataset, we followed an approach suggested by prior work [13, 91], i.”
BackgroundMethodInfluentialSelf-citationAn Empirical Study of Model-Agnostic Techniques for Defect Prediction Models
Jiarpakdee, J., Tantithamthavorn, C. K., Dam, H. K., & Grundy, J. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2020.2982385 · Cited by 148 · cited 2×
“a novel aspect of leveraging the state-of-theart model-agnostic technique (LIME) to identify defective lines, in addition to being used to explain the prediction of defective files from defect models [36]. Our framework will help developers effectively prioritize SQA effort. ACKNOWLEDGEMENT C. Tantithamthavorn was part…”
MethodInfluentialSelf-citationThe Impact of Correlated Metrics on the Interpretation of Defect Models
Jiarpakdee, J., Tantithamthavorn, C. K., & Hassan, A. E. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2891758 · Cited by 146
Self-citationA Large-Scale Study of the Impact of Feature Selection Techniques on Defect Classification Models
Ghotra, B., McIntosh, S., & Hassan, A. E. (2017) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2017.18 · Cited by 142
The Impact of Mislabelling on the Performance and Interpretation of Defect Prediction Models
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., Ihara, A., & Matsumoto, K. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.93 · Cited by 140 · cited 1×
“The predictive accuracy of the defect prediction model heavily relies on the modelling pipelines of defect prediction models [4, 22, 56, 73, 74, 76, 78].”
MethodSelf-citationThe Effects of Over and Under Sampling on Fault-prone Module Detection
Kamei, Y., Monden, A., Matsumoto, S., Kakimoto, T., & Matsumoto, K.-I. (2007) · First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007) · doi:10.1109/esem.2007.28 · Cited by 138 · cited 2×
“Defect models at various granularity levels have been proposed, e.g., packages [43], components [82], modules [44], files [43, 53], methods [28].”
BackgroundMethodSelf-citationThongtanunam, P., McIntosh, S., Hassan, A. E., & Iida, H. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884852 · Cited by 138
Self-citationModern Release Engineering in a Nutshell -- Why Researchers Should Care
Adams, B., & McIntosh, S. (2016) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2016.108 · Cited by 138 · cited 5×
“Figure 1 illustrates a simplified software engineering workflow that includes SQA activities [1].”
BackgroundMethodInfluentialExtended static checking for Java
Cormac Flanagan, A., Leino, K. R. M., Lillibridge, M., Nelson, G., Saxe, J. B., & Stata, R. (2002) · ACM SIGPLAN Notices · doi:10.1145/543552.512558 · Cited by 131
Correlation exploitation in error ranking
Kremenek, T., Ashcraft, K., Yang, J., & Engler, D. (2004) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1029894.1029909 · Cited by 128 · cited 2×
“Several studies proposed approaches to filter and prioritize warnings reported by SBF [29, 30, 48, 49, 70].”
BackgroundPredicting accurate and actionable static analysis warnings
Ruthruff, J. R., Penix, J., Morgenthaler, J. D., Elbaum, S. G., & Rothermel, G. (2008) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1368088.1368135 · Cited by 126 · cited 1×
“Several studies proposed approaches to filter and prioritize warnings reported by SBF [29, 30, 48, 49, 70].”
BackgroundInvestigating Code Review Practices in Defective Files: An Empirical Study of the Qt System
Thongtanunam, P., McIntosh, S., Hassan, A. E., & Iida, H. (2015) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr.2015.23 · Cited by 125 · cited 1×
“, code reviews), it is still possible that software defects still slip through to the official release of software products [81, 82].”
BackgroundSelf-citationSupervised vs Unsupervised Models: A Holistic Look at Effort-Aware Just-in-Time Defect Prediction
Huang, Q., Xia, X., & Lo, D. (2017) · 2017 IEEE International Conference on Software Maintenance and Evolution (ICSME) · doi:10.1109/icsme.2017.51 · Cited by 124 · cited 2×
“, the top k% of lines ranked by their defectproneness [34].”
BackgroundAn experience report on defect modelling in practice: pitfalls and challenges
Tantithamthavorn, C., & Hassan, A. E. (2018) · Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice · doi:10.1145/3183519.3183547 · Cited by 122 · cited 1×
“The predictive accuracy of the defect prediction model heavily relies on the modelling pipelines of defect prediction models [4, 22, 56, 73, 74, 76, 78].”
MethodSelf-citationComparing static bug finders and statistical prediction
Rahman, F., Khatri, S., Barr, E. T., & Dévanbu, P. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2568225.2568269 · Cited by 121 · cited 5×
“Prior work shows that a static analysis tool can be used to identify defect-prone lines [25, 42, 47, 64, 66, 67].”
BackgroundMethodResultInfluentialBugram: bug detection with n-gram language models
Wang, S., Chollak, D., Movshovitz-Attias, D., & Tan, L. (2016) · Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/2970276.2970341 · Cited by 118 · cited 2×
“Recent work leverages the n-gram language models to predict defect-prone tokens and lines [67, 88].”
BackgroundHow many of all bugs do we find? a study of static bug detectors
Habib, A., & Pradel, M. (2018) · Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering · doi:10.1145/3238147.3238213 · Cited by 108 · cited 2×
“Habib and Pradel [25] found that static bug detectors are certainly worthwhile to detect real-world bugs.”
BackgroundZ-Ranking: Using Statistical Analysis to Counter the Impact of Static Analysis Approximations
Kremenek, T., & Engler, D. (2003) · Lecture notes in computer science · doi:10.1007/3-540-44898-5_16 · Cited by 103 · cited 1×
“Several studies proposed approaches to filter and prioritize warnings reported by SBF [29, 30, 48, 49, 70].”
BackgroundHeckman, S. S., & Williams, L. A. (2008) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/1414004.1414013 · Cited by 99 · cited 1×
“Several studies proposed approaches to filter and prioritize warnings reported by SBF [29, 30, 48, 49, 70].”
BackgroundThe impact of automated feature selection techniques on the interpretation of defect models
Jiarpakdee, J., Tantithamthavorn, C. K., & Treude, C. (2020) · Empirical Software Engineering · doi:10.1007/s10664-020-09848-1 · Cited by 95
Self-citation(2007) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2007.80 · Cited by 91
The importance of accounting for real-world labelling when predicting software vulnerabilities
Jimenez, M., Rwemalika, R., Papadakis, M., Sarro, F., Traon, Y. L., & Harman, M. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338941 · Cited by 89 · cited 1×
“The chronological order of the data may impact the results of prediction models in the context of software vulnerability [41].”
AutoSpearman: Automatically Mitigating Correlated Software Metrics for Interpreting Defect Models
Jiarpakdee, J., Tantithamthavorn, C. K., & Treude, C. (2018) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2018.00018 · Cited by 84
Self-citationBuilding Useful Program Analysis Tools Using an Extensible Java Compiler
Aftandilian, E. E., Sauciuc, R., Priya, S., & Krishnan, S. (2012) · IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) · doi:10.1109/scam.2012.28 · Cited by 78 · cited 13×
“ErrorProne (EP): Recently, major companies, e.g., Google, use ErrorProne to identify defect-prone lines [2].”
MethodInfluentialAgrawal, A., & Menzies, T. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1705.03697 · Cited by 78 · cited 4×
“For example, feature selection techniques [23, 39, 40], collinearity analysis [37– 39], class rebalancing techniques [75], classification techniques [22], parameter optimization [4, 21, 77, 80], model validation [79], and model interpretation [36, 37].”
MethodInfluentialComments on “Researcher Bias: The Use of Machine Learning in Software Defect Prediction”
Tantithamthavorn, C., McIntosh, S., Hassan, A. E., & Matsumoto, K. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2553030 · Cited by 76
Self-citationMutation-aware fault prediction
Bowes, D., Hall, T., Harman, M., Jia, Y., Sarro, F., & Wu, F. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931039 · Cited by 71 · cited 2×
“TMI techniques can be used to identify the important features in the defect models [8].”
MethodHow to “DODGE” Complex Software Analytics
Agrawal, A., Fu, W., Chen, D., Shen, X., & Menzies, T. (2019) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2945020 · Cited by 62 · cited 1×
“ork, we use Distance-to-heaven (d2h) of Agrawal and Menzies [4] to combine the recall and FAR values. D2h is the root mean square of the recall and false alarm values (i.e.,q (1 Recall)2+(0 FAR)2 2 ) [3, 4]. A d2h value of 0 indicates that an approach achieves a perfect identification, i.e., an approach can identify al…”
MethodRahman, M., Palani, D., & Rigby, P. C. (2019) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2019.00022 · Cited by 62 · cited 4×
“As suggested by Rahman and Rigby [66], removing these non-alphanumeric characters will ensure that the analyzed code tokens will not be artificially repetitive.”
BackgroundMethodInfluentialStatic analysis tools as early indicators of pre-release defect density
Nagappan, N., & Ball, T. (2005) · Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005. · doi:10.1109/icse.2005.1553604 · Cited by 58
An Empirical Evaluation of Regression Testing Based on Fix-Cache Recommendations
Engström, E., Runeson, P., & Wikstrand, G. (2010) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2010.40 · Cited by 37
Adaptively ranking alerts generated from automated static analysis
Heckman, S. S. (2007) · XRDS Crossroads The ACM Magazine for Students · doi:10.1145/1349332.1349339 · Cited by 37 · cited 1×
“Several studies proposed approaches to filter and prioritize warnings reported by SBF [29, 30, 48, 49, 70].”
BackgroundA Study of Redundant Metrics in Defect Prediction Datasets
Jiarpakdee, J., Tantithamthavorn, C. K., Ihara, A., & Matsumoto, K. (2016) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew.2016.30 · Cited by 37
Self-citation“Bad smells” in software analytics papers
Menzies, T., & Shepperd, M. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2019.04.005 · Cited by 37 · cited 1×
“The predictive accuracy of the defect prediction model heavily relies on the modelling pipelines of defect prediction models [4, 22, 56, 73, 74, 76, 78].”
MethodCorrelation exploitation in error ranking
Kremenek, T., Ashcraft, K., Yang, J., & Engler, D. (2004) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1041685.1029909 · Cited by 36
The Impact of Correlated Metrics on Defect ModelsPreprint
Jiarpakdee, J., Tantithamthavorn, C. K., & Hassan, A. E. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1801.10271 · Cited by 34 · cited 1×
“For example, feature selection techniques [23, 39, 40], collinearity analysis [37– 39], class rebalancing techniques [75], classification techniques [22], parameter optimization [4, 21, 77, 80], model validation [79], and model interpretation [36, 37].”
MethodSelf-citationREMI: defect prediction for efficient API testing
Kim, M., Nam, J., Yeon, J., Choi, S., & Kim, S. (2015) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2786805.2804429 · Cited by 33 · cited 1×
“At the Tizenwearable project by Samsung Electronics [46], they found that prioritizing APIs based on their defect-proneness increases the number of discovered defects and reduces the cost required for executing test cases.”
BackgroundRelease Stabilization on Linux and Chrome
Rahman, M. T., & Rigby, P. C. (2015) · IEEE Software · doi:10.1109/ms.2015.31 · Cited by 31 · cited 1×
“In other words, the files that are changed during the software development need to be checked and stabilized to ensure that these changes will not impact the overall quality of the software systems [32, 50, 65].”
BackgroundUsing Pre-Release Test Failures to Build Early Post-Release Defect Prediction Models
Herzig, K. (2014) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2014.21 · Cited by 29 · cited 1×
“In other words, the files that are changed during the software development need to be checked and stabilized to ensure that these changes will not impact the overall quality of the software systems [32, 50, 65].”
BackgroundTantithamthavorn, C. K. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2889160.2889256 · Cited by 28 · cited 1×
“The predictive accuracy of the defect prediction model heavily relies on the modelling pipelines of defect prediction models [4, 22, 56, 73, 74, 76, 78].”
MethodSelf-citationRegression Analysis with Python
Massaron, L., & Boschetti, A. (2016) · Cited by 25
An experience report on defect modelling in practice pitfalls and challenges
Chakkrit, T., & Ahmed, E. (2018) · ACM Proceedings · Cited by 25
PyDriller: Python Framework for Mining Software Repositories
Spadini, D., Aniche, M. F., & Bacchelli, A. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1327411 · Cited by 17
PyDriller: Python Framework for Mining Software Repositories
Spadini, D., Aniche, M. F., & Bacchelli, A. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1327410 · Cited by 17
Research works printed but not linked: 5
- 10.1023/A:I0I0933404324 · doi:10.1023/a:i0i0933404324
- Copeland, 2005, PMD Applied, 10
- 2020, Release Management Process
- Ribeiro, Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discov. Data Mining, 1135
- Tomczak, 2014, The need to report effect size estimates revisited. An overview of some recommended measures of effect size, Trends Sport Sci., 21, 19
Cited by
LineVul: a transformer-based line-level vulnerability prediction
Fu, M., & Tantithamthavorn, C. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3528452 · Cited by 384
“[60] proposed a machine learning-based approach with LIME model-agnostic technique (BoW+RF+LIME) to predict which lines are likely to be defective in the future.”
BackgroundMethodResultInfluentialSelf-citationSemi-Supervised Log-Based Anomaly Detection via Probabilistic Label Estimation
Yang, L., Chen, J., Wang, Z., Wang, W., Jiang, J., Dong, X., & Zhang, W. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00130 · Cited by 244
“Moreover, recent studies have presented that making software analytics models interpretable to software practitioners is as important as achieving accurate prediction [75]–[77].”
BackgroundLineVD: statement-level vulnerability detection using graph neural networks
Hin, D., Kan, A., Chen, H., & Babar, M. A. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3527949 · Cited by 218
“Particularly, there have been works attempting to detecting lines-level information by leveraging explainable artificial intelligence for software engineering tasks, such as detecting source code lines for defect prediction [50, 58].”
BackgroundVulRepair: a T5-based automated software vulnerability repair
Fu, M., Tantithamthavorn, C., Le, T., Nguyen, V., & Phung, D. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549098 · Cited by 207
Self-citationCode Prediction by Feeding Trees to Transformers
Kim, S., Zhao, J., Tian, Y., & Chandra, S. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00026 · Cited by 209
“For example, [43], [44] used LIME [45], a model-agnostic explainability method, to pinpoint the defective lines of code from defect detection models.”
Method
Metadata
- Authors
- Wattanakriengkrai, S., Thongtanunam, P., Tantithamthavorn, C., Hata, H., & Matsumoto, K.
- Affiliations
- Nara Institute of Science and Technology (JP)
- The University of Melbourne (AU)
- Monash University (AU)
- ORCID
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2022-05-01
- Volume
- 48
- Issue
- 5
- Pages
- 1480-1496
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W3083954092
- Cited by
- 137
- References
- 109
- Citations by year
- 2026: 21 · 2025: 24 · 2024: 38 · 2023: 24 · 2022: 15
- FWCI
- 12.69
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- IEEE Transactions on Software Engineering (publishedVersion, cc-by)
- Institutional Repositories DataBase (IRDB) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- Monash University Research Portal (Monash University) (submittedVersion, other-oa)
- NAIST Digital Library (Nara Institute of Science and Technology) (submittedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (cc-by)
- Funders
- JSPS KAKENHI (16H05857, 20H05706)
- Australian Research Council (DE200100941, DE210101091)
- Monash-FIT Early Career Researcher
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Software › Software Reliability and Analysis Research
- Keywords
- model-agnostic explanation, lime, explainable AI, software defect prediction, software quality assurance
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.