The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.
A large-scale empirical study of just-in-time quality assurance
Abstract
Defect prediction models are a well-known technique for identifying defect-prone files or packages such that practitioners can allocate their quality assurance efforts (e.g., testing and code reviews). However, once the critical files or packages have been identified, developers still need to spend considerable time drilling down to the functions or even code snippets that should be reviewed or tested. This makes the approach too time consuming and impractical for large software systems. Instead, we consider defect prediction models that focus on identifying defect-prone (“risky”) software changes instead of files or packages. We refer to this type of quality assurance activity as “Just-In-Time Quality Assurance,” because developers can review and test these risky changes while they are still fresh in their minds (i.e., at check-in time). To build a change risk model, we use a wide range of factors based on the characteristics of a software change, such as the number of added lines, and developer experience. A large-scale study of six open source and five commercial projects from multiple domains shows that our models can predict whether or not a change will lead to a defect with an average accuracy of 68 percent and an average recall of 64 percent. Furthermore, when considering the effort needed to review changes, we find that using only 20 percent of the effort it would take to inspect all changes, we can identify 35 percent of all defect-inducing changes. Our findings indicate that “Just-In-Time Quality Assurance” may provide an effort-reducing way to focus on the most risky changes and thus reduce the costs of developing high-quality software.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
McCabe, T. J. (1976) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1976.233837 · Cited by 6,059
A metrics suite for object oriented design
Chidamber, S. R., & Kemerer, C. F. (1994) · IEEE Transactions on Software Engineering · doi:10.1109/32.295895 · Cited by 5,669
Estimating the error rate of a prediction rule: improvement on cross-validation
Efron, B. (1983) · Journal of the American Statistical Association · doi:10.1080/01621459.1983.10477973 · Cited by 2,300
The cathedral and the bazaar: musings on Linux and open source by an accidental revolutionary
(2002) · Choice Reviews Online · doi:10.5860/choice.39-2841 · Cited by 2,251
Mallows, C. L. (1995) · Technometrics · doi:10.1080/00401706.1995.10484370 · Cited by 2,083
Mallows, C. L. (1973) · Technometrics · doi:10.2307/1267380 · Cited by 1,975
A validation of object-oriented design metrics as quality indicators
Basili, V. R., Briand, L. C., & Melo, W. L. (1996) · IEEE Transactions on Software Engineering · doi:10.1109/32.544352 · Cited by 1,726
Two case studies of open source software development: Apache and Mozilla
Mockus, A., Fielding, R. T., & Herbsleb, J. D. (2002) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/567793.567795 · Cited by 1,685
Self-citationLessmann, S., Baesens, B., Mues, C., & Pietsch, S. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2008.35 · Cited by 1,226
A Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154
Empirical validation of object-oriented metrics on open source software for fault prediction
Gyimothy, T., Ferenc, R., & Siket, I. (2005) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2005.112 · Cited by 904
Śliwerski, J., Zimmermann, T., & Zeller, A. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1082983.1083147 · Cited by 805
Mining metrics to predict component failures
Nagappan, N., Ball, T., & Zeller, A. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134349 · Cited by 782
Use of relative code churn measures to predict system defect density
Nagappan, N., & Ball, T. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062514 · Cited by 775
Predicting fault incidence using software change history
Graves, T. L., Karr, A. F., Marron, J. S., & Siy, H. (2000) · IEEE Transactions on Software Engineering · doi:10.1109/32.859533 · Cited by 753
Predicting the location and number of faults in large software systems
Ostrand, T. J., Weyuker, E. J., & Bell, R. M. (2005) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2005.49 · Cited by 720
Predicting faults using the complexity of code changes
Hassan, A. E. (2009) · 2009 IEEE 31st International Conference on Software Engineering · doi:10.1109/icse.2009.5070510 · Cited by 708
Self-citationMoser, R., Pedrycz, W., & Succi, G. (2008) · Proceedings of the 13th international conference on Software engineering - ICSE '08 · doi:10.1145/1368088.1368114 · Cited by 701
Cross-project defect prediction: a large scale experiment on data vs. domain vs. process
Zimmermann, T., Nagappan, N., Gall, H. C., Giger, E., & Murphy, B. (2009) · Zurich Open Repository and Archive (University of Zurich) · doi:10.5167/uzh-25785 · Cited by 694
Classifying Software Changes: Clean or Buggy?
Kim, S., Whitehead, & Zhang, Y. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.70773 · Cited by 672
An extensive comparison of bug prediction approaches
D'Ambros, M., Lanza, M., & Robbes, R. (2010) · 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010) · doi:10.1109/msr.2010.5463279 · Cited by 637
Defect prediction from static code features: current results, limitations, new approaches
Menzies, T., Milton, Z., Turhan, B., Cukic, B., Jiang, Y., & Bener, A. (2010) · Automated Software Engineering · doi:10.1007/s10515-010-0069-5 · Cited by 466
Arisholm, E., Briand, L. C., & Johannessen, E. B. (2010) · Journal of Systems and Software · doi:10.1016/j.jss.2009.06.055 · Cited by 455
The detection of fault-prone programs
Munson, J. C., & Khoshgoftaar, T. M. (1992) · IEEE Transactions on Software Engineering · doi:10.1109/32.135775 · Cited by 442
Predicting risk of software changes
Mockus, A., & Weiss, D. M. (2002) · Bell Labs Technical Journal · doi:10.1002/bltj.2229 · Cited by 429
Self-citationReLink: recovering links between bugs and changes
Wu, R., Zhang, H., Kim, S., & Cheung, S.-C. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025120 · Cited by 412
Śliwerski, J., Zimmermann, T., & Zeller, A. (2005) · Proceedings of the 2005 international workshop on Mining software repositories - MSR '05 · doi:10.1145/1083142.1083147 · Cited by 395
Expertise Browser: a quantitative approach to identifying expertise
Mockus, A., & Herbsleb, J. D. (2003) · Proceedings of the 24th International Conference on Software Engineering. ICSE 2002 · doi:10.1109/icse.2002.1007994 · Cited by 383
Self-citationThe prediction of faulty classes using object-oriented design metrics
El Emam, K., Melo, W., & Machado, J. C. (2001) · Journal of Systems and Software · doi:10.1016/s0164-1212(00)00086-8 · Cited by 359
Predicting fault-prone software modules in telephone switches
Ohlsson, N., & Alberg, H. (1996) · IEEE Transactions on Software Engineering · doi:10.1109/32.553637 · Cited by 346
Don't touch my code!: examining the effects of ownership on software quality
Bird, C., Nagappan, N., Murphy, B., Gall, H., & Devanbu, P. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025119 · Cited by 340
Kwek, L. C., & Oh, C. H. (1998) · The European Physical Journal C · doi:10.1007/s100520050261 · Cited by 324
Menzies, T., Dekhtyar, A., Distefano, J., & Greenwald, J. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.70721 · Cited by 277
Software Dependencies, Work Dependencies, and Their Impact on Failures
Cataldo, M., Mockus, A., Roberts, J. A., & Herbsleb, J. D. (2009) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2009.42 · Cited by 271
Self-citationCharacterizing and predicting which bugs get fixed: an empirical study of Microsoft Windows
Guo, P. J., Zimmermann, T., Nagappan, N., & Murphy, B. (2010) · Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1 · doi:10.1145/1806799.1806871 · Cited by 265
Yin, Z., Yuan, D., Zhou, Y., Pasupathy, S., & Bairavasundaram, L. (2011) · Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering · doi:10.1145/2025113.2025121 · Cited by 249
Investigating quality factors in object-oriented designs: an industrial case study
Briand, L. C., Wüst, J., Ikonomovski, S. V., & Lounis, H. (1999) · Proceedings of the 21st international conference on Software engineering · doi:10.1145/302405.302654 · Cited by 235
Toward understanding the rhetoric of small source code changes
Purushothaman, R., & Perry, D. E. (2005) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2005.74 · Cited by 216
Preprocessing CVS data for fine-grained analysis
Zimmermann, T. (2004) · "International Workshop on Mining Software Repositories (MSR 2004)" W17S Workshop - 26th International Conference on Software Engineering · doi:10.1049/ic:20040466 · Cited by 214
Effort-Aware Defect Prediction Models
Mende, T., & Koschke, R. (2010) · 2010 14th European Conference on Software Maintenance and Reengineering · doi:10.1109/csmr.2010.18 · Cited by 209
Revisiting common bug prediction findings using effort-aware models
Kamei, Y., Matsumoto, S., Monden, A., Matsumoto, K.-I., Adams, B., & Hassan, A. E. (2010) · 2010 IEEE International Conference on Software Maintenance · doi:10.1109/icsm.2010.5609530 · Cited by 207
Self-citationUse of relative code churn measures to predict system defect density
Nagappan, N., & Ball, T. (2005) · Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005. · doi:10.1109/icse.2005.1553571 · Cited by 201
Expertise browser: a quantitative approach to identifying expertise
Mockus, A., & Herbsleb, J. D. (2002) · Proceedings of the 24th international conference on Software engineering - ICSE '02 · doi:10.1145/581339.581401 · Cited by 199
Self-citationChange Bursts as Defect Predictors
Nagappan, N., Zeller, A., Zimmermann, T., Herzig, K., & Murphy, B. (2010) · 2010 IEEE 21st International Symposium on Software Reliability Engineering · doi:10.1109/issre.2010.25 · Cited by 195
Predicting fault-prone components in a java legacy system
Arisholm, E., & Briand, L. C. (2006) · Proceedings of the 2006 ACM/IEEE international symposium on Empirical software engineering · doi:10.1145/1159733.1159738 · Cited by 187
Revisiting the evaluation of defect prediction models
Mende, T., & Koschke, R. (2009) · Proceedings of the 5th International Conference on Predictor Models in Software Engineering · doi:10.1145/1540438.1540448 · Cited by 179
Briand, L. C., Brasili, V. R., & Hetmanski, C. J. (1993) · IEEE Transactions on Software Engineering · doi:10.1109/32.256851 · Cited by 179
An Investigation into the Functional Form of the Size-Defect Relationship for Software Modules
Koru, A. G., Zhang, D., Emam, K. E., & Liu, H. (2008) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2008.90 · Cited by 156
Do time of day and developer experience affect commit bugginess?
Eyolfson, J., Tan, L., & Lam, P. (2011) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/1985441.1985464 · Cited by 154
The Effects of Over and Under Sampling on Fault-prone Module Detection
Kamei, Y., Monden, A., Matsumoto, S., Kakimoto, T., & Matsumoto, K. (2007) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2007.28 · Cited by 154
Self-citationCode churn: a measure for estimating the impact of code change
Munson, J. C., & Elbaum, S. G. (2002) · Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272) · doi:10.1109/icsm.1998.738486 · Cited by 153
Classification and evaluation of defects in a project retrospective
Leszak, M., Perry, D. E., & Stoll, D. (2002) · Journal of Systems and Software · doi:10.1016/s0164-1212(01)00146-7 · Cited by 145
A degree-of-knowledge model to capture source code familiarity
Fritz, T., Ou, J., Murphy, G. C., & Murphy-Hill, E. R. (2010) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1806799.1806856 · Cited by 128
Comparing design and code metrics for software quality prediction
Jiang, Y., Čukić, B., Menzies, T., et al. (2008) · doi:10.1145/1370788.1370793 · Cited by 127
On the relation of refactorings and software defect prediction
Ratzinger, J., Sigmund, T., & Gall, H. C. (2008) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/1370750.1370759 · Cited by 121
Organizational volatility and its effects on software defects
Mockus, A. (2010) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1882291.1882311 · Cited by 120
Self-citationAn analysis of developer metrics for fault prediction
Matsumoto, S., Kamei, Y., Monden, A., Matsumoto, K., & Nakamura, M. (2010) · doi:10.1145/1868328.1868356 · Cited by 115
Self-citationZhou, M., & Mockus, A. (2010) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/1882291.1882313 · Cited by 107
Self-citationShihab, E., Jiang, Z. M., Ibrahim, W. M., Adams, B., & Hassan, A. E. (2010) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1145/1852786.1852792 · Cited by 91
Self-citation(2007) · Proceedings of the ... ACM-IEEE International Symposium on Empirical Software Engineering and Measurement/International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2007.80 · Cited by 91
High-impact defects: a study of breakage and surprise defects
Shihab, E., Mockus, A., Kamei, Y., Adams, B., & Hassan, A. E. (2011) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2025113.2025155 · Cited by 85
Self-citationProgrammer-based fault prediction
Ostrand, T. J., Weyuker, E. J., & Bell, R. M. (2010) · doi:10.1145/1868328.1868357 · Cited by 83
Li, P. L., Herbsleb, J. D., Shaw, M., & Robinson, B. (2006) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/1134285.1134343 · Cited by 77
Towards a Theoretical Model for Software Growth
Herraiz, I., Gonzalez-Barahona, J. M., & Robles, G. (2007) · doi:10.1109/msr.2007.31 · Cited by 74
Towards a simplification of the bug report form in eclipse
Herraiz, I., Germán, D. M., Gonzalez-Barahona, J. M., & Robles, G. (2008) · doi:10.1145/1370750.1370786 · Cited by 66
Balancing Misclassification Rates in Classification-Tree Models of Software Quality
Khoshgoftaar, T. M., Yuan, X., & Allen, E. B. (2000) · Empirical Software Engineering · doi:10.1023/a:1009896203228 · Cited by 62
Learning from bug-introducing changes to prevent fault prone code
Aversano, L., Cerulo, L., & Grosso, C. D. (2007) · International Workshop on Principles of Software Evolution · doi:10.1145/1294948.1294954 · Cited by 58
Approximating Change Sets at Philips Healthcare: A Case Study
Vanya, A., Premraj, R., & van Vliet, H. (2011) · Euromicro Conference on Software Maintenance and Reengineering (CSMR) · doi:10.1109/csmr.2011.18 · Cited by 8
Cited by
Deep Learning for Just-in-Time Defect Prediction
Yang, X., Lo, D., Xia, X., Zhang, Y., & Sun, J. (2015) · 2015 IEEE International Conference on Software Quality, Reliability and Security · doi:10.1109/qrs.2015.14 · Cited by 352
McIntosh, S., Kamei, Y., Adams, B., & Hassan, A. E. (2014) · Proceedings of the 11th Working Conference on Mining Software Repositories · doi:10.1145/2597073.2597076 · Cited by 348
Self-citationAn empirical study of the impact of modern code review practices on software quality
McIntosh, S., Kamei, Y., Adams, B., & Hassan, A. E. (2016) · Empirical Software Engineering · doi:10.1007/s10664-015-9381-9 · Cited by 332
Self-citationTantithamthavorn, C., Hassan, A. E., & Matsumoto, K. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2876537 · Cited by 327
Self-citationDeep Semantic Feature Learning for Software Defect Prediction
Wang, S., Liu, T., Nam, J., & Tan, L. (2020) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2018.2877612 · Cited by 320
Metadata
- Authors
- Kamei, Y., Shihab, E., Adams, B., Hassan, A. E., Mockus, A., Sinha, A., & Ubayashi, N.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2013-06
- Volume
- 39
- Issue
- 6
- Pages
- 757-773
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W2147386665
- Cited by
- 715
- References
- 68
- FWCI
- 27.96
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.