The DOI matches its Crossref record; 45 of the 45 references it cites were found in a registry or an index.
Evaluating and Improving Fault Localization
Abstract
Most fault localization techniques take as input a faulty program, and produce as output a ranked list of suspicious code locations at which the program may be defective. When researchers propose a new fault localization technique, they typically evaluate it on programs with known faults. The technique is scored based on where in its output list the defective code appears. This enables the comparison of multiple fault localization techniques to determine which one is better. Previous research has evaluated fault localization techniques using artificial faults, generated either by mutation tools or manually. In other words, previous research has determined which fault localization techniques are best at finding artificial faults. However, it is not known which fault localization techniques are best at finding real faults. It is not obvious that the answer is the same, given previous work showing that artificial faults have both similarities to and differences from real faults. We performed a replication study to evaluate 10 claims in the literature that compared fault localization techniques (from the spectrum-based and mutation-based families). We used 2995 artificial faults in 6 real-world programs. Our results support 7 of the previous claims as statistically significant, but only 3 as having non-negligible effect sizes. Then, we evaluated the same 10 claims, using 310 real faults from the 6 programs. Every previous result was refuted or was statistically and practically insignificant. Our experiments show that artificial faults are not useful for predicting which fault localization techniques perform best on real faults. In light of these results, we identified a design space that includes many previously-studied fault localization techniques as well as hundreds of new techniques. We experimentally determined which factors in the design space are most important, using an overall set of 395 real faults. Then, we extended this design space with new techniques. Several of our novel techniques outperform all existing techniques, notably in terms of ranking defective code in the top-5 or top-10 reports.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
An Analysis and Survey of the Development of Mutation Testing
Jia, Y., & Harman, M. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.62 · Cited by 1,745
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
Self-citationEmpirical evaluation of the tarantula automatic fault-localization technique
Jones, J. A., & Harrold, M. J. (2005) · Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/1101908.1101949 · Cited by 1,238
A Survey on Software Fault Localization
Wong, W. E., Gao, R., Li, Y., Abreu, R., & Wotawa, F. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2521368 · Cited by 1,110
Self-citationSimplifying and isolating failure-inducing input
Zeller, A., & Hildebrandt, R. (2002) · IEEE Transactions on Software Engineering · doi:10.1109/32.988498 · Cited by 1,079
Is mutation an appropriate tool for testing experiments?
Andrews, J. H., Briand, L. C., & Labiche, Y. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062530 · Cited by 777
On the Accuracy of Spectrum-based Fault Localization
Abreu, R., Zoeteweij, P., & van Gemund, A. J. C. (2007) · Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007) · doi:10.1109/taic.part.2007.13 · Cited by 736
Self-citationAre mutants a valid substitute for real faults in software testing?
Just, R., Jalali, D., Inozemtseva, L., Ernst, M. D., Holmes, R., & Fraser, G. (2014) · Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering · doi:10.1145/2635868.2635929 · Cited by 595
Self-citationAre automated debugging techniques actually helping programmers?
Parnin, C., & Orso, A. (2011) · Proceedings of the 2011 International Symposium on Software Testing and Analysis · doi:10.1145/2001420.2001445 · Cited by 576
A model for spectra-based software diagnosis
Naish, L., Lee, H. J., & Ramamohanarao, K. (2011) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2000791.2000795 · Cited by 536
A practical evaluation of spectrum-based fault localization
Abreu, R., Zoeteweij, P., Golsteijn, R., & van Gemund, A. J. C. (2009) · Journal of Systems and Software · doi:10.1016/j.jss.2009.06.035 · Cited by 495
Self-citationUsing Mutation Analysis for Assessing and Comparing Testing Coverage Criteria
Andrews, J. H., Briand, L. C., Labiche, Y., & Namin, A. S. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.83 · Cited by 481
The DStar Method for Effective Software Fault Localization
Wong, W. E., Debroy, V., Gao, R., & Li, Y. (2014) · IEEE Transactions on Reliability · doi:10.1109/tr.2013.2285319 · Cited by 479
Spectrum-Based Multiple Fault Localization
Abreu, R., Zoeteweij, P., & van Gemund, A. J. C. (2009) · 2009 IEEE/ACM International Conference on Automated Software Engineering · doi:10.1109/ase.2009.25 · Cited by 401
Self-citationExperiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H., Goradia, T., & Ostrand, T. (2002) · Proceedings of 16th International Conference on Software Engineering · doi:10.1109/icse.1994.296778 · Cited by 375
Metallaxis‐FL: mutation‐based fault localization
Papadakis, M., & Le Traon, Y. (2015) · Software Testing, Verification and Reliability · doi:10.1002/stvr.1509 · Cited by 371
Is mutation an appropriate tool for testing experiments?
Andrews, J. H., Brand, L. C., & Labiche, Y. (2005) · Proceedings. 27th International Conference on Software Engineering, 2005. ICSE 2005. · doi:10.1109/icse.2005.1553583 · Cited by 368
Practitioners' expectations on automated fault localization
Kochhar, P. S., Xia, X., Lo, D., & Li, S. (2016) · Proceedings of the 25th International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931051 · Cited by 346
Weiser, M. (1979) · Deep Blue (University of Michigan) · doi:10.7302/11363 · Cited by 341
A brief survey of program slicing
Xu, B., Qian, J., Zhang, X., Wu, Z., & Chen, L. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1050849.1050865 · Cited by 313
Ask the Mutants: Mutating Faulty Programs for Fault Localization
Moon, S., Kim, Y., Kim, M., & Yoo, S. (2014) · 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation · doi:10.1109/icst.2014.28 · Cited by 291
Statistical Debugging: A Hypothesis Testing-Based Approach
Liu, C., Fei, L., Yan, X., Han, J., & Midkiff, S. P. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.105 · Cited by 288
On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques
Do, H., & Rothermel, G. (2006) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2006.92 · Cited by 287
Lightweight fault-localization using multiple coverage types
Santelices, R., Jones, J. A., Yu, Y., & Harrold, M. J. (2009) · 2009 IEEE 31st International Conference on Software Engineering · doi:10.1109/icse.2009.5070508 · Cited by 264
The major mutation framework: efficient and scalable mutation analysis for Java
Just, R. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628053 · Cited by 216
Self-citationGZoltar: an eclipse plug-in for testing and debugging
Campos, J., Riboira, A., Perez, A., & Abreu, R. (2012) · Proceedings of the 27th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/2351676.2351752 · Cited by 208
Self-citationAn Analysis of the Search Spaces for Generate and Validate Patch Generation Systems
Long, F., & Rinard, M. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884872 · Cited by 167
A Crosstab-based Statistical Method for Effective Fault Localization
Wong, E., Wei, T., Qi, Y., & Zhao, L. (2008) · 2008 International Conference on Software Testing, Verification, and Validation · doi:10.1109/icst.2008.65 · Cited by 166
Daran, M., & Thévenod-Fosse, P. (1996) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/226295.226313 · Cited by 154
Steimann, F., Frenkel, M., & Abreu, R. (2013) · International Symposium on Software Testing and Analysis · doi:10.1145/2483760.2483767 · Cited by 149
Self-citationMAJOR: An efficient and extensible tool for mutation analysis in a Java compiler
Just, R., Schweiggert, F., & Kapfhammer, G. M. (2011) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2011.6100138 · Cited by 136
Self-citationTest case purification for improving fault localization
Xuan, J., & Monperrus, M. (2014) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2635868.2635906 · Cited by 127
Automated Fault Localization Using Potential InvariantsPreprint
Pytlik, B., Renieris, M., Krishnamurthi, S., & Reiss, S. P. (2003) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0310040 · Cited by 120
4 Test Case Purification for Improving Fault Localization
Xuan, J., Monperrus, M., & Hal, H. I. (2014) · Cited by 100
Using automated program repair for evaluating the effectiveness of fault localization techniques
Qi, Y., Mao, X., Lei, Y., & Wang, C. (2013) · International Symposium on Software Testing and Analysis · doi:10.1145/2483760.2483785 · Cited by 99
A Low-Cost Approximate Minimal Hitting Set Algorithm and Its Application to Model-Based Diagnosis ∗
Abreu, R., & van Gemund, A. J. C. (2009) · Cited by 87
Self-citationTheory and Practice, Do They Match? A Case with Spectrum-Based Fault Localization
Le, T.-D. B., Thung, F., & Lo, D. (2013) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2013.52 · Cited by 76
HSFal: Effective fault localization using hybrid spectrum of full slices and execution slices
Ju, X., Jiang, S., Chen, X., Wang, X., Zhang, Y., & Cao, H. (2013) · Journal of Systems and Software · doi:10.1016/j.jss.2013.11.1109 · Cited by 71
Model-Based Debugging or How to Diagnose Programs Automatically
Wotawa, F., Stumptner, M., & Mayer, W. (2002) · Lecture notes in computer science · doi:10.1007/3-540-48035-8_72 · Cited by 71
Fault density, fault types, and spectra-based fault localization
DiGiuseppe, N., & Jones, J. A. (2014) · Empirical Software Engineering · doi:10.1007/s10664-014-9304-1 · Cited by 65
The use of mutation in testing experiments and its sensitivity to external threats
Namin, A. S., & Kakarla, S. (2011) · International Symposium on Software Testing and Analysis · doi:10.1145/2001420.2001461 · Cited by 65
Evaluating the Accuracy of Fault Localization Techniques
Ali, S., Andrews, J. H., Dhandapani, T., & Wang, W. (2009) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2009.89 · Cited by 62
Using HTML5 visualizations in software fault localization
Gouveia, C. M. D. S., Campos, J., & Abreu, R. (2013) · IEEE Working Conference on Software Visualization (VISSOFT) · doi:10.1109/vissoft.2013.6650539 · Cited by 57
Self-citationShould I follow this fault localization tool’s output?
Le, T.-D. B., Lo, D., & Thung, F. (2014) · Empirical Software Engineering · doi:10.1007/s10664-014-9349-1 · Cited by 53
Cifuentes, C., Hoermann, C., Keynes, N., Li, L., Long, S., Mealy, E., Mounteney, M., & Scholz, B. (2009) · doi:10.1145/1555860.1555866 · Cited by 47
Cited by
DeepFL: integrating multiple fault diagnosis dimensions for deep fault localization
Li, X., Li, W., Zhang, Y., & Zhang, L. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330574 · Cited by 336
Shaping program repair space with existing patches and similar code
Jiang, J., Xiong, Y., Zhang, H., Gao, Q., & Chen, X. (2018) · Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3213846.3213871 · Cited by 329
TBar: revisiting template-based automated program repair
Liu, K., Koyuncu, A., Kim, D., & Bissyandé, T. F. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330577 · Cited by 321
Boosting coverage-based fault localization via graph-based representation learning
Lou, Y., Zhu, Q., Dong, J., Li, X., Sun, Z., Hao, D., Zhang, L., & Zhang, L. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468580 · Cited by 163
ChangeRCA: finding root causes from software changes in large online systems
Yu, G., Chen, P., He, Z., Yan, Q., Luo, Y., Li, F., & Zheng, Z. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3643728 · Cited by 9
Metadata
- Authors
- Pearson, S., Campos, J., Just, R., Fraser, G., Abreu, R., Ernst, M. D., Pang, D., & Keller, B.
- Venue
- 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE)
- Publisher
- IEEE
- Published
- 2017-05
- Pages
- 609-620
- OpenAlex
- W2620081107
- Cited by
- 426
- References
- 45
- FWCI
- 35.60
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.