The DOI matches its Crossref record; 73 of the 73 references it cites were found in a registry or an index.
An Empirical Study of Fault Localization Families and Their Combinations
Open accessAbstract
The performance of fault localization techniques is critical to their adoption in practice. This paper reports on an empirical study of a wide range of fault localization techniques on real-world faults. Different from previous studies, this paper (1) considers a wide range of techniques from different families, (2) combines different techniques, and (3) considers the execution time of different techniques. Our results reveal that a combined technique significantly outperforms any individual technique (200 percent increase in faults localized in Top 1), suggesting that combination may be a desirable way to apply fault localization techniques and that future techniques should also be evaluated in the combined setting. Our implementation is publicly available for evaluating and combining fault localization techniques.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LIBSVM: A library for support vector machines
Chang, C.-C., & Lin, C.-J. (2011) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/1961189.1961199 · Cited by 40,918
Thirteen Ways to Look at the Correlation Coefficient
Lee Rodgers, J., & Nicewander, W. A. (1988) · The American Statistician · doi:10.1080/00031305.1988.10475524 · Cited by 4,052
Learning to rank using gradient descent
Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., & Hullender, G. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102363 · Cited by 2,836
Learning to rank for information retrieval
Liu, T.-Y. (2010) · Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1835449.1835676 · Cited by 1,856
An Analysis and Survey of the Development of Mutation Testing
Jia, Y., & Harman, M. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.62 · Cited by 1,745
Learning to Rank for Information Retrieval
Liu, T.-Y. (2009) · Foundations and Trends® in Information Retrieval · doi:10.1561/1500000016 · Cited by 1,568
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
Self-citationEmpirical evaluation of the tarantula automatic fault-localization technique
Jones, J. A., & Harrold, M. J. (2005) · Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/1101908.1101949 · Cited by 1,238
A Survey on Software Fault Localization
Wong, W. E., Gao, R., Li, Y., Abreu, R., & Wotawa, F. (2016) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2521368 · Cited by 1,110 · cited 2×
“report, rather than a set of test cases, and generate as output a list of relevant source code files [45].”
BackgroundDynamically discovering likely program invariants to support program evolution
Ernst, M. D., Cockrell, J., Griswold, W. G., & Notkin, D. (2001) · IEEE Transactions on Software Engineering · doi:10.1109/32.908957 · Cited by 1,050
Self-citationPinpoint: problem determination in large, dynamic Internet services
Chen, M. Y., Kiciman, E., Fratkin, E., Fox, A., & Brewer, E. (2003) · Proceedings International Conference on Dependable Systems and Networks · doi:10.1109/dsn.2002.1029005 · Cited by 835
Modern information retrieval: the concepts and technology behind search
(2011) · Choice Reviews Online · doi:10.5860/choice.48-6950 · Cited by 786
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 774
Korel, B., & Laski, J. (1988) · Information Processing Letters · doi:10.1016/0020-0190(88)90054-3 · Cited by 759
On the Accuracy of Spectrum-based Fault Localization
Abreu, R., Zoeteweij, P., & van Gemund, A. J. C. (2007) · Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007) · doi:10.1109/taic.part.2007.13 · Cited by 736
Moser, R., Pedrycz, W., & Succi, G. (2008) · Proceedings of the 13th international conference on Software engineering - ICSE '08 · doi:10.1145/1368088.1368114 · Cited by 701
Weiser, M. (1981) · doi:10.5555/800078.802557 · Cited by 695
Are automated debugging techniques actually helping programmers?
Parnin, C., & Orso, A. (2011) · Proceedings of the 2011 International Symposium on Software Testing and Analysis · doi:10.1145/2001420.2001445 · Cited by 576
Locating causes of program failures
Cleve, H., & Zeller, A. (2005) · Proceedings of the 27th international conference on Software engineering - ICSE '05 · doi:10.1145/1062455.1062522 · Cited by 550
A model for spectra-based software diagnosis
Naish, L., Lee, H. J., & Ramamohanarao, K. (2011) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2000791.2000795 · Cited by 536
Zhou, J., Zhang, H., & Lo, D. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227210 · Cited by 507
A practical evaluation of spectrum-based fault localization
Abreu, R., Zoeteweij, P., Golsteijn, R., & van Gemund, A. J. C. (2009) · Journal of Systems and Software · doi:10.1016/j.jss.2009.06.035 · Cited by 495
Predicting Faults from Cached History
Kim, S., Zimmermann, T., Whitehead Jr., E. J., & Zeller, A. (2007) · 29th International Conference on Software Engineering (ICSE'07) · doi:10.1109/icse.2007.66 · Cited by 486
The DStar Method for Effective Software Fault Localization
Wong, W. E., Debroy, V., Gao, R., & Li, Y. (2014) · IEEE Transactions on Reliability · doi:10.1109/tr.2013.2285319 · Cited by 479
Poshyvanyk, D., Gueheneuc, Y.-G., Marcus, A., Antoniol, G., & Rajlich, V. (2007) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2007.1016 · Cited by 469
Fault localization with nearest neighbor queries
Renieres, M., & Reiss, S. P. (2004) · 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings. · doi:10.1109/ase.2003.1240292 · Cited by 427
Improving bug localization using structured information retrieval
Saha, R. K., Lease, M., Khurshid, S., & Perry, D. E. (2013) · 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2013.6693093 · Cited by 390
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H., Goradia, T., & Ostrand, T. (2002) · Proceedings of 16th International Conference on Software Engineering · doi:10.1109/icse.1994.296778 · Cited by 375
Zhou, J. H., Zhang, H., & Lo, D. (2012) · doi:10.5555/2337223.2337226 · Cited by 372
Metallaxis‐FL: mutation‐based fault localization
Papadakis, M., & Le Traon, Y. (2015) · Software Testing, Verification and Reliability · doi:10.1002/stvr.1509 · Cited by 371
A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization
Xie, X., Chen, T. Y., Kuo, F.-C., & Xu, B. (2013) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/2522920.2522924 · Cited by 353
A Short Introduction to Learning to Rank
LI, H. (2011) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.e94.d.1854 · Cited by 348
Practitioners' expectations on automated fault localization
Kochhar, P. S., Xia, X., Lo, D., & Li, S. (2016) · Proceedings of the 25th International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931051 · Cited by 346
Shaping program repair space with existing patches and similar code
Jiang, J., Xiong, Y., Zhang, H., Gao, Q., & Chen, X. (2018) · Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3213846.3213871 · Cited by 329
Self-citationA brief survey of program slicing
Xu, B., Qian, J., Zhang, X., Wu, Z., & Chen, L. (2005) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/1050849.1050865 · Cited by 313
Precise Condition Synthesis for Program Repair
Xiong, Y., Wang, J., Yan, R., Zhang, J., Han, S., Huang, G., & Zhang, L. (2017) · 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2017.45 · Cited by 299
Self-citationAsk the Mutants: Mutating Faulty Programs for Fault Localization
Moon, S., Kim, Y., Kim, M., & Yoo, S. (2014) · 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation · doi:10.1109/icst.2014.28 · Cited by 291
Fault localization using execution slices and dataflow tests
Agrawal, H., Horgan, J. R., London, S., & Wong, W. E. (2002) · Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE'95 · doi:10.1109/issre.1995.497652 · Cited by 288
Locating faults through automated predicate switching
Zhang, X., Gupta, N., & Gupta, R. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134324 · Cited by 277
FLUCCS: using code and change metrics to improve fault localization
Sohn, J., & Yoo, S. (2017) · Proceedings of the 26th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3092703.3092717 · Cited by 242
Learning to Combine Multiple Ranking Metrics for Fault Localization
Xuan, J., & Monperrus, M. (2014) · 2014 IEEE International Conference on Software Maintenance and Evolution · doi:10.1109/icsme.2014.41 · Cited by 210
An empirical investigation of the relationship between spectra differences and regression faults
Harrold, M. J., Rothermel, G., Sayre, K., Wu, R., & Yi, L. (2000) · Software Testing, Verification and Reliability · doi:10.1002/1099-1689(200009)10:3<171::aid-stvr209>3.0.co;2-j · Cited by 205
Boosting Bug-Report-Oriented Fault Localization with Segmentation and Stack-Trace Analysis
Wong, C.-P., Xiong, Y., Zhang, H., Hao, D., Zhang, L., & Mei, H. (2014) · 2014 IEEE International Conference on Software Maintenance and Evolution · doi:10.1109/icsme.2014.40 · Cited by 203
Self-citationA learning-to-rank based fault localization approach using likely invariants
B. Le, T.-D., Lo, D., Le Goues, C., & Grunske, L. (2016) · Proceedings of the 25th International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931049 · Cited by 200
Effective Software Fault Localization Using an RBF Neural Network
Wong, W. E., Debroy, V., Golden, R., Xu, X., & Thuraisingham, B. (2012) · IEEE Transactions on Reliability · doi:10.1109/tr.2011.2172031 · Cited by 189
Precise dynamic slicing algorithms
Zhang, X., Gupta, R., & Zhang, Y. (2003) · doi:10.5555/776816.776855 · Cited by 186
Locus: locating bugs from software changes
Wen, M., Wu, R., & Cheung, S.-C. (2016) · Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering · doi:10.1145/2970276.2970359 · Cited by 172
A Crosstab-based Statistical Method for Effective Fault Localization
Wong, E., Wei, T., Qi, Y., & Zhao, L. (2008) · 2008 International Conference on Software Testing, Verification, and Validation · doi:10.1109/icst.2008.65 · Cited by 166
An empirical investigation of program spectra
Harrold, M. J., Rothermel, G., Wu, R., & Yi, L. (1998) · Proceedings of the 1998 ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering · doi:10.1145/277631.277647 · Cited by 164
Transforming programs and tests in tandem for fault localization
Li, X., & Zhang, L. (2017) · Proceedings of the ACM on Programming Languages · doi:10.1145/3133916 · Cited by 162
Self-citationIdentifying patch correctness in test-based program repair
Xiong, Y., Liu, X., Zeng, M., Zhang, L., & Huang, G. (2018) · Proceedings of the 40th International Conference on Software Engineering · doi:10.1145/3180155.3180182 · Cited by 162
Self-citationSteimann, F., Frenkel, M., & Abreu, R. (2013) · International Symposium on Software Testing and Analysis · doi:10.1145/2483760.2483767 · Cited by 149
CrashLocator: locating crashing faults based on crash stacks
Wu, R., Zhang, H., Cheung, S.-C., & Kim, S. (2014) · International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2610386 · Cited by 143
Evolving Human Competitive Spectra-Based Fault Localisation Techniques
Yoo, S. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-33119-0_18 · Cited by 142
A survey on bug-report analysis
Zhang, J. M., Wang, X.-Y., Hao, D., Xie, B., Zhang, L., & Mei, H. (2015) · Science China Information Sciences · doi:10.1007/s11432-014-5241-2 · Cited by 135
Self-citationAn efficient relevant slicing method for debugging
Gyimóthy, T., Beszédes, Á., & Forgács, I. (1999) · ACM SIGSOFT Software Engineering Notes · doi:10.1145/318774.319248 · Cited by 131
Meng, N., Kim, M., & McKinley, K. S. (2011) · ACM SIGPLAN Notices · doi:10.1145/1993316.1993537 · Cited by 112
BugCache for inspections: hit or miss?
Rahman, F., Posnett, D., Hindle, A., Barr, E. T., & Dévanbu, P. (2011) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/2025113.2025157 · Cited by 109
Xia, X., Bao, L., Lo, D., & Li, S. (2016) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme.2016.67 · Cited by 104
Zhang, J. M., Wang, Z., Zhang, L., Hao, D., Zang, L., Cheng, S., & Zhang, L. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931038 · Cited by 92
Self-citationProvably Optimal and Human-Competitive Results in SBSE for Spectrum Based Fault Localisation
Xie, X., Kuo, F., Chen, T. Y., Yoo, S., & Harman, M. (2013) · Lecture notes in computer science · doi:10.1007/978-3-642-39742-4_17 · Cited by 91
Precise dynamic slicing algorithms
Zhang, X., Gupta, R., & Zhang, Y. (2003) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2003.1201211 · Cited by 87
Using compressed bytecode traces for slicing Java programs
Wang, T., & Roychoudhury, A. (2004) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse.2004.1317473 · Cited by 87
Theory and Practice, Do They Match? A Case with Spectrum-Based Fault Localization
Le, T.-D. B., Thung, F., & Lo, D. (2013) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2013.52 · Cited by 76
A study of effectiveness of dynamic slicing in locating real faults
Zhang, X., Gupta, N., & Gupta, R. (2006) · Empirical Software Engineering · doi:10.1007/s10664-006-9007-3 · Cited by 71
Meng, N., Kim, M., & McKinley, K. S. (2011) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/1993498.1993537 · Cited by 71
Should I follow this fault localization tool’s output?
Le, T.-D. B., Lo, D., & Thung, F. (2014) · Empirical Software Engineering · doi:10.1007/s10664-014-9349-1 · Cited by 53
Heuristics for Automatic Localization of Software Faults
Pan, H., & Spafford, E. H. (1992) · Cited by 48
Kuo, T.-M., Lee, C.-P., & Lin, C. (2014) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611973440.93 · Cited by 41
Hao, D., Zhang, L., Zhong, H., Mei, H., & Sun, J. (2005) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2005.43 · Cited by 29
Self-citationGRT: An Automated Test Generator Using Orchestrated Program Analysis
Ma, L., Artho, C. V., Zhang, C., Satō, H., Gmeiner, J., & Ramler, R. (2015) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.102 · Cited by 20
Cost-effective testing based fault localization with distance based test-suite reduction
Wang, X., Jiang, S., Gao, P., Ju, X., Wang, R., & Zhang, Y. (2017) · Science China Information Sciences · doi:10.1007/s11432-016-9057-8 · Cited by 14
Lightweight fault localization combined with fault context to improve fault absolute rank
Wang, Y., Huang, Z., Li, Y., & Fang, B. (2017) · Science China Information Sciences · doi:10.1007/s11432-017-9112-2 · Cited by 11
Cited by
SEQUENCER: Sequence-to-Sequence Learning for End-to-End Program Repair
Chen, Z., Kommrusch, S. J., Tufano, M., Pouchet, L.-N., Poshyvanyk, D., & Monperrus, M. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2019.2940179 · Cited by 396
DeepFL: integrating multiple fault diagnosis dimensions for deep fault localization
Li, X., Li, W., Zhang, Y., & Zhang, L. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330574 · Cited by 336
Self-citationBoosting coverage-based fault localization via graph-based representation learning
Lou, Y., Zhu, Q., Dong, J., Li, X., Sun, Z., Hao, D., Zhang, L., & Zhang, L. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468580 · Cited by 163
Self-citationA Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization
Kang, S., An, G., & Yoo, S. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660771 · Cited by 128
“The precision of SBFL, MBFL, and IRFL was recalculated based on the artifacts of Zou et al. [53]; for other techniques, precision comes from the corresponding papers.”
InfluentialTowards Better Graph Neural Network-Based Fault Localization through Enhanced Code Representation
Rafi, M. N., Kim, D. J., Chen, A. R., Chen, T.-H. (., & Wang, S. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3660793 · Cited by 24
Metadata
- Authors
- Zou, D., Liang, J., Xiong, Y., Ernst, M. D., & Zhang, L.
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2021-02-01
- Volume
- 47
- Issue
- 2
- Pages
- 332-347
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- 2326-3881
- OpenAlex
- W2962715466
- Cited by
- 270
- References
- 73
- FWCI
- 75.08
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.