CoChat
Sign Up
Cited by 128Software Testing and Debugging TechniquesSoftware Engineering ResearchSoftware System Performance and Reliability

A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization

Open access
Kang, Sungmin, An, Gabin, Yoo, Shin·2024·Proceedings of the ACM on Software Engineering
DOI 10.1145/3660771
Open this paper in CoChat — free account

Abstract

Fault Localization (FL), in which a developer seeks to identify which part of the code is malfunctioning and needs to be fixed, is a recurring challenge in debugging. To reduce developer burden, many automated FL techniques have been proposed. However, prior work has noted that existing techniques fail to provide rationales for the suggested locations, hindering developer adoption of these techniques. With this in mind, we propose A uto FL, a Large Language Model (LLM)-based FL technique that generates an explanation of the bug along with a suggested fault location. A uto FL prompts an LLM to use function calls to navigate a repository, so that it can effectively localize faults over a large software repository and overcome the limit of the LLM context length. Extensive experiments on 798 real-world bugs in Java and Python reveal A uto FL improves method-level acc@1 by up to 233.3% over baselines. Furthermore, developers were interviewed on their impression of A uto FL-generated explanations, showing that developers generally liked the natural language explanations of A uto FL, and that they preferred reading a few, high-quality explanations instead of many.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3660771

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs do not generate explicit numeric scores, making MFR and MAR unsuitable for LLM-based FL.

    “These metrics are unsuitable for LLM-based approaches because LLMs are inherently language models that do not generate explicit numeric scores.”

    From Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization (2024) · cites this paper as [14]

    Section 3.3 (p. 7) says AutoFL, an LLM-based fault-localization technique, assigns numeric scores to candidate methods and ranks them by those scores. The evaluation also uses ranking metrics (RQ2, p. 9). The paper does not specifically discuss MFR or MAR, but its scored and ranked LLM-based results contradict the stated reason for calling those metrics unsuitable. (LLM, conf=0.84)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

  • Most LLM-based fault localization techniques either cannot perform repository-level fault localization due to limited context window or rely on complicated or costly agentic design.

    “Meanwhile, most such LLM-based techniques either cannot perform repository-level FL due to the limited context window of LLMs [Yang et al. 2024b], or rely on complicated/costly agentic design to navigate through the codebase [Kang et al. 2024;Qin et al. 2024].”

    From Demystifying LLM-Based Software Engineering Agents (2025) · cites this paper as Kang et al. 2024

    The Approach (p. 4) says AutoFL uses autonomous function calls to explore a large repository, supporting the claim that it relies on an agentic navigation design. But the paper describes its navigation functions as simple (p. 6) and its runtime as lightweight relative to a baseline (Results, p. 12); it does not support calling the design complicated or costly. The separate context-window claim is attached to Yang et al., not this source. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Most LLM-based fault localization techniques either cannot perform repository-level FL due to the limited context window of LLMs or rely on complicated/costly agentic design to navigate through the codebase.

    “Meanwhile, most such LLM-based techniques either cannot perform repository-level FL due to the limited context window of LLMs [98], or rely on complicated/costly agentic design to navigate through the codebase [48, 79].”

    From Agentless: Demystifying LLM-based Software Engineering Agents (2024) · cites this paper as [48]

    The Introduction (p. 1) says repository-scale fault localization is difficult for LLMs because repositories exceed their context limits, and describes AutoFL’s function calls as a solution. It does not establish that most LLM-based techniques cannot perform repository-level localization. The agentic-design clause is attached to other citations, not this source. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 43

  1. [1][1] Rui Abreu, Peter Zoeteweij, and Arjan J.C. van Gemund. 2007. On the Accuracy of Spectrum-based Fault Localization. In Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007) . 89-98. https://doi.org/10.1109/TAIC.PART.2007.13 - · DOI
  2. [2][2] H. Agrawal, J.R. Horgan, S. London, and W.E. Wong. 1995. Fault localization using execution slices and dataflow tests. In Proceedings of Sixth International Symposium on Software Reliability Engineering. ISSRE'95 . 143-151. https: //doi.org/10.1109/ISSRE.1995.497652 - · DOI
  3. [7][7] Thomas Bach, Artur Andrzejak, Changyun Seo, Christian Bierstedt, Christian Lemke, and Daniel Ritter et al. 2022. Testing Very Large Database Management Systems: The Case of SAP HANA. Datenbank-Spektrum 22, 3 (nov 2022), 195-215. https://doi.org/10.1007/s13222-022-00426-x - · DOI
  4. [8][8] Marcel Böhme, Ezekiel O. Soremekun, Sudipta Chattopadhyay, Emamurho Ugherughe, and Andreas Zeller. 2017. Where is the Bug and How is It Fixed? An Experiment with Practitioners. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering (Paderborn, Germany) (ESEC/FSE 2017) . Association for Computing Machinery, New York, NY, USA, 117-128. https://doi.org/10.1145/3106237.3106255 - · DOI
  5. [9][9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, and Prafulla et al. Dhariwal. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901. https://doi.org/10.5555/3495724.3495883 - · DOI
  6. [11][11] Xueying Du, Yiling Lou, Mingwei Liu, Xin Peng, and Tianyong Yang. 2023. KG4CraSolver: Recommending Crash Solutions via Knowledge Graph. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE '23) . ACM. https://doi.org/10.1145/3611643.3616317 - · DOI
  7. [12][12] Joseph L Fleiss, Bruce Levin, and Myunghee Cho Paik. 2013. Statistical methods for rates and proportions . John Wiley & Sons. https://doi.org/10.1002/0471445428 - · DOI · Paper page
  8. [13][13] Marko Ivankovic, Goran Petrovic, René Just, and Gordon Fraser. 2019. Code coverage at Google. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . 955-963. https://doi.org/10.1145/3338906.3340459 - · DOI
  9. [14][14] Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan. 2023. Impact of Code Language Models on Automated Program Repair. In Proceedings of the 45th International Conference on Software Engineering (Melbourne, Victoria, Australia) (ICSE '23) . IEEE Press, 1430-1442. https://doi.org/10.1109/ICSE48619.2023.00125 - · DOI · Paper page
  10. [16][16] Sungmin Kang, Bei Chen, Shin Yoo, and Jian-Guang Lou. 2023. Explainable Automated Debugging via Large Language Model-driven Scientific Debugging. https://doi.org/10.48550/arXiv.2304.02195 arXiv:2304.02195 [cs.SE] - · DOI
  11. [17][17] Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction. In Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE 2023) . https://doi.org/10.1109/ICSE48619.2023.00194 - · DOI · Paper page
  12. [18][18] Yong Woo Kim. 2003. Efficient use of code coverage in large-scale software development. In Proceedings of the 2003 conference of the Centre for Advanced Studies on Collaborative research . 145-155. -
  13. [19][19] Amy J. Ko, Thomas D. LaToza, and Margaret M. Burnett. 2015. A Practical Guide to Controlled Experiments of Software Engineering Tools with Human Participants. Empirical Softw. Engg. 20, 1 (feb 2015), 110-141. https: //doi.org/10.1007/s10664-013-9279-3 - · DOI
  14. [20][20] Pavneet Singh Kochhar, Xin Xia, David Lo, and Shanping Li. 2016. Practitioners' Expectations on Automated Fault Localization. In Proceedings of the 25th International Symposium on Software Testing and Analysis (Saarbrücken, Germany) (ISSTA 2016) . Association for Computing Machinery, New York, NY, USA, 165-176. https://doi.org/10.1145/ 2931037.2931051 -
  15. [21][21] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large Language Models are Zero-Shot Reasoners. https://doi.org/10.48550/arXiv.2205.11916 arXiv:2205.11916 [cs.CL] - · DOI · Paper page
  16. [22][22] Yiğit Küçük, Tim A. D. Henderson, and Andy Podgurski. 2021. Improving Fault Localization by Integrating Value and Predicate Based Causal Inference Techniques. In Proceedings of the 43rd International Conference on Software Engineering (Madrid, Spain) (ICSE '21) . IEEE Press, 649-660. https://doi.org/10.1109/ICSE43902.2021.00066 - · DOI
  17. [24][24] Yi Li, Shaohua Wang, and Tien N. Nguyen. 2021. Fault Localization with Code Coverage Representation Learning. In Proceedings of the 43rd International Conference on Software Engineering (Madrid, Spain) (ICSE '21) . IEEE Press, 661-673. https://doi.org/10.1109/ICSE43902.2021.00067 - · DOI
  18. [25][25] Yiling Lou, Ali Ghanbari, Xia Li, Lingming Zhang, Haotian Zhang, Dan Hao, and Lu Zhang. 2020. Can automated program repair refine fault localization? a unified debugging approach. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis . 75-87. https://doi.org/10.1145/3395363.3397351 - · DOI
  19. [26][26] Parvez Mahbub, Ohiduzzaman Shuvo, and Mohammad Masudur Rahman. 2023. Explaining Software Bugs Leveraging Code Structures in Neural Machine Translation. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE. https://doi.org/10.1109/icse48619.2023.00063 - · DOI
  20. [27][27] Alexandru Marginean, Johannes Bader, Satish Chandra, Mark Harman, Yue Jia, Ke Mao, Alexander Mols, and Andrew Scott. 2019. SapFix: Automated End-to-End Repair at Scale. In 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) . 269-278. https://doi.org/10.1109/ICSE-SEIP.2019.00039 - · DOI
  21. [28][28] Matias Martinez, Thomas Durieux, Romain Sommerard, Jifeng Xuan, and Martin Monperrus. 2017. Automatic repair of real bugs in java: A large-scale experiment on the defects4j dataset. Empirical Software Engineering 22 (2017), 1936-1964. https://doi.org/10.1007/s10664-016-9470-4 - · DOI
  22. [29][29] Seokhyeon Moon, Yunho Kim, Moonzoo Kim, and Shin Yoo. 2014. Ask the Mutants: Mutating Faulty Programs for Fault Localization. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation . 153-162. https://doi.org/10.1109/ICST.2014.28 - · DOI
  23. [30][30] OpenAI. 2023. GPT-4 Technical Report. https://doi.org/10.48550/arXiv.2303.08774 arXiv:2303.08774 [cs.CL] - · DOI · Paper page
  24. [32][32] Mike Papadakis and Yves Le Traon. 2015. Metallaxis-FL: Mutation-Based Fault Localization. Softw. Test. Verif. Reliab. 25, 5-7 (aug 2015), 605-628. https://doi.org/10.1002/stvr.1509 - · DOI
  25. [33][33] Chris Parnin and Alessandro Orso. 2011. Are Automated Debugging Techniques Actually Helping Programmers?. In Proceedings of the 2011 International Symposium on Software Testing and Analysis (Toronto, Ontario, Canada) (ISSTA '11) . Association for Computing Machinery, New York, NY, USA, 199-209. https://doi.org/10.1145/2001420.2001445 - · DOI
  26. [34][34] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face. https://doi.org/10.48550/arXiv.2303.17580 arXiv:2303.17580 [cs.CL] - · DOI · Paper page
  27. [36][36] Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. 2021. Retrieval Augmentation Reduces Hallucination in Conversation. https://doi.org/10.48550/arXiv.2104.07567 arXiv:2104.07567 [cs.CL] - · DOI
  28. [37][37] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, and Yasmine Babaei et al. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. https://doi.org/10.48550/arXiv.2307.09288 arXiv:2307.09288 [cs.CL] - · DOI · Paper page
  29. [38][38] Song Wang, Nishtha Shrestha, Abarna Kucheri Subburaman, Junjie Wang, Moshi Wei, and Nachiappan Nagappan. 2021. Automatic Unit Test Generation for Machine Learning Libraries: How Far Are We?. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) . 1548-1560. https://doi.org/10.1109/ICSE43902.2021.00138 - · DOI
  30. [39][39] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. In The Eleventh International Conference on Learning Representations . https://doi.org/10.48550/arXiv.2203.11171 - · DOI · Paper page
  31. [40][40] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, Quoc Le, and Denny Zhou. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. ArXiv abs/2201.11903 (2022). https: //doi.org/10.48550/arXiv.2201.11903 - · DOI · Paper page
  32. [41][41] Ratnadira Widyasari, Gede Artha Azriadi Prana, Stefanus Agus Haryono, Shaowei Wang, and David Lo. 2022. Real world projects, real faults: evaluating spectrum based fault localization techniques on Python projects. Empirical Software Engineering 27, 6 (2022), 147. https://doi.org/10.1007/s10664-022-10189-4 - · DOI
  33. [42][42] Ratnadira Widyasari, Sheng Qin Sim, Camellia Lok, Haodi Qi, Jack Phan, Qijin Tay, Constance Tan, Fiona Wee, Jodie Ethelda Tan, Yuheng Yieh, Brian Goh, Ferdian Thung, Hong Jin Kang, Thong Hoang, David Lo, and Eng Lieh Ouh. 2020. BugsInPy: A Database of Existing Bugs in Python Programs to Enable Controlled Testing and Debugging Studies. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Virtual Event, USA) (ESEC/FSE 2020) . Association for Computing Machinery, New York, NY, USA, 1556-1560. -
  34. [43][43] Emily Rowan Winter, Vesna Nowack, David Bowes, Steve Counsell, Tracy Hall, Sæmundur Haraldsson, John Woodward, Serkan Kirbas, Etienne Windels, Olayori McBello, Abdurahman Atakishiyev, Kevin Kells, and Matthew Pagano. 2022. Towards Developer-Centered Automatic Program Repair: Findings from Bloomberg. In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Singapore, Singapore) (ESEC/FSE 2022) . Association for Computing Machinery, New York, NY, USA, 1578-1588. https://doi.org/10.1145/3540250.3558953 - · DOI
  35. [44][44] W. Eric Wong, Vidroha Debroy, Ruizhi Gao, and Yihao Li. 2014. The DStar Method for Effective Software Fault Localization. IEEE Transactions on Reliability 63, 1 (2014), 290-308. https://doi.org/10.1109/TR.2013.2285319 - · DOI
  36. [45][45] W. Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016. A Survey on Software Fault Localization. IEEE Transactions on Software Engineering 42, 8 (2016), 707-740. https://doi.org/10.1109/TSE.2016.2521368 - · DOI · Paper page
  37. [47][47] Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2022. Practical Program Repair in the Era of Large Pre-trained Language Models. arXiv preprint arXiv:2210.14179 (2022). https://doi.org/10.48550/arXiv.2210.14179 - · DOI
  38. [48][48] Chunqiu Steven Xia and Lingming Zhang. 2023. Conversational Automated Program Repair. https://doi.org/10.48550/ arXiv.2301.13246 arXiv:2301.13246 [cs.SE] - · arXiv
  39. [49][49] Xin Xia, Lingfeng Bao, David Lo, and Shanping Li. 2016. 'Automated Debugging Considered Harmful' Considered Harmful: A User Study Revisiting the Usefulness of Spectra-Based Fault Localization Techniques with Professionals Using Real Bugs from Large Systems. In 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME) . 267-278. https://doi.org/10.1109/ICSME.2016.67 - · DOI
  40. [50][50] Jifeng Xuan and Martin Monperrus. 2014. Test case purification for improving fault localization. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (Hong Kong, China) (FSE 2014) . Association for Computing Machinery, New York, NY, USA, 52-63. https://doi.org/10.1145/2635868.2635906 - · DOI
  41. [51][51] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 (2022). https://doi.org/10.48550/arXiv.2210. 03629 - · DOI
  42. [52][52] Muhan Zeng, Yiqian Wu, Zhentao Ye, Yingfei Xiong, Xin Zhang, and Lu Zhang. 2022. Fault Localization via Efficient Probabilistic Modeling of Program Semantics. In Proceedings of the 44th International Conference on Software Engineering (Pittsburgh, Pennsylvania) (ICSE '22) . Association for Computing Machinery, New York, NY, USA, 958-969. https: //doi.org/10.1145/3510003.3510073 - · DOI
  43. [53][53] Daming Zou, Jingjing Liang, Yingfei Xiong, Michael D. Ernst, and Lu Zhang. 2019. An empirical study of fault localization families and their combinations. IEEE Transactions on Software Engineering 47 (Feb. 2019), 332-347. https://doi.org/10.1109/TSE.2019.2892102 · DOI

Cited by

Sorted by citations

Authors
Kang, S., An, G., & Yoo, S.
Affiliations
Korea Advanced Institute of Science and Technology (KR)
ORCID
Venue
Proceedings of the ACM on Software Engineering
Publisher
Association for Computing Machinery (ACM)
Published
2024-07-12
Volume
1
Issue
FSE
Pages
1424-1446
ISSN
2994-970X (ISSN-L)
DOI
10.1145/3660771
OpenAlex
W4400582623
Cited by
128
References
5
Citations by year
2026: 63 · 2025: 51 · 2024: 13 · 2023: 1
FWCI
130.22
Type
journal article
Language
en
Open access
Licence
Locations
  • Proceedings of the ACM on software engineering. (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Funders
  • National Research Foundation of Korea (RS-2023-00208998)
  • Engineering Research Center Program (2021R1A5A1021944)
  • Institute of Information & Communications Technology Planning & Evaluation (2021-0-01001)
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
Keywords
fault location, large language models, natural language explanations, developer interviews, developer adoption

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review