The DOI matches its Crossref record; 46 of the 48 references it cites were found in a registry or an index.
Large Language Models for Test-Free Fault Localization
Open accessAbstract
Fault Localization (FL) aims to automatically localize buggy lines of code, a key first step in many manual and automatic debugging tasks. Previous FL techniques assume the provision of input tests, and often require extensive program analysis, program instrumentation, or data preprocessing. Prior work on deep learning for APR struggles to learn from small datasets and produces limited results on real-world programs. Inspired by the ability of large language models (LLMs) of code to adapt to new tasks based on very few examples, we investigate the applicability of LLMs to line level fault localization. Specifically, we propose to overcome the left-to-right nature of LLMs by fine-tuning a small set of bidirectional adapter layers on top of the representations learned by LLMs to produce LLMAO, the first language model based fault localization approach that locates buggy lines of code without any test coverage information. We fine-tune LLMs with 350 million, 6 billion, and 16 billion parameters on small, manually curated corpora of buggy programs such as the Defects4J corpus. We observe that our technique achieves substantially more confidence in fault localization when built on the larger models, with bug localization performance scaling consistently with the LLM size. Our empirical evaluation shows that LLMAO improves the Top-1 results over the state-of-the-art machine learning fault localization (MLFL) baselines by 2.3%--54.4%, and Top-5 results by 14.4%-35.6%. LLMAO is also the first FL technique trained using a language model architecture that can detect security vulnerabilities down to the code line level.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Meanwhile, most such LLM-based techniques either cannot perform repository-level FL due to the limited context window of LLMs [98], or rely on complicated/costly agentic design to navigate through the codebase [48, 79].”
From Agentless: Demystifying LLM-based Software Engineering Agents (2024) · cites this paper as [98]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We use SWE-bench because it unites many separate SE tasks, such as automated program repair [10, 40, 55], bug localization [4, 58], and testing [22, 46, 56] under a single task formulation that faithfully mirrors practical SE.”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [58]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
The graph neural network model
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G. (2009) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2005605 · Cited by 9,582
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“A key property of LLMs is that their performance improves consistently with the scale of their computational budget [16], which is itself a function of the model and training data size.”
BackgroundLearning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,451
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
Empirical evaluation of the tarantula automatic fault-localization technique
Jones, J. A., & Harrold, M. J. (2005) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/1101908.1101949 · Cited by 1,274
Deep Learning Based Vulnerability Detection: Are We There Yet?
Chakraborty, S., Krishna, R., Ding, Y., & Ray, B. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3087402 · Cited by 970
Flagged first, then by citations
2 of the 48 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- L. Tunstall, L. Von Werra, and T. Wolf, Natural language processing with transformers. O'Reilly Media, Inc., 2022.
- B. Ray, V. Hellendoorn, S. Godhane, Z. Tu, A. Bacchelli, and P. Devanbu, "On the" naturalness" of buggy code," in Proceedings of the 38th International Conference on Software Engineering, pp. 428--439, 2016.
- L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al., "The pile: An 800gb dataset of diverse text for language modeling," arXiv preprint arXiv:2101.00027, 2020. · arXiv
- P. S. Kochhar, X. Xia, D. Lo, and S. Li, "Practitioners' expectations on automated fault localization," in Proceedings of the 25th International Symposium on Software Testing and Analysis, pp. 165--176, 2016.
- H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, "An empirical cyber-security evaluation of github copilot's code contributions," ArXiv abs/2108.09293, 2021. · arXiv
Cited by
Sorted by citations
Items 101–148 of 148
Improving fault localization with semantic enhancement and decision fusion
Liu, B., & He, P. (2025) · International Conference on Computing and Artificial Intelligence (ICCAI) · doi:10.1109/iccai66501.2025.00113
Diagnosing Deep Learning Errors with Reinforcement Learning-Driven Adversarial Examples
Shao, K., Lu, Y., Ai, L., & Sun, M. (2025) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs65678.2025.00026
A Hybrid Approach to Fault Localization: Integrating LLMs with IR-Based Methods
Nguyen, T. B., Cao, T. N., Nguyen, V. T., & Nguyen, N. T. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-00972-2_27
Fine-Grained Bug Localization in Software Projects Using Code Graph
Sunitha, E. V. (2026) · Lecture notes in networks and systems · doi:10.1007/978-981-96-9275-0_38
Explainable Fault Localization for Programming Assignments via LLM-Guided Annotation
Liu, F., Wang, T., Zhang, L., Yang, Z., Jiang, J., & Sun, Z. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00167 · Cited by 3
Jiao, R., Zhang, Y., Li, J., & Ma, Z. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00071 · Cited by 2
Polyglot: An Extensible Framework to Benchmark Code Translation with LLMs
Vieira, M., Shah, P. A., Shah, B., & Krasniqi, R. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00195
Human Cognitive Pattern Simulation for Crowdsourced Test Report Consistency Detection
Ling, Y., Yu, S., Chen, S., Wang, L., Fang, C., Guo, W., Liu, J., & Chen, Z. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3659293
FACKT: A Fault-Aware Model for Code Knowledge Tracing
Zhou, M., Zhou, Q., Xia, W., Jiang, Y., Lin, C., & Zhao, S. (2026) · Lecture notes in computer science · doi:10.1007/978-981-95-5719-6_5
Large Language Models-Driven Bug Localization Framework for Automated Debugging
Dhanalakshmi, J., Naik, D. A., Evangeline, C., Shanthi, A. V. K., Eapen, M., & Dhanraj, J. A. (2025) · International Conference on Artificial Intelligence and Knowledge Discovery in Concurrent Engineering (ICECONF) · doi:10.1109/iceconf65644.2025.11379560
Time Travel: LLM-Assisted Semantic Behavior Localization with Git Bisect
Wang, Y., & Hong, W. (2025) · IEEE Smart World Congress (SWC) · doi:10.1109/swc65939.2025.00299
Making Root Cause Localization on FPGA Simulation Tools Robust
Zhang, X., Guo, S., Fu, Z., Dong, P., Wang, N., Li, X., & Jiang, H. (2026) · ACM Transactions on Architecture and Code Optimization · doi:10.1145/3799984
Unleashing the Potential of Coverage Representation in Deep Learning-Based Fault Localization
Wang, J., Lei, Y., Xie, H., & Liu, C. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3674478
SLMFL: Structural and semantic fusion for fault localization based on Small Language Models
Fan, X., Hu, J., Zheng, W., Liu, Y., & Gao, S. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108143 · Cited by 1
An Analytical Study of Fine-Tuning Approaches for LLMs in Software Engineering
Zougari, O., Kharmoum, N., Rami, D., Kodad, M., & Ziti, S. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-18716-1_1
Adaptive Self-Prompting in Agentic LLM Frameworks for Code Fault Detection
Muhtadi, M., Mahmoud, Q. H., & Azim, A. (2026) · Software · doi:10.3390/software5020016
Arash : Token-Efficient LLM-Assisted Crash Root Cause Analysis in Fuzz Driver Generation
Xie, M., Li, K., Ge, J., Ma, W., Sun, Y., Kong, Z., Zhang, C., Liu, D., Chang, O., & Liu, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3811034
Zheng, W., Xu, A., Fan, X., Pan, H., Yang, F., & Xiao, P. (2026) · Software Quality Journal · doi:10.1007/s11219-026-09786-z
Iterative self-organizing clustering for software multiple-fault localisation
Cao, H., Jia, R., Tian, F., Xie, Y., chu, Y., Jia, J., Wang, F., & Ge, H. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00845-w
Evaluating Large Language Models for Automated Serverless Workflow Generation
Ristov, S., Gritsch, P., Unterhofer, F., & Breu, R. (2026) · ACM International Conference on Distributed and Event-Based Systems (DEBS) · doi:10.1145/3809481.3816480
Park, J., Kang, M., Jeon, S., & Hwang, S. O. (2026) · Sensors · doi:10.3390/s26134129
Wang, H., Yu, R., Pang, Y., Zhao, Y., Chen, J., Jiang, J., & Wang, Z. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3787792
TestForge: Benchmarking LLM-Based Test Case Generation
Vieira, M., Shah, B., & Shah, P. A. (2026) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner67736.2026.00014
Abductive Reasoning for Neurosymbolic Fault Localization
Le, M. T., Zheng, X., & Kang, H. J. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3786582.3786836
Semi, M. M. A., Rabbi, M. M. K., Ahmed, K. R., Islam, M. A., Rashid, M. M., Uddin, A., & Islam, R. (2026) · Multimedia · doi:10.3390/multimedia2030010
Adaptive Mixing of Embeddings from Multiple Code Language Models for Fault Localization
Yang, J., Park, E. Y., & Ko, I. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00044
Pinpointing Flakiness in Web Tests via DOM Event Analysis and LLM-based Reasoning
Pei, Y., Sohn, J., Habchi, S., & Papadakis, M. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00026
Chen, F., Yang, G., Lee, C., Jie, Z., & Chen, Y. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3831589
Test-Free Fault Localization Through Deep Source Code Analysis Using Encoder-Based Language Models
del-Hoyo-Gabaldon, J.-A., García‐Cabot, A., García‐López, E., & de-Fitero-Dominguez, D. (2026) · Applied Sciences · doi:10.3390/app16147170
Consistency-Based Software Diagnosis: Accuracy, Scalability, and Limitations
Sallinger, S., Graussam, L., Weißenbacher, G., Zuleger, F., & Ignatiev, A. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-32537-2_10
CoFiLoc: A Coarse-to-fine LLM-based framework for method-level bug localization
Cao, N., Nguyen, N. T., & Nguyen, T. B. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00649-2
Wu, L., Wei, J., Wu, Y., Du, B., Zhao, R., & Liu, Y. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113057
Coverage-Free LLM-Based Fault Localization via Semantic Reasoning Over Historical Bug Reports
Song, J., & Sohn, J. (2026) · IEEE Access · doi:10.1109/access.2026.3719447
LogHound: Coverage-Aware Path Reconstruction for Log-Based Fault Localization
Lin, Z., Feng, Y., Chen, J., Zhu, T., Chen, G., & Liu, J. (2026) · IEEE International Conference on Joint Cloud Computing (JCC) · doi:10.1109/jcc72984.2026.00026
An, D., Wang, S., Liu, B., & Zhu, L. (2026) · Mathematics · doi:10.3390/math14183297
Bug Whisperer: Unleashing LLMs to Craft Context-Aware Tests Guided by Path Constraints and Bug Clues
Xia, X., Zhang, J., Hu, X., & Li, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3846194
SPDZCoder: Combining Expert Knowledge with LLMs for Generating Privacy-Computing Code
Dong, X., Xin, P., Huang, L., Wang, Y., Li, J., & Xu, W. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3845985 · Cited by 1
From Greedy Steps to Global Optimization: Learning Sequential Test Suite Generation
Wang, G., Yang, C., Zhou, X., Sun, Z., Wang, B., Lo, D., & Hao, D. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832162 · Cited by 4
Li, C., Wang, F., Pan, M., Li, Z., Zeng, M., Zhang, B., Yu, X., Wang, B., Hua, K., & Li, X. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832097
Beyond Similarity Scores: Evidence-Based Third-Party Library Detection for C/C++ Binaries
Liu, C., Xu, Z., Zhang, L., Wu, J., Li, K., & Liu, Y. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832125
Yu, H., Wang, X., Zhang, J., Meng, X., Li, J., Liu, Y., & Hu, C. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832301
Zhu, X., Wu, Z., Wang, T., Yan, Y., Zhou, W., Wu, J., Jiang, Z., & Zheng, Z. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832292
Steering Tree-of-Thought Reasoning via Deductive Verification
Haoliang, C., Tang, E., Zhang, S., Mao, J., Fu, Y., Ma, J., Cui, K., Liu, Y., Tian, Y., Gao, X., et al. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3832200
HyTri: Hybrid Triage of CI Failures via LLM Guided Semantic Reasoning and Change Attribution
Broide, L., Mordoch, A., & Stern, R. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3837467
An Empirical Analysis of LLM Usage in Software Engineering Course Projects
Jeong, H., & Kim, J. (2026) · ACM Transactions on Computing Education · doi:10.1145/3859283
Code Graph based Bug Localization in Software Projects
Ajan, S., Gopikrishnan, S., Kaimal, R. R., & Sunitha, E. V. (2026) · International Conference on Trends in Engineering Systems and Technologies (ICTEST) · doi:10.1109/ictest69239.2026.11722873
DSLA: End-to-End Feature Fusion for Superior Fault Localization of Software
Xiao, Q., Zhou, M., Peng, J., Yang, S., Song, Z., Shi, Z., Li, Z., & Sun, L. (2026) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn69854.2026.11728715
Metadata
- Authors
- Yang, A. Z. H., Le Goues, C., Martins, R., & Hellendoorn, V.
- Affiliations
- Carnegie Mellon University (US)
- Carnegie Mellon University, Pittsburgh, Pennsylvania, USA
- Venue
- Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2024-02-06
- Pages
- 1-12
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4391579639
- Cited by
- 144
- References
- 48
- Citations by year
- 2026: 53 · 2025: 64 · 2024: 27
- FWCI
- 39.31
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- NSF (National Science Foundation) (CCF-1750116)
- National Science Foundation (CCF-1762363)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- large language models, LLM fine-tuning, Defects4J, vulnerability detection, automated program repair
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
