The DOI matches its Crossref record; 46 of the 48 references it cites were found in a registry or an index.
Large Language Models for Test-Free Fault Localization
Open accessAbstract
Fault Localization (FL) aims to automatically localize buggy lines of code, a key first step in many manual and automatic debugging tasks. Previous FL techniques assume the provision of input tests, and often require extensive program analysis, program instrumentation, or data preprocessing. Prior work on deep learning for APR struggles to learn from small datasets and produces limited results on real-world programs. Inspired by the ability of large language models (LLMs) of code to adapt to new tasks based on very few examples, we investigate the applicability of LLMs to line level fault localization. Specifically, we propose to overcome the left-to-right nature of LLMs by fine-tuning a small set of bidirectional adapter layers on top of the representations learned by LLMs to produce LLMAO, the first language model based fault localization approach that locates buggy lines of code without any test coverage information. We fine-tune LLMs with 350 million, 6 billion, and 16 billion parameters on small, manually curated corpora of buggy programs such as the Defects4J corpus. We observe that our technique achieves substantially more confidence in fault localization when built on the larger models, with bug localization performance scaling consistently with the LLM size. Our empirical evaluation shows that LLMAO improves the Top-1 results over the state-of-the-art machine learning fault localization (MLFL) baselines by 2.3%--54.4%, and Top-5 results by 14.4%-35.6%. LLMAO is also the first FL technique trained using a language model architecture that can detect security vulnerabilities down to the code line level.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Meanwhile, most such LLM-based techniques either cannot perform repository-level FL due to the limited context window of LLMs [98], or rely on complicated/costly agentic design to navigate through the codebase [48, 79].”
From Agentless: Demystifying LLM-based Software Engineering Agents (2024) · cites this paper as [98]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“We use SWE-bench because it unites many separate SE tasks, such as automated program repair [10, 40, 55], bug localization [4, 58], and testing [22, 46, 56] under a single task formulation that faithfully mirrors practical SE.”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [58]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155
The graph neural network model
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G. (2009) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2005605 · Cited by 9,582
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · doi:10.48550/arxiv.2001.08361 · Cited by 9,550 · cited 1×
“A key property of LLMs is that their performance improves consistently with the scale of their computational budget [16], which is itself a function of the model and training data size.”
BackgroundLearning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,451
Sparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
Empirical evaluation of the tarantula automatic fault-localization technique
Jones, J. A., & Harrold, M. J. (2005) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/1101908.1101949 · Cited by 1,274
Deep Learning Based Vulnerability Detection: Are We There Yet?
Chakraborty, S., Krishna, R., Ding, Y., & Ray, B. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3087402 · Cited by 970
On the Accuracy of Spectrum-based Fault Localization
Abreu, R., Zoeteweij, P., & van Gemund, A. V. (2007) · Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007) · doi:10.1109/taic.part.2007.13 · Cited by 939
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932
Woolson, R. F. (2008) · Wiley Encyclopedia of Clinical Trials · doi:10.1002/9780471462422.eoct979 · Cited by 925
Evaluating large language models trained on code
Chen, M. (2024) · TIB Data Manager · doi:10.57702/wbv3e61b · Cited by 737
Engler, D., & Ashcraft, K. (2003) · ACM SIGOPS Operating Systems Review · doi:10.1145/1165389.945468 · Cited by 678
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
An Evaluation of Similarity Coefficients for Software Fault Localization
Abreu, R., Zoeteweij, P., & van Gemund, A. J. C. (2006) · Pacific Rim International Symposium on Dependable Computing · doi:10.1109/prdc.2006.18 · Cited by 604
Are automated debugging techniques actually helping programmers?
Parnin, C., & Orso, A. (2011) · International Symposium on Software Testing and Analysis · doi:10.1145/2001420.2001445 · Cited by 593
Zhou, Y., Liu, S., Siow, J. K., Du, X., & Liu, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.03496 · Cited by 582
Code completion with statistical language models
Raychev, V., Vechev, M., & Yahav, E. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594321 · Cited by 555
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540
Self-citationMetallaxis‐FL: mutation‐based fault localization
Papadakis, M., & Traon, Y. L. (2015) · Software testing, verification & reliability · doi:10.1002/stvr.1509 · Cited by 384
DeepFL: integrating multiple fault diagnosis dimensions for deep fault localization
Li, X., Li, W., Zhang, Y., & Zhang, L. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330574 · Cited by 336
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330
Ask the Mutants: Mutating Faulty Programs for Fault Localization
Moon, S., Kim, Y., Kim, M., & Yoo, S. (2014) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2014.28 · Cited by 315
Effective Fault Localization using Code Coverage
Wong, W. E., Qi, Y., Zhao, L., & Cai, K. (2007) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2007.109 · Cited by 210
InCoder: A Generative Model for Code Infilling and Synthesis
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-T., Zettlemoyer, L., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2204.05999 · Cited by 187 · cited 1×
“Models such as Codex [14] and InCoder [48] can perform the opposite direction as well: generate natural language docstring from code snippets alone.”
MethodFault Localization with Code Coverage Representation Learning
Li, Y., Wang, S., & Nguyen, T. D. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00067 · Cited by 176
Li, Y., Wang, S., Nguyen, T. N., & Nguyen, S. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3360588 · Cited by 164
Boosting coverage-based fault localization via graph-based representation learning
Lou, Y., Zhu, Q., Dong, J., Li, X., Sun, Z., Hao, D., Zhang, L., & Zhang, L. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468580 · Cited by 163
Widyasari, R., Sim, S. Q., Lok, C., Qi, H., Phan, J., Tay, Q., Tan, C., Wee, F., Tan, J. E., Yieh, Y., et al. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417943 · Cited by 150
Where is the bug and how is it fixed? an experiment with practitioners
Böhme, M., Soremekun, E., Chattopadhyay, S., Ugherughe, E., & Zeller, A. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106255 · Cited by 136
Durieux, T., Madeiral, F., Martínez, M., & Abreu, R. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338911 · Cited by 130
Li, Y., Wang, S., & Nguyen, T. N. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510177 · Cited by 121
Test generation via Dynamic Symbolic Execution for mutation testing
Zhang, L., Xie, T., Zhang, L., Tillmann, N., de Halleux, J., & Mei, H. (2010) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2010.5609672 · Cited by 117
Program synthesis using natural language
Desai, A., Gulwani, S., Hingorani, V., Jain, N., Karkare, A., Marron, M., Sailesh, R., & Roy, S. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884786 · Cited by 116
Localizing failure-inducing program edits based on spectrum information
Zhang, L., Kim, M., & Sarfraz Khurshid, D. (2011) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2011.6080769 · Cited by 110
Using Machine Learning to Support Debugging with Tarantula
Briand, L., Labiche, Y., & Liu, X. (2007) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2007.31 · Cited by 99
Meng, X., Wang, X., Zhang, H., Sun, H., & Liu, X. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510147 · Cited by 93
GPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.06745 · Cited by 67
Deep Learning-Based Fault Localization with Contextual Information
Zhang, Z., Lei, Y., Tan, Q., Mao, X., Ping, Z. E. N. G., & CHANG, X. (2017) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2017edl8143 · Cited by 53
Injecting mechanical faults to localize developer faults for evolving software
Zhang, L., Zhang, L., & Khurshid, S. (2013) · ACM SIGPLAN Notices · doi:10.1145/2544173.2509551 · Cited by 40
Fault localization to detect co-change fixing locations
Li, Y., Wang, S., & Nguyen, T. N. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549137 · Cited by 25
A large-scale study of call graph-based impact prediction using mutation testing
Musco, V., Monperrus, M., & Preux, P. (2016) · Software Quality Journal · doi:10.1007/s11219-016-9332-8 · Cited by 24
Assessing and Comparing Mutation-based Fault Localization TechniquesPreprint
Chekam, T. T., Papadakis, M., & Traon, Y. L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.05512 · Cited by 15
Dodson, B. L., Hammett, P. C., & Klerx, R. (2014) · doi:10.1002/9781118796245.ch12 · Cited by 6
Research works printed but not linked: 5
- L. Tunstall, L. Von Werra, and T. Wolf, Natural language processing with transformers. O'Reilly Media, Inc., 2022.
- B. Ray, V. Hellendoorn, S. Godhane, Z. Tu, A. Bacchelli, and P. Devanbu, "On the" naturalness" of buggy code," in Proceedings of the 38th International Conference on Software Engineering, pp. 428--439, 2016.
- L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al., "The pile: An 800gb dataset of diverse text for language modeling," arXiv preprint arXiv:2101.00027, 2020. · arXiv
- P. S. Kochhar, X. Xia, D. Lo, and S. Li, "Practitioners' expectations on automated fault localization," in Proceedings of the 25th International Symposium on Software Testing and Analysis, pp. 165--176, 2016.
- H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, "An empirical cyber-security evaluation of github copilot's code contributions," ArXiv abs/2108.09293, 2021. · arXiv
Cited by
Sorted by citations
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1601 · Cited by 264
When LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 383
“Thereby, the model can understand and perform cybersecurity-related tasks, such as automated threat detection, vulnerability identification, and security policy recommendations.”
A Review on Edge Large Language Models: Design, Execution, and Applications
Zheng, Y., Chen, Y., Qian, B., Shi, X., Shu, Y., & Chen, J. (2025) · ACM Computing Surveys · doi:10.1145/3719664 · Cited by 169
Metadata
- Authors
- Yang, A. Z. H., Le Goues, C., Martins, R., & Hellendoorn, V.
- Affiliations
- Carnegie Mellon University (US)
- Carnegie Mellon University, Pittsburgh, Pennsylvania, USA
- Venue
- Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2024-02-06
- Pages
- 1-12
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4391579639
- Cited by
- 144
- References
- 48
- Citations by year
- 2026: 53 · 2025: 64 · 2024: 27
- FWCI
- 39.31
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- NSF (National Science Foundation) (CCF-1750116)
- National Science Foundation (CCF-1762363)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- large language models, LLM fine-tuning, Defects4J, vulnerability detection, automated program repair
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
