Large Language Models for Test-Free Fault Localization
Open accessAbstract
Fault Localization (FL) aims to automatically localize buggy lines of code, a key first step in many manual and automatic debugging tasks. Previous FL techniques assume the provision of input tests, and often require extensive program analysis, program instrumentation, or data preprocessing. Prior work on deep learning for APR struggles to learn from small datasets and produces limited results on real-world programs. Inspired by the ability of large language models (LLMs) of code to adapt to new tasks based on very few examples, we investigate the applicability of LLMs to line level fault localization. Specifically, we propose to overcome the left-to-right nature of LLMs by fine-tuning a small set of bidirectional adapter layers on top of the representations learned by LLMs to produce LLMAO, the first language model based fault localization approach that locates buggy lines of code without any test coverage information. We fine-tune LLMs with 350 million, 6 billion, and 16 billion parameters on small, manually curated corpora of buggy programs such as the Defects4J corpus. We observe that our technique achieves substantially more confidence in fault localization when built on the larger models, with bug localization performance scaling consistently with the LLM size. Our empirical evaluation shows that LLMAO improves the Top-1 results over the state-of-the-art machine learning fault localization (MLFL) baselines by 2.3%--54.4%, and Top-5 results by 14.4%-35.6%. LLMAO is also the first FL technique trained using a language model architecture that can detect security vulnerabilities down to the code line level.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Most LLM-based fault localization techniques either cannot perform repository-level FL due to the limited context window of LLMs or rely on complicated/costly agentic design to navigate through the codebase.
“Meanwhile, most such LLM-based techniques either cannot perform repository-level FL due to the limited context window of LLMs [98], or rely on complicated/costly agentic design to navigate through the codebase [48, 79].”
From Agentless: Demystifying LLM-based Software Engineering Agents (2024) · cites this paper as [98]
Only the first clause is attributed to this paper by citation [98]. Section 4.1.2 describes LLMAO as scoring lines within a code file, and Section 3.1 states that its model has a 2,048-token input limit. The paper does not establish that most LLM-based techniques cannot perform repository-level fault localization because of that limit; it says LLMAO uses windows to handle longer inputs. (LLM, conf=0.78)
Experimental — accuracy not yet measuredSWE-bench unites many separate software engineering tasks under a single task formulation that faithfully mirrors practical software engineering.
“We use SWE-bench because it unites many separate SE tasks, such as automated program repair [10, 40, 55], bug localization [4, 58], and testing [22, 46, 56] under a single task formulation that faithfully mirrors practical SE.”
From SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (2024) · cites this paper as [58]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
ImageNet classification with deep convolutional neural networks
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017) · Communications of the ACM · doi:10.1145/3065386 · Cited by 112,155 · see citing works
The graph neural network model
Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G. (2009) · IEEE Transactions on Neural Networks · doi:10.1109/tnn.2008.2005605 · Cited by 9,582 · see citing works
Learning long-term dependencies with gradient descent is difficult
Bengio, Y., Simard, P., & Frasconi, P. (1994) · IEEE Transactions on Neural Networks · doi:10.1109/72.279181 · Cited by 8,542 · see citing works
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,451 · see citing works
Scaling Laws for Neural Language Models
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020) · arXiv · Cited by 1,749 · cited 1× · see citing works
“A key property of LLMs is that their performance improves consistently with the scale of their computational budget [16], which is itself a function of the model and training data size.”
BackgroundSparks of Artificial General Intelligence: Early experiments with GPT-4
Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.12712 · Cited by 1,526 · see citing works
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447 · see citing works
Empirical evaluation of the tarantula automatic fault-localization technique
Jones, J. A., & Harrold, M. J. (2005) · International Conference on Automated Software Engineering · doi:10.1145/1101908.1101949 · Cited by 1,366 · see citing works
Training Compute-Optimal Large Language Models
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. D. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-2176 · Cited by 932 · see citing works
Woolson, R. F. (2008) · Wiley Encyclopedia of Clinical Trials · doi:10.1002/9780471462422.eoct979 · Cited by 925 · see citing works
On the Accuracy of Spectrum-based Fault Localization
Abreu, R., Zoeteweij, P., & Gemund, A. V. (2007) · Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007) · doi:10.1109/taic.part.2007.13 · Cited by 925 · see citing works
Deep Learning Based Vulnerability Detection: Are We There Yet?
Chakraborty, S., Krishna, R., Ding, Y., & Ray, B. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3087402 · Cited by 846 · see citing works
Evaluating large language models trained on code
Chen, M. (2024) · TIB Data Manager · doi:10.57702/wbv3e61b · Cited by 737 · see citing works
Engler, D., & Ashcraft, K. (2003) · ACM SIGOPS Operating Systems Review · doi:10.1145/1165389.945468 · Cited by 663 · see citing works
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660 · see citing works
Zhou, Y., Liu, S., Siow, J. K., Du, X., & Liu, Y. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.03496 · Cited by 582 · see citing works
Are automated debugging techniques actually helping programmers?
Parnin, C., & Orso, A. (2011) · International Symposium on Software Testing and Analysis · doi:10.1145/2001420.2001445 · Cited by 580 · see citing works
An Evaluation of Similarity Coefficients for Software Fault Localization
Abreu, R., Zoeteweij, P., & van Gemund, A. J. C. (2006) · Pacific Rim International Symposium on Dependable Computing · doi:10.1109/prdc.2006.18 · Cited by 569 · see citing works
Code completion with statistical language models
Raychev, V., Vechev, M., & Yahav, E. (2014) · Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation · doi:10.1145/2594291.2594321 · Cited by 555 · see citing works
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540 · see citing works
Self-citationMetallaxis‐FL: mutation‐based fault localization
Papadakis, M., & Traon, Y. L. (2015) · Software testing, verification & reliability · doi:10.1002/stvr.1509 · Cited by 384 · see citing works
DeepFL: integrating multiple fault diagnosis dimensions for deep fault localization
Li, X., Li, W., Zhang, Y., & Zhang, L. (2019) · Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis · doi:10.1145/3293882.3330574 · Cited by 336 · see citing works
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330 · see citing works
Ask the Mutants: Mutating Faulty Programs for Fault Localization
Moon, S., Kim, Y., Kim, M., & Yoo, S. (2014) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst.2014.28 · Cited by 298 · see citing works
Effective Fault Localization using Code Coverage
Wong, W. E., Qi, Y., Zhao, L., & Cai, K. (2007) · IEEE Annual International Computer Software and Applications Conference (COMPSAC) · doi:10.1109/compsac.2007.109 · Cited by 210 · see citing works
InCoder: A Generative Model for Code Infilling and Synthesis
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-T., Zettlemoyer, L., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2204.05999 · Cited by 187 · cited 1× · see citing works
“Models such as Codex [14] and InCoder [48] can perform the opposite direction as well: generate natural language docstring from code snippets alone.”
MethodLi, Y., Wang, S., Nguyen, T. N., & Nguyen, S. (2019) · Proceedings of the ACM on Programming Languages · doi:10.1145/3360588 · Cited by 164 · see citing works
Boosting coverage-based fault localization via graph-based representation learning
Lou, Y., Zhu, Q., Dong, J., Li, X., Sun, Z., Hao, D., Zhang, L., & Zhang, L. (2021) · Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3468264.3468580 · Cited by 163 · see citing works
Fault Localization with Code Coverage Representation Learning
Li, Y., Wang, S., & Nguyen, T. D. (2021) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse43902.2021.00067 · Cited by 159 · see citing works
Widyasari, R., Sim, S. Q., Lok, C., Qi, H., Phan, J., Tay, Q., Tan, C., Wee, F., Tan, J. E., Yieh, Y., et al. (2020) · Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3368089.3417943 · Cited by 150 · see citing works
Durieux, T., Madeiral, F., Martínez, M., & Abreu, R. (2019) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3338906.3338911 · Cited by 130 · see citing works
Where is the bug and how is it fixed? an experiment with practitioners
Böhme, M., Soremekun, E., Chattopadhyay, S., Ugherughe, E., & Zeller, A. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106255 · Cited by 122 · see citing works
Li, Y., Wang, S., & Nguyen, T. N. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510177 · Cited by 121 · see citing works
Test generation via Dynamic Symbolic Execution for mutation testing
Zhang, L., Xie, T., Zhang, L., Tillmann, N., de Halleux, J., & Mei, H. (2010) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2010.5609672 · Cited by 117 · see citing works
Program synthesis using natural language
Desai, A., Gulwani, S., Hingorani, V., Jain, N., Karkare, A., Marron, M., Sailesh, R., & Roy, S. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884786 · Cited by 116 · see citing works
Localizing failure-inducing program edits based on spectrum information
Zhang, L., Kim, M., & Sarfraz Khurshid, D. (2011) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsm.2011.6080769 · Cited by 110 · see citing works
Using Machine Learning to Support Debugging with Tarantula
Briand, L., Labiche, Y., & Liu, X. (2007) · Proceedings/Proceedings - International Symposium on Software Reliability Engineering · doi:10.1109/issre.2007.31 · Cited by 99 · see citing works
Meng, X., Wang, X., Zhang, H., Sun, H., & Liu, X. (2022) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3510003.3510147 · Cited by 93 · see citing works
GPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.06745 · Cited by 67 · see citing works
Deep Learning-Based Fault Localization with Contextual Information
Zhang, Z., Lei, Y., Tan, Q., Mao, X., Ping, Z. E. N. G., & CHANG, X. (2017) · IEICE Transactions on Information and Systems · doi:10.1587/transinf.2017edl8143 · Cited by 51 · see citing works
Injecting mechanical faults to localize developer faults for evolving software
Zhang, L., Zhang, L., & Khurshid, S. (2013) · ACM SIGPLAN Notices · doi:10.1145/2544173.2509551 · Cited by 40 · see citing works
Fault localization to detect co-change fixing locations
Li, Y., Wang, S., & Nguyen, T. N. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549137 · Cited by 25 · see citing works
A large-scale study of call graph-based impact prediction using mutation testing
Musco, V., Monperrus, M., & Preux, P. (2016) · Software Quality Journal · doi:10.1007/s11219-016-9332-8 · Cited by 24 · see citing works
Assessing and Comparing Mutation-based Fault Localization TechniquesPreprint
Chekam, T. T., Papadakis, M., & Traon, Y. L. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1607.05512 · Cited by 15 · see citing works
Dodson, B. L., Hammett, P. C., & Klerx, R. (2014) · doi:10.1002/9781118796245.ch12 · Cited by 6 · see citing works
Flagged first, then by citations
2 of the 48 references couldn’t be matched to a known paper.
Research works printed but not linked: 7
- Y. Li, S. Wang, and T. N. Nguyen, "Dear: A novel deep learning-based approach for automated program repair, " in Proceedings of the 44th International Conference on Software Engineering, pp. 511-523, 2022.
- L. Tunstall, L. Von Werra, and T. Wolf, Natural language processing with trans- formers. O'Reilly Media, Inc., 2022.
- B. Ray, V. Hellendoorn, S. Godhane, Z. Tu, A. Bacchelli, and P. Devanbu, "On the" naturalness" of buggy code," in Proceedings of the 38th International Conference on Software Engineering, pp. 428-439, 2016.
- D. Engler and K. Ashcraft, "Racerx: Effective, static detection of race conditions and deadlocks, " ACM SIGOPS operating systems review, vol. 37, no. 5, pp. 237-252, 2003.
- L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al., "The pile: An 800gb dataset of diverse text for language modeling, " arXiv preprint arXiv:2101.00027, 2020. · arXiv
- P. S. Kochhar, X. Xia, D. Lo, and S. Li, "Practitioners' expectations on automated fault localization, " in Proceedings of the 25th International Symposium on Software Testing and Analysis, pp. 165-176, 2016.
- H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, "An empirical cyber- security evaluation of github copilot's code contributions, " ArXiv abs/2108.09293, 2021. · arXiv
Cited by
Sorted by citations
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3695988 · Cited by 1,029 · see citing works
Software Testing With Large Language Models: Survey, Landscape, and Vision
Wang, J., Huang, Y., Chen, C., Liu, Z., Wang, S., & Wang, Q. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3368208 · Cited by 573 · see citing works
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · Advances in Neural Information Processing Systems 37 · doi:10.52202/079017-1601 · Cited by 264 · see citing works
When LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 220 · see citing works
“Thereby, the model can understand and perform cybersecurity-related tasks, such as automated threat detection, vulnerability identification, and security policy recommendations.”
A Review on Edge Large Language Models: Design, Execution, and Applications
Zheng, Y., Chen, Y., Qian, B., Shi, X., Shu, Y., & Chen, J. (2025) · ACM Computing Surveys · doi:10.1145/3719664 · Cited by 169 · see citing works
Towards an understanding of large language models in software engineering tasks
Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10602-0 · Cited by 149 · see citing works
Demystifying LLM-Based Software Engineering Agents
Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3715754 · Cited by 87 · see citing works
“Recently, researchers have proposed LLM-based FL [Kang et al. 2024; Qin et al. 2024; Wu et al. 2023; Yang et al. 2024b], which leverages the powerful code and natural language understanding of modern LLMs to directly localize bugs.”
Generative AI for self-adaptive systems: State of the art and research roadmap
Li, J., Zhang, M., Li, N., Weyns, D., Jin, Z., & Tei, K. (2024) · ACM Transactions on Autonomous and Adaptive Systems · doi:10.1145/3686803 · Cited by 77 · see citing works
Multitask-Based Evaluation of Open-Source LLM on Software Vulnerability
Yin, X., Ni, C., & Wang, S. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3470333 · Cited by 84 · see citing works
Jaffal, N. O., Alkhanafseh, M., & Mohaisen, D. (2025) · AI · doi:10.3390/ai6090216 · Cited by 69 · see citing works
Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities
Khare, A., Dutta, S., Li, Z., Solko-Breslin, A., Rajeev Alur, A., & Naik, M. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10988968 · Cited by 64 · see citing works
How secure is AI-generated code: a large-scale comparison of large language models
Tihanyi, N., Bisztray, T., Ferrag, M. A., Jain, R., & Cordeiro, L. C. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10590-1 · Cited by 63 · see citing works
Exploring the Capabilities of LLMs for Code-Change-Related Tasks
Fan, L., Liu, J., Liu, Z., Lo, D., Xia, X., & Li, S. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3709358 · Cited by 41 · see citing works
SoapFL: A Standard Operating Procedure for LLM-Based Method-Level Fault Localization
Qin, Y., Wang, S., Lou, Y., Dong, J., Wang, K., Li, X., & Mao, X. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3543187 · Cited by 32 · see citing works
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
Yang, J., Jimenez, C. E., Wettig, A., Lieret, K., Yao, S., Narasimhan, K., & Press, O. (2024) · arXiv · doi:10.48550/arxiv.2405.15793 · Cited by 31 · see citing works
vul-rag: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG
Du, X.-Y., Zheng, G., Wang, K.-X., Zou, Y., Wang, Y.-J., Deng, W.-T., Feng, J.-Y., Liu, M.-W., Chen, B., Peng, X., et al. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3797277 · Cited by 188 · see citing works
Reasoning Runtime Behavior of a Program with LLM: How Far are We?
Chen, J., Pan, Z., Hu, X., Li, Z., Li, G., & Xia, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00012 · Cited by 40 · see citing works
Agentless: Demystifying LLM-based Software Engineering Agents
Xia, C. S., Deng, Y., Dunn, S., & Zhang, L. (2024) · arXiv · doi:10.48550/arxiv.2407.01489 · Cited by 24 · see citing works
“Recently, researchers have proposed LLM-based FL [98, 91, 79, 48], which leverages the powerful code and natural language understanding of modern LLMs to directly localize bugs.”
BackgroundEvaluating Agent-Based Program Repair at Google
Rondon, P., Wei, R., Cambronero, J., Cito, J., Sun, A., Sanyam, S., Tufano, M., & Chandra, S. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00038 · Cited by 19 · see citing works
LLM for Mobile: An Initial Roadmap
Chen, D., Liu, Y., Zhou, M., Zhao, Y., Wang, H., Wang, S., Chen, X., Bissyandé, T. F., Klein, J., & Li, L. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708528 · Cited by 21 · see citing works
ChatDBG: Augmenting Debugging with Large Language Models
Levin, K. H., van Kempen, N., Berger, E. D., & Freund, S. N. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729355 · Cited by 16 · see citing works
Are Large Language Models Memorizing Bug Benchmarks?
Ramos, D., Mamede, C., Jain, K., Canelas, P., Gamboa, C., & Goues, C. L. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00005 · Cited by 16 · see citing works
Chakraborty, P., Alfadel, M., & Nagappan, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3579574 · Cited by 16 · see citing works
What You See is What You Get: Attention-Based Self-Guided Automatic Unit Test Generation
Yin, X., Ni, C., Xu, X., & Yang, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00105 · Cited by 14 · see citing works
Wang, T., Wang, P., Yang, F., Wang, S., Fang, Q., & Chi, M. (2024) · Journal of Process Control · doi:10.1016/j.jprocont.2024.103342 · Cited by 12 · see citing works
Large language models in source code static analysis
Ignatyev, V. N., Shimchik, N. V., Panov, D. D., & Mitrofanov, A. A. (2024) · Ivannikov Memorial Workshop (IVMEM) · doi:10.1109/ivmem63006.2024.10659715 · Cited by 11 · see citing works
Context-based Transfer Learning for Structuring Fault Localization and Program Repair Automation
Zhang, L., Guo, S., Guo, Y., Li, H., Chai, Y., Chen, R., Li, X., & Jiang, H. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3705302 · Cited by 10 · see citing works
Wang, X., Tian, Y., Huang, K., & Liang, B. (2024) · Computers & Security · doi:10.1016/j.cose.2024.104113 · Cited by 10 · see citing works
A Systematic Exploration of Mutation‐Based Fault Localization Formulae
Wang, B., Wei, J., Chen, M., Chen, C., Lin, Y., & Zhang, J. M. (2024) · Software Testing Verification and Reliability · doi:10.1002/stvr.1905 · Cited by 9 · see citing works
Vulnerability Handling of AI-Generated Code - Existing Solutions and Open Challenges
Kaniewski, S., Holstein, D., Schmidt, F., & Heer, T. (2024) · Conference on AI, Science, Engineering, and Technology (AIxSET) · doi:10.1109/aixset62544.2024.00026 · Cited by 9 · see citing works
Evaluating the Effectiveness of LLMs in Fixing Maintainability Issues in Real-World Projects
Nunes, H., Figueiredo, E., Rocha, L., Nadi, S., Ferreira, F., & Esteves, G. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00069 · Cited by 9 · see citing works
SmartFL: Semantics Based Probabilistic Fault Localization
Wu, Y., Liu, Y., Yin, Y., Zeng, M., Ye, Z., Zhang, X., Xiong, Y., & Zhang, L. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3574487 · Cited by 8 · see citing works
Steenhoek, B. J., Sivaraman, K., Gonzalez, R. S., Mohylevskyy, Y., Moghaddam, R. Z., & Le, W. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00126 · Cited by 8 · see citing works
How Execution Features Relate to Failures: An Empirical Study and Diagnosis Approach
Smytzek, M., Eberlein, M., Grunske, L., & Zeller, A. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3783989 · Cited by 8 · see citing works
Wang, B., Deng, M., Chen, M., Lin, Y., Zhou, J., & Zhang, J. M. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00549-x · Cited by 8 · see citing works
Traceability and Accountability in Role-Specialized Multi-Agent LLM Pipelines
Barrak, A. (2025) · 2025 40th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW) · doi:10.1109/asew67777.2025.00064 · Cited by 15 · see citing works
RunBugRun: An executable dataset for automated program repair
Prenner, J. A., & Robbes, R. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10790-3 · Cited by 11 · see citing works
Advancing software security: DCodeBERT for automatic vulnerability detection and repair
Bensaoud, A., & Kalita, J. K. (2025) · Journal of Industrial Information Integration · doi:10.1016/j.jii.2025.100834 · Cited by 6 · see citing works
Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industry
Du, X., Feng, J., Zou, Y., Xu, W., Ma, J., Zhang, W., Liu, S., Peng, X., & Lou, Y. (2026) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1145/3786583.3786910 · Cited by 6 · see citing works
FLAG: F inding L ine A nomalies (in RTL code) with G enerative AI
Ahmad, B., Ah-kiow, J., Tan, B., Karri, R., & Pearce, H. A. (2025) · ACM Transactions on Design Automation of Electronic Systems · doi:10.1145/3736411 · Cited by 6 · see citing works
Impact of Large Language Models of Code on Fault Localization
Ji, S., Lee, S.-H., Lee, C.-S., Han, Y.-S., & Im, H. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10989036 · Cited by 6 · see citing works
CodeDoctor: multi-category code review comment generation
Li, Y., Wu, Y., Wang, Z., Huang, L., Wang, J., Li, J. K., & Huang, M. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00491-y · Cited by 6 · see citing works
Semantic context based coincidental correct test cases detection for fault localization
Hu, J. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00466-5 · Cited by 5 · see citing works
LLM-BL: Large Language Models are Zero-Shot Rankers for Bug Localization
Li, Z., Jiang, Z., Huang, Q., & Gu, Q. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00064 · Cited by 4 · see citing works
Can test cases generated by large language models facilitate automated program repair?
Zhang, C., Wang, H., Xu, C., Liu, J., Liu, K., & Liu, Z. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10802-w · Cited by 4 · see citing works
Chang, J., Zhou, X., Wang, L.-L., Lo, D., & Li, B.-X. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3668601 · Cited by 17 · see citing works
Where Do AI Coding Agents Fail? An Empirical Study of Failed Agentic Pull Requests in GitHub
Ehsani, R., Pathak, S., Rawal, S., Mujahid, A. A., Imran, M. M., & Chatterjee, P. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793579 · Cited by 8 · see citing works
Malo in the Code Jungle: Explainable Fault Localization for Decentralized Applications
Zhang, H., Wu, J., Wu, Z., Chen, Z., Lin, D., Chen, J., Zhou, Y., & Zheng, Z. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3578816 · Cited by 4 · see citing works
Chang, B., Zhao, B., Zhang, Q., Liu, P., Tian, Y., Beyah, R., & Ji, S. (2025) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp61157.2025.00002 · Cited by 4 · see citing works
Zhang, Y., Zhao, Z., Zhang, D., & Dai, J. (2025) · International Conference on Power Electronics and Electric Drives (PEED) · doi:10.1109/peed63748.2025.00038 · Cited by 4 · see citing works
Enhanced Vulnerability Localization: Harmonizing Task-Specific Tuning and General LLM Prompting
Tian, W., Lin, Y., Gao, X., & Sun, H. (2025) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme64153.2025.00020 · Cited by 4 · see citing works
Automated program repair for variability bugs in software product line systems
Nguyen, T.-T., Zhang, X., Arcaini, P., Ishikawa, F., & Vo, H. D. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112152 · Cited by 4 · see citing works
Empirical Evaluation of Large Language Models for Novice Program Fault Localization
Liu, Y., Liu, H., Yang, Z., Li, Z., & Liu, Y. (2024) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs62785.2024.00027 · Cited by 4 · see citing works
Fang, C., Huang, C., Lee, S.-Y., Tseng, Y.-H., & Chu, C. W. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112388 · Cited by 4 · see citing works
Fault Localization via Fine-Tuning Large Language Models with Mutation Generated Stack Traces
Jambigi, N., Bogacz, B., Mueller, M., Bach, T., & Felderer, M. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10988982 · Cited by 4 · see citing works
SBEST: Spectrum-based fault localization without fault-triggering tests
Rafi, M. N., Pacheco, L. B. S., Chen, A. R., Yang, J., & Chen, T.-H. P. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10761-8 · Cited by 4 · see citing works
Systematic Analysis of Learning-Based Software Fault Localization
Zou, Y., Li, H., Li, D., Zhao, M., & Chen, Z. (2024) · International Symposium on System Security, Safety, and Reliability (ISSSR) · doi:10.1109/isssr61934.2024.00068 · Cited by 4 · see citing works
Calico: Automated Knowledge Calibration and Diagnosis for Elevating AI Mastery in Code Tasks
Qiu, Y., Hu, J., Zhang, Q., & Yin, H. (2024) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3650212.3680399 · Cited by 4 · see citing works
Evaluating Large Language Models for Line-Level Vulnerability Localization
Zhang, J., Wang, C., Li, A., Sun, W.-S., Zhang, C., Ma, W., & Liu, Y. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3649250 · Cited by 27 · see citing works
Combining Logic and Large Language Models for Assisted Debugging and Repair of ASP Programs
Brancas, R., Manquinho, V. M., & Martins, R. (2025) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst62969.2025.10988950 · Cited by 3 · see citing works
Self-citationDu, B., Kang, X., Xu, H., Wu, Y., & Liu, Y. (2025) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs65678.2025.00016 · Cited by 3 · see citing works
Improving Software Defect Detection With LSTM-Based Semantic Modeling and Class Imbalance Handling
de Andrade, H. G. I., Pombo, N., & Pais, S. (2025) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2025.3613134 · Cited by 3 · see citing works
Banitaan, S., Daoud, M., Alquran, H., & Akour, M. A. (2026) · Information · doi:10.3390/info17010073 · Cited by 5 · see citing works
Mahmud, A., Rawajfih, Y., & Arnold, R. (2025) · 2025 Annual Computer Security Applications Conference Workshops (ACSAC Workshops) · doi:10.1109/acsacw69556.2025.00074 · Cited by 3 · see citing works
Shahriar, A., Hisham, S. J., Rahman, K. M. A., Islam, R., Hossain, M. S., Hwang, R., & Lin, Y. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3586480 · Cited by 4 · see citing works
Two-Phase Software Fault Localization Based on Relational Graph Convolutional Neural Networks
Fan, X., Shu, J., Shen, Z., Ge, Y., & Fu, Z. (2024) · Computers, materials & continua/Computers, materials & continua (Print) · doi:10.32604/cmc.2024.057695 · Cited by 3 · see citing works
Uncertainty Estimation of Automatic Software Debugging in Open-Source Projects Hosting Platform
Liang, H., Guo, S., Li, H., & Li, C. H. (2025) · IEEE Transactions on Consumer Electronics · doi:10.1109/tce.2024.3524511 · Cited by 3 · see citing works
Revisiting Unnaturalness for Automated Program Repair in the Era of Large Language Models
Yang, A. Z. H., Kolak, S., Hellendoorn, V. J., Martins, R., & Goues, C. L. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00089 · Cited by 3 · see citing works
Self-citationA Study on C Code Defect Detection with Fine-Tuned Large Language Models
Wang, Y., Wang, X., Yu, H., Gao, F., Liu, X., & Wang, X. (2024) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec65559.2024.00055 · Cited by 2 · see citing works
So much more than test cases – An industrial study on testing of software units and components
Mårtensson, T. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112479 · Cited by 2 · see citing works
Graph neural network for fault localization in sequence-based models
Raza, M. A., & Wardat, M. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10666-6 · Cited by 2 · see citing works
Enhancing Fault Localization via Causal Measurement Features
Li, Y., Cai, B., Yang, H., Cai, C., Qiu, Y., Chen, T., & Zhang, C. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228088 · Cited by 2 · see citing works
Review of Large Language Model-Based Software Fault Localization Techniques
Tian, P., Liu, P., & Li, Y. (2025) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa66321.2025.00021 · Cited by 2 · see citing works
Large Language Models for Software Fault Localization: a Survey
Chang, X., Li, D., & Wong, W. E. (2025) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa66321.2025.00024 · Cited by 2 · see citing works
Liu, C., Lin, Y., Huang, Y., Chang, J., Qi, B., Jiang, B., Huang, Z., & Dong, J. S. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00117 · Cited by 2 · see citing works
Large Language Models for Fault Diagnosis
Qi, Z., Ren, J., Gan, W., & Yu, P. S. (2025) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata66926.2025.11402351 · Cited by 2 · see citing works
DCoL-A: Agentic dual chain of thinking helps LLMs pretend logic solvers
Chen, M., Wu, L.-I., Liu, R., Chang, X., Xue, J., & Li, G. (2026) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2026.103780 · Cited by 2 · see citing works
COSMosFL: Ensemble of Small Language Models for Fault Localisation
Cho, H., Kang, S., An, G., & Yoo, S. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00007 · Cited by 2 · see citing works
Analysis of the Effectiveness of Large Language Model Feature in Source Code Defect Detection
He, T., Yang, M., Hu, W. P., & Chen, Y. (2024) · International Conference on Artificial Intelligence and Computer Information Technology (AICIT) · doi:10.1109/aicit62434.2024.10730232 · Cited by 1 · see citing works
Can Language Models Pretend Solvers? Logic Code Simulation with LLMs
Chen, M., Li, G., Wu, L.-I., Liu, R., Su, Y., Chang, X., & Xue, J. (2024) · Lecture notes in computer science · doi:10.1007/978-981-96-0602-3_6 · Cited by 3 · see citing works
PAFL: Enhancing Fault Localizers by Leveraging Project-Specific Fault Patterns
Kim, D., Jeon, M., Hwang, D., & Oh, H. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3720526 · Cited by 1 · see citing works
SecureMind: A Framework for Benchmarking Large Language Models in Memory Bug Detection and Repair
Wang, H., Jacob, D., Kelly, D., Elkhatib, Y., Singer, J., & Wang, Z. (2025) · ACM SIGPLAN International Symposium on Memory Management (ISMM) · doi:10.1145/3735950.3735954 · Cited by 1 · see citing works
Simple Fault Localization Using Execution Traces
Prenner, J. A., & Robbes, R. (2025) · IEEE/ACM International Workshop on Automated Program Repair (APR) · doi:10.1109/apr66717.2025.00013 · Cited by 1 · see citing works
Artificial Intelligence Techniques-Driven Automated Localization of Software Faults
Pranav Raj, S., & Venkatraman, K. (2025) · International Conference on Intelligent Sustainable Systems (ICISS) · doi:10.1109/iciss63372.2025.11076249 · Cited by 1 · see citing works
Hou, Z., Duan, Z., Yang, M., Wu, M., & Yang, S. (2025) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs65678.2025.00015 · Cited by 1 · see citing works
Empirical Evaluation of LLMs for Automated Program Fault Localisation
Liu, Y., Wang, X., Liu, H., Huang, R., & Wu, Y. (2025) · IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) · doi:10.1109/qrs-c65679.2025.00063 · Cited by 1 · see citing works
Reinforcement Learning vs Supervised Learning: A tug of war to generate refactored code accurately
Palit, I., & Sharma, T. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3756681.3756971 · Cited by 3 · see citing works
Bug Localization Based on Context-Aware Code Translation and Feature Fusion
Cheng, W., Zhang, R., Wang, G., & Chen, Y. (2026) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194025501153 · Cited by 1 · see citing works
LOSVER: Line-Level Modifiability Signal-Guided Vulnerability Detection and Classification
Nam, D., & Baik, J. (2025) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase63991.2025.00092 · Cited by 1 · see citing works
Chowdhury, K., Banik, D., & Shamim, S. I. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3808241 · Cited by 1 · see citing works
Le, T.-H.-C., Hua, H.-Q.-B., & Le, X.-B. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834420 · Cited by 1 · see citing works
Towards Connecting Bugs and Architecture in Software Systems: A Perspective
Rao, A. E., & Chimalakonda, S. (2024) · IEEE International Conference on Software Architecture Companion (ICSA-C) · doi:10.1109/icsa-c63560.2024.00025 · Cited by 1 · see citing works
Improve Graph Learning Fault Localization Via Multimodal Code RepresentationPreprint
Yu, B., & Guo, J. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4840043 · Cited by 1 · see citing works
Rafi, M. N., Kim, D. J., Chen, T.-H., & Wang, S. (2024) · arXiv · doi:10.48550/arxiv.2412.18750
“Recent research has explored the use of LLMs in various software engineering tasks, including Fault Localization (FL) [14, 41] and Automatic Program Repair (APR) [39, 45], which show great potential for automatically resolving real-world issues in large codebases.”
It’s Organic: Software Testing of Emerging Domains (Keynote)
Cohen, M. B. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3674720
SpecNLP: A Pre-trained Model Enhanced with Spectrum Profile for Bug Localization
Farzandway, M., & Ghassemi, F. (2024) · IEEE International Conference on Artificial Intelligence Testing (AITest) · doi:10.1109/aitest62860.2024.00018
Automatically Learning a Precise Measurement for Fault Diagnosis Capability of Test Cases
Zhao, Y., Sun, Z., Wang, G., Liang, Q., Zhang, Y., Lou, Y., Hao, D., & Zhang, L. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712189
Statistics in Medicine: Tools for Health Research
Salem, S. M. A., Thajil, A. S. Q., lafta, N. S. T., Sahin, A. K. A., & alkarim, M. S. A. A. (2025) · Bright Sky Publications eBooks · doi:10.62906/bs.book.351
Fine-grained approach to metamorphic slicing: Enhancing fault localization in the absence of oracle
Ren, D., Zheng, Z., Liu, H., & Chen, T. Y. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108297
Improving fault localization with semantic enhancement and decision fusion
Liu, B., & He, P. (2025) · International Conference on Computing and Artificial Intelligence (ICCAI) · doi:10.1109/iccai66501.2025.00113
- Authors
- Yang, A. Z. H., Le Goues, C., Martins, R., & Hellendoorn, V.
- Affiliations
- Carnegie Mellon University (US)
- Carnegie Mellon University, Pittsburgh, Pennsylvania, USA
- Venue
- Proceedings of the IEEE/ACM 46th International Conference on Software Engineering
- Publisher
- ACM
- Published
- 2024-02-06
- Pages
- 1-12
- ISSN
- 0270-5257 (ISSN-L)
- OpenAlex
- W4391579639
- Cited by
- 144
- References
- 48
- Citations by year
- 2026: 53 · 2025: 64 · 2024: 27
- FWCI
- 39.31
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- NSF (National Science Foundation) (CCF-1750116)
- National Science Foundation (CCF-1762363)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- large language models, LLM fine-tuning, Defects4J, vulnerability detection, automated program repair
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
