CREF: An LLM-Based Conversational Software Repair Framework for Programming Tutors
Open accessAbstract
With the proven effectiveness of Large Language Models (LLMs) in code-related tasks, researchers have explored their potential for program repair. However, existing repair benchmarks might have influenced LLM training data, potentially causing data leakage. To evaluate LLMs’ realistic repair capabilities, (i) we introduce an extensive, non-crawled benchmark TutorCode, comprising 1,239 C++ defect codes and associated information such as tutor guidance, solution description, failing test cases, and the corrected code. Our work assesses LLM’s repair performance on TutorCode, measuring repair correctness (TOP-5 and AVG-5) and patch precision (RPSR). (ii) We then provide a comprehensive investigation into which types of extra information can help LLMs improve their repair performance. Among these types, tutor guidance was the most effective information. To fully harness LLMs’ conversational capabilities and the benefits of augmented information, (iii) we introduce a novel conversational semi-automatic repair framework CREF assisting human programming tutors. It demonstrates a remarkable AVG-5 improvement of 17.2%-24.6% compared to the baseline, achieving an impressive AVG-5 of 76.6% when utilizing GPT-4. These results highlight the potential for enhancing LLMs’ repair capabilities through tutor interactions and historical conversations. The successful application of CREF in a real-world educational setting demonstrates its effectiveness in reducing tutors’ workload and improving students’ learning experience, showing promise for code review and other software engineering tasks.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The successful use of zero-shot and one-shot learning to summarize the code and of conversational prompts for automatic program repair leads to new efficient prompting strategies in software engineering.
“The successful use of zero-shot and one-shot learning to summarize the code [229, 235] and of conversational prompts for automatic program repair [222-224, 230] leads to new efficient prompting strategies in software engineering.”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [230]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other
Mann, H. B., & Whitney, D. R. (1947) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177730491 · Cited by 14,320 · see citing works
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., & Zettlemoyer, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.703 · Cited by 8,821 · see citing works
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602 · see citing works
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208 · see citing works
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.05685 · Cited by 1,888 · see citing works
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495 · see citing works
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495 · see citing works
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778 · see citing works
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531 · see citing works
CodeT5+: Open Code Large Language Models for Code Understanding and Generation
Wang, Y., Le, H., Gotmare, A. D., Bui, N. D. Q., Li, J., & Hoi, S. C. H. (2023) · arXiv · doi:10.48550/arxiv.2305.07922 · Cited by 459
Automated Program Repair in the Era of Large Pre-trained Language Models
Xia, C. S., Wei, Y., & Zhang, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00129 · Cited by 433 · see citing works
Le Goues, C., Pradel, M., & Roychoudhury, A. (2019) · Communications of the ACM · doi:10.1145/3318162 · Cited by 410 · see citing works
Liu, J., Xia, C. S., Wang, Y., & Zhang, L. (2023) · arXiv · doi:10.48550/arxiv.2305.01210 · Cited by 390 · see citing works
Monperrus, M. (2018) · ACM Computing Surveys · doi:10.1145/3105906 · Cited by 386 · see citing works
Practitioners' expectations on automated fault localization
Kochhar, P. S., Xia, X., Lo, D., & Li, S. (2016) · International Symposium on Software Testing and Analysis · doi:10.1145/2931037.2931051 · Cited by 358 · see citing works
Le, X.-B. D., Lo, D., & Goues, C. L. (2016) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner.2016.76 · Cited by 341 · see citing works
An Analysis of the Automatic Bug Fixing Performance of ChatGPT
Sobania, D., Briesch, M., Hanna, C., & Petke, J. (2023) · 2023 IEEE/ACM International Workshop on Automated Program Repair (APR) · doi:10.1109/apr59189.2023.00012 · Cited by 338 · see citing works
Proceedings of the 44th International Conference on Software Engineering
(2022) · doi:10.1145/3510003 · Cited by 327 · see citing works
The ManyBugs and IntroClass benchmarks for automated repair of C programs
Le Goues, C., Holtschulte, N., Smith, E. K., Brun, Y., Devanbu, P., Forrest, S., & Weimer, W. (2015) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2015.2454513 · Cited by 326 · see citing works
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., & Kaplan, J. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2212.08073 · Cited by 326 · see citing works
StarCoder: may the source be with you!
Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.06161 · Cited by 261 · see citing works
Automated Repair of Programs from Large Language Models
Fan, Z., Gao, X., Mirchev, M., Roychoudhury, A., & Tan, S. H. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00128 · Cited by 256 · see citing works
VulRepair: a T5-based automated software vulnerability repair
Fu, M. C., Tantithamthavorn, C. K., Le, T., Nguyen, V., & Phung, D. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549098 · Cited by 224 · see citing works
ChatGPT is a Remarkable Tool—For Experts
Azaria, A., Azoulay, R., & Reches, S. (2023) · Data Intelligence · doi:10.1162/dint_a_00235 · Cited by 157 · see citing works
WizardCoder: Empowering Code Large Language Models with Evol-Instruct
Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q., & Jiang, D. (2023) · arXiv · doi:10.48550/arxiv.2306.08568 · Cited by 120 · see citing works
Is ChatGPT the Ultimate Programming Assistant -- How far is it?
Tian, H., Lu, W., Li, T. O., Tang, X., Cheung, S.-C., Klein, J., & Bissyandé, T. F. (2023) · arXiv · doi:10.48550/arxiv.2304.11938 · Cited by 114 · see citing works
Self-citationA feasibility study of using automated program repair for introductory programming assignments
Yi, J., Ahmed, U. Z., Karkare, A., Tan, S. H., & Roychoudhury, A. (2017) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3106237.3106262 · Cited by 113 · see citing works
Conversational Automated Program RepairPreprint
Xia, C., & Zhang, L.-M. (2023) · arXiv.org · doi:10.48550/arxiv.2301.13246 · Cited by 103 · see citing works
Tian, H., Liu, K., Kaboré, A. K., Koyuncu, A., Li, L., Klein, J., & Bissyandé, T. F. (2020) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3324884.3416532 · Cited by 101 · see citing works
Self-citationCan OpenAI's codex fix bugs?: an evaluation on QuixBugs
Prenner, J. A., Babii, H., & Robbes, R. (2022) · Proceedings of the Third International Workshop on Automated Program Repair · doi:10.1145/3524459.3527351 · Cited by 99 · see citing works
Re-Factoring Based Program Repair Applied to Programming Assignments
Yang, H., Ahmed, U. Z., Mechtaev, S., Leong, B., & Roychoudhury, A. (2019) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2019.00044 · Cited by 92 · see citing works
Ni, C., Wang, W., Yang, K., Xia, X., Liu, K., & Lo, D. (2022) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3540250.3549165 · Cited by 79 · see citing works
Can we trust the evaluation on ChatGPT?
Aiyappa, R., An, J., Kwak, H., & Ahn, Y.-Y. (2023) · Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023) · doi:10.18653/v1/2023.trustnlp-1.5 · Cited by 72 · see citing works
Repairing Bugs in Python Assignments Using Large Language ModelsPreprint
Zhang, J.-L., Cambronero, J., Gulwani, S., Le, V., Piskac, R., Soares, G., & Verbruggen, G. (2022) · arXiv.org · doi:10.48550/arxiv.2209.14876 · Cited by 68 · see citing works
Context-Aware Code Change Embedding for Better Patch Correctness Assessment
Lin, B., Wang, S., Wen, M., & Mao, X. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3505247 · Cited by 63 · see citing works
StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of CodePreprint
Babe, H. M., Nguyen, S., Zi, Y., et al. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2306.04556 · Cited by 56 · see citing works
Automated clustering and program repair for introductory programming assignments
Gulwani, S., Radiček, I., & Zuleger, F. (2018) · ACM SIGPLAN Notices · doi:10.1145/3296979.3192387 · Cited by 55 · see citing works
Goues, C. L., Pradel, M., Roychoudhury, A., & Chandra, S. (2021) · IEEE Software · doi:10.1109/ms.2021.3072577 · Cited by 40 · see citing works
Verifix: Verified Repair of Programming Assignments
Ahmed, U. Z., Fan, Z., Yi, J., Al-Bataineh, O. I., & Roychoudhury, A. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3510418 · Cited by 33 · see citing works
A study on Prompt Design, Advantages and Limitations of ChatGPT for Deep Learning Program RepairPreprint
Cao, J., Li, M., Wen, M., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.08191 · Cited by 33 · see citing works
Let’s Talk With Developers, Not About Developers: A Review of Automatic Program Repair Research
Winter, E., Nowack, V., Bowes, D., Counsell, S., Hall, T., Haraldsson, S. O., & Woodward, J. R. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3152089 · Cited by 27 · see citing works
Automated Feedback Generation for Competition-Level Code
Zhang, J., Li, D., Kolesar, J. C., Shi, H., & Piskač, R. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3560425 · Cited by 22 · see citing works
Lee, C., Myung, J., Han, J., Jin, J., & Oh, A. H. (2023) · arXiv.org · doi:10.48550/arxiv.2309.10419 · Cited by 18 · see citing works
Deep Just-In-Time Defect Localization
Qiu, F., Gao, Z., Xia, X., Lo, D., Grundy, J., & Wang, X. (2021) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2021.3135875 · Cited by 17 · see citing works
Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models
Phung, T., Cambronero, J., Gulwani, S., Kohn, T., Majumdar, R., Singla, A., & Soares, G. (2023) · arXiv · doi:10.48550/arxiv.2302.04662 · Cited by 16 · see citing works
Is this Change the Answer to that Problem?
Tian, H., Tang, X., Habib, A., Wang, S., Liu, K., Xia, X., Klein, J., & Bissyandé, T. F. (2022) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3551349.3556914 · Cited by 15 · see citing works
Self-citationChatGPT is a Remarkable Tool -- For ExpertsPreprint
Azaria, A., Azoulay, R., & Reches, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.03102 · Cited by 11 · see citing works
Interactive Patch Generation and Suggestion
Gao, X., & Roychoudhury, A. (2020) · doi:10.1145/3387940.3392179 · Cited by 7 · see citing works
Automated Program Repair from Fuzzing Perspective
Kim, Y., Han, S., Khamit, A. Y., & Yi, J. (2023) · ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) · doi:10.1145/3597926.3598101 · Cited by 7 · see citing works
Exploring the Effectiveness of LLMs in Automated Logging Generation: An Empirical StudyPreprint
Li, Y., Huo, Y., Jiang, Z., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.05950 · Cited by 7 · see citing works
Priority Measurement of Patches for Program Repair Based on Semantic Distance
Dong, Y., Wu, M., Zhang, L., Yin, W., Wu, M., & Li, H. (2020) · Symmetry · doi:10.3390/sym12122102 · Cited by 4 · see citing works
Generation of C++ Code from Isabelle/HOL Specification
Jiang, D., & Xu, B. (2022) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194022500401 · Cited by 4 · see citing works
STEAM: Simulating the InTeractive BEhavior of ProgrAMmers for Automatic Bug FixingPreprint
Zhang, Y., Jin, Z. G., Xing, Y., & Li, G. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.14460 · Cited by 3 · see citing works
Program RepairPreprint
Gao, X. Y., Noller, Y., & Roychoudhury, A. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2211.12787 · Cited by 3 · see citing works
KNOD: Domain Knowledge Distilled Tree Decoder for Automated Program RepairPreprint
Jiang, N., Lutellier, T., Lou, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.01857 · Cited by 2 · see citing works
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.08774
Flagged first, then by citations
Research works printed but not linked: 15
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877-1901.
- Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2022. InCoder: A Generative Model for Code Infilling and Synthesis. In The Eleventh International Conference on Learning Representations.
- Li Ge, Peng Xin, Wang Qianxiang, Xie Tao, Jin Zhi, Wang Ji, Ma Xiaoxing, and Li Xuandong. 2023. Challenges from LLMs as a Natural Language Based Human- machine Collaborative Tool for Software Development and Evolution. In Journal of Software, 2023, 34(10). 4601-4606.
- Ronald Kemker, Marc McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. 2018. Measuring catastrophic forgetting in neural networks. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32.
- Sophia D Kolak, Ruben Martins, Claire Le Goues, and Vincent Josua Hellendoorn. 2022. Patch generation with language models: Feasibility and scaling behavior. In Deep Learning for Code Workshop.
- Qingyuan Li, Wenkang Zhong, Chuanyi Li, Jidong Ge, and Bin Luo. 2024. Empir- ical Study on the Data Leakage Problem in Neural Program Repair. Journal of Software 35, 7 (2024), 0-0.
- Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. ICLR (2023).
- Nikhil Parasaram, Earl T Barr, and Sergey Mechtaev. 2023. Rete: Learning Names- pace Representation for Program Repair. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1264-1276.
- Weiguo Pian, Hanyu Peng, Xunzhu Tang, Tiezhu Sun, Haoye Tian, Andrew Habib, Jacques Klein, and Tegawendé F Bissyandé. 2023. MetaTPTrans: A meta learning approach for multilingual code representation learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 5239-5247.
- Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485-5551.
- Atsushi Shirafuji, Md Mostafizer Rahman, Md Faizul Ibne Amin, and Yutaka Watanobe. 2023. Program repair with minimal edits using codet5. arXiv preprint arXiv:2309.14760 (2023). · arXiv
- Bjarne Stroustrup. 2013. The C++ programming language. Pearson Education.
- Chunqiu Steven Xia and Lingming Zhang. 2023. Keep the Conversation Going: Fixing 162 out of 337 bugs for $0.42 each using ChatGPT. arXiv:2304.00385 [cs.SE] · arXiv
- Boyang Yang, Haoye Tian, Jiadong Ren, Hongyu Zhang, Jacques Klein, Tegawendé F. Bissyandé, Claire Le Goues, and Shunfu Jin. 2024. Multi-Objective Fine-Tuning for Enhanced Program Repair with LLMs. arXiv:2404.12636 · arXiv
- Michihiro Yasunaga and Percy Liang. 2020. Graph-based, self-supervised program repair from diagnostic feedback. In International Conference on Machine Learning. PMLR, 10799-10808.
Cited by
Sorted by citations
Training LLM-based Tutors to Improve Student Learning Outcomes in Dialogues
Scarlatos, A., Liu, N.-M., Lee, J., Baraniuk, R. G., & Lan, A. S. (2025) · International Conference on Artificial Intelligence in Education · doi:10.1007/978-3-031-98414-3_18 · Cited by 75 · see citing works
DeclarUI: Bridging Design and Development with Automated Declarative UI Code Generation
Zhou, T., Zhao, Y., Hou, X., Sun, X., Chen, K., & Wang, H. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3715726 · Cited by 19 · see citing works
Self-citationAdvanced large language models and visualization tools for data analytics learning
Valverde-Rebaza, J., González, A., Navarro-Hinojosa, O., & Noguez, J. (2024) · Frontiers in Education · doi:10.3389/feduc.2024.1418006 · Cited by 22 · see citing works
Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey
She, X., Liu, Y., Zhao, Y., He, Y., Li, L., Tantithamthavorn, C. K., Qin, Z., & Wang, H. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3748647 · Cited by 23 · see citing works
Self-citationA 2030 Roadmap for Software Engineering
Pezzè, M., Abrahão, S., Penzenstadler, B., Poshyvanyk, D., Roychoudhury, A., & Yue, T. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3731559 · Cited by 13 · see citing works
Context-aware prompting for LLM-based program repair
Li, Y., Cai, M., Chen, J., Xu, Y., Huang, L., & Li, J. K. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00512-w · Cited by 10 · see citing works
Reihanian, I., Hou, Y., & Sun, Q. (2025) · AI · doi:10.3390/ai7010006 · Cited by 10 · see citing works
Automated Testing of the GUI of a Real-Life Engineering Software using Large Language Models
Rosenbach, T., Heidrich, D., & Weinert, A. (2025) · IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) · doi:10.1109/icstw64639.2025.10962502 · Cited by 6 · see citing works
Nongkhai, L. N., Wang, J., Wynn, A., & Mendori, T. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100526 · Cited by 6 · see citing works
Towards Detecting Prompt Knowledge Gaps for Improved LLM-guided Issue Resolution
Ehsani, R., Pathak, S., & Chatterjee, P. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00107 · Cited by 9 · see citing works
Koohestani, R., de Bekker, P., Koç, B., & Izadi, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3644183 · Cited by 5 · see citing works
Can AI Fix Buggy Code? Exploring the Use of Large Language Models in Automated Program Repair
Zhang, L., Singhal, A., Zou, Q., Sun, X., Liu, P., & Lin, H.-Y. (2025) · Computer · doi:10.1109/mc.2025.3527407 · Cited by 4 · see citing works
Aruleba, K., Oyelere, S. S., Obaido, G., & Sanusi, I. T. (2025) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3754508.3754517 · Cited by 4 · see citing works
Du, B., Kang, X., Xu, H., Wu, Y., & Liu, Y. (2025) · IEEE International Conference on Software Quality, Reliability and Security (QRS) · doi:10.1109/qrs65678.2025.00016 · Cited by 3 · see citing works
AI-driven self-healing across the edge–cloud continuum: A systematic literature review
Bonilla, L., Díaz-de-Arcaya, J., Aguirre-Usandizaga, J., & Almeida, A. (2026) · Information and Software Technology · doi:10.1016/j.infsof.2026.108232 · Cited by 3 · see citing works
You Don’t Have to Say Where to Edit! jLED—Joint Learning to Localize and Edit Source Code
Pian, W., Li, Y., Tian, H., Sun, T., Song, Y., Tang, X., Habib, A., Klein, J., & Bissyandé, T. F. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3712187 · Cited by 2 · see citing works
Self-citationFu, A., Xu, P., Li, J., Kuang, B., & Gao, Y. (2025) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2025.3618407 · Cited by 1 · see citing works
Misconception-Aware LLM Programming Tutor: Lessons Learned from Student-Tutor Interactions
Niousha, R., Smith, S. F., O’Neill, A., Zamfirescu-Pereira, J. D., DeNero, J., & Norouzi, N. (2026) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3770761.3777289 · Cited by 1 · see citing works
HybridLog: LLM-Powered Log Anomaly Analysis with Heterogeneous Semantic Collaboration Training
Zuo, H., Tang, F., Li, Y., Zhao, M., Wang, R., & Liu, J. (2025) · IEEE Smart World Congress (SWC) · doi:10.1109/swc65939.2025.00123 · Cited by 1 · see citing works
Fayed, M. S., & Fayed, A. S. (2026) · Electronics · doi:10.3390/electronics15040903 · Cited by 1 · see citing works
Towards defect-type-aware adaptive program repair: A stage-wise approach with large language models
Zhang, R., Zhang, S., & Xie, L. (2026) · Software Quality Journal · doi:10.1007/s11219-026-09751-w · Cited by 3 · see citing works
Debugging Engine Enhanced by Prior Knowledge: Can We Teach LLM How to Debug?
Li, K., Wu, S., Tang, X., Yao, C., Bu, S., Xu, Q., & Chen, G. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797110 · Cited by 1 · see citing works
SWE-Bench+: Enhanced LLM Coding Benchmark
Xue, H., Aleithan, R., Enan, N., Uddin, G., & Wang, S. (2026) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3805760.3814924 · Cited by 1 · see citing works
LLM-Enabled Educational Artefacts in Computer Science Education: A Systematic Mapping Study
Sanchez, F. L., Alario‐Hoyos, C., Guamán, D. S., & Caiza, J. C. (2026) · Applied Sciences · doi:10.3390/app16199778
SPVR: syntax-to-prompt vulnerability repair based on large language models
Wang, R., Li, Z., Gao, C., Wang, C., Yang, X. H., & Wang, X. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00579-5 · Cited by 1 · see citing works
Exploration Study About LLM with Proper Prompt in Automated Program Repair
Liang, H., Gao, C., & Jia, Y. F. (2025) · Communications in computer and information science · doi:10.1007/978-981-96-4509-1_28
冯, 丙. (2025) · Advances in Education · doi:10.12677/ae.2025.15101822
PCRepair: A Context-Aware Template-Based Approach for Automated Program Repair
Cao, H., Zhang, W., Wang, Y., Chu, Y., Deng, M., & He, Z. (2025) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194025500731
How Efficient is Your Clone? Evaluating LLM Capabilities for Runtime and MemoryAware Code SelectionPreprint
RAZZAQ, A., Ramaswamy, P., Akbar, M. A., & Rao, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5679011
HLSDebugger: Identification and Correction of Logic Bugs in HLS Code with LLM Solutions
Wang, J., Liu, S., Lu, Y., & Xie, Z. (2025) · doi:10.1109/iccad66269.2025.11240664
SHARE-RAG: A Structure-Aware and Hypothesis-Guided Retrieval-AugmentedPreprint
Liu, Y. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6365614
PRLBridge: A Legality-Aware Interface between Compilers and LLMs for Secure DNN Obfuscation
Li, Y., Li, M., & Fang, L. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3817605
SmarTracker: Leveraging LLMs to Automate User Behavior Tracking in Mobile Apps
Wu, S., Zhou, H., Zhao, K., Yu, L., Bao, S., & Luo, X. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3701153
How efficient is your clone? Evaluating LLM capabilities for runtime and memory-aware code selection
Ramaswamy, P., Akbar, M. A., Rao, M., & Razzaq, A. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112973
DebugTA: An LLM-Based Agent for Simplifying Debugging and Teaching in Programming Education
Fu, L., Chen, D., Yuan, H., Li, Q., Liu, W., Zhang, W., & Yu, Y. H. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29755-6_35
Automated Repair of Requirements for Cyber-Physical Systems in Simulink Requirements Tables
Babikian, A. A., Sandro, A. D., Formica, F., Menghi, C., & Chećhik, M. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3808208
Mining Long Tail Bugs: Identifying Rare and Overlooked Issues in Code
Liang, W., Wu, Y., Ling, X., Luo, T., Liu, D., Zhang, H., & Wu, J. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797071
Li, X., Ye, S., Li, W., & Li, Z. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3805558
Wu, L., Wei, J., Wu, Y., Du, B., Zhao, R., & Liu, Y. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113057
Messarhi, O., Kerdoudi, M. L., & Tibermacine, O. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113101
InfraAPR: An Agent-Based Repair Framework for Cloud Infrastructure-as-Code Programs
Chen, Q., Liu, Z., Li, C., Zhang, Y., Hu, N., & Liao, Q. (2026) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3832783.3834526
- Authors
- Yang, B., Tian, H., Pian, W., Yu, H., Wang, H., Klein, J., Bissyandé, T. F., & Jin, S.
- Affiliations
- Yanshan University (CN)
- The University of Melbourne (AU)
- University of Luxembourg (LU)
- Beijing JudaoYouda Network Technology Co. Ltd., Beijing, China
- ORCID
- Venue
- Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis
- Publisher
- ACM
- Published
- 2024-09-11
- Pages
- 882-894
- OpenAlex
- W4402457600
- Cited by
- 41
- References
- 38
- Citations by year
- 2026: 18 · 2025: 22 · 2024: 1
- FWCI
- 15.61
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Open Repository and Bibliography (University of Luxembourg) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Funders
- European Commission (949014, HORIZON2020)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- program repair, large language models, tutor guidance, GPT-4, code review
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
