Helping LLMs improve code generation using feedback from testing and static analysis
Open accessPublished version of a preprint
The DOI matches its Crossref record; 36 of the 36 references it cites were found in a registry or an index; none of the 19 sources its claims cite is checked for support yet.
Abstract
Large Language Models (LLMs) are one of the most promising developments in the field of artificial intelligence, and the software engineering community has readily noticed their potential role in the software development life-cycle. Developers routinely ask LLMs to generate code snippets, increasing productivity but also potentially introducing ownership, privacy, correctness, and security issues. Previous work highlighted how code generated by mainstream commercial LLMs is often not safe, containing vulnerabilities, bugs, and code smells. In this paper, we present a framework that leverages testing and static analysis to assess the quality, and guide the self-improvement, of code generated by general-purpose, open-source LLMs. First, we ask LLMs to generate C code to solve a number of programming tasks. Then we employ ground-truth tests to assess the (in)correctness of the generated code, and a static analysis tool to detect potential safety vulnerabilities. Next, we assess the models ability to evaluate the generated code, by asking them to detect errors and vulnerabilities. Finally, we test the models ability to fix the generated code, providing the reports produced during the static analysis and incorrectness evaluation phases as feedback. Our results show that models often produce incorrect code, and that the generated code can include safety issues. Moreover, they perform very poorly at detecting either issue. On the positive side, we observe a substantial ability to fix flawed code when provided with information about failed tests or potential vulnerabilities, indicating a promising avenue for improving the safety of LLM-based code generation tools.
Overview
CoChat Summary
TL;DR
The paper evaluates four open-source LLMs on C code generation, self-evaluation, and repair using unit tests plus Infer static analysis on an MBPP-derived benchmark of 89 tasks. It finds that generated code is more often functionally incorrect than insecure, self-evaluation is poor, and repair improves both correctness and safety when the models are given test or analyzer feedback.
Key findings
- After benchmark creation and cleaning, 89 tasks compiled for all models, yielding a GEN dataset of 356 files; across these, correctness ranged from 46% for gemma-7b-it to 65% for llama3-70b-8192.
- Infer found 44 vulnerabilities in total on the GEN dataset; safe files were 328/356 overall, and by model the safe-file counts were 85, 84, 82, and 77 for gemma-7b-it, llama3-8b-8192, llama3-70b-8192, and mixtral-8x7b-32768, respectively.
- Self-evaluation performed poorly: for correctness, F1-scores were all reported as poor, with one model often replying almost always YES and another almost always NO; for safety, all F1-scores were under 0.3.
- Repair improved outcomes: the best correctness repair result was 96 out of 155 incorrect files (62%), and the best safety repair result fixed 39 out of 44 vulnerabilities (89%); overall, the paper says about 77% of previously detected vulnerabilities were fixed.
Methods
- Built a benchmark by randomly selecting 100 MBPP tasks, translating them to C, and manually adapting unit tests; after compilation, 89 tasks remained in the GEN dataset.
- Evaluated four open-source instruction-tuned models: gemma-7b-it, llama3-8b-8192, llama3-70b-8192, and mixtral-8x7b-32768, queried through the Groq API at temperature 0.2.
- Ran a three-phase pipeline: code generation, self-evaluation, and repair; correctness was checked with MBPP-derived unit tests, and safety with Infer on compiling programs, with up to 6 repair iterations.
Limitations
- The paper states that using a crafted C version of MBPP does not entirely eliminate memorization risk, because parts of the task formulation may still overlap with training data.
- It also states that Infer is neither complete nor sound, so it can produce false positives and false negatives; the authors manually inspected detected vulnerabilities.
- Evident from the text: the benchmark was reduced from 100 selected tasks to 89 compiling tasks, and the study is limited to C and to the four chosen open-source models, so the results may not generalize beyond that setting.
Model-written summary of a shortened version of the paper’s full text (it is longer than we read). Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 Valid · 2 Not valid · 12 Not checked · 100 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results show that models often produce incorrect code, and that the generated code can include safety issues.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find that only 46% to 65% of the generated code does not show correctness issues.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find instead a strong tendency for one-sided predictions in particular for two models.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unsurprisingly, none of the generated code compiled as-is due to the presence of textual reasoning at the end of the code sections, as required by the prompt.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results shows how, with some effort and intervention, the code generated by each model can be made to compile in 93% or more of the cases”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
44 more claims
Create a free account to see all 114 claims and to verify the 19 references still open.
- It may state two separate claims as one. 37
- Probably not a substantial claim. 41
- The quote only partly states this claim. 3
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ I. INTRODUCTION
“The best model for correctness manages to completely repair only 96 out of 155 incorrect files (62%).”
§ Repair
“| llama3-70b-8192 | 92 | 8 | 52 | 2 | 59% |”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| | llama3-70b-8192 | 5/6 | 83% | 8% | 71% | 14% |”
Reported 71 · recomputed 83.333333
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Repair
“For all the models, the highest percentage of success in fixing the vulnerabilities is obtained when repairing the code generated by llama3-70b-8192 , with percentages ranging from 90% to 100%.”
§ Repair
“| mixtral-8x7b-32768 | gemma-7b-it | 7 | 0 | 0 | | | llama3-8b-8192 | 8 | 0 | 0 | | | llama3-70b-8192 | 9 | 1 | 0 |”
Citations
Cites
Cousot, P., & Cousot, R. (1977) · Proceedings of the 4th ACM SIGACT-SIGPLAN symposium on Principles of programming languages - POPL '77 · doi:10.1145/512950.512973 · Cited by 6,279 · cited 1×
“…sized allocations), an out-of-bound array access, or an integer computation leading to overflow using an analysis based on the domain of intervals [14]; • linters , which implements a declarative-based lint-ing framework providing syntax-based checkers tracking, e.g., dangerous pointer…”
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions
Pearce, H., Ahmad, B., Tan, B., Dolan-Gavitt, B., & Karri, R. (2022) · 2022 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp46214.2022.9833571 · Cited by 476
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436 · cited 1×
“Despite that, programmers frequently use LLM-powered tools (e.g., GitHub Copilot [2]) for speeding up the development process or to get a starting point for finding solutions [3], be that for simple code snippets, specific functions or entire software applications.”
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. D. L., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.06825 · Cited by 340 · cited 1×
“This model has the same architecture as Mistral 7B [10] but each layer is composed of 8 feed-forward blocks (i.e., experts) and for every token, at each layer, a router network selects two experts to process the current state and combines their outputs, eventually with different experts at every…”
Gemma: Open Models Based on Gemini Research and Technology
Gemma Team, Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivière, M., Kale, M. S., Love, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2403.08295 · Cited by 284 · cited 2×
“For our experiments we consider four LLMs: Llama 3 8B and 70B [5], Gemma 7B [6], and Mixtral 8X7B [7].”
IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities
Li, Z., Dutta, S., & Naik, M. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2405.17238 · Cited by 260
InferFix: End-to-End Program Repair with LLMs
Jin, M., Shahriar, S., Tufano, M., Shi, X., Lu, S., Sundaresan, N., & Svyatkovskiy, A. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3613892 · Cited by 191 · cited 1×
“Frameworks for program repair based on static analyzers like InferFix [19] employ LLMs fine-tuned on bug repair patches to solve core repairing tasks for three types of bugs.”
Infer: An Automatic Program Verifier for Memory Safety of C Programs
Calcagno, C., & Distefano, D. (2011) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-20398-5_33 · Cited by 180 · cited 1×
“In order to automatically detect safety issues such as memory management, resource leaks, and access control in LLM-generated C code, we use Infer [8], a state-of-the-art open-source static analysis tool by Meta.”
Lu, G., Ju, X., Chen, X., Pei, W., & Cai, Z. (2024) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112031 · Cited by 179
Research works printed but not linked: 7
- Gartner: gartner hype cycle shows ai practices and platform engineering will reach mainstream adoption in software engineering in two to five years 2024. https://www.gartner.com/en/newsroom/press-releases/2023-11-28-gartner-hype-cycle-shows-ai-practices-and-platform-engineering-will-reach-mainstream-adoption-in-software-engineering-in-two-to-five-years.
- Austin J, Odena A, Nye MI, Bosma M, Michalewski H, Dohan D, Jiang E, Cai CJ, Terry M, Le QV, Sutton C. Program synthesis with large language models. 2021 CoRR arXiv:abs/2108.07732.
- Wei J, Wang X, Schuurmans D, Bosma M, Ichter B, Xia F, Chi EH, Le QV, Zhou D. Chain-of-thought prompting elicits reasoning in large language models. In: Koyejo S, Mohamed S, Agarwal A, Belgrave D, Cho K, Oh A. (eds.) Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022 2022. http://papers.nips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html.
- Zhou Y, Muresanu AI, Han Z, Paster K, Pitis S, Chan H, Ba J. Large language models are human-level prompt engineers. In: The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1–5, 2023. OpenReview.net, 2023. https://openreview.net/forum?id=92gvk82DE-.
- Olausson TX, Inala JP, Wang C, Gao J, Solar-Lezama A. Is self-repair a silver bullet for code generation? In: The Twelfth International Conference on Learning Representations, 2023.
- Islam NT, Khoury J, Seong A, Bou-Harb E, Najafirad P. Enhancing source code security with llms: demystifying the challenges and generating reliable repairs. In: Network and Distributed System Security (NDSS) Symposium 2024, 2024.
- He J, Vero M, Krasnopolska G, Vechev MT. Instruction tuning for secure code generation. In: Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21–27, 2024. OpenReview.net, 2024. https://openreview.net/forum?id=MgTzMaYHvG.
Cited by
A Systematic Literature Review on Detecting Software Vulnerabilities with Large Language Models
Kaniewski, S., Schmidt, F., Enzweiler, M., Menth, M., & Heer, T. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3815425 · Cited by 38
Position: Vibe Coding Needs Vibe Reasoning: Improving Vibe Coding with Formal Verification
Mitchell, J., & Shaaban, Y. (2025) · Proceedings of the 1st ACM SIGPLAN International Workshop on Language Models and Programming Languages · doi:10.1145/3759425.3763390 · Cited by 8
Sokol, O. (2025) · Technologies and Engineering · doi:10.30857/2786-5371.2025.1.3 · Cited by 2
“Finally, G. Dolcetti et al. (2024) emphasised the importance of feedback from testing and static analysis.”
Adaptive Self-Prompting in Agentic LLM Frameworks for Code Fault Detection
Muhtadi, M., Mahmoud, Q. H., & Azim, A. (2026) · Software · doi:10.3390/software5020016
“Dolcetti et al. [30] propose a framework that combines prompt engineering with feedback from testing and static analysis to improve LLM-based code generation and repair.”
A Generative AI Tool for Constructing Web Frontend with Integration of Quality Feedback Loop
Walloppornlapak, W., & Senivongse, T. (2026) · doi:10.1145/3816713.3818803
Metadata
- Authors
- Dolcetti, G., Arceri, V., Iotti, E., Maffeis, S., Cortesi, A., & Zaffanella, E.
- Affiliations
- Ca' Foscari University of Venice (IT)
- University of Parma (IT)
- Imperial College London (GB)
- Corresponding author
- Dolcetti, Greta
- Venue
- Discover Artificial Intelligence
- Published
- 2026-03-05
- Volume
- 6
- Issue
- 1
- ISSN
- 2731-0809 (ISSN-L)
- OpenAlex
- W4405634796
- Cited by
- 8
- References
- 26
- Citations by year
- 2026: 6
- FWCI
- 51.04
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by-nc-nd
- creativecommons.org/licenses/by-nc-nd/4.0 (tdm)
- creativecommons.org/licenses/by-nc-nd/4.0 (vor, 39 d)
- Locations
- Discover Artificial Intelligence (publishedVersion, cc-by-nc-nd)
- arXiv (Cornell University) (submittedVersion, cc-by)
- ARCA (Università Ca' Foscari Venezia) (submittedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University) (cc-by)
- Funders
- Università degli Studi di Parma
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- large language models, code generation, static analysis, software testing, code safety, vulnerability detection, bug detection, code repair, C programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Helping LLMs improve code generation using feedback from testing and static analysis. https://app.cochat.ai/papers/10.1007/s44163-026-01009-5
- The paper
- Dolcetti, G., Arceri, V., Iotti, E., Maffeis, S., Cortesi, A., & Zaffanella, E. (2026). Helping LLMs improve code generation using feedback from testing and static analysis. Discover Artificial Intelligence. https://doi.org/10.1007/s44163-026-01009-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.