CoChat
Sign Up
Cited by 4Software Engineering ResearchSoftware Testing and Debugging TechniquesWeb Application Security Vulnerabilities

The DOI matches its Crossref record.

SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories

Open access
Shen, Chihao, Dilgren, Connor, Chiniya, Purva, Griffith, Luke, Ding, Yu, Chen, Yizheng·2026·Proceedings of the 3rd International Workshop on Large Language Models For Code
DOI 10.1145/3786181.3788703
Open this paper in CoChat — free account

Abstract

This paper introduces SecRepoBench, a benchmark to evaluate code agents on secure code completion in real-world repositories. SecRepoBench has 318 code completion tasks in 27 C/C++ repositories, covering 15 CWEs. We evaluate 29 standalone LLMs and 15 code agents across 3 state-of-the-art agent frameworks using our benchmark. We find that state-of-the-art LLMs struggle with generating correct and secure code completions. However, code agents significantly outperform standalone LLMs. We show that SecRepoBench is more difficult than the prior state-of-the-art benchmark. Finally, our comprehensive analysis provides insights into potential directions for enhancing the ability of code agents to write correct and secure code in real-world repositories.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3786181.3788703

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

No references could be resolved yet.

Research works printed but not linked: 34

  1. Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et al. 2021. Program Synthesis with Large Language Models. arXiv preprint arXiv:https://arXiv.org/abs/2108.07732 (2021). · arXiv
  2. Manish Bhatt Sahana Chennabasappa Cyrus Nikolaidis Shengye Wan Ivan Evtimov Dominik Gabi Daniel Song Faizan Ahmad Cornelius Aschermann Lorenzo Fontana et al. 2023. Purple llama cyberseceval: A secure coding benchmark for language models. arXiv preprint arXiv:https://arXiv.org/abs/2312.04724 (2023). · arXiv
  3. Egor Bogomolov Aleksandra Eliseeva Timur Galimzyanov Evgeniy Glukhov Anton Shapkin Maria Tigina Yaroslav Golubev Alexander Kovrigin Arie van Deursen Maliheh Izadi et al. 2024. Long Code Arena: a Set of Benchmarks for Long-Context Code Models. arXiv preprint arXiv:https://arXiv.org/abs/2406.11612 (2024). · arXiv
  4. Mark Chen Jerry Tworek Heewoo Jun Qiming Yuan Henrique Ponde de Oliveira Pinto Jared Kaplan Harri Edwards Yuri Burda Nicholas Joseph Greg Brockman et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:https://arXiv.org/abs/2107.03374 (2021). · arXiv
  5. 10.1145/3607199.3607242 · DOI
  6. 10.1109/ICSE55347.2025.00038 · DOI
  7. 10.52202/075280-2023 · DOI
  8. Yanjun Fu Ethan Baker Yu Ding and Yizheng Chen. 2024. Constrained Decoding for Secure Code Generation. arXiv:https://arXiv.org/abs/2405.00218 (2024). · arXiv
  9. Alex Gu Baptiste Rozière Hugh Leather Armando Solar-Lezama Gabriel Synnaeve and Sida I Wang. 2024. Cruxeval: A benchmark for code reasoning understanding and execution. arXiv preprint arXiv:https://arXiv.org/abs/2401.03065 (2024). · arXiv
  10. 10.1109/SaTML59370.2024.00040 · DOI
  11. 10.1145/3576915.3623175 · DOI
  12. Jingxuan He, Mark Vero, Gabriela Krasnopolska, and Martin Vechev. 2024. Instruction Tuning for Secure Code Generation. In Proceedings of the International Conference on Machine Learning (ICML).
  13. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, et al. 2021. Measuring Coding Challenge Competence With APPS. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  14. Ivan Homoliak Martin Perešíni Aleš Smrčka Kamil Malinka and Petr Hanacek. 2024. Enhancing Security of AI-Based Code Synthesis with GitHub Copilot via Cheap and Efficient Prompt-Engineering. arXiv preprint arXiv:https://arXiv.org/abs/2403.12671 (2024). · arXiv
  15. Nafis Tanveer Islam and Peyman Najafirad. 2024. Code Security Vulnerability Repair Using Reinforcement Learning with Large Language Models. In Proceedings of the AAAI Conference on Artificial Intelligence Workshop.
  16. Naman Jain King Han Alex Gu Wen-Ding Li Fanjia Yan Tianjun Zhang Sida Wang Armando Solar-Lezama Koushik Sen and Ion Stoica. 2024. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv preprint arXiv:https://arXiv.org/abs/2403.07974 (2024). · arXiv
  17. 10.1145/3650212.3680371 · DOI
  18. Yujia Li David Choi Junyoung Chung Nate Kushman Julian Schrittwieser Rémi Leblond Tom Eccles James Keeling Felix Gimeno Agustin Dal Lago et al. 2022. Competition-level code generation with alphacode. Science 378 6624 (2022) 1092–1097. · DOI · Paper page
  19. Shanchao Liang Yiran Hu Nan Jiang and Lin Tan. 2024. Can Language Models Replace Programmers? REPOCOD Says’ Not Yet’. arXiv preprint arXiv:https://arXiv.org/abs/2410.21647 (2024). · arXiv
  20. Jiawei Liu Chunqiu Steven Xia Yuyao Wang and Lingming Zhang. 2024. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems 36 (2024).
  21. Tianyang Liu, Canwen Xu, and Julian McAuley. 2024. RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems. In The Twelfth International Conference on Learning Representations.
  22. 10.1145/3658644.3690298 · DOI
  23. Yuzhou Nie, Zhun Wang, Yu Yang, Ruizhe Jiang, Yuheng Tang, Xander Davies, Yarin Gal, Bo Li, Wenbo Guo, and Dawn Song. 2025. SecCodePLT: A Unified Benchmark for Evaluating the Security Risks and Capabilities of Code Agents. In The Thirty-Ninth Annual Conference on Neural Information Processing Systems.
  24. 10.1109/SP46214.2022.9833571 · DOI · Paper page
  25. 10.1109/SP46215.2023.10179324 · DOI
  26. 10.1109/LLM4Code66737.2025.00009 · DOI
  27. 10.1145/3433210.3437528 · DOI
  28. 10.1145/3549035.3561184 · DOI · Paper page
  29. Mark Vero Niels Mündler Victor Chibotaru Veselin Raychev Maximilian Baader Nikola Jovanović Jingxuan He and Martin Vechev. 2025. BaxBench: Can LLMs Generate Correct and Secure Backends?arXiv preprint arXiv:https://arXiv.org/abs/2502.11844 (2025). · arXiv
  30. Jiexin Wang Liuwen Cao Xitong Luo Zhiping Zhou Jiayuan Xie Adam Jatowt and Yi Cai. 2023. Enhancing Large Language Models for Secure Code Generation: A Dataset-driven Study on Vulnerability Mitigation. arXiv preprint arXiv:https://arXiv.org/abs/2310.16263 (2023). · arXiv
  31. Xingyao Wang Boxuan Li Yufan Song Frank F. Xu Xiangru Tang Mingchen Zhuge Jiayi Pan Yueqi Song Bowen Li Jaskirat Singh Hoang H. Tran Fuqiang Li Ren Ma Mingzhang Zheng Bill Qian Yanjun Shao Niklas Muennighoff Yizhe Zhang Binyuan Hui Junyang Lin Robert Brennan Hao Peng Heng Ji and Graham Neubig. 2024. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. arxiv:https://arXiv.org/abs/2407.16741 [cs.SE] https://arxiv.org/abs/2407.16741 · arXiv
  32. Colin White Samuel Dooley Manley Roberts Arka Pal Ben Feuer Siddhartha Jain Ravid Shwartz-Ziv Neel Jain Khalid Saifullah Siddartha Naidu et al. 2024. Livebench: A challenging contamination-free llm benchmark. arXiv preprint arXiv:https://arXiv.org/abs/2406.19314 (2024). · arXiv
  33. Chunqiu Steven Xia Yinlin Deng and Lingming Zhang. 2024. Top Leaderboard Ranking= Top Coding Proficiency Always? EvoEval: Evolving Coding Benchmarks via LLM. Proceedings of the First Conference on Language Modeling (2024).
  34. 10.1609/aaai.v38i19.30185 · DOI

Cited by

Sorted by citations

Metadata

Authors
Shen, C., Dilgren, C., Chiniya, P., Griffith, L., Ding, Y., & Chen, Y.
Affiliations
  • University of Maryland, College Park (US)
  • Google (United States) (US)
ORCID
Venue
Proceedings of the 3rd International Workshop on Large Language Models For Code
Publisher
ACM
Published
2026-04-12
Pages
159-166
DOI
10.1145/3786181.3788703
OpenAlex
W4416017103
Cited by
4
References
0
Citations by year
2026: 3
FWCI
32.02
Type
conference paper
Language
en
Open access
Licence
Locations
  • International Workshop on Large Language Models for Code (LLM4Code) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • NSF (CNS-244271)
  • Open Philanthropy (No award number)
  • OpenAI (No award number)
  • Google DeepMind (No award number)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
Keywords
coding agents, benchmarking, Common Weakness Enumeration, large language models, code generation, software security

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review