eyeballvul: a future-proof benchmark for vulnerability detection in the wild
Open accessAbstract
Long contexts of recent LLMs have enabled a new use case: asking models to find security vulnerabilities in entire codebases. To evaluate model performance on this task, we introduce eyeballvul: a benchmark designed to test the vulnerability detection capabilities of language models at scale, that is sourced and updated weekly from the stream of published vulnerabilities in open-source repositories. The benchmark consists of a list of revisions in different repositories, each associated with the list of known vulnerabilities present at that revision. An LLM-based scorer is used to compare the list of possible vulnerabilities returned by a model to the list of known vulnerabilities for each revision. As of July 2024, eyeballvul contains 24,000+ vulnerabilities across 6,000+ revisions and 5,000+ repositories, and is around 55GB in size.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Eyeballvul is a benchmark designed to test the vulnerability detection capabilities of language models at scale, containing 24,000+ vulnerabilities across 6,000+ revisions and 5,000+ repositories.
“Eyeballvul (Chauvin 2024) is a benchmark designed to test the vulnerability detection capabilities of language models at scale, which have contained 24,000 + vulnerabilities across 6,000 + revisions and 5,000 + repositories.”
From When LLMs meet cybersecurity: a systematic literature review (2025) · cites this paper as Chauvin 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
No references could be resolved yet.
Research works printed but not linked: 7
- Bhatt, M., Chennabasappa, S., Li, Y., Nikolaidis, C., Song, D., Wan, S., Ahmad, F., Aschermann, C., Chen, Y., Kapil, D., Molnar, D., Whitman, S., & Saxe, J. (2024). Cyberseceval 2: A wide-ranging cyber- security evaluation suite for large language models. https://arxiv.org/abs/2404.13161 · arXiv
- Chen, Y., Ding, Z., Alowain, L., Chen, X., & Wagner, D. (2023). Diversevul: A new vulnerable source code dataset for deep learning based vulnerability detection. https://arxiv.org/abs/2304.00409 · arXiv
- Phuong, M., Aitchison, M., Catt, E., Cogan, S., Kaskasoli, A., Krakovna, V., Lindner, D., Rahtz, M., Assael, Y., Hodkinson, S., Howard, H., Lieberum, T., Kumar, R., Raad, M. A., Webson, A., Ho, L., Lin, S., Farquhar, S., Hutter, M., . . . Shevlane, T. (2024). Evaluating frontier models for dangerous capabilities. https://arxiv.org/abs/2403.13793 · arXiv
- Serebryany, K. OSS-Fuzz -google's continuous fuzzing service for open source software. In: Vancouver, BC: USENIX Association, 2017, August.
- Shao, M., Chen, B., Jancheska, S., Dolan-Gavitt, B., Garg, S., Karri, R., & Shafique, M. (2024). An empir- ical evaluation of llms for solving offensive security challenges. https://arxiv.org/abs/2402.11814 · arXiv
- Shao, M., Jancheska, S., Udeshi, M., Dolan-Gavitt, B., Xi, H., Milner, K., Chen, B., Yin, M., Garg, S., Krishnamurthy, P., Khorrami, F., Karri, R., & Shafique, M. (2024). Nyu ctf dataset: A scalable open-source benchmark dataset for evaluating llms in offensive security. https://arxiv.org/abs/ 2406.05590
- Wen, X.-C., Wang, X., Chen, Y., Hu, R., Lo, D., & Gao, C. (2024). Vuleval: Towards repository-level evaluation of software vulnerability detection. https://arxiv.org/abs/2404.15596 · arXiv
Cited by
Sorted by citations
When LLMs meet cybersecurity: a systematic literature review
Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., & Meng, D. (2025) · Cybersecurity · doi:10.1186/s42400-025-00361-w · Cited by 220 · see citing works
- Authors
- Chauvin, T.
- Corresponding author
- Chauvin, Timothee
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2024
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2407.08708
- OpenAlex
- W4400612098
- Cited by
- 0
- References
- 0
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Web Application Security Vulnerabilities
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Keywords
- vulnerability detection, security vulnerabilities, large language models, benchmark, open source software, software repositories, software security
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
