The DOI matches its Crossref record; 1 of the 1 references it cites were found in a registry or an index.
GitBug-Java: A Reproducible Benchmark of Recent Java Bugs
Open accessAbstract
Bug-fix benchmarks are essential for evaluating methodologies in automatic program repair (APR) and fault localization (FL). However, existing benchmarks, exemplified by Defects4J, need to evolve to incorporate recent bug-fixes aligned with contemporary development practices. Moreover, reproducibility, a key scientific principle, has been lacking in bug-fix benchmarks. To address these gaps, we present GitBug-Java, a reproducible benchmark of recent Java bugs. GitBug-Java features 199 bugs extracted from the 2023 commit history of 55 notable open-source repositories. The methodology for building GitBug-Java ensures the preservation of bug-fixes in fully-reproducible environments. We publish GitBug-Java at https://github.com/gitbugactions/gitbug-java.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
To preserve integrity and prevent leakage to AI assistants' training data, the code was not hosted in a public git repository.
“Finally, to preserve the integrity of the study, i.e., preventing any leakage to AI assistants' training data (Silva et al. 2024), we chose not to host the code in a public git repository.”
From Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability (2026) · cites this paper as Silva et al. 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Defects4J: a database of existing faults to enable controlled testing studies for Java programs
Just, R., Jalali, D., & Ernst, M. D. (2014) · Proceedings of the 2014 International Symposium on Software Testing and Analysis · doi:10.1145/2610384.2628055 · Cited by 1,447
Flagged first, then by citations
Research works printed but not linked: 23
- 10.1109/ASE56229.2023.00116 · DOI
- 10.1109/ICSE-Companion.2019.00035 · DOI
- 10.1145/3524842.3528480 · DOI
- Thomas Durieux and Rui Abreu. 2019. Critical review of bugswarm for fault localization and program repair. arXiv preprint arXiv:1905.09375 (2019). · arXiv
- 10.1109/ICST.2019.00019 · DOI
- Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. arXiv preprint arXiv:2305.10160 (2023). · arXiv
- 10.1109/ICSE48619.2023.00125 · DOI · Paper page
- Jae Yong Lee, Sungmin Kang, Juyeon Yoon, and Shin Yoo. 2023. The GitHub Recent Bugs Dataset for Evaluating LLM-based Debugging Applications. arXiv preprint arXiv:2310.13229 (2023). · arXiv
- 10.1145/3135932.3135941 · DOI · Paper page
- 10.1109/ASE.2015.87 · DOI
- 10.1145/2635868.2635920 · DOI
- 10.1109/SANER.2019.8667991 · DOI
- 10.1016/j.cola.2019.03.006 · DOI
- Julian Aron Prenner and Romain Robbes. 2023. RunBugRun-An Executable Dataset for Automated Program Repair. arXiv preprint arXiv:2304.01102 (2023). · arXiv
- Nuno Saavedra, André Silva, and Martin Monperrus. 2024. GitBug-Actions: Building Reproducible Bug-Fix Benchmarks with GitHub Actions. In 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE.
- 10.1145/3196398.3196473 · DOI
- André Silva, Matias Martinez, Benjamin Danglot, Davide Ginelli, and Martin Monperrus. 2021. Flacoco: Fault localization for java based on industry-grade coverage. arXiv preprint arXiv:2111.12513 (2021). · arXiv
- 10.1109/ICSE.2003.1201189 · DOI
- Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. DebugBench: Evaluating Debugging Capability of Large Language Models. arXiv preprint arXiv:2401.04621 (2024). · arXiv
- 10.1109/ICSE.2019.00048 · DOI
- 10.1145/1882362.1882446 · DOI
- Quanjun Zhang Tongke Zhang Juan Zhai Chunrong Fang Bowen Yu Weisong Sun and Zhenyu Chen. 2023. A Critical Review of Large Language Model on Software Engineering: An Example from ChatGPT and Automated Program Repair. arXiv:2310.08879 [cs.SE] · arXiv
- 10.1109/ICSE48619.2023.00195 · DOI
Cited by
Sorted by citations
Zhou, X., Weyssow, M., Widyasari, R., Zhang, T., He, J., Lyu, Y., Chang, J., Zhang, B., Huang, D., & Lo, D. (2025) · arXiv · doi:10.48550/arxiv.2502.06215 · Cited by 3
Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability
Borg, M., Hewett, D., Hagatulah, N., Couderc, N., Söderberg, E., Graham, D., Kini, U., & Farley, D. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10889-1 · Cited by 4
Metadata
- Authors
- Silva, A., Saavedra, N., & Monperrus, M.
- Affiliations
- Instituto de Engenharia de Sistemas e Computadores Investigação e Desenvolvimento (PT)
- Venue
- Proceedings of the 21st International Conference on Mining Software Repositories
- Publisher
- ACM
- Published
- 2024-04-15
- Pages
- 118-122
- OpenAlex
- W4400242157
- Cited by
- 23
- References
- 1
- Citations by year
- 2026: 9 · 2025: 13 · 2024: 1
- FWCI
- 6.28
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- International Conference on Mining Software Repositories (MSR) (publishedVersion, cc-by-sa)
- arXiv (Cornell University) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- Fundação para a Ciência e a Tecnologia (BD/04736/2023, UIDB/50021/2020)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Computer Networks and Communications › Software System Performance and Reliability
- Keywords
- reproducibility, automated program repair, fault location, Defects4J, open-source repositories, commit history
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
