QuixBugs: a multi-lingual program repair benchmark set based on the quixey challenge
Open accessAbstract
Recent years have seen an explosion of work in automated program repair. While previous work has focused exclusively on tools for single languages, recent work in multi-language transformation has opened the door for multi-language program repair tools. Evaluating the performance of such a tool requires having a benchmark set of similar buggy programs in different languages. We present QuixBugs, consisting of 40 programs translated to both Python and Java, each with a bug on a single line. The QuixBugs benchmark suite is based on problems from the Quixey Challenge, where programmers were given a short buggy program and 1 minute to fix the bug.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Code repair is commonly evaluated on Quixbugs or Python bugs.
“Code repair is commonly evaluated on Quixbugs (Lin et al., 2017;Prenner & Robbes, 2021;Ye et al., 2021;Xia & Zhang, 2023;Jiang et al., 2023;Sobania et al., 2023) or Python bugs (He et al., 2022;Bradley et al., 2023).”
From OctoPack: Instruction Tuning Code Large Language Models (2023) · cites this paper as Lin et al., 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 7 · Cited by 242
Research works printed but not linked: 6
- Dongsun Kim, Jaechang Nam, Jaewoo Song, and Sunghun Kim. 2013. Automatic Patch Generation Learned from Human-Written Patches. In Proceedings of the 2013 International Conference on Software Engineering. IEEE Press, 802-811.
- James Koppel. 2017. "Incremental Parametric Syntax for Multi-Language Transformation". Master's thesis. MIT.
- Claire Le Goues, Thanh Vu Nguyen, Stephanie Forrest, and Westley Weimer. 2012. Genprog: A Generic Method for Automatic Software Repair. IEEE Transactions on Software Engineering 38, 1 (2012), 54-72.
- Fan Long and Martin Rinard. 2016. Automatic Patch Generation by Learning Correct Code. In Proceedings of the 43rd Annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages.
- Matias Martinez and Martin Monperrus. 2016. ASTOR: A Program Repair Library for Java. In Proceedings of the 25th International Sympo- sium on Software Testing and Analysis. ACM, 441-444.
- Shin Hwei Tan, Jooyong Yi, Sergey Mechtaev, Abhik Roychoudhury, et al. 2017. "Codeflaws: A Programming Competition Benchmark for Evaluating Automated Program Repair Tools". In Proceedings of the 39th International Conference on Software Engineering Companion. IEEE Press, 180-182.
- Authors
- Lin, D., Koppel, J., Chen, A., & Solar-Lezama, A.
- Affiliations
- n.n., n.n.
- Massachusetts Institute of Technology (US)
- Google (United States) (US)
- Venue
- Proceedings Companion of the 2017 ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity
- Publisher
- ACM
- Published
- 2017-10-22
- Pages
- 55-56
- OpenAlex
- W2762550985
- Cited by
- 242
- References
- 7
- Citations by year
- 2026: 32 · 2025: 45 · 2024: 47 · 2023: 38 · 2022: 28 · 2021: 19 · 2020: 16 · 2019: 14 · 2018: 3
- FWCI
- 6.69
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM SIGPLAN International Conference on Systems, Programming, Languages, and Applications: Software for Humanity (SPLASH) (publishedVersion, cc-by-nc-sa)
- Funders
- National Science Foundation (1122374)
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Logic, programming, and type systems
- Keywords
- automated program repair, benchmark suite, Python, Java, program translation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
