The DOI matches its Crossref record; 75 of the 76 references it cites were found in a registry or an index.
The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs
Abstract
The field of automated software repair lacks a set of common benchmark problems. Although benchmark sets are used widely throughout computer science, existing benchmarks are not easily adapted to the problem of automatic defect repair, which has several special requirements. Most important of these is the need for benchmark programs with reproducible, important defects and a deterministic method for assessing if those defects have been repaired. This article details the need for a new set of benchmarks, outlines requirements, and then presents two datasets, ManyBugs and IntroClass, consisting between them of 1,183 defects in 15 C programs. Each dataset is designed to support the comparative evaluation of automatic repair algorithms asking a variety of experimental questions. The datasets have empirically defined guarantees of reproducibility and benchmark quality, and each study object is categorized to facilitate qualitative evaluation and comparisons by category of bug or program. The article presents baseline experimental results on both datasets for three existing repair methods, GenProg, AE, and TrpAutoRepair, to reduce the burden on researchers who adopt these datasets for their own comparative evaluations.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“For the second case, a finite test suite necessarily under-approximates an executable's full specification (Liu et al., 2023; Le Goues et al., 2015).”
From ProgramBench: Can Language Models Rebuild Programs From Scratch? (2026) · cites this paper as Le Goues et al., 2015
In the citing paper
“The program repair community had previously proposed the Manybugs and IntroClass benchmarks for repairing C programs [64], while a large number of research publications have looked at the Defects4J benchmark [84] for repairing Java programs.”
From A 2030 Roadmap for Software Engineering (2025) · cites this paper as [64]
Citations
Cites
The PARSEC benchmark suite: characterization and architectural implications
Bienia, C., Kumar, S., Singh, J. P., & Li, K. (2008) · Proceedings of the 17th international conference on Parallel architectures and compilation techniques · doi:10.1145/1454115.1454128 · Cited by 3,444
KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs
Cadar, C., Dunbar, D., & Engler, D. (2008) · doi:10.5555/1855741.1855756 · Cited by 2,620
An Analysis and Survey of the Development of Mutation Testing
Jia, Y., & Harman, M. (2011) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.62 · Cited by 1,745
The DaCapo benchmarks: java benchmarking development and analysis
Blackburn, S. M., Garner, R., Hoffmann, C., Khang, A. M., McKinley, K. S., Bentzur, R., Diwan, A., Feinberg, D., Frampton, D., Guyer, S. Z., et al. (2006) · Proceedings of the 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applications · doi:10.1145/1167473.1167488 · Cited by 1,600
Do, H., Elbaum, S., & Rothermel, G. (2005) · Empirical Software Engineering · doi:10.1007/s10664-005-3861-2 · Cited by 1,146
GenProg: A Generic Method for Automatic Software Repair
Le Goues, C., Nguyen, T., Forrest, S., & Weimer, W. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.104 · Cited by 1,085
Self-citationAnvik, J., Hiew, L., & Murphy, G. C. (2006) · Proceedings of the 28th international conference on Software engineering · doi:10.1145/1134285.1134336 · Cited by 924
Experiments on the effectiveness of dataflow- and control-flow-based test adequacy criteria
Hutchins, M., Foster, H. D., Goradia, T., & Ostrand, T. J. (1994) · doi:10.5555/257734.257766 · Cited by 774
Automatically finding patches using genetic programming
Weimer, W., Nguyen, T., Le Goues, C., & Forrest, S. (2009) · 2009 IEEE 31st International Conference on Software Engineering · doi:10.1109/icse.2009.5070536 · Cited by 761
Self-citationScalable statistical bug isolation
Liblit, B., Naik, M., Zheng, A. X., Aiken, A., & Jordan, M. I. (2005) · ACM SIGPLAN Notices · doi:10.1145/1064978.1065014 · Cited by 729
Research works printed but not linked: 6
- lu, 0, Bugbench: Benchmarks for evaluating bug detection tools, Workshop on the Evaluation of Software Defect Detection Tools, 16
- 2013, Economist, 409
- bradbury, 2010, Automatic repair of concurrency bugs, Proc Int Symp Search Based Softw Eng —Fast Abstracts, 1
- järvisalo, 0, Proc Int Conf Theory and Appl Satisfiability Testing
- egly, 0, Proc Int Conf Theory and Appl Satisfiability Testing
- durieux, 2015, CoRR
Cited by
Automated Vulnerability Detection in Source Code Using Deep Representation Learning
Russell, R., Kim, L., Hamilton, L., Lazovich, T., Harer, J., Ozdemir, O., Ellingwood, P., & McConley, M. (2018) · 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA) · doi:10.1109/icmla.2018.00120 · Cited by 623
“2 package from the ManyBugs dataset [17] took nearly a day of effort.”
MethodAngelix: scalable multiline program patch synthesis via symbolic analysis
Mechtaev, S., Yi, J., & Roychoudhury, A. (2016) · Proceedings of the 38th International Conference on Software Engineering · doi:10.1145/2884781.2884807 · Cited by 515
“However, we note that the GenProg benchmark we used for our experiments is one of the most extensive one available in the literature (the Many-Bugs benchmark [15] has also been released very recently as an extension of the GenProg benchmark).”
MethodNopol: Automatic Repair of Conditional Statement Bugs in Java Programs
Xuan, J., Martinez, M., DeMarco, F., Clement, M., Marcote, S. L., Durieux, T., Le Berre, D., & Monperrus, M. (2017) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2016.2560811 · Cited by 430
An analysis of patch plausibility and correctness for generate-and-validate patch generation systems
Qi, Z., Long, F., Achour, S., & Rinard, M. (2015) · Proceedings of the 2015 International Symposium on Software Testing and Analysis · doi:10.1145/2771783.2771791 · Cited by 420
Automated Program Repair in the Era of Large Pre-trained Language Models
Xia, C. S., Wei, Y., & Zhang, L. (2023) · 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE) · doi:10.1109/icse48619.2023.00129 · Cited by 433
“3) ManyBugs [64]: A C repair dataset consisting of 185 bugs gathered from 9 open-source projects with developer written tests.”
Metadata
- Authors
- Le Goues, C., Holtschulte, N., Smith, E. K., Brun, Y., Devanbu, P., Forrest, S., & Weimer, W.
- Affiliations
- University of New Mexico (US)
- University of Massachusetts Amherst (US)
- University of Virginia (US)
- Venue
- IEEE Transactions on Software Engineering
- Published
- 2015-12-01
- Volume
- 41
- Issue
- 12
- Pages
- 1236-1256
- ISSN
- 0098-5589 (ISSN-L)
- 1939-3520
- OpenAlex
- W841012168
- Cited by
- 326
- References
- 76
- Citations by year
- 2026: 16 · 2025: 31 · 2024: 30 · 2023: 29 · 2022: 34 · 2021: 29 · 2020: 26 · 2019: 35 · 2018: 46 · 2017: 27 · 2016: 17 · 2015: 6
- FWCI
- 134.58
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- Locations
- IEEE Transactions on Software Engineering (publishedVersion)
- Funders
- AFOSR (FA9550-07-1-0532, FA9550-10-1-0277)
- US Defense Advanced Research Projects Agency (DARPA) (P-1070-113237)
- US Department of Energy (DOE) (DE-AC02-05CH11231)
- US National Science Foundation (NSF) (CCF-0729097, CCF-0905236, CCF-1446683, CNS-0905222))
- Topics
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
- Keywords
- benchmark dataset, C programs, reproducibility, ae
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs. https://app.cochat.ai/papers/10.1109/tse.2015.2454513
- The paper
- Le Goues, C., Holtschulte, N., Smith, E. K., Brun, Y., Devanbu, P., Forrest, S., & Weimer, W. (2015). The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2015.2454513
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.