Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 75 of the 76 references it cites were found in a registry or an index.

The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs

Le Goues, Claire, Holtschulte, Neal, Smith, Edward K., Brun, Yuriy, Devanbu, Premkumar, Forrest, Stephanie, Weimer, Westley · 2015 · IEEE Transactions on Software Engineering

Abstract

The field of automated software repair lacks a set of common benchmark problems. Although benchmark sets are used widely throughout computer science, existing benchmarks are not easily adapted to the problem of automatic defect repair, which has several special requirements. Most important of these is the need for benchmark programs with reproducible, important defects and a deterministic method for assessing if those defects have been repaired. This article details the need for a new set of benchmarks, outlines requirements, and then presents two datasets, ManyBugs and IntroClass, consisting between them of 1,183 defects in 15 C programs. Each dataset is designed to support the comparative evaluation of automatic repair algorithms asking a variety of experimental questions. The datasets have empirically defined guarantees of reproducibility and benchmark quality, and each study object is categorized to facilitate qualitative evaluation and comparisons by category of bug or program. The article presents baseline experimental results on both datasets for three existing repair methods, GenProg, AE, and TrpAutoRepair, to reduce the burden on researchers who adopt these datasets for their own comparative evaluations.

Overview

Integrity signals

Registry

Crossref · 10.1109/tse.2015.2454513

Cites

75 of 75 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 76 references

Research works printed but not linked: 6

  1. lu, 0, Bugbench: Benchmarks for evaluating bug detection tools, Workshop on the Evaluation of Software Defect Detection Tools, 16
  2. 2013, Economist, 409
  3. bradbury, 2010, Automatic repair of concurrency bugs, Proc Int Symp Search Based Softw Eng —Fast Abstracts, 1
  4. järvisalo, 0, Proc Int Conf Theory and Appl Satisfiability Testing
  5. egly, 0, Proc Int Conf Theory and Appl Satisfiability Testing
  6. durieux, 2015, CoRR

Cited by

See all 328

Metadata

Authors
Le Goues, C., Holtschulte, N., Smith, E. K., Brun, Y., Devanbu, P., Forrest, S., & Weimer, W.
Affiliations
  • University of New Mexico (US)
  • University of Massachusetts Amherst (US)
  • University of Virginia (US)
Venue
IEEE Transactions on Software Engineering
Published
2015-12-01
Volume
41
Issue
12
Pages
1236-1256
ISSN
  • 0098-5589 (ISSN-L)
  • 1939-3520
OpenAlex
W841012168
Cited by
326
References
76
Citations by year
2026: 16 · 2025: 31 · 2024: 30 · 2023: 29 · 2022: 34 · 2021: 29 · 2020: 26 · 2019: 35 · 2018: 46 · 2017: 27 · 2016: 17 · 2015: 6
FWCI
134.58
Type
journal article
Language
en
Open access
closed
Locations
IEEE Transactions on Software Engineering (publishedVersion)
Funders
  • AFOSR (FA9550-07-1-0532, FA9550-10-1-0277)
  • US Defense Advanced Research Projects Agency (DARPA) (P-1070-113237)
  • US Department of Energy (DOE) (DE-AC02-05CH11231)
  • US National Science Foundation (NSF) (CCF-0729097, CCF-0905236, CCF-1446683, CNS-0905222))
Topics
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Evolutionary Algorithms and Applications
Keywords
benchmark dataset, C programs, reproducibility, ae

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs. https://app.cochat.ai/papers/10.1109/tse.2015.2454513
The paper
Le Goues, C., Holtschulte, N., Smith, E. K., Brun, Y., Devanbu, P., Forrest, S., & Weimer, W. (2015). The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs. IEEE Transactions on Software Engineering. https://doi.org/10.1109/tse.2015.2454513

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.