CoChat

Better Patching Using LLM Prompting, via Self-Consistency

Open access
Ahmed, Toufique, Devanbu, Premkumar · 2023 · 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)
Cited by 41

The DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.

Abstract

Large Language models (LLMs) can be induced to solve non-trivial problems with “few-shot” prompts including illustrative problem-solution examples. Now if the few-shots also include “chain of thought” ($\mathcal{C}oT$) explanations, which are of the form problem-explanation-solution, LLMs will generate a “explained” solution, and perform even better. Recently an exciting, substantially better technique, self-consistency [1] ($\mathcal{S}-C$) has emerged, based on the intuition that there are many plausible explanations for the right solution; when the LLM is sampled repeatedly to generate a pool of explanation-solution pairs, for a given problem, the most frequently occurring solutions in the pool (ignoring the explanations) tend to be even more likely to be correct! Unfortunately, the use of this highly-performant$\mathcal{S}-C$(or even$\mathcal{C}oT$) approach in software engineering settings is hampered by the lack of explanations; most software datasets lack explanations. In this paper, we describe an application of the$\mathcal{S}-C$approach to program repair, using the commit log on the fix as the explanation, only in the illustrative few-shots. We achieve state-of-the art results, beating previous approaches to prompting-based program repair, on the MODIT dataset; we also find evidence suggesting that the correct commit messages are helping the LLM learn to produce better patches.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/ase56229.2023.00065

Cites

17 of 17 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 17 references

Research works printed but not linked: 11

  1. berabi, 2021, Tfix: Learning to fix coding errors with a text-to-text transformer, International Conference on Machine Learning, 780
  2. 10.1109/ASE51524.2021.9678559 · DOI · Paper page
  3. wang, 2022, Self-consistency improves chain of thought reasoning in language models, ArXiv Preprint
  4. 10.1109/ICSE43902.2021.00107 · DOI · Paper page
  5. chen, 2019, Sequencer: Sequence-to-sequence learning for end-to-end program repair, IEEE Transactions on Software Engineering, 47, 1943
  6. 10.1145/3510003.3510222 · DOI · Paper page
  7. 10.1145/3395363.3397369 · DOI · Paper page
  8. 10.1145/3551349.3556955 · DOI · Paper page
  9. 10.1561/1500000019 · DOI · Paper page
  10. bareiß, 2022, Code generation tools (almost) for free? a study of few-shot, pre-trained language models on code, ArXiv Preprint
  11. ahmed, 2023, Improving few-shot prompts with relevant static analysis products, ArXiv Preprint

Cited by

See all 41

Metadata

Authors
Ahmed, T., & Devanbu, P.
Venue
2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE)
Publisher
IEEE
Published
2023-09-11
Pages
1742-1746
OpenAlex
W4388483039
Cited by
41
References
17
FWCI
10.93
Type
conference paper
Language
en
Open access
green
Funders
National Science Foundation (2107592)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Better Patching Using LLM Prompting, via Self-Consistency. https://app.cochat.ai/papers/10.1109/ase56229.2023.00065
The paper
Ahmed, T., & Devanbu, P. (2023). Better Patching Using LLM Prompting, via Self-Consistency. 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). https://doi.org/10.1109/ase56229.2023.00065

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.