Learning Performance-Improving Code Edits

Shypula, Alexander; Madaan, Aman; Zeng, Yimeng; Alon, Uri; Gardner, Jacob; Hashemi, Milad; Neubig, Graham; Ranganathan, Parthasarathy; Bastani, Osbert; Yazdanbakhsh, Amir

doi:10.48550/arxiv.2302.07867

PreprintCited by 32Software Engineering ResearchParallel Computing and Optimization TechniquesTeaching and Learning Programming

Learning Performance-Improving Code Edits

Open access

Shypula, Alexander, Madaan, Aman, Zeng, Yimeng, Alon, Uri, Gardner, Jacob, Hashemi, Milad, Neubig, Graham, Ranganathan, Parthasarathy +2 more·2023·arXiv

DOI 10.48550/arxiv.2302.07867arXiv 2302.07867

Open this paper in CoChat — free account Open PDF

Abstract

With the decline of Moore's law, optimizing program performance has become a major focus of software research. However, high-level optimizations such as API and algorithm changes remain elusive due to the difficulty of understanding the semantics of code. Simultaneously, pretrained large language models (LLMs) have demonstrated strong capabilities at solving a wide range of programming tasks. To that end, we introduce a framework for adapting LLMs to high-level program optimization. First, we curate a dataset of performance-improving edits made by human programmers of over 77,000 competitive C++ programming submission pairs, accompanied by extensive unit tests. A major challenge is the significant variability of measuring performance on commodity hardware, which can lead to spurious "improvements." To isolate and reliably evaluate the impact of program optimizations, we design an environment based on the gem5 full system simulator, the de facto simulator used in academia and industry. Next, we propose a broad range of adaptation strategies for code optimization; for prompting, these include retrieval-based few-shot prompting and chain-of-thought, and for finetuning, these include performance-conditioned generation and synthetic data augmentation based on self-play. A combination of these techniques achieves a mean speedup of 6.86 with eight generations, higher than average optimizations from individual programmers (3.66). Using our model's fastest generations, we set a new upper limit on the fastest speedup possible for our dataset at 9.64 compared to using the fastest human submissions available (9.56).

AI snapshot

Written from the paper’s full text.

TL;DR

The paper introduces PIE, a benchmark of over 77K C++ competitive-programming submission pairs with gem5-measured execution times, and adapts LLMs to generate performance-improving code edits. Its best model, GPT-3.5 fine-tuned with self-play synthetic data, achieves 6.86× mean speedup with eight generations, exceeding the 3.66× average human speedup and setting a 9.64× upper limit versus 9.56× for fastest human submissions.

Key findings

PIE contains 77,967 training pairs from 1,474 problems, 2,544 validation pairs from 77 problems, and 978 test pairs from 41 problems, with over 42.8 million gem5 simulations used to relabel execution times.
GPT-3.5 augmented with synthetic data from self-play achieved 87.63% %OPT, 6.86× SPEEDUP, and 95.09% %Correct for Best@8, compared with the human reference average speedup of 3.66×.
Fine-tuned performance-conditioned CodeLlama 13B achieved 66.56% %OPT, 5.65× SPEEDUP, and 70.96% %Correct for Best@8.
Using the model’s fastest generations, the paper reports a new upper limit of 9.64× speedup on the dataset, slightly above the fastest human submissions at 9.56×.

Methods

The authors curated PIE from CodeNet C++ competitive-programming submissions, tracking chronological accepted submissions and keeping pairs with more than 10% relative time improvement; execution time was measured deterministically with the gem5 full-system simulator using an Intel Skylake configuration.
They evaluated prompting strategies (instruction prompting, few-shot prompting, chain-of-thought, and dynamic retrieval-based few-shot prompting using CodeBertScore and FAISS) and fine-tuning strategies (standard fine-tuning, high-quality subset, performance-conditioned generation, and synthetic data from self-play) on CodeLlama 7B/13B/34B and OpenAI GPT-3.5/GPT-4.
Metrics were %OPT, SPEEDUP, and %Correct, with BEST@k sampling at temperature 0.7; all C++ programs were compiled with GCC 9.3.0, C++17, and -O3.

Limitations

The paper states that its gem5 experiments use a two-minute timeout, which may introduce slight variability, and that altering this timeout could change the results.
The paper states that test cases may not perfectly cover all cases; in a manual analysis of 120 pairs, 10 speedups came from reducing array allocation constants and may not reflect genuine algorithmic improvements.
The paper reports a precision-recall style trade-off: self-play improved speedup from 6.74 to 6.86 and %OPT from 86.71 to 87.63 but slightly dropped correctness, and performance-conditioning may restrict correct programs (mixing 10/10, 9/10, and 8/10 tags increased %Correct from 59.95% to 64.36%). Evident from the text, the benchmark is limited to C++ competitive programming and a gem5 Skylake model, so generalization to other languages, domains, or hardware is not established.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2302.07867

Cites

85 of 85 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

3 of 15 cited sources checked · 3 supported · 0 flagged

Findingcitation
The PIE dataset is constructed from performance-improving edits made by human programmers in competitive programming tasks from CodeNet.
“Our dataset is constructed based on performance-improving edits (PIE) made by human programmers in a range of competitive programming tasks from CodeNet (Puri et al., 2021).”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites Puri et al., 2021 source
Findingown data
The average individual human sampled in the test set achieved an average speedup of 3.66×.
“the average individual human sampled in our test set achieved an average speedup of 3.66×.”
Quote in the paperSubstantial claim
Findingown data
Without data-driven methods that leverage PIE, the strongest baseline COT only warrants a 1.60× average speedup over 8 submissions, compared with the 3.66× human reference.
“Without data-driven methods that leverage PIE, our strongest baseline COT only warrants a 1.60× average speedup over 8 submissions vs. the 3.66× human reference.”
Quote in the paperSubstantial claim
Findingown data
A fine-tuned performance-conditioned version of CODELLAMA 13B can achieve an average speedup of 5.65× on the test set when taking 8 samples and filtering for correctness and execution time.
“When allowing a model to take 8 samples and filtering for correctness and execution time, our fine-tuned performance-conditioned version of CODELLAMA 13B can achieve an average speedup of 5.65× on our test set”
- Also stated in: § 4.3 DISCUSSION AND KEY TAKEAWAYS · plain
  “CODELLAMA 13B with performance-conditioned generation (66.56% %OPT, 5.65× SPEEDUP for BEST@8)”
Quote in the paperSubstantial claim
Findingown data
Benchmarking on real hardware can lead to large, phantom performance improvements due only to random chance.
“Indeed, we find that benchmarking on real hardware can lead to large, phantom performance “improvements” due only to random chance.”
- Also stated in: § 2 PERFORMANCE IMPROVING EDITS (PIE) DATASET · plain
  “If benchmarking is not performed carefully, it is easy to mistakenly over-report program optimization results.”
Quote in the paperSubstantial claim

Findingown data
A fine-tuned version of GPT-3.5 augmented with synthetic data via self-play achieves an average speedup of 6.86×.
“a fine-tuned version of GPT-3.5 augmented with synthetic data via self-play achieves an average speedup of 6.86×”
- Also stated in: § 1 INTRODUCTION · plain
  “our best model, GPT-3.5 augmented with synthetic data obtained from self-play, achieves an average speedup of 6.86×”
Quote in the paperSubstantial claim
Findingown data
Pre-trained code LLMs are limited in their ability to optimize code without a dataset like PIE.
“Our results indicate that pre-trained code LLMs are limited in their ability to optimize code without a dataset like PIE.”
- Also stated in: § 1 INTRODUCTION · plain
  “We find that LLMs are limited in the challenging task of code optimization.”
Quote in the paperSubstantial claim
Findingown data
The best model, GPT-3.5 augmented with synthetic data from self-play, optimizes 87.63% of the test set by at least 10%.
“Overall, our best model, GPT-3.5 augmented with synthetic data obtained from self-play, achieves an average speedup of 6.86×, and optimizes 87.63% of the test set by at least 10%.”
- Also stated in: § 4.3 DISCUSSION AND KEY TAKEAWAYS · plain
  “GPT-3.5 with synthetic data (87.63%%OPT, 6.86× SPEEDUP for BEST@8)”
Quote in the paperSubstantial claim
Findingown data
Synthetic data from self-play marginally improves generalization.
“Synthetic data from self-play marginally improves generalization.”
Quote in the paperSubstantial claim
Findingown data
The transformations in the sampled 120 (source, optimized) program pairs can be broadly categorized into four kinds: Algorithmic changes, Input/Output operations, Data Structure modifications, and Miscellaneous adjustments.
“We find that the transformations can be broadly categorized into four kinds: Algorithmic changes, Input/Output operations (IO), Data Structure modifications, and Miscellaneous adjustments.”
Quote in the paperNot a substantial claim
Findingnone given
Submissions in PIE are not uniformly distributed across problems.
“Furthermore, submissions are not uniformly distributed across problems.”
Quote in the paperSubstantial claim
Findingcitation
Fine-tuning on the smaller, high-quality dataset yields substantially stronger results than traditional fine-tuning on all models.
“We see substantially stronger results when fine-tuning on the smaller, high-quality dataset.”
Quote in the paperSubstantial claim
Findingown data
Few-shot prompts often yield similar results compared to simple instruction-prompting.
“We find that few-shot prompts often yield similar results compared to simple instruction-prompting.”
Quote in the paperSubstantial claim
Findingcitation
Additional test cases from AlphaCode were generated with a fine-tuned LLM.
“To improve coverage, we include additional test cases from AlphaCode (Li et al., 2022) generated with a fine-tuned LLM.”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites Li et al., 2022
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Findingown data
The CodeNet dataset's CPU time information is inconsistent.
“The CodeNet dataset includes CPU time, but we found the information to be inconsistent (see Appendix A.9).”
Quote in the paperSubstantial claim
Findingown data
Algorithmic changes are the most common transformation category, comprising approximately 34.15% of changes.
“Algorithmic changes (complex modifications, such as changing recursive methods to dynamic programming, and unexpected ones, such as omitting Binary Indexed Trees for simpler constructs) are most common, comprising ~34.15% of changes;”
Quote in the paperNot a substantial claim
Findingown data
As more samples are drawn, the performance of HQ and HQ + Self-play gradually converge to a similar level.
“However, as more samples are drawn, the performance of HQ and HQ + Self-play gradually converge to a similar level of performance.”
Quote in the paperNot a substantial claim
Findingown data
Data Structure modifications comprised approximately 21.14% of changes.
“Data Structures (e.g., switching from vectors to arrays) comprised ~21.14%,”
Quote in the paperNot a substantial claim
Findingown data
Fine-tuning significantly outperforms prompting for models of the same size.
“Our results demonstrate that while prompting can be an effective way to adapt models (with retrieval), fine-tuning significantly outperforms prompting for models of the same size.”
Quote in the paperSubstantial claim
Findingown data
Fine-tuning GPT-3.5 on the top 5,793 examples from PIE with a maximum of 8 duplicates, instead of the 5,570 pairs that included synthetic programs, saw BEST@1 performance degrade.
“For GPT-3.5, to be sure the increases came from the type of data and not the quantity of data, we performed an ablation by fine-tuning on the top 5,793 examples from PIE with a maximum of 8 duplicates (instead of the 5,570 pairs that included synthetic programs), and we saw BEST@1 performance degrade”
Quote in the paperSubstantial claim
Findingown data
When prompted with instructions alone, both GPT-3.5 and CODELLAMA 34B demonstrated better %OPT and SPEEDUP metrics than few-shot prompting.
“For instance, when prompted with instructions alone, both GPT-3.5 and CODEL-LAMA 34B demonstrated better %OPT and SPEEDUP metrics.”
Quote in the paperSubstantial claim
Findingown data
In 10 out of the 120 cases examined, the speedup stemmed from reducing the constants used to allocate arrays.
“Specifically, in 10 out of the 120 cases we examined, the speedup stemmed from reducing the constants used to allocate arrays.”
Quote in the paperNot a substantial claim
Findingown data
When the model gets the program wrong, it most often misses most test cases.
“Additionally, with test cases, we find that when the model gets the program wrong, it seems to most often get it quite wrong by missing most test cases.”
Quote in the paperNot a substantial claim
Findingown data
Performance-conditioned generation improved CODELLAMA 13B from 47.75% %OPT, 3.43× SPEEDUP to 66.56% %OPT, 5.65× SPEEDUP for BEST@8.
“it improved the performance of CODEL-LAMA 13B from 47.75% %OPT, 3.43× SPEEDUP to 66.56% %OPT, 5.65× SPEEDUP for BEST@8.”
Quote in the paperSubstantial claim
Findingown data
Performance-conditioned training outperforms fine-tuning.
“Performance-conditioned training outperforms fine-tuning.”
Quote in the paperSubstantial claim
Findingown data
Fine-tuning with PIE substantially improves all models.
“Fine-tuning with PIE substantially improves all models.”
Quote in the paperSubstantial claim
Findingown data
Comparing 6 generations from the 7B performance-conditioned model conditioned on 10/10 versus combining first 2 generations conditioned on 10/10, 9/10, and 8/10 increased %Correct from 59.95% to 64.36%.
“we experimented using the first 6 generations from the 7b Performance-Conditioned model when conditioned on 10/10 versus combining the first 2 generations when conditioned on 10/10, 9/10, and 8/10 (i.e.comparing 6 total generations from one strategy vs. 6 total generations across difference strategies).When we did this, we saw a %Correct increase from 59.95% to 64.36%.”
Quote in the paperSubstantial claim
Findingown data
The gap between GPT-3.5 fine-tuned on HQ data and HQ + Self-Play seems to diminish with more generations.
“The data in Section 4.2 shows that the gap between GPT-3.5 fine-tuned on HQ data and HQ + Self-Play seems to diminish with more generations.”
Quote in the paperNot a substantial claim
Findingown data
Input/Output operations comprised approximately 26.02% of changes.
“Input/Output operations (e.g., changing 'cin/cout' to 'scanf/printf', efficiently reading strings) comprised ~26.02%;”
Quote in the paperNot a substantial claim
Interpretationown data
Data-driven methods using PIE, such as retrieval-based few-shot prompting and fine-tuning, are highly effective at achieving strong optimization abilities in LLMs.
“We find that data-driven methods using PIE, like retrieval-based few-shot prompting and fine-tuning, are highly effective at achieving strong optimization abilities in LLMs.”
Quote in the paperSubstantial claim
Interpretationown data
Conditioning the model only to generate programs with a 10/10 optimization rate may constrain the number of optimizations available for any given input.
“We believe that conditioning the model only to generate programs with a 10/10 optimization rate may constrain the number of optimizations available for any given input.”
Quote in the paperSubstantial claim
Interpretationown data
The PIE training dataset targets just 1,474 problems, which may limit the learned model's ability to generalize to new programs.
“While we have tens of thousands of slow-fast pairs in the PIE training dataset, these submissions target just 1,474 problems, which may limit the learned model's ability to generalize to new programs.”
Quote in the paperSubstantial claim
Interpretationown data
PIE effectively adapts LLMs for program optimization in few-shot settings.
“showing that PIE effectively adapts LLMs for program optimization in few-shot settings.”
Quote in the paperSubstantial claim
Premisecitation
The performance of PEFT may be heavily task-dependent.
“Together, these works indicate that the performance of PEFT may be heavily task-dependent.”
Quote in the paperSubstantial claim0 of 2 sources checked
- cites He et al. (2021)
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Niederfahrenhorst et al. (2023)
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Interpretationown data
The vast majority of speedups do not suffer from the issue of spurious speedups from reduced array allocation constants.
“Nevertheless, our analysis shows that the vast majority of speedups do not suffer from this issue, supporting our strong empirical results.”
Quote in the paperSubstantial claim
Interpretationreasoning
Training with HQ data helps increase the model's coverage, allowing it to optimize a large number of programs with just a single greedy sample.
“Training with HQ data helps increase the model's coverage, allowing it to optimize a large number of programs with just a single greedy sample.”
Quote in the paperNot a substantial claim
Interpretationown data
Performance labels may restrict the set of generated programs that are correct.
“These results support the explanation that performance labels may restrict the set of generated programs that are correct.”
Quote in the paperSubstantial claim
Interpretationown data
Openly available models such as CODELLAMA can be competitive with GPT-3.5.
“Our results demonstrate that openly available models such as CODELLAMA can be competitive with GPT-3.5.”
Quote in the paperSubstantial claim

Classified as probably not a substantial claim. 15
May combine two claims. 33
The quote only partly states this claim. 2

Classified as probably not a substantial claim.
Premisecitation
Pretrained large language models have demonstrated strong capabilities at solving a wide range of programming tasks.
“Simultaneously, pretrained large language models (LLMs) have demonstrated strong capabilities at solving a wide range of programming tasks.”
- Also stated in: § 1 INTRODUCTION · boosted
  “Recently, pre-trained large language models (LLMs) have demonstrated impressive performance at a wide range of programming tasks (Chen et al., 2021b;Fried et al., 2022;Xu et al., 2022;Nijkamp et al., 2022).”
Quote in the paperSubstantial claim1 of 3 sources checked
- cites Chen et al., 2021b
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Fried et al., 2022 source
- cites Xu et al., 2022
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
Classified as probably not a substantial claim.
Premisecitation
Retrieval-based mechanisms can improve language models for various tasks requiring factual or procedural knowledge.
“Recent work has demonstrated that retrieval-based mechanisms can improve language models for various tasks requiring factual or procedural knowledge (Liu et al., 2021;Poesia et al., 2021;Rubin et al., 2022;Madaan et al., 2022;Shrivastava et al., 2023).”
Quote in the paperSubstantial claim2 of 3 sources checked
- cites Liu et al., 2021 source
- cites Poesia et al., 2021
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Rubin et al., 2022 source
May combine two claims.
Findingown data
Benchmarking 500 identical program pairs with HYPERFINE yielded a mean speedup of 1.12×, standard deviation 0.36, and top 5% speedup 1.91×.
“We benchmarked 500 programs “pairs” where the “slow” and “fast” programs are identical.Ideally, we should have source time target time = 1 (i.e., the two programs have identical performance).However, we observed a mean speedup of 1.12×, with a standard deviation of 0.36, and the top 5% of pairs exhibited a speedup of 1.91×.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
Aggregating over all human submissions in the test set with a higher sampling budget, GPT-3.5 achieved a speedup of 9.64× and surpassed the best human submissions across all programmers with a speedup of 9.56×.
“Aggregating over all human submissions in the test set with a higher sampling budget, GPT-3.5 achieved a speedup of 9.64× and surpassed the best human submissions across all programmers with a speedup of 9.56×.”
- Also stated in: § 4.2 RESULTS FOR FINETUNING · plain
  “For the PIE dataset, the best human speedup achieved was 9.56×, while our model achieved a slightly higher speedup of 9.64×, setting a new upper limit.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
PIE is a dataset of over 77K competitive C++ programming submission pairs with execution time annotations from gem5 and extensive unit tests.
“First, we curate a dataset of performance-improving edits made by human programmers of over 77 K competitive C++ programming submission pairs, accompanied by extensive unit tests.”
- Also stated in: § 1 INTRODUCTION · plain
  “We introduce a new code dataset of more than 77 K C++ program pairs, named PIE, with execution time annotations collected from the gem5 simulator.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
Error analysis of GPT-3.5 fine-tuned with self-play shows about 60% of failures break a unit test, about 30% produce correct but slower or below-threshold programs, and about 10% have syntax errors.
“Table 5 shows that a large fraction ∼60% of the failures happen because the proposed changes break a unit test.In about 30% of the cases, the model produces a correct, but the generated program is either slower (10%) or doesn't meet our threshold for speedup (10%).Finally, in about 10% of the cases, the generated program has a syntax error.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
There is a mild negative correlation between problem description length and average accuracy (-0.15) and between source program length and average accuracy (-0.26).
“The results show a mild negative correlation between the problem description length and average accuracy (-0.15) and between the source program length and average accuracy (-0.26), suggesting longer inputs slightly reduce accuracy.”
Quote in the paperSubstantial claim
The quote only partly states this claim.
Findingown data
Dynamic retrieval-based few-shot prompting with K=4 retrieved prompts outperforms all baseline variants.
“Table 2 (bottom) shows results using our dynamic retrieval-based few-shot prompting strategy, with a preferable setting at K = 4 retrieved prompts.”
- Also stated in: § 4.1 RESULTS FOR FEW-SHOT PROMPTING · plain
  “The results show that dynamic few-shot prompting outperforms all the baseline variants”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
For CODELLAMA and GPT-3.5, adding synthetic data improves %OPT and often SPEEDUP, particularly with BEST@1.
“For CODELLAMA and GPT-3.5, compared to using no synthetic data, the additional data improves both %OPT and often SPEEDUP, particularly with BEST@1.”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
For problem number p03160, between submissions s766827701 and s964782197, a speedup of 2.44× is reported despite the programs and environments being identical.
“For problem number p03160 and between submission s766827701 and s964782197 a speedup of 2.44× is reported, despite the programs and environments being identical.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
In the prompt (private) regime, GPT-3.5 with CoT achieved 43.05% %Opt, 1.60× speedup, and 91.72% %Correct.
“| Prompt (P) | GPT-3.5, CoT | 43.05% | 1.60× | 91.72% |”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
In the prompt (open-access) regime, CODELLAMA 34B with CoT achieved 19.63% %Opt, 1.30× speedup, and 78.73% %Correct.
“| Prompt (O) | CODELLAMA 34B, CoT | 19.63% | 1.30× | 78.73% |”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Findingown data
The PIE dataset was relabeled using gem5, requiring over 42.8 million simulations.
“Thus, we relabel the execution time using gem5 as described below; to create these annotated runtimes, we performed over 42.8 million simulations in our gem5 environment.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
In the retrieval (open-access) regime, CODELLAMA 34B achieved 42.54% %Opt, 2.43× speedup, and 73.62% %Correct.
“| Retrieval (O) | CODELLAMA 34B | 42.54% | 2.43× | 73.62% |”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
In the fine-tuning (private) regime, GPT-3.5 with Self-Play achieved 87.63% %Opt, 6.86× speedup, and 95.09% %Correct.
“| FineTune (P) | GPT-3.5, Self-Play | 87.63% | 6.86× | 95.09% |”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
Miscellaneous optimizations comprised approximately 18.70% of changes.
“and Miscellaneous (e.g., code cleanups and constant optimizations) comprised ~18.70%.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
The generation and equivalence checking required executing over 1.4 million binary input pairs.
“In total, this required executing over 1.4 million binary input pairs.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
The high-quality subset contains 4,085 slow-fast pairs, with no more than 4 submission pairs per problem, averaging 2.77 pairs per problem.
“we additionally fine-tune with a subset of 4,085 “high-quality” slow-fast pairs-in particular, we take examples with the highest speedup and disallow more than 4 submission pairs per problem, for an average of 2.77 pairs per problem.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
GPT-4 outperforms GPT-3.5 Best@1 and under-performs GPT-3.5 Best@8.
“we note that GPT-4 outperforms GPT-3.5 Best@1 and under-performs GPT-3.5 Best@8”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
Synthetic data generation yielded 3,314 unique synthetic programs and many thousand more duplicates.
“we obtain 3,314 unique synthetic programs and many thousand more duplicates.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
Average speedup has a mild negative correlation with problem description length (-0.16) and source program length (-0.11).
“Additionally, the average speedup has a mild negative correlation with both the problem description length (-0.16) and the source program length (-0.11), indicating a minimal impact of length on speedup compared to correctness.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
COT prompting improves performance over instruction-tuned and fixed prompt setups, but only for the larger CODELLAMA (13B and 34B) and GPT-3.5 models.
“we observe improvements with this approach over both instructiontuned and fixed prompt setups, but notably only for the larger CODELLAMA (13B and 34B) and GPT-3.5 models.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
The human reference solutions achieved 100.00% %Opt, 3.66× speedup, and 100.00% %Correct.
“| Human reference | | 100.00% | 3.66× | 100.00% |”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
After attempting to generate 10,000 new programs through the prompting strategy, 6,553 were not in the training/validation/test set of PIE.
“We found that after attempting to generate 10,000 new programs through the prompting strategy, 6,553 were not in the training/validation/test set of PIE.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
Dynamic retrieval provides enormous gains over all other prompting approaches; it improved CODELLAMA 34B from 19.63% %OPT, 1.30× SPEEDUP to 34.25% %OPT, 2.28× SPEEDUP for BEST@8.
“Our results show that dynamic retrieval provides enormous gains over all other prompting approaches; for instance, it improved the performance of CODELLAMA 34B from 19.63 %OPT, 1.30× SPEEDUP to 34.25% %OPT, 2.28× SPEEDUP for BEST@8.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
After excluding test cases that would cause timeouts above 2 minutes in gem5, the median number of test cases per problem is 82.5 in training, 75 in validation, and 104 in test.
“after excluding them, we obtain a median of 82.5 test cases per problem in our training set, 75 test cases per problem in our validation set, and 104 test cases per problem for our test set (Appendix A.5).”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
After filtering, 1,485 optimized synthetic examples were kept where the optimized program is at least 5× faster and semantic duplicates limited to three.
“Finally, we keep pairs where the optimized program is at least 5× faster and limit semantic duplicates to three, resulting in 1,485 optimized synthetic examples.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
After training with Self-Play, there is a slight drop in correctness, but speedup and correctness increase from 6.74 to 6.86 and from 86.71 to 87.63.
“Additionally, there is a slight drop in correctness after training with Self-Play; however, the speedup and correctness increase from 6.74 → 6.86 and 86.71 → 87.63.”
Quote in the paperNot a substantial claim
The quote only partly states this claim.
Findingreasoning
Both fine-tuned CODELLAMA models (7B and 13B) show significant improvements in %OPT and SPEEDUP with performance-conditioned generation.
“Both fine-tuned CODELLAMA models (7B and 13B) show significant improvements in %OPT and SPEEDUP.”
Quote in the paperSubstantial claim
May combine two claims.
Findingown data
LoRA fine-tuning of CODELLAMA 7B on All data achieved 1.12% Best@1 %Opt, 1.01× speedup, and 45.82% %Correct.
“| All | CODELLAMA 7B | 1.12% | 1.01× | 45.82% | 9.57% | 1.17× | 87.47% |”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
The PIE dataset splits contain 77,967 training pairs from 1,474 problems, 2,544 validation pairs from 77 problems, and 978 test pairs from 41 problems.
“We obtain a training set of 77,967 pairs from 1,474 problems, a validation set of 2,544 pairs from 77 problems, and a test set of 978 pairs from 41 problems.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
In the fine-tuning (open-access) regime, CODELLAMA 13B with Perf-Cond achieved 66.56% %Opt, 5.65× speedup, and 70.96% %Correct.
“| FineTune (O) | CODELLAMA 13B, Perf-Cond | 66.56% | 5.65× | 70.96% |”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).
Findingnone given
Increased speedup from dynamic retrieval may come with some cost of correctness.
“We note that increased speedup may, however, come with some cost of correctness.”
Quote in the paperNot a substantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown data
Of the 6,553 generated programs not in PIE, 3,314 equivalence sets were found.
“of these 6,553 generations we found 3,314 equivalence sets.”
Quote in the paperNot a substantial claim
May combine two claims.
Findingown data
In the retrieval (private) regime, GPT-4 achieved 76.07% %Opt, 3.93× speedup, and 95.71% %Correct.
“| Retrieval (P) | GPT-4 | 76.07% | 3.93× | 95.71% |”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation
The gem5 full system simulator provides detailed microarchitectural emulation of modern processors and is the gold standard CPU simulator in academia and industry.
“we evaluate performance using the gem5 CPU simulator (Binkert et al., 2011), the gold standard CPU simulator in academia and industry, and models state-of-the-art general-purpose processors.”
- Also stated in: § 2 PERFORMANCE IMPROVING EDITS (PIE) DATASET · plain
  “we measure program performance using the gem5 (Binkert et al., 2011) full system simulator, which provides detailed microarchitectural emulation of modern processors.”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites Binkert et al., 2011 source
May combine two claims.
Study design and conductreasoning
Performance measurement using the gem5 full system simulator is deterministic and provides reproducible, denoised performance results.
“This evaluation strategy is entirely deterministic, ensuring both reliability and reproducibility.”
- Also stated in: § 2 PERFORMANCE IMPROVING EDITS (PIE) DATASET · plain
  “Executing deterministic programs in gem5 provides deterministic performance results.This enables reproducibility in research and denoised performance measurements.2”
Quote in the paperSubstantial claim0 of 1 sources checked
- cites Binkert et al., 2011 source
May combine two claims.
Interpretationreasoning
PIE enables reproducible evaluation of LLMs for program optimization and reliable performance annotations for training.
“PIE enables reproducible evaluation of LLMs for program optimization and reliable performance annotations for training.”
Quote in the paperSubstantial claim
Classified as probably not a substantial claim.
Premisecitation
The effectiveness of parameter-efficient methods depends on the training data.
“Recent work has shown that the effectiveness of parameter-efficient methods depends on the training data.”
Quote in the paperSubstantial claim0 of 2 sources checked
- cites He et al. (2021)
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- cites Niederfahrenhorst et al. (2023)
  Couldn’t be matched to a paper in Crossref, so it can’t be checked.
May combine two claims.
Interpretationown data
Language models struggle to generate a correct program when faced with larger source programs and challenging problems, but their ability to optimize programs is minimally impacted.
“Overall, this analysis reveals that language models struggle to generate a correct program when faced with larger source programs and challenging problems, but their ability to optimize programs is minimally impacted.”
Quote in the paperSubstantial claim
May combine two claims.
Interpretationreasoning
Program optimization is a non-trivial task requiring knowledge of algorithms, data structures, and programming grounded in performance; retrieving highly relevant examples may improve an LLM's optimization ability.
“Program optimization is a non-trivial task requiring knowledge of algorithms, data structures, and programming grounded in performance; thus, retrieving highly relevant examples may improve an LLM's optimization ability.”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone given
The experiments use a two-minute timeout, which may introduce slight variability.
“Our experiments use a two-minute timeout, which may introduce slight variability.”
Quote in the paperNot a substantial claim
- The authors state this as an assumption: gem5 terminates.
May combine two claims.
Interpretationown data
The model trained on synthetic data learns to try novel optimization strategies, but that comes at the cost of making more mistakes, revealing a precision-recall style trade-off.
“This reveals a precision-recall style trade-off: the model trained on synthetic data learns to try novel optimization strategies, but that comes at the cost of making more mistakes.”
Quote in the paperNot a substantial claim
Classified as probably not a substantial claim. May combine two claims.
Study design and conductreasoning
All C++ programs were compiled with GCC version 9.3.0 and C++17 with -O3 optimization flag, so reported improvements are on top of the optimizing compiler.
“We compile all C++ programs with GCC version 9.3.0 and C++17 as well as the -O3 optimization flag; therefore, any reported improvements would be those on top of the optimizing compiler.”
Quote in the paperSubstantial claim
Probably not a substantial claim: may not pass S3 (load-bearing).
Premisecitation
CodeNet includes an average of 4 test cases per problem.
“CodeNet includes an average of 4 test cases per problem.”
Quote in the paperNot a substantial claim0 of 1 sources checked
- cites Puri et al., 2021 source
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductown data
The error analysis was performed on a randomly sampled set of 120 (source, optimized) program pairs.
“We manually analyze a randomly sampled set of 120 (source, optimized) program pairs to understand the algorithmic and structural changes responsible for the performance gains.”
Quote in the paperNot a substantial claim
May combine two claims.
Interpretationown data
The LLM is capable of performing sophisticated optimizations while preserving functionality.
“These findings show the LLM's capability to perform sophisticated optimizations while preserving functionality.”
Quote in the paperSubstantial claim

47 of 47

The paper relies on this without support.
High-level optimizations such as API and algorithm changes remain elusive due to the difficulty of understanding the semantics of code.
“However, high-level optimizations such as API and algorithm changes remain elusive due to the difficulty of understanding the semantics of code.”
The paper relies on this without support.
Lack of open datasets and reliable performance measurement techniques has stymied research in automating performance optimization.
“However, these techniques are either narrow or difficult to build on due to the lack of open datasets and lack of reliable performance measurement techniques, which has stymied research in this direction.”
The paper relies on this without support.
Programs can typically be written in many ways with different performance profiles.
“Programs can typically be written in many ways with different performance profiles.”
The authors state this as an assumption.
gem5 terminates.
“This assumes gem5 terminates.”

4 of 4

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

Two different Best@8 retrieval results for CODELLAMA 34B
Major
The discussion gives CODELLAMA 34B with dynamic retrieval a Best@8 result of 42.54% optimized and 2.43× speedup, but the later takeaway gives 34.25% and 2.28× for the same model, method, and sampling budget.
Failure categories do not account for the stated 30%
Minor
The two categories given for correct but insufficiently optimized programs are 10% slower and 10% below the speedup threshold. They add to 20%, not the stated approximately 30%.
Reported 30 · recomputed 20
Optimization percentage is described as correctness
Minor
The text calls the increase from 86.71 to 87.63 an increase in correctness. Table 1 instead identifies 87.63% as %Opt and reports 95.09% as %Correct for the self-play model.

3 of 3

Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1× · see citing works
“Next, we use few-shot prompting (Brown et al., 2020).”
Decoupled Weight Decay Regularization
Loshchilov, I., & Hutter, F. (2017) · arXiv · doi:10.48550/arxiv.1711.05101 · Cited by 12,522 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
(Preprint)
Buffel, C., van Aalst, J., Bangels, A., Toelen, J., Allegaert, K., Verschueren, S., & Stichele, G. V. (2018) · doi:10.2196/preprints.12431 · Cited by 5,871
The gem5 simulator
Binkert, N., Beckmann, B., Black, G., Reinhardt, S. K., Saidi, A., Basu, A., Hestness, J., Hower, D. R., Krishna, T., Sardashti, S., et al. (2011) · ACM SIGARCH Computer Architecture News · doi:10.1145/2024716.2024718 · Cited by 4,930 · cited 2× · see citing works
“To address this challenge, we measure program performance using the gem5 (Binkert et al., 2011) full system simulator, which provides detailed microarchitectural emulation of modern processors.”
Method
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · cited 2× · see citing works
“First, we consider baseline prompting approaches, including techniques such as chain-of-thought (Wei et al., 2022b) (CoT).”
BackgroundMethod
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228 · cited 1× · see citing works
“We use FAISS (Johnson et al., 2019) to retrieve K closest programs from the training set; and to construct a “slow 1 → fast 1 || ...” style prompt on the fly (See Figure 15 in Appendix A.11).”
Method
LoRA: Low-Rank Adaptation of Large Language Models
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2106.09685 · Cited by 2,523 · cited 1× · see citing works
“Our results pave an exciting path for improving computing efficiency post Moore’s law.”
Background
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
A survey of software refactoring
Mens, T., & Tourwé, T. (2004) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2004.1265817 · Cited by 1,367 · cited 1× · see citing works
“Machine learning has been applied to improve performance by refactoring code (Mens & Tourwé, 2004; Ag-nihotri & Chug, 2020), identify compiler transformations (Bacon et al., 1994; Talaashrafi, 2022; Mammadli et al., 2021), perform parameter search (Hamadi, 2013; Huang et al., 2019; Kaufman et al.,…”
Method
Emergent Abilities of Large Language Models
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022) · arXiv · doi:10.48550/arxiv.2206.07682 · Cited by 1,264 · cited 1× · see citing works
“Finally, in line with the findings of Wei et al. (2022a) that identified C O T prompting as an emergent capability, we observe improvements with this approach over both instruction-tuned and fixed prompt setups, but notably only for the larger CODELLAMA (13B and 34B) and GPT -3.5 models.”
ResultInfluential
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 4× · see citing works
“To improve coverage, we include additional test cases from AlphaCode (Li et al., 2022) generated with a fine-tuned LLM.”
BackgroundMethodInfluential
The Flan Collection: Designing Data and Methods for Effective Instruction TuningPreprint
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. (2023) · International Conference on Machine Learning · doi:10.48550/arxiv.2301.13688 · Cited by 1,029 · cited 1× · see citing works
“We use prompts instructing the LLM to improve the performance of the given program, an approach commonly referred to as instruction prompting (Mishra et al., 2021; Gupta et al., 2022; Longpre et al., 2023); details on the prompt are in Figure 12 in Appendix A.11.”
Method
Compiler transformations for high-performance computing
Bacon, D. F., Graham, S. L., & Sharp, O. J. (1994) · ACM Computing Surveys · doi:10.1145/197405.197406 · Cited by 824 · cited 1× · see citing works
“…has been applied to improve performance by refactoring code (Mens & Tourwé, 2004; Ag-nihotri & Chug, 2020), identify compiler transformations (Bacon et al., 1994; Talaashrafi, 2022; Mammadli et al., 2021), perform parameter search (Hamadi, 2013; Huang et al., 2019; Kaufman et al., 2021;…”
Method
DeepSpeed: System Optimizations Enable Training Deep Learning Models with Over 100 Billion Parameters
Rasley, J., Rajbhandari, S., Ruwase, O., & He, Y. (2020) · Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining · doi:10.1145/3394486.3406703 · Cited by 823 · see citing works
Decision Transformer: Reinforcement Learning via Sequence ModelingPreprint
Chen, L.-L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., & Mordatch, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.01345 · Cited by 677 · cited 1× · see citing works
“Inspired by recent prompting strategies (Zhang et al., 2023) and offline-rl (Chen et al., 2021a), we introduce performance tags during training by associating each “fast” program with a tag indicating the optimal achievable performance across all solutions in the dataset.”
Method
DeepCoder: Learning to Write ProgramsPreprint
Balog, M., Gaunt, A. L., Brockschmidt, M., Nowozin, S., & Tarlow, D. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01989 · Cited by 641 · cited 2× · see citing works
“Recent work leverages machine learning to assist program synthesis (Balog et al., 2016; Bunel et al., 2018; Devlin et al., 2017a;b; Kalyan et al., 2018; Parisotto et al., 2016; Odena & Sutton, 2020; Nye et al., 2019; Meng et al., 2011; Jain et al., 2022; Austin et al., 2021; Xiong & Wang, 2022; Cambronero et al., 2023…”
Background
What Makes Good In-Context Examples for GPT-3?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2022) · Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures · doi:10.18653/v1/2022.deelio-1.10 · Cited by 563 · cited 2× · see citing works
“Next we consider a retrieval-based prompting approach where retrieval is used to select examples most similar to the current one (Liu et al., 2021; Poesia et al., 2021).”
MethodInfluential
A systematic evaluation of large language models of code
Xu, F. F., Alon, U., Neubig, G., & Hellendoorn, V. J. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534862 · Cited by 540 · cited 3× · see citing works
“In contrast, we focus on adapting pre-trained LLMs (Chen et al., 2021b; Nijkamp et al., 2022; Tunstall et al., 2022; Xu et al., 2022; Fried et al., 2022) to performance optimization.”
BackgroundMethodSelf-citation
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536 · cited 1× · see citing works
“, 2022), to clone detection, defect detection, and much more (Lu et al., 2021; Xu et al., 2022; Zhou et al., 2022; Austin et al., 2021).”
Background
Code Llama: Open Foundation Models for Code
Rozière, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y., Liu, J., Sauvestre, R., Remez, T., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.12950 · Cited by 531 · cited 1× · see citing works
“This methodology aligns with self-play and self-instruct approachs in neural program synthesis (Haluptzok et al., 2022; Rozière et al., 2023).”
Method
Towards a Unified View of Parameter-Efficient Transfer LearningPreprint
He, J., Zhou, C., Ma, X., Berg-Kirkpatrick, T., & Neubig, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04366 · Cited by 514 · see citing works
Self-citation
Calibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513 · cited 1× · see citing works
“This observation aligns with the findings of Zhao et al. (2021), which highlighted that few-shot examples can sometimes bias the model and lead to an incorrect understanding of the task.”
Result
There’s plenty of room at the Top: What will drive computer performance after Moore’s law?
Leiserson, C. E., Thompson, N., Emer, J., Kuszmaul, B. C., Lampson, B., Sánchez, D., & Schardl, T. B. (2020) · Science · doi:10.1126/science.aam9744 · Cited by 473 · cited 1× · see citing works
“…2022; Kumar et al., 2022), auto-vectorize code (Nuzman et al., 2006; Mendis et al., 2019), optimize GPU code (Liou et al., 2020; Cummins et al., 2021), and automatically select algorithms (Kotthoff, 2016; Kerschke et al., 2019). and room at the top (Leiserson et al., 2020; Sherry & Thompson, 2021).”
Method
Automated Algorithm Selection: Survey and Perspectives
Kerschke, P., Hoos, H. H., Neumann, F., & Trautmann, H. (2018) · Evolutionary Computation · doi:10.1162/evco_a_00242 · Cited by 440 · cited 1× · see citing works
“…2022; Kumar et al., 2022), auto-vectorize code (Nuzman et al., 2006; Mendis et al., 2019), optimize GPU code (Liou et al., 2020; Cummins et al., 2021), and automatically select algorithms (Kotthoff, 2016; Kerschke et al., 2019). and room at the top (Leiserson et al., 2020; Sherry & Thompson, 2021).”
Method
Program Synthesis with Large Language Models
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07732 · Cited by 388 · cited 3× · see citing works
“Recent work leverages machine learning to assist program synthesis (Balog et al., 2016; Bunel et al., 2018; Devlin et al., 2017a;b; Kalyan et al., 2018; Parisotto et al., 2016; Odena & Sutton, 2020; Nye et al., 2019; Meng et al., 2011; Jain et al., 2022; Austin et al., 2021; Xiong & Wang, 2022; Cambronero et al., 2023…”
Background
Learning To Retrieve Prompts for In-Context Learning
Rubin, O., Herzig, J., & Berant, J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.191 · Cited by 374 · cited 1× · see citing works
“Recent work has demonstrated that retrieval-based mechanisms can improve language models for various tasks requiring factual or procedural knowledge (Liu et al., 2021; Poesia et al., 2021; Rubin et al., 2022; Madaan et al., 2022; Shrivastava et al., 2023).”
Background
A Survey of Automatic Parameter Tuning Methods for Metaheuristics
Huang, C., Li, Y., & Yao, X. (2019) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2019.2921598 · Cited by 373 · cited 3× · see citing works
“, 1994; Talaashrafi, 2022), parameter search (Hamadi & H madi, 2013; Huang et al., 2019; Kaufman et al., 2021), auto-vectorization (Nuzman et al.”
BackgroundInfluential
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., & Xiong, C. (2022) · arXiv · doi:10.48550/arxiv.2203.13474 · Cited by 330 · cited 2× · see citing works
“In contrast, we focus on adapting pre-trained LLMs (Chen et al., 2021b; Nijkamp et al., 2022; Tunstall et al., 2022; Xu et al., 2022; Fried et al., 2022) to performance optimization.”
BackgroundMethod
GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
Black, S., Leo, G., Wang, P., Leahy, C., & Biderman, S. (2021) · Zenodo · doi:10.5281/zenodo.5297715 · Cited by 324 · see citing works
Reframing Instructional Prompts to GPTk’s LanguagePreprint
Mishra, S., Khashabi, D., Baral, C., Choi, Y., & Hajishirzi, H. (2021) · Findings · doi:10.48550/arxiv.2109.07830 · Cited by 287 · see citing works
Stochastic superoptimization
Schkufza, E., Sharma, R., & Aiken, A. (2013) · International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS) · doi:10.1145/2451116.2451150 · Cited by 279 · see citing works
LIMA: Less Is More for Alignment
Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., Ma, X., Efrat, A., Yu, P., Yu, L., et al. (2023) · arXiv · doi:10.48550/arxiv.2305.11206 · Cited by 275 · see citing works
Do Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023) · Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3576915.3623157 · Cited by 259 · see citing works
Auto-vectorization of interleaved data for SIMD
Nuzman, D., Rosen, I., & Zaks, A. (2006) · ACM-SIGPLAN Symposium on Programming Language Design and Implementation · doi:10.1145/1133981.1133997 · Cited by 239 · cited 1×
“…perform parameter search (Hamadi, 2013; Huang et al., 2019; Kaufman et al., 2021; Seshadri et al., 2022; Kumar et al., 2022), auto-vectorize code (Nuzman et al., 2006; Mendis et al., 2019), optimize GPU code (Liou et al., 2020; Cummins et al., 2021), and automatically select algorithms…”
Method
Genetic Improvement of Software: A Comprehensive Survey
Petke, J., Haraldsson, S. O., Harman, M., Langdon, W. B., White, D. R., & Woodward, J. R. (2017) · IEEE Transactions on Evolutionary Computation · doi:10.1109/tevc.2017.2693219 · Cited by 230 · cited 2× · see citing works
“Using machine learning algorithms and heuristics (Petke et al., 2017) for compiler optimization has been explored extensively in code refactoring (Mens & Tourwé, 2004; Agnihotri & Chug, 2020), compiler transformation (Bacon et al.”
Method
Neuro-Symbolic Program SynthesisPreprint
Parisotto, E., Mohamed, A., Singh, R., Li, L. C., Zhou, D., & Kohli, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01855 · Cited by 227 · cited 2× · see citing works
“Recent work leverages machine learning to assist program synthesis (Balog et al., 2016; Bunel et al., 2018; Devlin et al., 2017a;b; Kalyan et al., 2018; Parisotto et al., 2016; Odena & Sutton, 2020; Nye et al., 2019; Meng et al., 2011; Jain et al., 2022; Austin et al., 2021; Xiong & Wang, 2022; Cambronero et al., 2023…”
Background
Towards a holistic approach to auto-parallelization
Tournavitis, G., Wang, Z., Franke, B., & O’boyle, M. F. P. (2009) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/1542476.1542496 · Cited by 204 · see citing works
InCoder: A Generative Model for Code Infilling and Synthesis
Fried, D., Aghajanyan, A., Lin, J., Wang, S., Wallace, E., Shi, F., Zhong, R., Yih, W.-T., Zettlemoyer, L., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2204.05999 · Cited by 187 · see citing works
Faster sorting algorithms discovered using deep reinforcement learning
Mankowitz, D. J., Michi, A., Zhernov, A., Gelmi, M., Selvi, M., Paduraru, C., Leurent, E., Iqbal, S., Lespiau, J.-B., Ahern, A., et al. (2023) · Nature · doi:10.1038/s41586-023-06004-9 · Cited by 177 · cited 1× · see citing works
“Recent work has demonstrated the promise of deep learning for automating performance optimization (Garg et al., 2022; Mankowitz et al., 2023).”
Background
Algorithm Selection for Combinatorial Search Problems: A Survey
Kotthoff, L. (2014) · AI Magazine · doi:10.1609/aimag.v35i3.2460 · Cited by 173 · cited 3× · see citing works
“, 2021), automatic algorithm selection (Kotthoff, 2016; Kerschke t al., 2019), and room at the top (Leiserson t al.”
Method
Unit Test Case Generation with Transformers and Focal ContextPreprint
Tufano, M., Drain, D., Svyatkovskiy, A., Deng, S. K., & Sundaresan, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05617 · Cited by 158 · see citing works
What Makes Good In-Context Examples for GPT-$3$?
Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., & Chen, W. (2021) · arXiv · doi:10.48550/arxiv.2101.06804 · Cited by 148 · see citing works
Reframing Instructional Prompts to GPTk’s Language
Mishra, S. R., Khashabi, D., Baral, C. R., Choi, Y., & Hajishirzi, H. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.50 · Cited by 140 · cited 1× · see citing works
“We use prompts instructing the LLM to improve the performance of the given program, an approach commonly referred to as instruction prompting (Mishra et al., 2021; Gupta et al., 2022; Longpre et al., 2023); details on the prompt are in Figure 12 in Appendix A.11.”
Method
Untitled work
(2021) · doi:10.1145/3510003.3510203 · Cited by 139 · cited 2× · see citing works
“Recent work leverages machine learning to assist program synthesis (Balog et al., 2016; Bunel et al., 2018; Devlin et al., 2017a;b; Kalyan et al., 2018; Parisotto et al., 2016; Odena & Sutton, 2020; Nye et al., 2019; Meng et al., 2011; Jain et al., 2022; Austin et al., 2021; Xiong & Wang, 2022; Cambronero et al., 2023…”
Background
An Evaluation of Edge TPU Accelerators for Convolutional Neural Networks
Seshadri, K., Akin, B., Laudon, J., Narayanaswami, R., & Yazdanbakhsh, A. (2022) · 2022 IEEE International Symposium on Workload Characterization (IISWC) · doi:10.1109/iiswc55918.2022.00017 · Cited by 138 · cited 1× · see citing works
“…(Bacon et al., 1994; Talaashrafi, 2022; Mammadli et al., 2021), perform parameter search (Hamadi, 2013; Huang et al., 2019; Kaufman et al., 2021; Seshadri et al., 2022; Kumar et al., 2022), auto-vectorize code (Nuzman et al., 2006; Mendis et al., 2019), optimize GPU code (Liou et al., 2020;…”
BackgroundSelf-citation
RobustFill: Neural Program Learning under Noisy I/OPreprint
Devlin, J., Uesato, J., Bhupatiraju, S., Singh, R., Mohamed, A., & Kohli, P. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1703.07469 · Cited by 127 · see citing works
CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code
Zhou, S., Alon, U., Agarwal, S., & Neubig, G. (2023) · arXiv · doi:10.48550/arxiv.2302.05527 · Cited by 100 · cited 1× · see citing works
“Inspired by recent prompting strategies (Zhang et al., 2023) and offline-rl (Chen et al., 2021a), we introduce performance tags during training by associating each “fast” program with a tag indicating the optimal achievable performance across all solutions in the dataset.”
MethodSelf-citation
Souper: A Synthesizing SuperoptimizerPreprint
Sasnauskas, R., Chen, Y., Collingbourne, P., et al. (2017) · arXiv.org · doi:10.48550/arxiv.1711.04422 · Cited by 99 · see citing works
Autonomous Search
Hamadi, Y., Monfroy, É., & Saubion, F. (2012) · Springer Berlin Heidelberg · doi:10.1007/978-3-642-21434-9 · Cited by 95
Leveraging Grammar and Reinforcement Learning for Neural Program SynthesisPreprint
Bunel, R., Hausknecht, M., Devlin, J., Singh, R., & Kohli, P. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.04276 · Cited by 94 · cited 2× · see citing works
“Recent work leverages machine learning to assist program synthesis (Balog et al., 2016; Bunel et al., 2018; Devlin et al., 2017a;b; Kalyan et al., 2018; Parisotto et al., 2016; Odena & Sutton, 2020; Nye et al., 2019; Meng et al., 2011; Jain et al., 2022; Austin et al., 2021; Xiong & Wang, 2022; Cambronero et al., 2023…”
Background
CodeT: Code Generation with Generated Tests
Chen, B., Zhang, F., Nguyen, A., Zan, D., Lin, Z., Lou, J.-G., & Chen, W. (2022) · arXiv · doi:10.48550/arxiv.2207.10397 · Cited by 92 · see citing works
CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks
Puri, R., Kung, D. S., Janssen, G., Zhang, W., Domeniconi, G., Zolotov, V., Dolby, J., Chen, J., Choudhury, M., Decker, L., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.12655 · Cited by 92 · cited 5×
“We also calculated the upper limit for speedup for the test set by aggregating the fastest submission across all submissions from CodeNet (118,841 accepted solutions) versus the fastest generation from our fastest model when increasing samples to 40 in Section 3.2.”
MethodInfluential
Graph-based, Self-Supervised Program Repair from Diagnostic Feedback
Yasunaga, M., & Liang, P. (2020) · arXiv · doi:10.48550/arxiv.2005.10636 · Cited by 81 · see citing works
Neural-Guided Deductive Search for Real-Time Program Synthesis from ExamplesPreprint
Kalyan, A., Mohta, A., Polozov, O., Batra, D., Jain, P., & Gulwani, S. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.01186 · Cited by 74 · cited 1× · see citing works
“Recent work leverages machine learning to assist program synthesis (Balog et al., 2016; Bunel et al., 2018; Devlin et al., 2017a;b; Kalyan et al., 2018; Parisotto et al., 2016; Odena & Sutton, 2020; Nye et al., 2019; Meng et al., 2011; Jain et al., 2022; Austin et al., 2021; Xiong & Wang, 2022; Cambronero et al., 2023…”
Background
Neural Program Meta-Induction
Devlin, J., Bunel, R., Singh, R., et al. (2017) · Neural Information Processing Systems · doi:10.48550/arxiv.1710.04157 · Cited by 74
Synchromesh: Reliable code generation from pre-trained language models
Poesia, G., Polozov, O., Le, V., Tiwari, A., Soares, G., Meek, C., & Gulwani, S. (2022) · arXiv · doi:10.48550/arxiv.2201.11227 · Cited by 72 · cited 2× · see citing works
“Recent work has demonstrated that retrieval-based mechanisms can improve language models for various tasks requiring factual or procedural knowledge (Liu et al., 2021; Poesia et al., 2021; Rubin et al., 2022; Madaan et al., 2022; Shrivastava et al., 2023).”
BackgroundMethod
Systematic editing
Meng, N., Kim, M., & McKinley, K. S. (2011) · ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI) · doi:10.1145/1993498.1993537 · Cited by 71 · cited 2× · see citing works
“Recent work leverages machine learning to assist program synthesis (Balog et al., 2016; Bunel et al., 2018; Devlin et al., 2017a;b; Kalyan et al., 2018; Parisotto et al., 2016; Odena & Sutton, 2020; Nye et al., 2019; Meng et al., 2011; Jain et al., 2022; Austin et al., 2021; Xiong & Wang, 2022; Cambronero et al., 2023…”
BackgroundMethod
Memory-assisted prompt editing to improve GPT-3 after deployment
Madaan, A., Tandon, N., Clark, P., & Yang, Y. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.183 · Cited by 60 · cited 1× · see citing works
“Given the high costs of training models through the OpenAI API, we also use this dataset as a base for fine-tuning experiments with GPT -3.5.”
MethodInfluentialSelf-citation
A Systematic Literature Survey of Software Metrics, Code Smells and Refactoring Techniques
Agnihotri, M., & Chug, A. (2020) · Journal of Information Processing Systems · doi:10.3745/jips.04.0184 · Cited by 50 · cited 1×
“Overall, our best model, GPT -3.5 augmented with synthetic data obtained from self-play, achieves an average speedup of 6.86 × , and optimizes 87.63 % of the test set by at least 10%.”
MethodInfluential
Learning to Infer Program SketchesPreprint
Nye, M., Hewitt, L., Tenenbaum, J. B., & Solar-Lezama, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.06349 · Cited by 49 · cited 2× · see citing works
“Recent work leverages machine learning to assist program synthesis (Balog et al., 2016; Bunel et al., 2018; Devlin et al., 2017a;b; Kalyan et al., 2018; Parisotto et al., 2016; Odena & Sutton, 2020; Nye et al., 2019; Meng et al., 2011; Jain et al., 2022; Austin et al., 2021; Xiong & Wang, 2022; Cambronero et al., 2023…”
Background
InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning
Gupta, P., Jiao, C., Yeh, Y.-T., Mehri, S., Eskénazi, M., & Bigham, J. P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.33 · Cited by 46 · cited 1× · see citing works
“Dataset imbalance.”
FlashFill++: Scaling Programming by Example by Cutting to the Chase
Cambronero, J., Gulwani, S., Le, V., et al. (2023) · Proc. ACM Program. Lang. · doi:10.1145/3571226 · Cited by 46 · cited 2×
“Recent work leverages machine learning to assist program synthesis (Balog et al., 2016; Bunel et al., 2018; Devlin et al., 2017a;b; Kalyan et al., 2018; Parisotto et al., 2016; Odena & Sutton, 2020; Nye et al., 2019; Meng et al., 2011; Jain et al., 2022; Austin et al., 2021; Xiong & Wang, 2022; Cambronero et al., 2023…”
Background
TrojanPuzzle: Covertly Poisoning Code-Suggestion Models
Aghakhani, H., Dai, W., Manoel, A., Fernandes, X., Kharkar, A., Kruegel, C., Vigna, G., Evans, D., Zorn, B., & Sim, R. (2024) · IEEE Symposium on Security and Privacy (S&P) · doi:10.1109/sp54263.2024.00140 · Cited by 45 · see citing works
A Learned Performance Model for Tensor Processing UnitsPreprint
Kaufman, S. J., Phothilimthana, P. M., Zhou, Y., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.01040 · Cited by 40 · see citing works
Triangulating Python Performance Issues with ScalenePreprint
Berger, E., Stern, S., & Pizzorno, J. A. (2022) · USENIX Symposium on Operating Systems Design and Implementation · doi:10.48550/arxiv.2212.07597 · Cited by 28 · cited 1× · see citing works
“SCALENE (Berger et al., 2022) designs a low-overhead Python-profiler that helps the developers to identify inefficiencies in their Python code and apply relevant optimization.”
Method
Repository-Level Prompt Generation for Large Language Models of Code
Shrivastava, D., Larochelle, H., & Tarlow, D. (2022) · arXiv · doi:10.48550/arxiv.2206.12839 · Cited by 27 · cited 1× · see citing works
“Recent work has demonstrated that retrieval-based mechanisms can improve language models for various tasks requiring factual or procedural knowledge (Liu et al., 2021; Poesia et al., 2021; Rubin et al., 2022; Madaan et al., 2022; Shrivastava et al., 2023).”
Background
L2S: A Framework for Synthesizing the Most Probable Program under a Specification
Xiong, Y., & Wang, B. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487570 · Cited by 25 · see citing works
Data Mining and Constraint Programming: Foundations of a Cross-Disciplinary Approach
Bessiere, C., Raedt, L. D., Kotthoff, L., et al. (2016) · Data Mining and Constraint Programming · doi:10.1007/978-3-319-50137-6 · Cited by 22 · cited 1×
“…2022; Kumar et al., 2022), auto-vectorize code (Nuzman et al., 2006; Mendis et al., 2019), optimize GPU code (Liou et al., 2020; Cummins et al., 2021), and automatically select algorithms (Kotthoff, 2016; Kerschke et al., 2019). and room at the top (Leiserson et al., 2020; Sherry & Thompson, 2021).”
AlpaGasus: Training A Better Alpaca with Fewer DataPreprint
Chen, L., Li, S., Yan, J., Wang, H., Gunaratna, K., Yadav, V., Tang, Z., Srinivasan, V., Zhou, T., Huang, H., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.08701 · Cited by 22 · see citing works
MAGPIE: Machine Automated General Performance Improvement via Evolution of Software
Blot, A., & Petke, J. (2022) · arXiv.org · doi:10.48550/arxiv.2208.02811 · Cited by 21 · cited 2×
“Additionally, Chen et al. (2022) uses a discrete variational auto-encoder, each latent representation maps to a different category of code edits, and canonicalized code representations to automatically suggest performance improvements, Shypula et al. (2021) trains seq2seq models from scratch on optimization data to su…”
Method
Language Models Can Teach Themselves to Program Better
Haluptzok, P., Bowers, M., & Kalai, A. T. (2022) · arXiv · doi:10.48550/arxiv.2207.14502 · Cited by 19 · cited 2× · see citing works
“This methodology aligns with self-play and self-instruct approachs in neural program synthesis (Haluptzok et al., 2022; Rozière et al., 2023).”
Method
Learning to make compiler optimizations more effective
Mammadli, R., Selakovic, M., Wolf, F., & Pradel, M. (2021) · MAPS@PLDI · doi:10.1145/3460945.3464952 · Cited by 19 · cited 1× · see citing works
“…by refactoring code (Mens & Tourwé, 2004; Ag-nihotri & Chug, 2020), identify compiler transformations (Bacon et al., 1994; Talaashrafi, 2022; Mammadli et al., 2021), perform parameter search (Hamadi, 2013; Huang et al., 2019; Kaufman et al., 2021; Seshadri et al., 2022; Kumar et al., 2022),…”
Method
Learning to Represent Programs with Property SignaturesPreprint
Odena, A., & Sutton, C. A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.09030 · Cited by 18 · see citing works
How Fast Do Algorithms Improve? [Point of View]
Sherry, Y. M., & Thompson, N. (2021) · Proceedings of the IEEE · doi:10.1109/jproc.2021.3107219 · Cited by 15 · cited 1× · see citing works
“…2022; Kumar et al., 2022), auto-vectorize code (Nuzman et al., 2006; Mendis et al., 2019), optimize GPU code (Liou et al., 2020; Cummins et al., 2021), and automatically select algorithms (Kotthoff, 2016; Kerschke et al., 2019). and room at the top (Leiserson et al., 2020; Sherry & Thompson, 2021).”
Method
The Wisdom of Hindsight Makes Language Models Better Instruction Followers
Zhang, T., Liu, F., Wong, J., Abbeel, P., & Gonzalez, J. E. (2023) · arXiv · doi:10.48550/arxiv.2302.05206 · Cited by 13 · cited 1× · see citing works
“Table 2 (bottom) shows results using our dynamic retrieval-based few-shot prompting strategy, with a preferable setting at K = 4 retrieved prompts.”
Method
DeepPERF: A Deep Learning-Based Approach For Improving Software Performance
Garg, S., Moghaddam, R. Z., Clement, C. B., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2206.13619 · Cited by 12 · cited 5×
“DeepPERF (Garg et al., 2022) uses a transformer-based model fine-tuned for the task of generating performance improvement patches for C# applications.”
BackgroundMethodInfluential
Data-Driven Offline Optimization For Architecting Hardware AcceleratorsPreprint
Kumar, A., Yazdanbakhsh, A., Hashemi, M., Swersky, K., & Levine, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.11346 · Cited by 11 · cited 1× · see citing works
“…Talaashrafi, 2022; Mammadli et al., 2021), perform parameter search (Hamadi, 2013; Huang et al., 2019; Kaufman et al., 2021; Seshadri et al., 2022; Kumar et al., 2022), auto-vectorize code (Nuzman et al., 2006; Mendis et al., 2019), optimize GPU code (Liou et al., 2020; Cummins et al., 2021), and…”
BackgroundSelf-citation
Magpie
Gao, G. (2024) · The Massachusetts review · doi:10.1353/mar.2024.a922965 · Cited by 11
Learning to Improve Code Efficiency
Chen, B., Tarlow, D., Swersky, K., et al. (2022) · arXiv.org · doi:10.48550/arxiv.2208.05297 · Cited by 11
Self-citation
Synthesizing code quality rules from examples
Garg, P., & Sengamedu, S. H. (2022) · Proceedings of the ACM on Programming Languages · doi:10.1145/3563350 · Cited by 9 · see citing works
Combinatorial Search: From Algorithms to Systems
Hamadi, Y. (2013) · Springer Berlin Heidelberg · doi:10.1007/978-3-642-41482-4 · Cited by 7 · cited 1×
“…(Mens & Tourwé, 2004; Ag-nihotri & Chug, 2020), identify compiler transformations (Bacon et al., 1994; Talaashrafi, 2022), perform parameter search (Hamadi, 2013; Huang et al., 2019; Kaufman et al., 2021; Seshadri et al., 2022; Kumar et al., 2022), auto-vectorize code (Nuzman et al., 2006; Mendis…”
Learning to Superoptimize Real-world ProgramsPreprint
Shypula, A., Yin, P., Lacomis, J., Goues, C. L., Schwartz, E. J., & Neubig, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.13498 · Cited by 2 · cited 1× · see citing works
“…representation maps to a different category of code edits, and canonicalized code representations to automatically suggest performance improvements, Shypula et al. (2021) trains seq2seq models from scratch on optimization data to superoptimize assembly programs after compilation, Shi et al. (2019)…”
BackgroundSelf-citation
Self-supervised Monocular Depth Estimation with Multiple Resolution Images
Ryu, G., & Kim, W. (2024) · Journal of Broadcast Engineering · doi:10.5909/jbe.2024.29.4.399
Do Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2022) · arXiv

Flagged first, then by citations

9 of the 94 references couldn’t be matched to a known paper.

Research works printed but not linked: 8

Alfred V Aho, Ravi Sethi, and Jeffrey D Ullman. Compilers: Principles, Techniques, and Tools, volume 2. Addison-wesley Reading, 2007.
Chris Cummins, Zacharias V Fisches, Tal Ben-Nun, Torsten Hoefler, Michael FP O'Boyle, and Hugh Leather. ProGraML: A Graph-based Program Representation for Data Flow Analysis and Compiler Op- timizations. In ICLR, 2021.
Jhe-Yu Liou, Xiaodong Wang, Stephanie Forrest, and Carole-Jean Wu. GEVO: GPU Code Optimization using Evolutionary Computation. TACO, 2020. · DOI
Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. MemPrompt: Memory-assisted Prompt Edit- ing with User Feedback. In EMNLP, 2022.
Charith Mendis, Cambridge Yang, Yewen Pu, Dr Amarasinghe, Michael Carbin, et al. Compiler Auto- Vectorization with Imitation Learning. NeurIPS, 2019.
Hui Shi, Yang Zhang, Xinyun Chen, Yuandong Tian, and Jishen Zhao. Deep Symbolic Superoptimization without Human Knowledge. In ICLR, 2019.
Delaram Talaashrafi. Advances in the Automatic Detection of Optimization Opportunities in Computer Programs. PhD thesis, Western University, 2022.
Lewis Tunstall, Leandro Von Werra, and Thomas Wolf. Natural Language Processing with Transformers. "O'Reilly Media, Inc.", 2022.

Sorted by citations

Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · Advances in Neural Information Processing Systems 36 · doi:10.52202/075280-2019 · Cited by 755 · see citing works
Self-citation
Mathematical discoveries from program search with large language models
Romera-Paredes, B., Barekatain, M., Novikov, A., Balog, M., Kumar, M. P., Dupont, E., Ruiz, F. J. R., Ellenberg, J. S., Wang, P., Fawzi, O., et al. (2024) · Nature · doi:10.1038/s41586-023-06924-6 · Cited by 544 · see citing works
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650 · see citing works
“In the work by Madaan et al. [169], the authors use CODEGEN and CodeX to suggest functionally correct, Performance-Improving Edits (PIEs), improving execution time of Python and C++ (already pre-optimised with the maximally optimising compiler option -O3 ).”
Method
Self-Refine: Iterative Refinement with Self-Feedback
Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang, Y., et al. (2023) · arXiv · doi:10.48550/arxiv.2303.17651 · Cited by 219 · see citing works
“We evaluate SELF-REFINE on 7 diverse tasks: Dialogue Response Generation (Appendix M; Mehri and Eskenazi, 2020), Code Optimization (Appendix N; Madaan et al., 2023), Code Readability Improvement (Appendix L; Puri et al.”
BackgroundMethodInfluentialSelf-citation
Teaching Large Language Models to Self-Debug
Chen, X., Lin, M., Schärli, N., & Zhou, D. (2023) · arXiv · doi:10.48550/arxiv.2304.05128 · Cited by 77 · see citing works
“For code generation, a number of works have trained models to perform code optimization [34], interact with users for multi-turn code generation [62, 58, 38], and fix generated programs based on human feedback [25, 16, 6].”
Method
Generative Artificial Intelligence for Software Engineering - a Research Agenda
NguyenDuc, A., Cabrero-Daniel, B., Arora, C., Przybylek, A., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K.-K., et al. (2023) · doi:10.2139/ssrn.4622517 · Cited by 71 · see citing works
Refactoring Programs Using Large Language Models with Few-Shot Examples
Shirafuji, A., Oda, Y., Suzuki, J., Morishita, M., & Watanobe, Y. (2023) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec60848.2023.00025 · Cited by 77 · see citing works
“As the most related work, Madaan et al. [16] demonstrated the capability of LLMs (i.e., Codex [2] and CodeGen [7]) to improve the time efficiency of programs by proposing a large-scale dataset consisting of (slower, faster) pairs of programs written by the same user.”
BackgroundMethod
Generative Artificial Intelligence for Software Engineering—A Research Agenda
Nguyen‐Duc, A., Cabrero‐Daniel, B., Przybylek, A., Arora, C., Khanna, D., Herda, T., Rafiq, U., Melegati, J., Guerra, E., Kemell, K., et al. (2025) · Software: Practice and Experience · doi:10.1002/spe.70005 · Cited by 60 · see citing works
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
Gou, Z., Shao, Z., Gong, Y., Shen, Y., Yang, Y., Duan, N., & Chen, W. (2023) · arXiv · doi:10.48550/arxiv.2305.11738 · Cited by 57 · see citing works
“[17] Aman Madaan, Alexander Shypula, Uri Alon, Milad Hashemi, Parthasarathy Ranganathan, Yiming Yang, Graham Neubig, and Amir Yazdanbakhsh.”
BackgroundSelf-citation
DreamCodeVR: Towards Democratizing Behavior Design in Virtual Reality with Speech-Driven Programming
Giunchi, D., Numan, N., Gatti, E., & Steed, A. J. (2024) · IEEE Conference on Virtual Reality and 3D User Interfaces (VR) · doi:10.1109/vr58804.2024.00078 · Cited by 46 · see citing works
“Moreover, comprehending and optimizing the LLMs’ code performance represents an unresolved issue [51].”
Prompting Techniques for Secure Code Generation: A Systematic Investigation
Tony, C., Ferreyra, N. E. D., Mutas, M., Dhiff, S., & Scandariato, R. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3722108 · Cited by 43 · see citing works
“5 , G P T - 4 , C o d e x Y e l p r e v i e w s [ 116 ] , F E D [ 65 ] , P I E [ 62 ] , C o d e N e t [ 77 ] , G S M 8 K [ 1 7 ] , A c r o n y m s , C o mm o n G e n [ 57 ] 2023 G P T - 3 , G P T - 3 .”
CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation
Peng, J., Cui, L., Huang, K., Yang, J., & Ray, B. (2025) · International Workshop on Large Language Models for Code (LLM4Code) · doi:10.1109/llm4code66737.2025.00009 · Cited by 29 · see citing works
“While most emphasize functional correctness—a key criteria for acceptance by human developers—recent efforts have begun exploring efficiency evaluation, collectively aiming to encompass all critical aspects of benchmarking LLM-generated code [12], [13].”
Reasoning Runtime Behavior of a Program with LLM: How Far are We?
Chen, J., Pan, Z., Hu, X., Li, Z., Li, G., & Xia, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00012 · Cited by 40 · see citing works
“As a result, considering these dimensions we select several state-of-the-art code LLMs that have been applied to various code related tasks [46], [47], [48].”
Transformers in source code generation: A comprehensive survey
Ghaemi, H., Alizadehsani, Z., Shahraki, A., & Corchado, J. M. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103193 · Cited by 23 · see citing works
OctoPack: Instruction Tuning Code Large Language Models
Muennighoff, N., Liu, Q., Zebaze, A., Zheng, Q., Hui, B., Zhuo, T. Y., Singh, S., Tang, X., von Werra, L., & Longpre, S. (2023) · arXiv · doi:10.48550/arxiv.2308.07124 · Cited by 17 · see citing works
“Future work may consider more models or extending H UMAN E VAL P ACK to new languages or tasks, such as code efﬁciency (Madaan et al., 2023a; Yetistiren et al., 2022) or code classiﬁcation (Khan et al., 2023).”
Method
Supersonic: Learning to Generate Source Code Optimizations in C/C++
Chen, Z., Fang, S., & Monperrus, M. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3423769 · Cited by 21 · see citing works
“In contrast, most related works [6], [7] employ a full program output representation.”
BackgroundMethodInfluential
Let’s Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs
Aggarwal, P., Madaan, A., Yang, Y., & Mausam, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.761 · Cited by 29 · see citing works
Self-citation
Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization
Yao, W., Heinecke, S., Niebles, J. C., Liu, Z., Feng, Y., Xue, L., Murthy, R., Chen, Z., Zhang, J., Arpit, D., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.02151 · Cited by 10 · see citing works
“…rewards, to enhance the agent’s behaviors, resulting in performances that are solely dependent on the quality of the pre-trained LLM. Self-refine (Madaan et al., 2023a) addresses this limitation by employing a single LLM as a generator, refiner, and provider of feedback, allowing for iterative…”
Background
Program Repair with Minimal Edits Using CodeT5
Shirafuji, A., Rahman, M. M., Amin, M. F. I., & Watanobe, Y. (2023) · International Conference on Awareness Science and Technology (iCAST) · doi:10.1109/icast57874.2023.10359288 · Cited by 10 · see citing works
“Not only fixing syntactic or semantic bugs, but Codex has also shown the ability to fix security bugs [44], improving time performance [10] and code readability [11].”
BackgroundMethod
Challenges and Paths Towards AI for Software EngineeringPreprint
Gu, A., Jain, N., Li, W., Shetty, M., Shao, Y., Li, Z., Yang, D., Ellis, K., Sen, K., & Solar-Lezama, A. (2025) · Qeios · doi:10.32388/vv1661 · Cited by 24 · see citing works
Exploring the Potential of Llama Models in Automated Code Refinement: A Replication Study
Caumartin, G., Qin, Q., Chatragadda, S., Panjrolia, J., Li, H., & Costa, D. E. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00070 · Cited by 8 · see citing works
“LLMs can also improve conciseness, elevate quality, and suppress complicity by refining the original codes [10]–[12].”
Using GPT-4 to Generate Failure Logic
Clegg, K. D., Habli, I., & McDermid, J. A. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-68738-9_11 · Cited by 6 · see citing works
Efficacy of Various Large Language Models in Generating Smart Contracts
Chatterjee, S., & Ramamurthy, B. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-85363-0_31 · Cited by 6 · see citing works
Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering
(2024) · doi:10.1145/3650105 · Cited by 4 · see citing works
Investigating Execution-Aware Language Models for Code Optimization
Menna, F. D., Traini, L., Bavota, G., & Cortellessa, V. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00029 · Cited by 3 · see citing works
“We evaluate the effectiveness of each language model using the PIE test set [10].”
Influential
Optimizing Code Runtime Performance Through Context-Aware Retrieval-Augmented Generation
Acharya, M., Zhang, Y., Leach, K., & Huang, Y. (2025) · IEEE International Conference on Program Comprehension (ICPC) · doi:10.1109/icpc66645.2025.00028 · Cited by 11 · see citing works
“While traditional compilers excel at tasks like instruction scheduling and register allocation, higher-level optimizations such as restructuring logic, refining control flows, and addressing inefficiencies still depend heavily on human expertise [2].”
Artemis AI: Multi-LLM Framework for Code Optimisation
Giavrimis, R., Basios, M., Wu, F., Kanthan, L., & Bauer, R. (2025) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai64502.2025.00057 · Cited by 2 · see citing works
“Garg et al. [18] and Shypula et al. [19] used the performance commits in open source projects to fine-tune LLMs, aiming to enhance the models’ capabilities in suggesting performance-improving code edits.”
Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation
Zelikman, E., Lorch, E., Mackey, L., & Kalai, A. T. (2023) · arXiv · doi:10.48550/arxiv.2310.02304 · Cited by 4 · see citing works
“Prior work, such as STaR (Zelikman et al., 2022 work has explored teaching language models to debug or optimize code (Chen et al., 2023b; Shypula et al., 2023).”
Method
RunBugRun: An executable dataset for automated program repair
Prenner, J. A., & Robbes, R. (2026) · Empirical Software Engineering · doi:10.1007/s10664-025-10790-3 · Cited by 11 · see citing works
Translating and Optimising Computational Microscopy Algorithms with Large Language Models
Guzzi, F., Kourousias, G., Pugliese, R., Gianoncelli, A., & Billè, F. (2024) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro60963.2024.10569791
“The topic raised high interest in 2023: in [20] the authors propose i) a dataset, ii) a framework to estimate the code speedup and iii) a model based on CodeLlama13B [21], that is a specific foundational model trained for code-based prompts; similarly, in [12] the authors analyze the limits of…”
Background
VerilogASTBench: Benchmark Construction of Verilog AST Dataset with Dual-Stage AST Semantic Enhancement Framework
Zhang, L., Chen, C., Yan, D., Xu, H., Cao, M., Song, J., Cai, Z., & Guo, Y. (2026) · Proceedings of the ACM on software engineering. · doi:10.1145/3797073
Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?
Chen, Z., Sun, Z., Shi, Y., Lo, D., & Jiang, L. (2026) · arXiv · doi:10.48550/arxiv.2607.01211

Authors: Shypula, A., Madaan, A., Zeng, Y., Alon, U., Gardner, J., Hashemi, M., Neubig, G., Ranganathan, P., Bastani, O., & Yazdanbakhsh, A.
Affiliations: University of Pennsylvania (US)
Google (United States) (US)
ORCID: Shypula, Alexander
Madaan, Aman
Alon, Uri
Hashemi, Milad
Neubig, Graham
Ranganathan, Parthasarathy
Yazdanbakhsh, Amir
Venue: arXiv
Publisher: arXiv
Published: 2023
ISSN: 2331-8422 (ISSN-L)
DOI: 10.48550/arxiv.2302.07867
arXiv: 2302.07867
OpenAlex: W4321276180
Semantic Scholar: 1786a2f9140ed7211b21302977de64e948b92308
Cited by: 23
References: 94
Citations by year: 2026: 2 · 2025: 10 · 2024: 6 · 2023: 5
Type: posted content
Language: en
Open access: green
arxiv.org
Licence: other-oa
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations: arXiv (Cornell University) (submittedVersion, other-oa)
arXiv (Cornell University)
Funders: DeepMind
Topics: Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Physical Sciences › Computer Science › Hardware and Architecture › Parallel Computing and Optimization Techniques
Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords: program optimization, large language models, performance evaluation, chain-of-thought prompting, synthetic data augmentation, fine-tuning, competitive programming

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-10.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-10.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Learning Performance-Improving Code Edits

Abstract

AI snapshot

TL;DR

Key findings

Methods

Limitations

Integrity signals

Claims the paper makes

Self-check

Cites

Cited by

Related

Record

About these checks