CoChat

Choose your programming copilot: a comparison of the program synthesis performance of github copilot and genetic programming

Sobania, Dominik, Briesch, Martin, Rothlauf, Franz · 2022 · Proceedings of the Genetic and Evolutionary Computation Conference
Cited by 91

The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.

Abstract

GitHub Copilot, an extension for the Visual Studio Code development environment powered by the large-scale language model Codex, makes automatic program synthesis available for software developers. This model has been extensively studied in the field of deep learning, however, a comparison to genetic programming, which is also known for its performance in automatic program synthesis, has not yet been carried out. In this paper, we evaluate GitHub Copilot on standard program synthesis benchmark problems and compare the achieved results with those from the genetic programming literature. In addition, we discuss the performance of both approaches. We find that the performance of the two approaches on the benchmark problems is quite similar, however, in comparison to GitHub Copilot, the program synthesis approaches based on genetic programming are not yet mature enough to support programmers in practical software development. Genetic programming usually needs a huge amount of expensive hand-labeled training cases and takes too much time to generate solutions. Furthermore, source code generated by genetic programming approaches is often bloated and difficult to understand. For future work on program synthesis with genetic programming, we suggest researchers to focus on improving the execution time, readability, and usability.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3512290.3528700

Cites

5 of 5 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 5 references

Research works printed but not linked: 30

  1. Francisco Baeta , João Correia , Tiago Martins , and Penousal Machado . 2021. TensorGP-Genetic Programming Engine in TensorFlow .. In EvoApplications . Springer , 763--778. Francisco Baeta, João Correia, Tiago Martins, and Penousal Machado. 2021. TensorGP-Genetic Programming Engine in TensorFlow.. In EvoApplications. Springer, 763--778.
  2. Dzmitry Bahdanau , Kyunghyun Cho , and Yoshua Bengio . 2015 . Neural Machine Translation by Jointly Learning to Align and Translate . In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7--9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1409 .0473 Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7--9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.). http://arxiv.org/abs/1409.0473 · arXiv
  3. Markus F Brameier and Wolfgang Banzhaf . 2007. Linear genetic programming . Springer Science & Business Media . Markus F Brameier and Wolfgang Banzhaf. 2007. Linear genetic programming. Springer Science & Business Media.
  4. Tom Brown , Benjamin Mann , Nick Ryder , Melanie Subbiah , Jared D Kaplan , Prafulla Dhariwal , Arvind Neelakantan , Pranav Shyam , Girish Sastry , Amanda Askell , Sandhini Agarwal , Ariel Herbert-Voss , Gretchen Krueger , Tom Henighan , Rewon Child , Aditya Ramesh , Daniel Ziegler , Jeffrey Wu , Clemens Winter , Chris Hesse , Mark Chen , Eric Sigler , Mateusz Litwin , Scott Gray , Benjamin Chess , Jack Clark , Christopher Berner , Sam McCandlish , Alec Radford , Ilya Sutskever , and Dario Amodei . 2020 . Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H . Lin (Eds.) , Vol. 33 . Curran Associates, Inc. , 1877--1901. https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877--1901. https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  5. Nicholas Carlini , Florian Tramèr , Eric Wallace , Matthew Jagielski , Ariel Herbert-Voss , Katherine Lee , Adam Roberts , Tom Brown , Dawn Song , Úlfar Erlingsson , Alina Oprea , and Colin Raffel . 2021 . Extracting Training Data from Large Language Models . In 30th USENIX Security Symposium (USENIX Security 21) . USENIX Association, 2633--2650. https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting Training Data from Large Language Models. In 30th USENIX Security Symposium (USENIX Security 21). USENIX Association, 2633--2650. https://www.usenix.org/conference/usenixsecurity21/presentation/carlini-extracting
  6. Mark Chen , Jerry Tworek , Heewoo Jun , Qiming Yuan , Henrique Ponde de Oliveira Pinto , Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021 . Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). · arXiv
  7. 10.3115/v1/D14-1179 · DOI · Paper page
  8. 10.18653/v1/2020.emnlp-main.728 · DOI · Paper page
  9. Nichael Lynn Cramer . 1985 . A representation for the adaptive generation of simple sequential programs . In proceedings of an International Conference on Genetic Algorithms and the Applications. 183--187 . Nichael Lynn Cramer. 1985. A representation for the adaptive generation of simple sequential programs. In proceedings of an International Conference on Genetic Algorithms and the Applications. 183--187.
  10. 10.1007/978-3-319-55696-3_17 · DOI
  11. 10.1007/978-3-319-99253-2_16 · DOI
  12. 10.1145/1836089.1836091 · DOI · Paper page
  13. 10.1145/3377929.3389987 · DOI
  14. 10.1145/3449639.3459285 · DOI
  15. 10.1145/3071178.3071330 · doi:10.1145/3071178.3071330
  16. 10.1145/3205455.3205603 · DOI
  17. 10.1145/2739480.2754769 · DOI · Paper page
  18. Thomas Helmuth and Lee Spector . 2021. Problem-solving benefits of down-sampled lexicase selection. arXiv preprint arXiv:2106.06085 ( 2021 ). Thomas Helmuth and Lee Spector. 2021. Problem-solving benefits of down-sampled lexicase selection. arXiv preprint arXiv:2106.06085 (2021). · arXiv
  19. 10.1145/3321707.3321865 · DOI
  20. John R Koza . 1992. Genetic programming: on the programming of computers by means of natural selection . Vol. 1 . MIT press . John R Koza. 1992. Genetic programming: on the programming of computers by means of natural selection. Vol. 1. MIT press.
  21. 10.1145/3319619.3321892 · DOI
  22. Md Omar Faruk Rokon , Risul Islam , Ahmad Darki , Evangelos E. Papalexakis , and Michalis Faloutsos . 2020 . SourceFinder: Finding Malware Source-Code from Publicly Available Repositories in GitHub . In 23rd International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2020 ). USENIX Association, San Sebastian, 149--163. https://www.usenix.org/conference/raid 2020/presentation/omar Md Omar Faruk Rokon, Risul Islam, Ahmad Darki, Evangelos E. Papalexakis, and Michalis Faloutsos. 2020. SourceFinder: Finding Malware Source-Code from Publicly Available Repositories in GitHub. In 23rd International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2020). USENIX Association, San Sebastian, 149--163. https://www.usenix.org/conference/raid2020/presentation/omar
  23. 10.1007/978-3-030-72812-0_9 · DOI
  24. 10.1145/3321707.3321738 · DOI
  25. 10.1007/978-3-030-44094-7_14 · doi:10.1007/978-3-030-44094-7_14
  26. 10.1145/3449639.3459305 · doi:10.1145/3449639.3459305
  27. 10.1109/TEVC.2022.3162324 · DOI
  28. 10.1145/1068009.1068292 · DOI · Paper page
  29. 10.1023/A:1014538503543 · DOI · Paper page
  30. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008. Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008.

Cited by

See all 91

Metadata

Authors
Sobania, D., Briesch, M., & Rothlauf, F.
Venue
Proceedings of the Genetic and Evolutionary Computation Conference
Publisher
ACM
Published
2022-07-08
Pages
1019-1027
OpenAlex
W4285734663
Cited by
91
References
5
FWCI
11.98
Type
conference paper
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Choose your programming copilot: a comparison of the program synthesis performance of github copilot and genetic programming. https://app.cochat.ai/papers/10.1145/3512290.3528700
The paper
Sobania, D., Briesch, M., & Rothlauf, F. (2022). Choose your programming copilot: a comparison of the program synthesis performance of github copilot and genetic programming. Proceedings of the Genetic and Evolutionary Computation Conference. https://doi.org/10.1145/3512290.3528700

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.