CoChat
Sign Up
PreprintCited by 18Software Engineering ResearchTopic ModelingTeaching and Learning Programming

A Comparative Study of Code Generation using ChatGPT 3.5 across 10 Programming Languages

Open access
Buscemi, Alessio·2023·arXiv
DOI 10.48550/arxiv.2308.04477arXiv 2308.04477
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) are advanced Artificial Intelligence (AI) systems that have undergone extensive training using large datasets in order to understand and produce language that closely resembles that of humans. These models have reached a level of proficiency where they are capable of successfully completing university exams across several disciplines and generating functional code to handle novel problems. This research investigates the coding proficiency of ChatGPT 3.5, a LLM released by OpenAI in November 2022, which has gained significant recognition for its impressive text generating and code creation capabilities. The skill of the model in creating code snippets is evaluated across 10 various programming languages and 4 different software domains. Based on the findings derived from this research, major unexpected behaviors and limitations of the model have been identified. This study aims to identify potential areas for development and examine the ramifications of automated code generation on the evolution of programming languages and on the tech industry.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2308.04477

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Discrepancies between AI-assisted and manual analysis can highlight either shortcomings in the AI-generated approach or opportunities to refine the analysis.

    “Discrepancies can highlight either shortcomings in the AI-generated approach or opportunities to refine the analysis 24 .”

    From Reimagining biomedical science workflows in the age of large language models (2026) · cites this paper as [24]

    The Introduction describes comparing ChatGPT-generated code across programming languages, and the Conclusion identifies limitations in that code. Neither section addresses discrepancies between AI-assisted and manual analysis or says such discrepancies can guide refinement. Searches for the claim’s phrasing and for a comparison with manually written work found no supporting passage; page numbers were not available in our copy. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 33

  1. K. Chowdhary and K. Chowdhary, "Natural language processing," Fundamentals of artificial intelligence, pp. 603-649, 2020.
  2. J. Wei, Y. Tay, R. Bommasani, et al., "Emergent abilities of large language models," arXiv preprint arXiv:2206.07682, 2022. · arXiv
  3. T. Brown, B. Mann, N. Ryder, et al., "Language models are few-shot learners," Advances in neural information processing systems, vol. 33, pp. 1877-1901, 2020.
  4. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding," arXiv preprint arXiv:1810.04805, 2018. · arXiv
  5. A. Vaswani, N. Shazeer, N. Parmar, et al., "Attention is all you need," Advances in neural information processing systems, vol. 30, 2017.
  6. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, "Deep reinforcement learning from human preferences," Advances in neural information processing systems, vol. 30, 2017.
  7. K. Weiss, T. M. Khoshgoftaar, and D. Wang, "A survey of transfer learning," Journal of Big data, vol. 3, no. 1, pp. 1-40, 2016.
  8. P. Liang, R. Bommasani, T. Lee, et al., "Holistic evaluation of language models," arXiv preprint arXiv:2211.09110, 2022. · arXiv
  9. P. Vaithilingam, T. Zhang, and E. L. Glassman, "Expectation vs. experi- ence: Evaluating the usability of code generation tools powered by large language models," in Chi conference on human factors in computing systems extended abstracts, 2022, pp. 1-7.
  10. L. Floridi and M. Chiriatti, "GPT-3: Its nature, scope, limits, and consequences," Minds and Machines, vol. 30, pp. 681-694, 2020.
  11. H. Touvron, L. Martin, K. Stone, et al., "Llama 2: Open Foundation and Fine-Tuned Chat Models," arXiv preprint arXiv:2307.09288, 2023. · arXiv
  12. R. Anil, A. M. Dai, O. Firat, et al., "Palm 2 technical report," arXiv preprint arXiv:2305.10403, 2023. · arXiv
  13. R. Thoppilan, D. De Freitas, J. Hall, et al., "Lamda: Language models for dialog applications," arXiv preprint arXiv:2201.08239, 2022. · arXiv
  14. H. Tian, W. Lu, T. O. Li, et al., "Is ChatGPT the Ultimate Programming Assistant-How far is it?" arXiv preprint arXiv:2304.11938, 2023. · arXiv
  15. Y. Cao, S. Li, Y. Liu, et al., "A comprehensive survey of ai-generated content (aigc): A history of generative ai from gan to chatgpt," arXiv preprint arXiv:2303.04226, 2023. · arXiv
  16. I. Goodfellow, J. Pouget-Abadie, M. Mirza, et al., "Generative adversar- ial nets," Advances in neural information processing systems, vol. 27, 2014.
  17. K. Gregor, I. Danihelka, A. Mnih, C. Blundell, and D. Wierstra, "Deep autoregressive networks," in International Conference on Machine Learning, PMLR, 2014, pp. 1242-1250.
  18. R. Gozalo-Brizuela and E. C. Garrido-Merchán, "A survey of Generative AI Applications," arXiv preprint arXiv:2306.02781, 2023. · arXiv
  19. A. Gillioz, J. Casas, E. Mugellini, and O. Abou Khaled, "Overview of the Transformer-based Models for NLP Tasks," in 2020 15th Conference on Computer Science and Information Systems (FedCSIS), IEEE, 2020, pp. 179-183.
  20. B. Stroustrup, The C++ Programming Language Fourth Edition, 2013.
  21. J. Meyerson, "The go programming language," IEEE software, vol. 31, no. 5, pp. 104-104, 2014.
  22. J. Bezanson, A. Edelman, S. Karpinski, and V. B. Shah, "Julia: A fresh approach to numerical computing," SIAM review, vol. 59, no. 1, pp. 65- 98, 2017.
  23. G. Van Rossum and F. L. Drake Jr, Python tutorial. Centrum voor Wiskunde en Informatica Amsterdam, The Netherlands, 1995, vol. 620.
  24. H. Wickham, M. Çetinkaya-Rundel, and G. Grolemund, R for data science. " O'Reilly Media, Inc.", 2023.
  25. D. Flanagan and Y. Matsumoto, The Ruby Programming Language: Everything You Need to Know. " O'Reilly Media, Inc.", 2008.
  26. A. Goldberg and D. Robson, Smalltalk-80: the language and its imple- mentation. Addison-Wesley Longman Publishing Co., Inc., 1983.
  27. J. Austin, A. Odena, M. Nye, et al., "Program synthesis with large language models," arXiv preprint arXiv:2108.07732, 2021. · arXiv
  28. M. Chen, J. Tworek, H. Jun, et al., "Evaluating large language models trained on code," arXiv preprint arXiv:2107.03374, 2021. · arXiv
  29. T. Yu, R. Zhang, K. Yang, et al., "Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task," arXiv preprint arXiv:1809.08887, 2018. · arXiv
  30. Q. Zheng, X. Xia, X. Zou, et al., "Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x," arXiv preprint arXiv:2303.17568, 2023. · arXiv
  31. Y. Li, D. Choi, J. Chung, et al., "Competition-level code generation with AlphaCode," Science, vol. 378, no. 6624, pp. 1092-1097, 2022.
  32. F. Cassano, J. Gouwar, D. Nguyen, et al., "MultiPL-E: a scalable and polyglot approach to benchmarking neural code generation," IEEE Transactions on Software Engineering, 2023.
  33. J. Liu, C. S. Xia, Y. Wang, and L. Zhang, "Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation," arXiv preprint arXiv:2305.01210, 2023. · arXiv

Cited by

Sorted by citations

Authors
Buscemi, A.
Corresponding author
Buscemi, Alessio
ORCID
Buscemi, Alessio
Venue
arXiv
Publisher
arXiv
Published
2023
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2308.04477
arXiv
2308.04477
OpenAlex
W4385749752
Cited by
18
References
0
Citations by year
2026: 3 · 2025: 8 · 2024: 5 · 2023: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
ChatGPT-3.5, large language models, automatic code generation, programming languages, code generation evaluation, software domains, model limitations

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review