Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues

May combine two claims.

66% of Python programs and 69% of Java programs generated by ChatGPT passed all provided test cases.

“66% and 69% of Python and Java programs generated by ChatGPT are functionally-correct programs, i.e., programs that pass all given test cases”

May combine two claims.

The study's dataset consists of 2,033 LeetCode programming tasks and 4,066 ChatGPT-generated code snippets in Java and Python.

“Introduce a new time-sensitive dataset comprising 2033 programming tasks and 4066 ChatGPT-generated code snippets implemented in two popular programming languages: Java and Python”

• Also stated in: § 3.2 Constructing Benchmark Dataset · plain

  “we successfully collected 2,033 programming tasks from LeetCode”

Probably not a substantial claim: may not pass S3 (load-bearing).

The 2,033 tasks comprised 501 easy, 1,064 medium, and 468 hard tasks.

“out of the 2,033 tasks in our dataset, we found that 501 were classiied as easy, 1,064 as medium, and 468 as hard”

• AssumptionRests on an assumption: LeetCode's difficulty labels accurately reflect the difficulty of programming tasks.

May combine two claims.

Among ChatGPT-generated code that passed test cases, 53% of Java code and 37% of Python code had code style and maintainability issues.

“among ChatGPT-generated code that passed the test cases, 53% of the Java code and 37% of the Python code exhibited code style and maintainability issues”

May combine two claims.

ChatGPT solved more than half of hard-level tasks before June 2021, but its performance dropped to nearly 0.1 for later periods.

“ChatGPT can solve more than half of the hard-level code tasks before June 2021, but its performance reduces drastically to nearly 0.1 for the subsequent time periods”

May combine two claims.

For Java, ChatGPT's pass@1 accuracy was 0.860 on easy tasks, 0.710 on medium tasks, and 0.468 on hard tasks.

“for Java, the model attains a pass@1 accuracy of 0.860 for easy tasks, 0.710 for medium tasks, and 0.468 for hard tasks”

• AssumptionRests on an assumption: LeetCode's difficulty labels accurately reflect the difficulty of programming tasks.

May combine two claims. The quote only partly states this claim.

The percentage of clean code decreased as task difficulty increased for both Python and Java; for Java, it dropped from 54% for easy tasks to 45% for medium and 33% for hard tasks.

“the percentage of clean Java code decreases from 54% for easy tasks to 45% for medium tasks, and further drops to 33% for hard tasks”

• AssumptionRests on an assumption: LeetCode's difficulty labels accurately reflect the difficulty of programming tasks.

May combine two claims.

The majority of generated code snippets were 10-20 lines for Python and 20-30 lines for Java.

“the majority of generated code snippets falling into the 10-20 lines range for Python and the 20-30 lines range for Java”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). The quote only partly states this claim.

The dataset contains more than 400 programming tasks introduced after 2021.

“while most tasks are from before 2021, there are still more than 400 test cases for evaluating ChatGPT's code generation capabilities”

May combine two claims.

The differences in pass rates by code length were statistically significant (p < 0.01) with a small to medium effect size.

“the Mann-Whitney U test conirms the signiicance of the diferences (p-value < 0.01) with a small to medium efect size”

May combine two claims.

For easy Python tasks, 7 out of 501 generated snippets (1%) have compilation and runtime errors, while for hard Python tasks, 46 out of 468 (10%) have execution errors.

“For example, 7 out of 501 generated code snippets (i.e., 1%) for easy Python tasks exhibit compilation and runtime errors, while the number of execution errors increases to 46 out of 468 (i.e., 10%) for hard Python tasks.”

May combine two claims.

Although simple feedback may be better for some issues, iterative feedback with static analysis and runtime errors performs much better than simple feedback as iteration rounds progress.

“Moreover, although our previous experiments indicated that simple feedback may be better for some issues, iterative feedback with static analysis and runtime errors performs much better than simple feedback as the iteration rounds progress.”

May combine two claims.

Wrong output and code style and maintainability issues are more prevalent and are the most common challenges in ChatGPT-generated code.

“However, Wrong Output and Code Style & Maintainability issues are more prevalent and tend to be the most common challenges faced by the generated code.”

May combine two claims.

Illegal Index errors are prevalent in both languages, particularly Java; 45 of 97 compilation and runtime errors in Java (46.4%) are due to invalid index use.

“We also notice that Illegal Index errors are quite prevalent in both languages, particularly in Java.In fact, out of the 97 compilation and runtime errors encountered in Java, 45 of them (46.4%) are attributed to using an invalid index.”

May combine two claims.

Prompts with detailed feedback can effectively assist ChatGPT in self-repairing code quality issues, whereas ambiguous feedback may negatively impact ChatGPT's performance.

“Finding 9: Prompts with detailed feedback can efectively assist ChatGPT in self-repairing code quality issues, whereas ambiguous feedback may have a negative impact on ChatGPT's performance.”

May combine two claims.

Feedback with static analysis and runtime errors is more effective in fixing code style and maintainability, while simple feedback performs better for the remaining quality issues in both Java and Python.

“we observed that feedback with static analysis and runtime errors is more efective in ixing code style and maintainability while simple feedback performs better in the remaining quality issues in both Java and Python.”

May combine two claims.

For LeetCode Problem 746 with input [10,15,20], ChatGPT-generated code produced 25 instead of the expected 15 because the loop iterated one step too many and included the last step's cost.

“However, the ChatGPT-generated code produces an output of 25.This error occurs because the loop iterates one step more than necessary, causing the last step's cost to be included in the calculation even when it should not be.”

May combine two claims.

Of 1,310 code snippets that failed test cases, 14% have compilation and runtime errors, 83% wrong outputs, 4% performance or efficiency issues, and 52% code style and maintainability issues on top of functional errors.

“Regarding the 1,310 code that failed test cases, 14% encounter compilation and runtime errors, 83% exhibit wrong outputs, 4% exhibit performance or eiciency issues, and notably, 52% exhibited issues related to code style and maintainability on top”

Probably not a substantial claim: may not pass S3 (load-bearing).

Table 4 compares common compilation and runtime error categories in Java and Python programs, based on 80 Python and 97 Java programs with errors.

“Table 4 presents a comparison of common compilation and runtime errors error categories in Java and Python programs (i.e., 80 Python and 97 Java programs with the errors).”

Probably not a substantial claim: may not pass S3 (load-bearing).

For LeetCode Problem 1982, ChatGPT-generated code caused a TIMEOUT due to inefficient loop control.

“when using ChatGPT to solve LeetCode Problem 1982, titled “Find Array Given Subset Sums”, the compiler outputs a łTIMEOUT” error due to ineicient loop control within the generated code.”

May combine two claims.

Performance stabilizes in later rounds, indicating potential upper bounds to iterative gains.

“However, our results show performance stabilizing in later rounds, indicating potential upper bounds to iterative gains.”

May combine two claims.

Static analysis feedback works better for code style and maintainability issues, while simple feedback is more effective for addressing execution errors and wrong outputs.

“For instance, static analysis feedback works better for code style and maintainability issues, while simple feedback is more efective for addressing execution errors and wrong outputs.”

May combine two claims.

The effectiveness of ChatGPT's self-mitigating capabilities depends on the type of feedback provided, the programming language, and the specific code quality issue.

“Our results suggest that the efectiveness of ChatGPT's self-mitigating capabilities depends on the type of feedback provided, the programming language, and the speciic code quality issue.”

May combine two claims.

For errors related to wrong outputs or efficiency, simpler prompts worked better in single-round feedback, likely because compiler feedback might be ambiguous or unhelpful.

“However, for errors related to wrong outputs or eiciency, simpler prompts worked better in single-round feedback, likely because the prompts collected from the compiler feedback might be ambiguous or unhelpful.”

May combine two claims.

Static feedback and runtime errors supplied detailed information and resulted in a correct fix at line 41, changing a for-loop to a foreach-loop.

“Static feedback and runtime errors, on the other hand, provide detailed information, i.e., leading to a correct ix in line 41 which change the for-loop to foreach-loop.”

May combine two claims.

Code produced by ChatGPT frequently has quality problems, including compilation and runtime errors, incorrect outputs, maintainability issues, and performance inefficiencies.

“Our indings demonstrate that while ChatGPT can generate functional code for various programming tasks, the generated code often sufers from quality issues, such as compilation and errors, wrong outputs, maintainability problems, and performance ineiciencies.”

• Also stated in: § Additional information · hedged

  “However, the generated code sometimes sufers from diferent code quality issues, such as execution errors, wrong outputs, and maintainability problems.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The ChatGPT model used was ChatGPT-March-23, trained on data up to 2021.

“we used the ChatGPT-March-23 version [41], which was trained on data up to 2021”

• cites [41]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The difference in code length distributions between Python and Java reflects differences in verbosity and structure between the two languages.

“This discrepancy highlights the diferences in verbosity and structure between the two programming languages”

May combine two claims.

Code quality issues in ChatGPT-generated code can affect code comprehension, introduce bugs, or create security vulnerabilities.

“Code quality issues in ChatGPT-generated code, if not properly identiied and addressed, may unduly afect code comprehension, introduce bugs, or create security vulnerabilities in users' projects [47].”

• cites [47]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The inverse relationship between code length and pass rate may be due to increased complexity and more interactions between code elements as code size grows.

“This could be due to the increased complexity, and the greater number of potential interactions between code elements as the code size grows”

Probably not a substantial claim: may not pass S3 (load-bearing).

In the static analysis, messages related to style issues such as whitespace, newline, and invalid naming conventions were ignored to reduce false positives, consistent with prior work.

“we choose to ignore messages related to style issues such as whitespace, newline, and invalid naming conventions, which is consistent with the approach taken in prior work [48]”

• cites [48] sourceNot checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

To mitigate randomness, ChatGPT was made deterministic by setting temperature to 0 and running the model once per task, using the first generated output for evaluation.

“To mitigate the randomness of ChatGPT, we make ChatGPT deterministic by setting the temperature to 0 and running the model once for each task, using the irst generated output for evaluation.”

May combine two claims.

Simple feedback is general, so ChatGPT is sometimes misguided, resulting in new problems while plausibly solving original issues.

“This is because simple feedback is quite general so ChatGPT is sometimes misguided, resulting in introducing new problems though plausibly solving the original issues.”

May combine two claims.

Compilation and runtime errors and performance and efficiency issues are relatively less frequent in ChatGPT-generated code, indicating ChatGPT is generally successful in generating syntactically correct and efficient code.

“From the table, it is evident that Compilation & Runtime Errors and Performance & Eiciency issues are relatively less frequent, indicating that ChatGPT is generally successful in generating syntactically correct and eicient code.”

Classified as probably not a substantial claim.

The introduction of noise by static analysis and runtime error feedback can confuse ChatGPT and lead to incorrect patches.

“This is possibly due to the introduction of noise by static analysis and runtime error feedback, which can confuse ChatGPT and lead to incorrect patches.”

May combine two claims.

The probabilistic nature of ChatGPT, which predicts subsequent tokens from preceding ones, enables it to understand common program semantics but captures semantics implicitly from training data, leading to misunderstandings and semantically-related compilation and runtime errors.

“These observations can be explained by the probabilistic nature of the ChatGPT model, which predicts subsequent tokens based on preceding ones.This nature enables ChatGPT to understand the semantics of common programs that appear frequently in the training set.However, the model captures the semantics implicitly from the training data, leading to misunderstandings of program semantics and subs”

• AssumptionRests on an assumption: ChatGPT is a probabilistic model that predicts subsequent tokens based on preceding ones.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Python's dynamic typing system allows more flexibility in variable types but can lead to unexpected type-related issues at runtime.

“This observation could be due to Python's dynamic typing system, which allows for more lexibility in variable types, but can also lead to unexpected type-related issues at runtime.”

• AssumptionRests on an assumption: Python's dynamic typing system allows more flexibility in variable types but can lead to unexpected type-related issues at runtime.

Classified as probably not a substantial claim.

To make the iterative repairing experiment more rigorous, the authors used only 402 new LeetCode test examples published in 2022, after GPT-3.5's pretraining knowledge cutoff.

“To make the experiment more rigorous, we only use the 402 new test examples from LeetCode that were published in the year 2022 (after GPT-3.5'spre-training knowledge cutof).”

• AssumptionRests on an assumption: GPT-3.5's pretraining knowledge cutoff occurred before the year 2022.

Probably not a substantial claim: may not pass S3 (load-bearing).

The study's dataset of 2,033 programming tasks from LeetCode may not represent all possible code generation tasks encountered in real-world software development.

“Our study is based on a dataset of 2,033 programming tasks from LeetCode, which may not represent all possible code generation tasks encountered in real-world software development.”

Probably not a substantial claim: may not pass S3 (load-bearing).

There is a large number of studies investigating code quality issues from human-written code.

“There is a large number of studies investigating code quality issues from humanwritten code.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Keuning et al. investigated code quality issues in student programs.

“Keuning et al. [26] investigate code quality issues in student programs.”

• cites Keuning et al. [26]Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim. May combine two claims.

To address variability in ChatGPT-generated code due to different prompts, the authors followed a similar methodology used by Fan et al. and Tian et al., using a consistent set of prompts across tasks.

“To address this concern, we followed the similar methodology used by Fan et al. [15] and Tian et al. [55], which ensures that our results are reliable by using a consistent set of prompts across diferent tasks.”

• cites Fan et al. [15]Not checked yet

  Not checked in the free run.

• cites Tian et al. [55]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Kochhar et al. conducted a large-scale empirical investigation on the code quality of open-source projects implemented in 17 programming languages.

“Kochhar et al. [27] conducted a large-scale empirical investigation on the code quality of open-source projects implemented in 17 programming languages.”

• cites Kochhar et al. [27]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

The quality of ChatGPT-generated code is a critical concern for top software companies when deciding whether to adopt it in practice.

“However, the quality of the ChatGPT-generated code is the critical concern for top software companies when deciding whether to adopt it in practice or not [10,21,44].”

• cites [10]Not checked yet

  Not checked in the free run.

• cites [21]Not checked yet

  Not checked in the free run.

• cites [44]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Saboury et al. empirically investigated code smells in 537 releases of five popular JavaScript applications.

“Saboury et al. [46] empirically investigate code smells in 537 releases of ive popular Javascript applications.”

• cites Saboury et al. [46]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

This paper is the first to systematically analyze and characterize code quality issues in ChatGPT-generated code and explore potential solutions to repair them.

“Moreover, it is the irst to systematically analyze and characterize the code quality issues in ChatGPT-generated code and explore potential solutions to repair them.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The incompleteness of the test suite could potentially result in missed program bugs.

“A possible threat arises from the incompleteness of the test suite, which could potentially result in missed program bugs.”

Probably not a substantial claim: may not pass S3 (load-bearing).

To eliminate potential bias in manual analysis and categorization, the authors conducted a sorting discussion among three annotators.

“To eliminate the potential bias, we conducted a sorting discussion among three annotators.”

May combine two claims.

Repeated interactions enable ChatGPT to refine its understanding, adjust based on user feedback, and converge towards more accurate solutions.

“This evidence spotlights how repeated interactions enable ChatGPT to reine its understanding, adjust based on user feedback, and converge towards more accurate solutions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The findings may not be directly applicable to programming languages other than Java and Python.

“Additionally, we focus on Java and Python, two popular programming languages; however, our indings may not be directly applicable to other programming languages.”

Probably not a substantial claim: may not pass S4 (based). May combine two claims.

The original test suite from LeetCode is carefully designed and widely recognized, so the risk of missed bugs is minimal.

“In our experiments, we use the original test suite from LeetCode, which is carefully designed and widely recognized.Thus, we believe this risk is minimal.”

• AssumptionRests on an assumption: The original LeetCode test suite is carefully designed and widely recognized.

May combine two claims.

To minimize the non-deterministic nature of ChatGPT, the authors set the temperature parameter to 0, ensuring consistent outputs for the same input and reducing variability.

“In addition, to minimize the non-deterministic nature of ChatGPT, we set the temperature parameter to 0 in our experiments.This approach ensures that ChatGPT produces consistent outputs for the same input, thereby reducing variability and enhancing the internal validity of our results.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The authors released their analysis and categorization results for public verification.

“We also release our analysis and categorization results for public veriication.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

To mitigate the risk of bugs in their code, the authors carefully checked their code and made it publicly available.

“To mitigate this risk, we have carefully checked our code and made our code publicly available [4].”

• cites [4]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

This paper is the first to conduct a time-sensitive evaluation of ChatGPT on a code generation task.

“To the best of our knowledge, this paper is the irst to conduct a time-sensitive evaluation of ChatGPT on a code generation task.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Xia et al. observed that ChatGPT's performance in generating correct patches improves notably as the number of iterations increases, with significant improvement around three iterations.

“Xia et al. [61] observed that ChatGPT's performance in generating correct patches improves notably as the number of iterations increases, with a signiicant improvement observed around three iterations.”

• cites Xia et al. [61]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Fan et al. examined common bugs in Codex-generated code across 113 programming tasks and tested automated program repair tools on those mistakes.

“Fan et al. [15] further analyze common bugs in code generated by Codex, the backend of Copilot, on 113 programming tasks and benchmark automated program repair tools on ixing the mistakes.”

• cites [15] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Gao et al. demonstrated that incorporating additional examples into the prompt could potentially enhance performance in bug fixing tasks.

“For example, Gao et al. [18] demonstrated that incorporating additional examples into the prompt could potentially enhance performance in bug ixing tasks.”

• cites Gao et al. [18]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S4 (based).

A possible explanation for the lack of overlap is that Python and Java have distinct code styles and common practices.

“The possible reason is that Python and Java have very diferent code styles and common practices.”

May combine two claims.

Nguyen et al. concentrate on correctness of AI-generated code, while Fan et al. aim to benchmark automated program repair tools for fixing mistakes.

“Moreover, Nguyen et al. [37] focus on analyzing the correctness of AI-generated code while Fan et al. [15] target to benchmark automated program repair tools on ixing mistakes.”

• cites [37] sourceNot checked yet

  Not checked in the free run.

• cites [15] sourceNot checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Ahmed et al. showed that in code summarization tasks, augmenting a prompt with explicit semantic facts can significantly improve performance.

“Meanwhile, Ahmed et al. [1] showed that in code summarization tasks, augmenting a prompt with explicit semantic facts can signiicantly improve performance.”

• cites Ahmed et al. [1]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Recent research highlights the crucial role of prompt engineering in enhancing the performance of large language models like ChatGPT for software engineering tasks.

“Recent research highlights the crucial role of prompt engineering in enhancing the performance of large language models (LLMs) like ChatGPT for software engineering tasks [23].”

• cites [23]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

ChatGPT serves a much broader audience than Codex and GitHub Copilot, including novice programmers and non-coders, with about 1.6 billion visits in April 2023.

“Unlike Codex and GitHub Copilot, which target experienced professional developers and programmers, ChatGPT caters to a much wider audience, with approximately 1.6 billion visits in April 2023 [49], including novice programmers and non-coders.”

• cites [49]Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Nguyen et al. assessed GitHub Copilot's performance on 33 programming tasks.

“Nguyen et al. [37] evaluated the performance of Copilot on 33 programming tasks.”

• cites [37] sourceNot checked yet

  Not checked in the free run.

May combine two claims.

The quality of ChatGPT-generated code in both Python and Java is imperfect and can be improved.

“The presence of these issues indicates that the code quality of ChatGPT-generated code for both Python and Java is not perfect and could be improved.”