The DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index; none of the 27 sources its claims cite is checked for support yet.
Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
Open accessAbstract
Since its introduction in November 2022, ChatGPT has rapidly gained popularity due to its remarkable ability in language understanding and human-like responses. ChatGPT, based on GPT-3.5 architecture, has shown great promise for revolutionizing various research fields, including code generation. However, the reliability and quality of code generated by ChatGPT remain unexplored, raising concerns about potential risks associated with the widespread use of ChatGPT-driven code generation. In this article, we systematically study the quality of 4,066 ChatGPT-generated programs of code implemented in two popular programming languages, i.e., Java and Python, for 2,033 programming tasks. The goal of this work is threefold. First, we analyze the correctness of ChatGPT on code generation tasks and uncover the factors that influence its effectiveness, including task difficulty, programming language, time that tasks are introduced, and program size. Second, we identify and characterize potential issues with the quality of ChatGPT-generated code. Last, we provide insights into how these issues can be mitigated. Experiments highlight that out of 4,066 programs generated by ChatGPT, 2,756 programs are deemed correct, 1,082 programs provide wrong outputs, and 177 programs contain compilation or runtime errors. Additionally, we further analyze other characteristics of the generated code through static analysis tools, such as code style and maintainability, and find that 1,930 ChatGPT-generated code snippets suffer from maintainability issues. Subsequently, we investigate ChatGPT’s self-repairing ability and its interaction with static analysis tools to fix the errors uncovered in the previous step. Experiments suggest that ChatGPT can partially address these challenges, improving code quality by more than 20%, but there are still limitations and opportunities for improvement. Overall, our study provides valuable insights into the current limitations of ChatGPT and offers a roadmap for future research and development efforts to enhance the code generation capabilities of artificial intelligence models such as ChatGPT.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This paper empirically evaluates ChatGPT-generated solutions for 2,033 LeetCode tasks in Java and Python, finding that many correct programs still carry quality problems. It also tests whether feedback prompts can help ChatGPT repair those issues, with partial success.
Key findings
- Of 4,066 generated programs, 2,756 passed all tests, 1,082 produced wrong outputs, and 177 had compilation or runtime errors. Overall pass@1 was 0.664 for Python and 0.691 for Java, with performance dropping on harder tasks and newer tasks.
- Code style and maintainability issues were common: 1,930 of 4,066 programs (47%) had such issues, including 1,243 of 2,756 passing programs (45%). Wrong outputs were also frequent, affecting 1,082 programs (27%).
- ChatGPT could partially self-repair issues, with reported fix rates from about 20% to 60%. Feedback containing static-analysis and runtime-error details helped more for style and maintainability, while simple feedback sometimes worked better for other issue types.
- Iterative repair improved code quality over rounds, but new issues were sometimes introduced in the fixed versions.
Methods
- The authors built a benchmark of 2,033 LeetCode tasks (501 easy, 1,064 medium, 468 hard) and prompted ChatGPT-March-23 with zero-shot prompts, temperature 0, one run per task, generating Java and Python code.
- Correctness was measured with pass@1 against LeetCode test suites; static analysis used PMD and Checkstyle for Java, and Pylint and Flake8 for Python.
- Issues were categorized through open card sorting on a sample of 154 tasks, then quantified. Repair experiments compared simple feedback against feedback augmented with static-analysis and runtime-error information, including iterative rounds on 402 tasks released after 2022.
Limitations
- The paper states external-validity limits: the dataset comes only from LeetCode and covers only Java and Python, so findings may not generalize to other tasks, languages, or project types.
- The paper states internal-validity concerns around possible bugs in the authors’ code and manual categorization bias, mitigated by public code, three annotators, and temperature 0.
- The paper states construct-validity concerns about pass@1 relying on LeetCode test suites and prompt variability; it also notes possible data leakage or memorization for older tasks. Evident from the text: only one ChatGPT version and one generation per task were used, which limits claims about other models or repeated sampling.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
77 Own result · 25 Own interpretation · 21 Not checked yet · 9 No basis given · 6 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“out of 4,066 programs generated by ChatGPT, 2,756 programs are deemed correct”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“1,082 programs provide wrong outputs”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“177 programs contain compilation or runtime errors”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“1,930 ChatGPT-generated code snippets sufer from maintainability issues”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT's performance drops up to ive times on new programming tasks introduced after January 2022”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT can partially ix code quality issues in the generated code with feedback from static analysis tools and runtime errors”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 1 INTRODUCTION
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Code quality issues commonly happen in both code that pass or failed test cases”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 5 RQ2: BUGS AND ISSUES 5.1 Static Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For Python, the pass@1 rate starts at 0.872 for code snippets with less than 10 lines and gradually decreases to 0.265 for code snippets with more than 50 lines”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 4 RQ1: PERFORMANCE
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT performs better on easy tasks than on medium and hard tasks”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 4 RQ1: PERFORMANCE
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: LeetCode's difficulty labels accurately reflect the difficulty of programming tasks.
In the paper
“their diference in performance is not signiicant with a p-value of at least 0.53 and an efect size value less than 0.02”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 4 RQ1: PERFORMANCE
What the sources say
- Not checked yetcites [35] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: LeetCode's difficulty labels accurately reflect the difficulty of programming tasks.
- It may state two separate claims as one. 39
- Probably not a substantial claim. 35
- The quote only partly states this claim. 2
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“66% and 69% of Python and Java programs generated by ChatGPT are functionally-correct programs, i.e., programs that pass all given test cases”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Introduce a new time-sensitive dataset comprising 2033 programming tasks and 4066 ChatGPT-generated code snippets implemented in two popular programming languages: Java and Python”
- Also stated in: § 3.2 Constructing Benchmark Dataset · plain
“we successfully collected 2,033 programming tasks from LeetCode”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § 3.2 Constructing Benchmark Dataset · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“out of the 2,033 tasks in our dataset, we found that 501 were classiied as easy, 1,064 as medium, and 468 as hard”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 3.2 Constructing Benchmark Dataset
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Rests on an assumption: LeetCode's difficulty labels accurately reflect the difficulty of programming tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“among ChatGPT-generated code that passed the test cases, 53% of the Java code and 37% of the Python code exhibited code style and maintainability issues”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“ChatGPT can solve more than half of the hard-level code tasks before June 2021, but its performance reduces drastically to nearly 0.1 for the subsequent time periods”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 4 RQ1: PERFORMANCE
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“for Java, the model attains a pass@1 accuracy of 0.860 for easy tasks, 0.710 for medium tasks, and 0.468 for hard tasks”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 4 RQ1: PERFORMANCE
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: LeetCode's difficulty labels accurately reflect the difficulty of programming tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the percentage of clean Java code decreases from 54% for easy tasks to 45% for medium tasks, and further drops to 33% for hard tasks”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 5 RQ2: BUGS AND ISSUES 5.1 Static Analysis
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one. The quote only partly states this claim.
- Rests on an assumption: LeetCode's difficulty labels accurately reflect the difficulty of programming tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the majority of generated code snippets falling into the 10-20 lines range for Python and the 20-30 lines range for Java”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 4 RQ1: PERFORMANCE
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“while most tasks are from before 2021, there are still more than 400 test cases for evaluating ChatGPT's code generation capabilities”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 3.2 Constructing Benchmark Dataset
plainn = 400- Asserted: fails — The authors may not put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The authors may not put it forward as true. The paper’s conclusions may not depend on it. The paper may give no reason why it holds. The quote only partly states this claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the Mann-Whitney U test conirms the signiicance of the diferences (p-value < 0.01) with a small to medium efect size”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 4 RQ1: PERFORMANCE
boostedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, 7 out of 501 generated code snippets (i.e., 1%) for easy Python tasks exhibit compilation and runtime errors, while the number of execution errors increases to 46 out of 468 (i.e., 10%) for hard Python tasks.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 5.3.1 Overall Analysis.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, although our previous experiments indicated that simple feedback may be better for some issues, iterative feedback with static analysis and runtime errors performs much better than simple feedback as the iteration rounds progress.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 6.2 Iterative Repairing
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, Wrong Output and Code Style & Maintainability issues are more prevalent and tend to be the most common challenges faced by the generated code.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 5.3.1 Overall Analysis.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also notice that Illegal Index errors are quite prevalent in both languages, particularly in Java.In fact, out of the 97 compilation and runtime errors encountered in Java, 45 of them (46.4%) are attributed to using an invalid index.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > Finding 5:
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Finding 9: Prompts with detailed feedback can efectively assist ChatGPT in self-repairing code quality issues, whereas ambiguous feedback may have a negative impact on ChatGPT's performance.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > Feedback with Static Analysis and Runtime Errors:
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“we observed that feedback with static analysis and runtime errors is more efective in ixing code style and maintainability while simple feedback performs better in the remaining quality issues in both Java and Python.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > Feedback with Static Analysis and Runtime Errors:
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, the ChatGPT-generated code produces an output of 25.This error occurs because the loop iterates one step more than necessary, causing the last step's cost to be included in the calculation even when it should not be.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 5.2 Open Card Sorting Discussion
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Regarding the 1,310 code that failed test cases, 14% encounter compilation and runtime errors, 83% exhibit wrong outputs, 4% exhibit performance or eiciency issues, and notably, 52% exhibited issues related to code style and maintainability on top”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 5.3.1 Overall Analysis.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Table 4 presents a comparison of common compilation and runtime errors error categories in Java and Python programs (i.e., 80 Python and 97 Java programs with the errors).”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 5.3.2 Analysis on Compilation & Runtime Errors.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“when using ChatGPT to solve LeetCode Problem 1982, titled “Find Array Given Subset Sums”, the compiler outputs a łTIMEOUT” error due to ineicient loop control within the generated code.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 5.2 Open Card Sorting Discussion
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, our results show performance stabilizing in later rounds, indicating potential upper bounds to iterative gains.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.1.2 Iterative Interactions.
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, static analysis feedback works better for code style and maintainability issues, while simple feedback is more efective for addressing execution errors and wrong outputs.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.2 Lessons Learned
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results suggest that the efectiveness of ChatGPT's self-mitigating capabilities depends on the type of feedback provided, the programming language, and the speciic code quality issue.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.2 Lessons Learned
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“However, for errors related to wrong outputs or eiciency, simpler prompts worked better in single-round feedback, likely because the prompts collected from the compiler feedback might be ambiguous or unhelpful.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.1.1 Prompt Optimization.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Static feedback and runtime errors, on the other hand, provide detailed information, i.e., leading to a correct ix in line 41 which change the for-loop to foreach-loop.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > Additional information
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our indings demonstrate that while ChatGPT can generate functional code for various programming tasks, the generated code often sufers from quality issues, such as compilation and errors, wrong outputs, maintainability problems, and performance ineiciencies.”
- Also stated in: § Additional information · hedged
“However, the generated code sometimes sufers from diferent code quality issues, such as execution errors, wrong outputs, and maintainability problems.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 9 CONCLUSION
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Additional information · hedged
In the paper
“we used the ChatGPT-March-23 version [41], which was trained on data up to 2021”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 3.3 The ChatGPT Model
What the sources say
- Not checked yetcites [41]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This discrepancy highlights the diferences in verbosity and structure between the two programming languages”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 4 RQ1: PERFORMANCE
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Code quality issues in ChatGPT-generated code, if not properly identiied and addressed, may unduly afect code comprehension, introduce bugs, or create security vulnerabilities in users' projects [47].”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 1 INTRODUCTION
What the sources say
- Not checked yetcites [47]
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This could be due to the increased complexity, and the greater number of potential interactions between code elements as the code size grows”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 4 RQ1: PERFORMANCE
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“we choose to ignore messages related to style issues such as whitespace, newline, and invalid naming conventions, which is consistent with the approach taken in prior work [48]”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 5 RQ2: BUGS AND ISSUES 5.1 Static Analysis
What the sources say
- Not checked yetcites [48] source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To mitigate the randomness of ChatGPT, we make ChatGPT deterministic by setting the temperature to 0 and running the model once for each task, using the irst generated output for evaluation.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 3.3 The ChatGPT Model
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is because simple feedback is quite general so ChatGPT is sometimes misguided, resulting in introducing new problems though plausibly solving the original issues.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > Feedback with Static Analysis and Runtime Errors:
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“From the table, it is evident that Compilation & Runtime Errors and Performance & Eiciency issues are relatively less frequent, indicating that ChatGPT is generally successful in generating syntactically correct and eicient code.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 5.3.1 Overall Analysis.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This is possibly due to the introduction of noise by static analysis and runtime error feedback, which can confuse ChatGPT and lead to incorrect patches.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > Feedback with Static Analysis and Runtime Errors:
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“These observations can be explained by the probabilistic nature of the ChatGPT model, which predicts subsequent tokens based on preceding ones.This nature enables ChatGPT to understand the semantics of common programs that appear frequently in the training set.However, the model captures the semantics implicitly from the training data, leading to misunderstandings of program semantics and subs”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 5.3.2 Analysis on Compilation & Runtime Errors.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: ChatGPT is a probabilistic model that predicts subsequent tokens based on preceding ones.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This observation could be due to Python's dynamic typing system, which allows for more lexibility in variable types, but can also lead to unexpected type-related issues at runtime.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > Finding 5:
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Rests on an assumption: Python's dynamic typing system allows more flexibility in variable types but can lead to unexpected type-related issues at runtime.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To make the experiment more rigorous, we only use the 402 new test examples from LeetCode that were published in the year 2022 (after GPT-3.5'spre-training knowledge cutof).”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 6.2 Iterative Repairing
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Rests on an assumption: GPT-3.5's pretraining knowledge cutoff occurred before the year 2022.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our study is based on a dataset of 2,033 programming tasks from LeetCode, which may not represent all possible code generation tasks encountered in real-world software development.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.3.1 External validity.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“There is a large number of studies investigating code quality issues from humanwritten code.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 8.2 Code uality Issues
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Keuning et al. [26] investigate code quality issues in student programs.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 8.2 Code uality Issues
What the sources say
- Not checked yetcites Keuning et al. [26]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“To address this concern, we followed the similar methodology used by Fan et al. [15] and Tian et al. [55], which ensures that our results are reliable by using a consistent set of prompts across diferent tasks.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.3.3 Construct validity.
What the sources say
- Not checked yetcites Fan et al. [15]
- Not checked yetcites Tian et al. [55]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
In the paper
“Kochhar et al. [27] conducted a large-scale empirical investigation on the code quality of open-source projects implemented in 17 programming languages.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 8.2 Code uality Issues
What the sources say
- Not checked yetcites Kochhar et al. [27]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“However, the quality of the ChatGPT-generated code is the critical concern for top software companies when deciding whether to adopt it in practice or not [10,21,44].”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 8.1 Large Language Model for Code Generation
What the sources say
- Not checked yetcites [10]
- Not checked yetcites [21]
- Not checked yetcites [44]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Saboury et al. [46] empirically investigate code smells in 537 releases of ive popular Javascript applications.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 8.2 Code uality Issues
What the sources say
- Not checked yetcites Saboury et al. [46]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, it is the irst to systematically analyze and characterize the code quality issues in ChatGPT-generated code and explore potential solutions to repair them.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 8.1 Large Language Model for Code Generation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A possible threat arises from the incompleteness of the test suite, which could potentially result in missed program bugs.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.3.3 Construct validity.
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To eliminate the potential bias, we conducted a sorting discussion among three annotators.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.3.2 Internal validity.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This evidence spotlights how repeated interactions enable ChatGPT to reine its understanding, adjust based on user feedback, and converge towards more accurate solutions.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.1.2 Iterative Interactions.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additionally, we focus on Java and Python, two popular programming languages; however, our indings may not be directly applicable to other programming languages.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.3.1 External validity.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In our experiments, we use the original test suite from LeetCode, which is carefully designed and widely recognized.Thus, we believe this risk is minimal.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.3.3 Construct validity.
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds. It may state two separate claims as one.
- Rests on an assumption: The original LeetCode test suite is carefully designed and widely recognized.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“In addition, to minimize the non-deterministic nature of ChatGPT, we set the temperature parameter to 0 in our experiments.This approach ensures that ChatGPT produces consistent outputs for the same input, thereby reducing variability and enhancing the internal validity of our results.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.3.2 Internal validity.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We also release our analysis and categorization results for public veriication.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.3.2 Internal validity.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“To mitigate this risk, we have carefully checked our code and made our code publicly available [4].”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.3.2 Internal validity.
What the sources say
- Not checked yetcites [4]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“To the best of our knowledge, this paper is the irst to conduct a time-sensitive evaluation of ChatGPT on a code generation task.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 8.1 Large Language Model for Code Generation
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
In the paper
“Xia et al. [61] observed that ChatGPT's performance in generating correct patches improves notably as the number of iterations increases, with a signiicant improvement observed around three iterations.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.1.2 Iterative Interactions.
What the sources say
- Not checked yetcites Xia et al. [61]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Fan et al. [15] further analyze common bugs in code generated by Codex, the backend of Copilot, on 113 programming tasks and benchmark automated program repair tools on ixing the mistakes.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 8.3 Code uality Issues of AI-generated Code
What the sources say
- Not checked yetcites [15] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“For example, Gao et al. [18] demonstrated that incorporating additional examples into the prompt could potentially enhance performance in bug ixing tasks.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.1.1 Prompt Optimization.
What the sources say
- Not checked yetcites Gao et al. [18]
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The possible reason is that Python and Java have very diferent code styles and common practices.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > Additional information
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. The paper may give no reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“Moreover, Nguyen et al. [37] focus on analyzing the correctness of AI-generated code while Fan et al. [15] target to benchmark automated program repair tools on ixing mistakes.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 8.3 Code uality Issues of AI-generated Code
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“Meanwhile, Ahmed et al. [1] showed that in code summarization tasks, augmenting a prompt with explicit semantic facts can signiicantly improve performance.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.1.1 Prompt Optimization.
What the sources say
- Not checked yetcites Ahmed et al. [1]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Recent research highlights the crucial role of prompt engineering in enhancing the performance of large language models (LLMs) like ChatGPT for software engineering tasks [23].”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 7.1.1 Prompt Optimization.
What the sources say
- Not checked yetcites [23]
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Unlike Codex and GitHub Copilot, which target experienced professional developers and programmers, ChatGPT caters to a much wider audience, with approximately 1.6 billion visits in April 2023 [49], including novice programmers and non-coders.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 8.3 Code uality Issues of AI-generated Code
What the sources say
- Not checked yetcites [49]
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Nguyen et al. [37] evaluated the performance of Copilot on 33 programming tasks.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 8.3 Code uality Issues of AI-generated Code
What the sources say
- Not checked yetcites [37] source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The presence of these issues indicates that the code quality of ChatGPT-generated code for both Python and Java is not perfect and could be improved.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > Additional information
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
No prior study formally investigates the reliability and quality of ChatGPT-generated code.
“there exists no study that formally investigates the reliability and quality of code generated by ChatGPT”
The paper relies on this without support.
This work is the first formal study of the reliability and quality of ChatGPT-generated code.
“we are the irst to formally study the reliability and quality of ChatGPT-generated code”
The paper relies on this without support.
LeetCode's difficulty labels accurately reflect the difficulty of programming tasks.
“501 classiied as easy, 1,064 as medium, and 468 as hard”
The paper relies on this without support.
Code quality issues are the most important concern because a single quality issue can lead to monetary and reputation costs.
“Code quality issues are the most important concern, as one quality issue (aka.software defect) could lead to monetary and reputation costs.”
The paper relies on this without support.
ChatGPT is a probabilistic model that predicts subsequent tokens based on preceding ones.
“These observations can be explained by the probabilistic nature of the ChatGPT model, which predicts subsequent tokens based on preceding ones.”
The paper relies on this without support.
Python's dynamic typing system allows more flexibility in variable types but can lead to unexpected type-related issues at runtime.
“This observation could be due to Python's dynamic typing system, which allows for more lexibility in variable types, but can also lead to unexpected type-related issues at runtime.”
The paper relies on this without support.
The original LeetCode test suite is carefully designed and widely recognized.
“In our experiments, we use the original test suite from LeetCode, which is carefully designed and widely recognized.”
The paper relies on this without support.
GPT-3.5's pretraining knowledge cutoff occurred before the year 2022.
“we only use the 402 new test examples from LeetCode that were published in the year 2022 (after GPT-3.5'spre-training knowledge cutof).”
The paper relies on this without support.
ChatGPT can learn from user interactions and improve its outputs based on feedback.
“Unlike traditional code generation tools, ChatGPT has the potential to learn from user interactions and reine its outputs based on the feedback it receives.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“ChatGPT is the most widely adopted LLM in code generation in the literature [15, 16, 23, 40, 42, 63, 70]”
From An Empirical Study of the Non-Determinism of ChatGPT in Code Generation (2025) · cites this paper as [42]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > Finding 1:
“The performance of ChatGPT is signiicantly and substantially afected by task diiculty, time that tasks are introduced, program size and programming languages.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 4 RQ1: PERFORMANCE
“their diference in performance is not signiicant with a p-value of at least 0.53 and an efect size value less than 0.02 [35].”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 6.2 Iterative Repairing
“we only use the 402 new test examples from LeetCode that were published in the year 2022”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 6.2 Iterative Repairing
“Figures 10, 11, and 12 present our iterative repairing results on 402 new LeetCode tasks released after 2022.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“The expected output is 15, achieved by climbing the steps with costs of 10 and 20, while skipping the step with a cost of 15.”
§ Refining ChatGPT-Generated Code: Characterizing and Mitigating Code uality Issues > 5.2 Open Card Sorting Discussion
Reported 15 · recomputed 30
Citations
Cites
On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other
Mann, H. B., & Whitney, D. R. (1947) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177730491 · Cited by 14,320
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.685 · Cited by 1,495
Wang, Y., Wang, W., Joty, S., & Hoi, S. C. H. (2021) · arXiv · doi:10.48550/arxiv.2109.00859 · Cited by 1,495
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043
Evaluating large language models trained on code
Chen, M. (2024) · TIB Data Manager · doi:10.57702/wbv3e61b · Cited by 737
Learning to summarize from human feedback
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020) · arXiv · doi:10.48550/arxiv.2009.01325 · Cited by 550
Cliff´s Delta Calculator: A non-parametric effect size program for two groups of observations
Macbeth, G., Razumiejczyk, E., & Ledesma, R. D. (2010) · Universitas Psychologica · doi:10.11144/javeriana.upsy10-2.cdcp · Cited by 442
Research works printed but not linked: 9
- Amazon. 2023. Amazon CodeWhisperer. Retrieved from https://aws.amazon.com/codewhisperer/
- Tom Copeland. 2005. PMD applied. Vol. 10. Centennial Books San Francisco.
- Ian Cordasco and Tarek Ziade. 2010. Flake8: Your tool for style guide enforcement. Programa De Computador (2010).
- Shuzheng Gao, Xin-Cheng Wen, Cuiyun Gao, Wenxuan Wang, and Michael R. Lyu. 2023. Constructing effective in-context demonstration for code intelligence tasks: An empirical study. arXiv preprint arXiv:2304.07575 (2023). · arXiv
- LeetCode. 2023. 1093. Statistics from a Large Sample. Retrieved from https://leetcode.com/problems/statistics-from-a-large-sample/description/
- Carlini Nicholas, Ippolito Daphne, Jagielski Matthew, Lee Katherine, Tramer Florian, and Zhang Chiyuan. 2023. Quantifying memorization across neural language models. In 11th International Conference on Learning Representations. 70–80.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI Blog 1, 8 (2019), 9.
- SimilarWeb. 2023. ChatGPT’s Traffic Overview. Retrieved from https://www.similarweb.com/website/chat.openai.com
- Donna Spencer. 2009. Card Sorting: Designing Usable Categories. Rosenfeld Media.
Cited by
Sorted by citations
An Empirical Study of the Non-Determinism of ChatGPT in Code Generation
Ouyang, S., Zhang, J. M., Harman, M., & Wang, M. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3697010 · Cited by 258
“Section 2 introduces the main procedure of our study.”
No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPT
Liu, Z., Tang, Y., Luo, X., Zhou, Y., & Zhang, L. F. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3392499 · Cited by 128
Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Study
Fu, Y., Liang, P., Tahir, A., Li, Z., Shahin, M., Yu, J., & Chen, J. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3716848 · Cited by 205
Bugs in large language models generated code: an empirical study
Tambon, F., Dakhel, A. M., Nikanjam, A., Khomh, F., Desmarais, M. C., & Antoniol, G. (2025) · Empirical Software Engineering · doi:10.1007/s10664-025-10614-4 · Cited by 70
On the use of large language models in model-driven engineering
Rocco, J. D., Ruscio, D. D., Sipio, C. D., Nguyen, P. T., & Rubei, R. (2025) · Software & Systems Modeling · doi:10.1007/s10270-025-01263-8 · Cited by 73
Metadata
- Authors
- Liu, Y., Le-Cong, T., Widyasari, R., Tantithamthavorn, C., Li, L., Le, X.-B. D., & Lo, D.
- Affiliations
- Monash University (AU)
- The University of Melbourne (AU)
- Singapore Management University (SG)
- Beihang University (CN)
- ORCID
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2024-06-04
- Volume
- 33
- Issue
- 5
- Pages
- 1-26
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4391282616
- Cited by
- 149
- References
- 43
- Citations by year
- 2026: 36 · 2025: 75 · 2024: 37
- FWCI
- 55.65
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Transactions on Software Engineering and Methodology (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Singapore Management University Institutional Knowledge (InK) (Singapore Management University) (submittedVersion, cc-by-nc-nd)
- Adelaide Research & Scholarship (AR&S) (University of Adelaide) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Research Foundation (NRF-NRFI08-2022-0002)
- Australian Research Council’s Discovery Early Career Researcher Award (DE200100941)
- Australian Government through the Australian Research Council’s Discovery Early Career Researcher Award (DE220101057)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- ChatGPT, large language models, code generation, code quality, program correctness, static analysis, software maintainability, code style, self-healing, Java, Python, compilation errors, runtime errors, task difficulty
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
