The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Does ChatGPT Help Novice Programmers Write Better Code? Results From Static Code Analysis
Open accessAbstract
In the realm of AI-enhanced programming education, there is growing interest in using such tools to help students understand good coding principles. This study investigates the impact of ChatGPT on code quality among part-time undergraduate students in introductory Java programming courses, who lack prior Java experience. The source code of 16 students from the control group (without ChatGPT) and 22 students from the treatment group (with ChatGPT) who completed identical programming exercises focused on coding conventions was analyzed. Static code analysis tools assessed adherence to a common coding convention ruleset and calculated cyclomatic and cognitive complexity metrics. The comparative analysis shows that the ChatGPT-assisted group significantly improved code quality, with fewer rule violations and reduced cyclomatic and cognitive complexities. The treatment group adhered more closely to coding standards and produced less complex code. Violations primarily occurred in line length, final parameters, and the extensibility of object-oriented programming (OOP). These findings suggest that ChatGPT can be beneficial in programming education by helping students write cleaner, less complex code and adhere to coding conventions. However, the study’s limitations, such as the small sample size and novice status of participants, call for further research with larger, more diverse populations and different educational contexts.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
McCabe, T. J. (1976) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1976.233837 · Cited by 6,059
ChatGPT for Education and Research: Opportunities, Threats, and Strategies
Rahman, M. M., & Watanobe, Y. (2023) · Applied Sciences · doi:10.3390/app13095783 · Cited by 1,080
Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366 · cited 1×
“In a recent study, Kazemitabaar et al. [24] examined the suitability of OpenAI Codex on supporting introductory programming for novice programmers.”
BackgroundAn empirical evaluation of GitHub copilot's code suggestions
Nguyen, N., & Nadi, S. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3528470 · Cited by 313 · cited 2×
“Nguyen and Nadi [16] assessed GitHub Copilot’s code recommen-dations’ accuracy and comprehensibility across different programming languages using LeetCode questions.”
BackgroundHow ChatGPT Will Change Software Engineering Education
Daun, M., & Brings, J. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588815 · Cited by 222 · cited 2×
“This supports the findings of Jacques [25] and Daun and Brings [26], who found that LLM-based tools can enhance students’ understanding of coding concepts and design principles.”
BackgroundResultOzkaya, I. (2023) · IEEE Software · doi:10.1109/ms.2023.3248401 · Cited by 206 · cited 1×
“It is essential that students possess the ability to evaluate the quality of AI-generated code while also The associate editor coordinating the review of this manuscript and approving it for publication was Alba Amato . enhancing their programming fluency and critical thinking skills [2].”
BackgroundAssessing the quality of GitHub copilot’s code generation
Yetiştiren, B., Ozsoy, I., & Tüzün, E. (2022) · International Conference on Predictive Models in Software Engineering · doi:10.1145/3558489.3559072 · Cited by 193 · cited 1×
“Yetistiren et al. [17] used the HumanEval dataset with 164 coding problems to evaluate GitHub Copilot’s code generation based on validity, correctness, and efficiency.”
MethodCognitive complexity: an overview and evaluation
Campbell, G. A. (2018) · IEEE/ACM International Conference on Technical Debt (TechDebt) · doi:10.1145/3194164.3194186 · Cited by 182
Students’ Experiences of Using ChatGPT in an Undergraduate Programming Course
Haindl, P., & Weinberger, G. (2024) · IEEE Access · doi:10.1109/access.2024.3380909 · Cited by 157 · cited 1×
“It complements our previous qualitative study regarding the suitability of ChatGPT for programming exercises and typical application scenarios from students’ perspective [8].”
ResultSelf-citationRefining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues
Liu, Y., Le-Cong, T., Widyasari, R., Tantithamthavorn, C., Li, L., Le, X.-B. D., & Lo, D. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3643674 · Cited by 149 · cited 2×
“Liu et al. [14] systematically studied the quality of ChatGPT-generated code and found that although it can generate correct code for a significant number of tasks, there are still issues with wrong outputs, compilation or runtime errors, and maintainability.”
BackgroundResultChatGPT in Teaching and Learning: A Systematic Review
Ali, D., Fatemi, Y., Boskabadi, E., Nikfar, M., Ugwuoke, J., & Ali, H. (2024) · Education Sciences · doi:10.3390/educsci14060643 · Cited by 146 · cited 2×
“This cautionary note is consistent with findings by Idrisov and Schlippe [15] and Ali et al. [27], who noted that while AI tools can be beneficial, their impact varies and must be evaluated in broader contexts.”
MethodResultExploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study
Guo, Q., Cao, J., Xie, X., Liu, S., Li, X., Chen, B., & Peng, X. (2023) · arXiv · doi:10.48550/arxiv.2309.08221 · Cited by 140
Code Quality Issues in Student Programs
Keuning, H., Heeren, B. J., & Jeuring, J. T. (2017) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3059009.3059061 · Cited by 138 · cited 1×
“Brown et al. [19], Karnalim et al. [20], and Keuning et al. [21], [22] provide insights into the frequent types of errors and code quality problems students encounter, including neglect of coding conventions and difficulties with program flow and modularization.”
BackgroundHartley, K., Hayak, ., & Ko, U. H. (2024) · Education Sciences · doi:10.3390/educsci14020120 · Cited by 137 · cited 2×
“Hartley et al. [29] assessed ChatGPT’s role in supporting independent coding learning, examining its effectiveness in delivering instructional materials, feedback, and planning.”
BackgroundResultExploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study
Guo, Q., Cao, J., Xie, X., Liu, S., Li, X., Chen, B., & Peng, X. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623306 · Cited by 136
Program Code Generation with Generative AIs
Idrisov, B., & Schlippe, T. (2024) · Algorithms · doi:10.3390/a17020062 · Cited by 118 · cited 2×
“Idrisov and Schlippe [15] evaluated the performance of generative AIs, including ChatGPT, BingAI Chat, GitHub Copilot, StarCoder, Code Llama, CodeWhis-perer, and InstructCodeT5 + , for six LeetCode problems of varying difficulty in Java, Python, and C ++ based on metrics such as correctness,…”
MethodResultNovice Java Programming Mistakes: Large-Scale Data vs. Educator Beliefs
Brown, N. C. C., & Altadmri, A. (2017) · ACM Transactions on Computing Education · doi:10.1145/2994154 · Cited by 107
Investigating novice programming mistakes: educator beliefs vs. student data
Brown, N. C. C., & Altadmri, A. (2014) · Proceedings of the tenth annual conference on International computing education research · doi:10.1145/2632320.2632343 · Cited by 89 · cited 1×
“Previous studies have analyzed the static code quality of student code [3], [4], [5], common mistakes made by students when learning to program [6], and the semantic style of code [7].”
BackgroundChen, J., Zhuo, Z., & Lin, J. (2023) · Sustainability · doi:10.3390/su152416928 · Cited by 84 · cited 2×
“This aligns with findings from Chen et al. [28] and Rahman and Watanobe [30], which highlighted potential drawbacks of over-reliance on AI tools, such as reduced critical thinking and the risk of not developing independent problem-solving skills.”
MethodResult"I know it when I see it" Perceptions of Code Quality: ITiCSE '17 Working Group Report
Börstler, J., Störrle, H., Toll, D., van Assema, J., Duran, R., Hooshangi, S., Jeuring, J. T., Keuning, H., Kleiner, C., & MacKellar, B. (2018) · doi:10.1145/3174781.3174785 · Cited by 72 · cited 2×
“Studies by Izu and Mirolo [4] and Börstler et al. [18] revealed disparities in the definition and prioritization of code quality among novice programmers, with aspects such as performance, structure, conciseness, and comprehensibility being evaluated differently.”
BackgroundMethodA Systematic Mapping Study of Code Quality in Education
Keuning, H., Jeuring, J. T., & Heeren, B. J. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588777 · Cited by 66 · cited 1×
“Brown et al. [19], Karnalim et al. [20], and Keuning et al. [21], [22] provide insights into the frequent types of errors and code quality problems students encounter, including neglect of coding conventions and difficulties with program flow and modularization.”
BackgroundUnderstanding semantic style by analysing student code
Ruvo, G. D., Tempero, E. D., Luxton-Reilly, A., Rowe, G. B., & Giacaman, N. (2018) · Australasian Computing Education Conference · doi:10.1145/3160489.3160500 · Cited by 64
Teaching CS-101 at the Dawn of ChatGPT
Jacques, L. (2023) · ACM Inroads · doi:10.1145/3595634 · Cited by 46 · cited 2×
“This supports the findings of Jacques [25] and Daun and Brings [26], who found that LLM-based tools can enhance students’ understanding of coding concepts and design principles.”
BackgroundResultMeasuring static quality of student code
Breuker, D. M., Derriks, J., & Brunekreef, J. (2011) · doi:10.1145/1999747.1999754 · Cited by 41 · cited 2×
“However, Breuker et al. [3] revealed that despite these efforts, there is no significant difference in code quality between first-year and second-year students, suggesting a plateau in learning that needs to be addressed through more effective teaching strategies.”
BackgroundNovice Use of the Java Programming Language
Brown, N. C. C., Weill-Tessier, P., Sekula, M., Costache, A.-L., & Kölling, M. (2022) · ACM Transactions on Computing Education · doi:10.1145/3551393 · Cited by 32 · cited 1×
“Brown et al. [19], Karnalim et al. [20], and Keuning et al. [21], [22] provide insights into the frequent types of errors and code quality problems students encounter, including neglect of coding conventions and difficulties with program flow and modularization.”
BackgroundDiscriminating Human-authored from ChatGPT-Generated Code Via Discernable Feature Analysis
Li, K., Hong, S., Fu, C., Zhang, Y., & Liu, M. (2023) · 2023 IEEE 34th International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew60843.2023.00059 · Cited by 23 · cited 1×
“In a study conducted by Li et al. [12], the authors examined the differences between human-authored code and code generated by ChatGPT.”
BackgroundÖstlund, L., Wicklund, N., & Glassey, R. J. (2023) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545945.3569829 · Cited by 18 · cited 2×
“Östlund et al. [5] and Brown and Altadmri [23] point to the impact of teaching assistants and changing instructional practices on improvements in code quality.”
BackgroundWork-In-Progress: Code Quality Issues of Computing Undergraduates
Karnalim, O., Simon, & Chivers, W. (2022) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon52537.2022.9766807 · Cited by 11 · cited 1×
“Brown et al. [19], Karnalim et al. [20], and Keuning et al. [21], [22] provide insights into the frequent types of errors and code quality problems students encounter, including neglect of coding conventions and difficulties with program flow and modularization.”
Background
Research works printed but not linked: 9
- 2024, OpenAI
- Izu, Exploring CS1 students notions of code quality, 12
- 2024, Checkstyle—Checkstyle 10.14.1
- 2024, PMD
- 2024, Code Conventions for the Java Programming Language: Contents
- 2024, Checkstyle—Checks
- 2024, Checkstyle—Suns Java Style
- 2024, Checkstyle/src/main/resources/sun.xml At Master
- 2024, Code Quality Tool & Secure Analysis With SonarQube
Cited by
Yang, T.-C., Hsu, Y.-C., & Wu, J.-Y. (2025) · Interactive Learning Environments · doi:10.1080/10494820.2025.2450659 · Cited by 80
“When recent research (Haindl & Weinberger, 2024) highlighted the necessity of considering the learning experience when attempting to explain the impact of ChatGPT on learning programming, the results of our research offers some insight.”
Misiejuk, K., López-Pernas, S., Kaliisa, R., & Saqr, M. (2025) · Journal of Learning Analytics · doi:10.18608/jla.2025.8591 · Cited by 30
PyChatAI: Enhancing Python Programming Skills—An Empirical Study of a Smart Learning System
Alanazi, M., Soh, B., Samra, H., & Li, A. (2025) · Computers · doi:10.3390/computers14050158 · Cited by 15
Qiao, Y., Shihab, M. I. H., & Hundhausen, C. D. (2025) · ACM Transactions on Computing Education · doi:10.1145/3785366 · Cited by 31
“The first two authors first familiarized themselves with the data by reading the papers.”
Stoyanova, D., Stoyanova-Petrova, S. V., Shotarova, S., Lyubomirov, S. Y., & Mileva, N. S. (2025) · Education Sciences · doi:10.3390/educsci16010019 · Cited by 15
Metadata
- Authors
- Haindl, P., & Weinberger, G.
- Affiliations
- University of Applied Sciences St Pölten (AT)
- Venue
- IEEE Access
- Published
- 2024
- Volume
- 12
- Pages
- 114146-114156
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W4401687261
- Cited by
- 45
- References
- 26
- Citations by year
- 2026: 25 · 2025: 20
- FWCI
- 9.82
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- Preprints.org (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, novice programmers, code quality, static code analysis, coding conventions, cyclomatic complexity, cognitive complexity, Java programming, introductory programming courses, part-time undergraduate students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.