The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Does ChatGPT Help Novice Programmers Write Better Code? Results From Static Code Analysis
Open accessAbstract
In the realm of AI-enhanced programming education, there is growing interest in using such tools to help students understand good coding principles. This study investigates the impact of ChatGPT on code quality among part-time undergraduate students in introductory Java programming courses, who lack prior Java experience. The source code of 16 students from the control group (without ChatGPT) and 22 students from the treatment group (with ChatGPT) who completed identical programming exercises focused on coding conventions was analyzed. Static code analysis tools assessed adherence to a common coding convention ruleset and calculated cyclomatic and cognitive complexity metrics. The comparative analysis shows that the ChatGPT-assisted group significantly improved code quality, with fewer rule violations and reduced cyclomatic and cognitive complexities. The treatment group adhered more closely to coding standards and produced less complex code. Violations primarily occurred in line length, final parameters, and the extensibility of object-oriented programming (OOP). These findings suggest that ChatGPT can be beneficial in programming education by helping students write cleaner, less complex code and adhere to coding conventions. However, the study’s limitations, such as the small sample size and novice status of participants, call for further research with larger, more diverse populations and different educational contexts.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
McCabe, T. J. (1976) · IEEE Transactions on Software Engineering · doi:10.1109/tse.1976.233837 · Cited by 6,059
ChatGPT for Education and Research: Opportunities, Threats, and Strategies
Rahman, M. M., & Watanobe, Y. (2023) · Applied Sciences · doi:10.3390/app13095783 · Cited by 1,080
Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580919 · Cited by 366 · cited 1×
“In a recent study, Kazemitabaar et al. [24] examined the suitability of OpenAI Codex on supporting introductory programming for novice programmers.”
BackgroundAn empirical evaluation of GitHub copilot's code suggestions
Nguyen, N., & Nadi, S. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3528470 · Cited by 313 · cited 2×
“Nguyen and Nadi [16] assessed GitHub Copilot’s code recommen-dations’ accuracy and comprehensibility across different programming languages using LeetCode questions.”
BackgroundExploring the Potential of ChatGPT in Automated Code Refinement: An Empirical Study
Guo, Q., Cao, J., Xie, X., Liu, S., Li, X., Chen, B., & Peng, X. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3623306 · Cited by 235
Ozkaya, I. (2023) · IEEE Software · doi:10.1109/ms.2023.3248401 · Cited by 206 · cited 1×
“It is essential that students possess the ability to evaluate the quality of AI-generated code while also The associate editor coordinating the review of this manuscript and approving it for publication was Alba Amato . enhancing their programming fluency and critical thinking skills [2].”
BackgroundAssessing the quality of GitHub copilot’s code generation
Yetiştiren, B., Ozsoy, I., & Tüzün, E. (2022) · International Conference on Predictive Models in Software Engineering · doi:10.1145/3558489.3559072 · Cited by 193 · cited 1×
“Yetistiren et al. [17] used the HumanEval dataset with 164 coding problems to evaluate GitHub Copilot’s code generation based on validity, correctness, and efficiency.”
MethodCognitive complexity: an overview and evaluation
Campbell, G. A. (2018) · IEEE/ACM International Conference on Technical Debt (TechDebt) · doi:10.1145/3194164.3194186 · Cited by 182
How ChatGPT Will Change Software Engineering Education
Daun, M., & Brings, J. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587102.3588815 · Cited by 177 · cited 2×
“This supports the findings of Jacques [25] and Daun and Brings [26], who found that LLM-based tools can enhance students’ understanding of coding concepts and design principles.”
BackgroundResultStudents’ Experiences of Using ChatGPT in an Undergraduate Programming Course
Haindl, P., & Weinberger, G. (2024) · IEEE Access · doi:10.1109/access.2024.3380909 · Cited by 157 · cited 1×
“It complements our previous qualitative study regarding the suitability of ChatGPT for programming exercises and typical application scenarios from students’ perspective [8].”
ResultSelf-citation
Research works printed but not linked: 9
- 2024, OpenAI
- Izu, Exploring CS1 students notions of code quality, 12
- 2024, Checkstyle—Checkstyle 10.14.1
- 2024, PMD
- 2024, Code Conventions for the Java Programming Language: Contents
- 2024, Checkstyle—Checks
- 2024, Checkstyle—Suns Java Style
- 2024, Checkstyle/src/main/resources/sun.xml At Master
- 2024, Code Quality Tool & Secure Analysis With SonarQube
Cited by
Yang, T.-C., Hsu, Y.-C., & Wu, J.-Y. (2025) · Interactive Learning Environments · doi:10.1080/10494820.2025.2450659 · Cited by 80
“When recent research (Haindl & Weinberger, 2024) highlighted the necessity of considering the learning experience when attempting to explain the impact of ChatGPT on learning programming, the results of our research offers some insight.”
Misiejuk, K., López-Pernas, S., Kaliisa, R., & Saqr, M. (2025) · Journal of Learning Analytics · doi:10.18608/jla.2025.8591 · Cited by 30
PyChatAI: Enhancing Python Programming Skills—An Empirical Study of a Smart Learning System
Alanazi, M., Soh, B., Samra, H., & Li, A. (2025) · Computers · doi:10.3390/computers14050158 · Cited by 15
Qiao, Y., Shihab, M. I. H., & Hundhausen, C. D. (2025) · ACM Transactions on Computing Education · doi:10.1145/3785366 · Cited by 31
“The first two authors first familiarized themselves with the data by reading the papers.”
Stoyanova, D., Stoyanova-Petrova, S. V., Shotarova, S., Lyubomirov, S. Y., & Mileva, N. S. (2025) · Education Sciences · doi:10.3390/educsci16010019 · Cited by 13
Philbin, C. A., & Sentance, S. (2025) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3754508.3754513 · Cited by 6
Joshi, O., Mundra, K., Yadav, D., & Das, S. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-99261-2_17 · Cited by 5
Comparative Analysis of AI Models for Python Code Generation: A HumanEval Benchmark Study
Bayram, A., Dalveren, G. G. M., & Derawi, M. O. (2025) · Applied Sciences · doi:10.3390/app15189907 · Cited by 9
Misiejuk, K., López‐Pernas, S., Kaliisa, R., & Saqr, M. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5145012 · Cited by 4
Aruleba, K., Oyelere, S. S., Obaido, G., & Sanusi, I. T. (2025) · Conference on United Kingdom & Ireland Computing Education Research (UKICER) · doi:10.1145/3754508.3754517 · Cited by 6
Annuš, N. (2025) · Education Sciences · doi:10.3390/educsci15091152 · Cited by 15
ÖZGÜL, E. (2026) · Kastamonu Eğitim Dergisi · doi:10.24106/kefdergi.1878122 · Cited by 6
Deng, H., Chen, H., & Dong, Y. (2026) · Journal of Educational Computing Research · doi:10.1177/07356331261424211 · Cited by 7
“Consequently, a growing body of empirical studies reports positive impacts of AI chatbots on programming learning outcomes compared to traditional methods (Fauzi et al., 2023; Haindl & Weinberger, 2024; Sun et al., 2024).”
A Systematic Survey on Large Language Models for Static Code Analysis
Salih, H. A. M., & Sarhan, Q. I. (2025) · ARO-The Scientific Journal of Koya University · doi:10.14500/aro.12082 · Cited by 3
Ru, Q., Wang, Y., Huang, C., & Luo, H. (2025) · International Conference on Computer Science and Technologies in Education (CSTE) · doi:10.1109/cste64638.2025.11092232 · Cited by 1
“Over 80% of the research shows that students who interact with LLMs (such as ChatGPT) during or after class, using them for tasks like code generation, debugging, explanations, and generating personalized materials, significantly improve their programming knowledge and skills [26-27].”
Insecure by design? A human-centric security perspective on AI-assisted software development
Glas, M., Nirschl, C., Lanyado, B., & van Niekerk, J. (2026) · Computers & Security · doi:10.1016/j.cose.2026.104842 · Cited by 8
AI-assisted code refactoring: Where can it be helpful and where do humans outperform it?
Ampatzoglou, A., Arvanitou, E.-M., Almpanopoulos, S., Mittas, N., & Chatzigeorgiou, A. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.112862 · Cited by 1
Generative AI in higher education: A systematic review of empirical uses, outcomes, and risks
Hawi, N. S., & Samaha, M. (2026) · International Journal of Educational Research Open · doi:10.1016/j.ijedro.2026.100584 · Cited by 2
A Systematic Mapping Study on Chatbots in Programming Education
Garcia, M., Garcia, R., Parizotto, A., Mendes, A., Valle, P. H. D., Vilela, R., Balancieri, R., & Silva, W. (2025) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2025.12323 · Cited by 2
Wu, M., & Ouyang, F. (2026) · Educational Psychology Review · doi:10.1007/s10648-026-10211-x
“…Choi & Kim, 2025 ; Cirett-Galán et al., 2024; Cubillos et al., 2025 ; Doleck et al., 2025 ; Dong et al., 2025 ; Er et al., 2024 ; Fan et al., 2025 ; Haindl & Weinberger, 2024 ; Huang et al., 2025a ; Huang et al., 2025b ; Huesca et al., 2024; Jalon et al., 2024; Johnson et al., 2024 ; Kosar et al.,…”
Gutek: Intelligent Revision Algorithms
Gałka, Ł. (2025) · IEEE Access · doi:10.1109/access.2025.3570624
“Lastly, the integration of artificial intelligence into software development processes has introduced new possibilities, allowing developers to leverage large language models (LLMs) for code generation and optimization, further streamlining development work-flows [31].”
Harnessing AIGC on Programming Learning in Higher Education: A Systematic Review
Li, M., Hou, D., Zhang, W., Tian, K., & Cheng, W. (2025) · International Conference on Distance Education and Learning (ICDEL) · doi:10.1109/icdel65868.2025.11193517
“Haindl and Weinberger [13] collected students' feedback and evaluations on using ChatGPT through an anonymous online survey, assessing the impact of ChatGPT on learning progress in programming education, its suitability, and exploring students' experiences with using ChatGPT.”
WIP: Guiding, Not Giving: Analyzing the Effectiveness of Guided Problem-Solving in CS1
Deoraj, R., Kurivella, S., & Moraes, M. C. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328186
“Haindl et al. [10] supported this concern in a study of novice programmers, where ChatGPT-assisted students produced cleaner code with fewer rule violations but struggled to adapt AI-generated content to specific assignment contexts.”
Comparative Analysis of Generative AI Performance in University Programming Courses
Suomalainen, E., Vanhala, E., & Ikonen, J. (2026) · Lecture notes in business information processing · doi:10.1007/978-3-032-14518-5_16
Dorokhov, M., Lepp, M., & Bardone, E. (2025) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale66047.2025.11346705
“While large language model (LLM)-based tools are not a threat in themselves and can serve as valuable support for novice programmers [20], it became important to ensure that students fully understood the solutions they submitted.”
Mapeamento Sistemático sobre o Uso de IAs Generativas no Ensino de Programação
Ferreira de Sousa, J., & Silveira, F. R. D. V. (2026) · Revista Eletrônica de Iniciação Científica em Computação · doi:10.5753/reic.2026.7169
From Correctness to Consistency: Redefining Reliability for the Agentware Era
Qin, X., & Gruppi, M. (2026) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1145/3805760.3814904 · Cited by 1
“The proliferation of Large Language Models (LLMs) like GitHub Copilot [10] and ChatGPT [18] is transforming software construction, enabling non-experts to build complex systems via natural language [11, 17].”
Álvarez-Jiménez, M., Pomagualli, A. G. C., Gamboa, D., & Oscullo, D. S. P. (2026) · Computers · doi:10.3390/computers15070466 · Cited by 2
“While GenAI performs adequately in tasks involving minor modifications or pattern-based responses, it shows clear limitations in assessments that demand abstraction, transfer, or real-world application, which are central to higher-order cognitive skills [55–57].”
Camilo, H. T., & Hauck, J. C. R. (2026) · Anais do Workshop sobre Educação em Computação (WEI) · doi:10.5753/wei.2026.20394
Kerimbayev, N., Shadiev, R., & Menlibay, Z. (2026) · Education and Information Technologies · doi:10.1007/s10639-026-14125-7
Katona, J., & Gyönyörű, K. I. K. (2026) · Heliyon · doi:10.1016/j.heliyon.2026.e45361 · Cited by 2
Metadata
- Authors
- Haindl, P., & Weinberger, G.
- Affiliations
- University of Applied Sciences St Pölten (AT)
- Venue
- IEEE Access
- Published
- 2024
- Volume
- 12
- Pages
- 114146-114156
- ISSN
- 2169-3536 (ISSN-L)
- OpenAlex
- W4401687261
- Cited by
- 45
- References
- 26
- Citations by year
- 2026: 25 · 2025: 20
- FWCI
- 9.82
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- IEEE Access (publishedVersion, cc-by)
- Preprints.org (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- ChatGPT, novice programmers, code quality, static code analysis, coding conventions, cyclomatic complexity, cognitive complexity, Java programming, introductory programming courses, part-time undergraduate students
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.