The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
The Impact of Large Language Models on Programming Education and Student Learning Outcomes
Open accessAbstract
Recent advancements in Large Language Models (LLMs) like ChatGPT and Copilot have led to their integration into various educational domains, including software development education. Regular use of LLMs in the learning process is still not well-researched; thus, this paper intends to fill this gap. The paper explores the nuanced impact of informal LLM usage on undergraduate students’ learning outcomes in software development education, focusing on React applications. We carefully designed an experiment involving thirty-two participants over ten weeks where we examined unrestricted but not specifically encouraged LLM use and their correlation with student performance. Our results reveal a significant negative correlation between increased LLM reliance for critical thinking-intensive tasks such as code generation and debugging and lower final grades. Furthermore, a downward trend in final grades is observed with increased average LLM use across all tasks. However, the correlation between the use of LLMs for seeking additional explanations and final grades was not as strong, indicating that LLMs may serve better as a supplementary learning tool. These findings highlight the importance of balancing LLM integration with the cultivation of independent problem-solving skills in programming education.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Empirical evidence shows that increased reliance on LLMs, particularly for tasks requiring critical thinking such as code generation and debugging, may negatively affect student performance, while their use for explanations appears more beneficial, indicating a complex and non-trivial role in educational contexts [3].”
From An Integrated AHP–Fuzzy AHP Evaluation Framework for Large Language Models in Software Engineering Education (2026) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature
Lo, C. K. (2023) · Education Sciences · doi:10.3390/educsci13040410 · Cited by 2,024
Grassini, S. (2023) · Education Sciences · doi:10.3390/educsci13070692 · Cited by 1,048
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
ChatGPT in education: Strategies for responsible implementation
Halaweh, M. (2023) · Contemporary Educational Technology · doi:10.30935/cedtech/13036 · Cited by 607
Impact of the Implementation of ChatGPT in Education: A Systematic Review
Montenegro-Rueda, M., Fernández-Cerero, J., Fernández-Batanero, J. M., & López-Meneses, E. (2023) · Computers · doi:10.3390/computers12080153 · Cited by 509
GitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
Fuchs, K. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1166682 · Cited by 353
Denny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287
Using GitHub Copilot to Solve Simple Programming Problems
Wermelinger, M. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569830 · Cited by 191
Cambaz, D., & Zhang, X. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630958 · Cited by 99
Program Code Generation with Generative AIs
Idrisov, B., & Schlippe, T. (2024) · Algorithms · doi:10.3390/a17020062 · Cited by 92
Improving Debugging Skills in the Classroom: The Effects of Teaching a Systematic Debugging Process
Michaeli, T., & Romeike, R. (2019) · Workshop in Primary and Secondary Computing Education · doi:10.1145/3361721.3361724 · Cited by 77
Github copilot in the classroom: learning to code with AI assistance
Puryear, B., & Sprint, G. (2022) · Journal of Computing Sciences in Colleges (JCSC; Formerly: Journal of Computing in Small Colleges) · doi:10.5555/3575618.3575622 · Cited by 76
Tan, K., Pang, T., Fan, C., & Yu, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.03433 · Cited by 56
The Potential Use of ChatGPT for Debugging and Bug Fixing
Haque, M. A., & Li, S. (2023) · EAI Endorsed Transactions on AI and Robotics · doi:10.4108/airo.v2i1.3276 · Cited by 49
From Copilot to Pilot: Towards AI Supported Software DevelopmentPreprint
Pudari, R., & Ernst, N. A. (2023) · arXiv.org · doi:10.48550/arxiv.2303.04142 · Cited by 37
Hliš, T., Četina, L., Beranič, T., & Pavlič, L. (2023) · Applied Sciences · doi:10.3390/app132413061 · Cited by 20
Flagged first, then by citations
Research works printed but not linked: 3
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention Is All You Need. Adv. Neural Inf. Process. Syst., 30.
- Lazuardy, 2022, Modern front end web architectures with react.js and next.js, Int. Res. J. Adv. Eng. Sci., 7, 132
- Carver, M.S., and Risinger, S.C. (1987). Empirical Studies of Programmers: Second Workshop, Ablex Publishing Corp.
Cited by
Sorted by citations
Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies
Deng, R., Jiang, M., Yu, X., Lu, Y., & Liu, S. (2025) · Computers & Education · doi:10.1016/j.compedu.2024.105224 · Cited by 531
Wang, J., & Fan, W. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04787-y · Cited by 322
“Therefore, studies with non-experimental or non-quasi-experimental designs were excluded, such as Jo š t et al. (2024).”
RetractedThe Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196
Prather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3689187.3709614 · Cited by 167
Large Language Models in Computer Science Education: A Systematic Literature Review
Raihan, M. N., Siddiq, M. L., Santos, J. C. S., & Zampieri, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701863 · Cited by 162
“From the perspective of the instructors, studies have indicated that CS in-structors found a negative correlation between the usage of LLMs and students’ grades [59].”
Metadata
- Authors
- Jošt, G., Taneski, V., & Karakatič, S.
- Affiliations
- University of Maribor (SI)
- Corresponding author
- Jošt, Gregor
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2024-05-13
- Volume
- 14
- Issue
- 10
- Pages
- 4115
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W4396896417
- Cited by
- 143
- References
- 16
- Citations by year
- 2026: 66 · 2025: 69 · 2024: 8
- FWCI
- 53.95
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- dCOBISS.SI Digital Repository (publishedVersion, cc-by)
- University of Maribor digital library (University of Maribor) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Slovenian Research and Innovation Agency (ARIS) (P2-0057)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- large language models, programming education, learning outcomes, code generation, debugging, React applications, undergraduate students, critical thinking, problem-solving skills, ChatGPT, Copilot
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
