The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
The Impact of Large Language Models on Programming Education and Student Learning Outcomes
Open accessAbstract
Recent advancements in Large Language Models (LLMs) like ChatGPT and Copilot have led to their integration into various educational domains, including software development education. Regular use of LLMs in the learning process is still not well-researched; thus, this paper intends to fill this gap. The paper explores the nuanced impact of informal LLM usage on undergraduate students’ learning outcomes in software development education, focusing on React applications. We carefully designed an experiment involving thirty-two participants over ten weeks where we examined unrestricted but not specifically encouraged LLM use and their correlation with student performance. Our results reveal a significant negative correlation between increased LLM reliance for critical thinking-intensive tasks such as code generation and debugging and lower final grades. Furthermore, a downward trend in final grades is observed with increased average LLM use across all tasks. However, the correlation between the use of LLMs for seeking additional explanations and final grades was not as strong, indicating that LLMs may serve better as a supplementary learning tool. These findings highlight the importance of balancing LLM integration with the cultivation of independent problem-solving skills in programming education.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Empirical evidence shows that increased reliance on LLMs, particularly for tasks requiring critical thinking such as code generation and debugging, may negatively affect student performance, while their use for explanations appears more beneficial, indicating a complex and non-trivial role in educational contexts [3].”
From An Integrated AHP–Fuzzy AHP Evaluation Framework for Large Language Models in Software Engineering Education (2026) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature
Lo, C. K. (2023) · Education Sciences · doi:10.3390/educsci13040410 · Cited by 2,024
Grassini, S. (2023) · Education Sciences · doi:10.3390/educsci13070692 · Cited by 1,048
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
ChatGPT in education: Strategies for responsible implementation
Halaweh, M. (2023) · Contemporary Educational Technology · doi:10.30935/cedtech/13036 · Cited by 607
Impact of the Implementation of ChatGPT in Education: A Systematic Review
Montenegro-Rueda, M., Fernández-Cerero, J., Fernández-Batanero, J. M., & López-Meneses, E. (2023) · Computers · doi:10.3390/computers12080153 · Cited by 509
GitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
Fuchs, K. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1166682 · Cited by 353
Denny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287
Using GitHub Copilot to Solve Simple Programming Problems
Wermelinger, M. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569830 · Cited by 191
Flagged first, then by citations
Research works printed but not linked: 3
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention Is All You Need. Adv. Neural Inf. Process. Syst., 30.
- Lazuardy, 2022, Modern front end web architectures with react.js and next.js, Int. Res. J. Adv. Eng. Sci., 7, 132
- Carver, M.S., and Risinger, S.C. (1987). Empirical Studies of Programmers: Second Workshop, Ablex Publishing Corp.
Cited by
Sorted by citations
Items 101–147 of 147
Stamatakis, M., Gavali, O., Berger, J., Wartena, C., Hoppe, A., & Ewerth, R. (2026) · arXiv.org · doi:10.48550/arxiv.2606.15919
“For instance, research in programming education suggests that extensive reliance on LLM-based assistance may negatively affect final grades and decrease independent problem-solving abilities of undergraduate students [10].”
GAI-supported skill development in design engineer scheduling optimization
Matsuno, K., Lin, K.-Y., Liyanaarachchi, N., & Weng, J. (2026) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-026-18424-8
Fujs, D., Vavpotič, D., & Poženel, M. (2026) · Automated Software Engineering · doi:10.1007/s10515-026-00633-w
“A similar pattern appears in recent work by Jošt et al. (2024), who found that students relying heavily on LLMs for difficult programming tasks, such as generating or debugging code, tend to perform worse overall.”
Understanding chatbot-assisted collaborative learning among female undergraduate students
Kuhail, M. A., Shuhaiber, A., Salman, S., & Alturki, N. M. (2026) · Discover Artificial Intelligence · doi:10.1007/s44163-026-01416-8
“Additionally, LLMs, trained on extensive datasets, offer students immediate access to a wealth of relevant information [49], effectively addressing the issue of resource overload that often accompanies online programming learning.”
Patterns of Learner-AI Interaction and Academic Performance in an Object-Oriented Programming CoursePreprint
Lepp, M. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.24755
Belščak, J. (2026) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro70003.2026.11591964
Niemann, S., & Schmid, U. (2026) · International Conference on Information Technology Based Higher Education and Training (ITHET) · doi:10.1109/ithet69978.2026.11585137
“What started out as a fear of students losing or never learning crucial skills and the possibility of cheating [1; 2; 3] soon turned into a broader discussion, including the potential that genAI might bring.”
InfluentialIntegrasi Moodle API dan LLM dalam Otomasi Monitoring Capaian Pembelajaran Lulusan Berbasis OBE
Sulistyo, W. Y., & Supriyanto, J. (2026) · JITU : Journal Informatic Technology And Communication · doi:10.36596/jitu.v10i1.2307
“Selain itu, penggunaan pustaka visualisasi seperti Mermaid.js diperlukan untuk mentransformasi struktur pemetaan yang kompleks menjadi representasi grafis yang denotatif dan mudah dipahami [17][18][19].”
Khan, N. A., Rizwan, M., & Khan, N. (2026) · Responsible Education Learning and Teaching in Emerging Economies · doi:10.26710/relate.v8i1.4009
“Students' final grades were determined to have a significant negative correlation with greater LLM use of critical thinking-oriented tasks, such as code generation and debugging (Jošt et al., 2024).”
Erazo-Arteaga, V. A., Palacios, S. M. V., & Rúales, M. C. (2026) · Revista de Ciencias Sociales · doi:10.31876/rcs.v32i2.45578 · Cited by 3
Basbas, M., Djamaa, B., & Senouci, M. R. (2026) · Security and Privacy · doi:10.1002/spy2.70223 · Cited by 2
Evaluating LLM-Generated Instructional Content for Parallel Computing Education
Yi, X., & Yan, Y. (2026) · IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum · doi:10.1109/ipdpsw71298.2026.00128
Cheng, Y., Yang, L., & Guile, D. (2026) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon67543.2026.11574423
Impact of AI Explanation Styles on IT Students’ Debugging Accuracy, Confidence, and Reliance
Dzvapatsva, G. P., Ncube, P. D. N., Chinhamo, E., & Matobobo, C. (2026) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon67543.2026.11574352
“Recent advances in large language models (LLMs) and AI-driven code assistants have transformed programming education, offering new avenues for scaffolding, real-time feedback, and iterative problem solving [9], [10], [11].”
Sun, H. (2026) · International Conference on Educational Digitalization and Computer Science in the Guangdong-Hong Kong-Macao Greater Bay Area · doi:10.1145/3828183.3828468
“Jošt et al. found that large language models can influence programming learning outcomes and reshape students’ working processes in software-development education [ 8 ] .”
Gardella, N., Prather, J., Leinonen, J., Denny, P., Pettit, R., & Riggs, S. (2026) · International Computing Education Research Workshop · doi:10.48550/arxiv.2604.18538
Prompting Without Principles: Are Students Transferring Software Engineering Knowledge to LLM Use?
Sousa, L., Ozkaya, I., Ivers, J., Cywinska, C., Xie, B., Kostial, M., Dey, T., & Edman, R. (2026) · SEET@ICSE · doi:10.1145/3786580.3786990
Knowledge Markers: An AI-Agnostic Concept for the Design of Programming Courses
Mayr, C. M. (2026) · arXiv.org · doi:10.48550/arxiv.2604.06331
“For example, Jošt et al. [2024] reported lower final grades when students relied more on large language models for tasks that require critical thinking, such as code generation and debugging.”
Velychko, V. Y. (2026) · CTE Workshop Proceedings · doi:10.55056/cte.1404
“…or automated repairs on demand – the evidence points to weakened learning and heightened dependence: trivial program generation displaces the productive struggle on which programming pedagogy relies [ 49 ] , and unstructured use correlates with poorer outcomes and over-reliance [ 27, 69 ] .”
Tompkins, M., Agarwal, N., Soneji, A., Wasinger, R., Nelson, C., Leach, K., Hasan, R., Doupé, A., Votipka, D., Shoshitaishvili, Y., et al. (2026) · doi:10.1145/3830454.3846694
“This is consistent with the longstanding finding that question quality, not frequency, correlates with achievement [14], and with traditional CS domains where unproductive help-seeking behaviors such as hint abuse predict poorer performance [20, 26, 30].”
Learner-Customized Algorithm Visualization Using Generative AI
Kang, E., Devi, S., & Kim, S. (2026) · International Conference on Software and Information Engineering · doi:10.1145/3789595.3789607
“Jošt et al. (2024) analyzed the impact of LLM use on learning outcomes in actual programming classes [8].”
Ganuthula, V. R. R., & Kumari, N. (2026) · Human Behavior and Emerging Technologies · doi:10.1155/hbe2/6725387
Troussas, C., Papakostas, C., Krouska, A., Phivos Mylonas, B., & Sgouropoulou, C. (2026) · International Journal of Intelligent Systems · doi:10.1155/int/9444834
PCEval: A Benchmark for Evaluating Physical Computing Capabilities of Large Language ModelsPreprint
Song, I., Jeon, E., & Lee, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2601.02404
“Prior studies already caution that students may rely on LLMs for “ready-made answers,” which can hinder deeper engagement and learning (Anson, 2024; Jošt et al., 2024).”
Choi, W. C., Chang, C. I., & Choi, H. F. (2025) · Artificial Intelligence and Cloud Computing Conference (AICCC) · doi:10.1145/3789982.3790043
“In summary, the literature has suggested that ChatGPT can significantly enhance learning achievement when used as a source of hints, feedback, and Socratic questioning rather than as a source of ready-made solutions [18].”
InfluentialDewan, P., George, S. D., Jain, Y., & Laney, M. (2025) · 2025 IEEE 32nd International Conference on High Performance Computing, Data and Analytics Workshop (HiPCW) · doi:10.1109/hipcw66559.2025.00011
“Thus, even for students who already know programming, a programming-free design remains worthwhile because it allows broader—though less deep—exploration of the intended concepts. underlying concepts [7].”
L’intelligenza artificiale nella didattica. Opportunità, rischi e regole
Galli, F. (2025) · Labour & Law Issues · doi:10.60923/issn.2421-2695/23590
An Intimacy-Aware Peer AI to Support Classroom Learning
Shitara, K., & Takano, K. (2025) · International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale66047.2025.11346770
“Jošt et al. investigated the nuanced impact of informal LLM use on undergraduate students’ learning outcomes in software development education, showing that increased average LLM use across tasks was associated with a decline in final grades [8].”
More polished, not necessarily more learned: LLMs and perceived text quality in higher education
Tärning, B., Tjøstheim, T. A., & Wallin, A. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1653992 · Cited by 2
“…by, or in cooperation with LLMs) (Deep and Chen, 2025; Kosmyna et al., 2025), in programming (with LLMs generating software codes based on prompts) (Jošt et al., 2024; Lyu et al., 2024) and in independent thinking (with material selected by and adapted to the relevant context by LLMs), (Deep and…”
Detection of AI-Generated texts with BERT variants
Balara, V., & Machová, K. (2025) · International Conference on Emerging eLearning Technologies and Applications (ICETA) · doi:10.1109/iceta67772.2025.11280096
“In many fields, LLMs have already changed the workflow of these professions, such as translation [1], education [2], programming [3] and search engines [4].”
The Impact of a GenAI Integration on First-Year Engineering Students' Programming Outcomes
Das, U. K., Yates, M. S., Bloch, K. A., Hagan, L., Thomas, E. L., Gatos, G., Rajkumar, A. K., Valavala, B., Franco, S., Divvela, S. K., et al. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328708 · Cited by 2
“Jošt et al. (2024) identified a negative correlation between increased use of large language models and final grades and critical thinking skills in early engineering students [24].”
History-grounded Design Speculation as a Method for AIImpact Anticipation
Lupetti, M. L., van der Maden, W., Sanni, J., Martinez, L., Cavallin, E., Cangiano, S., Mela, A., & Murray-Rust, D. (2025) · ACM SIGCHI Italian Chapter International Conference on Computer-Human Interaction · doi:10.1145/3750069.3750151
“The rise of what is colloquially called “vibe coding”—rapid, LLM-assisted programming that prioritises exploration over technical understanding—has certainly transformed how students approach these projects [24].”
FABRIC - A Fault-aware BiLSTM Approach for Programming Bug Detection and Corrective Suggestion
Le-Duy, A., Nguyen, D., Tran, B.-N. N., Tran, H., & Vu-Van, T. (2025) · International Conference on Information and Communicatiaon Technology · doi:10.1109/icoict66265.2025.11193025
Paiva, J., Virginio, J., Albuquerque, D., Rodrigues, G., & Sousa, I. (2025) · Anais do XXXIII Workshop sobre Educação em Computação (WEI 2025) · doi:10.5753/wei.2025.8709
Kamberovic, M. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3699682.3727571 · Cited by 1
Raj, J., Muppa, A., Dipukumar, A., Nirmal, R., Laddha, A., Kamath, T., Hong, S., Potla, M., & Boicu, M. (2024) · IEEE MIT Undergraduate Research Technology Conference (URTC) · doi:10.1109/urtc65039.2024.10937532 · Cited by 2
Alidadi, M., Taghiyareh, F., & Shahhoseini, N. (2024) · International Symposium on Telecommunications · doi:10.1109/ist64061.2024.10843532
Meta-Analysis on the Effect Size of Chatbot Integration in Student Science Performance
Villanueva, J. C. P. M., Atalia, G. D. N., & Villamin, J. L. A. (2026) · International Journal of Information and Education Technology · doi:10.18178/ijiet.2026.16.6.2633
“Students in the experimental group who regularly used chatbots completed their tasks faster than average but demonstrated less depth of comprehension and critical thinking compared to those in the control group [11].”
InfluentialTompkins, M., Agarwal, N., Soneji, A., Wasinger, R., Nelson, C., Leach, K., Hasan, R., Doupé, A., Votipka, D., Shoshitaishvili, Y., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2602.17448
“This is consistent with the longstanding finding that question quality, not frequency, correlates with achievement [14], and with traditional CS domains where unproductive help-seeking behaviors such as hint abuse predict poorer performance [20, 26, 30].”
Feedback Systems on Computer Programming Courses: A Systematic Literature Review
Castellanos, H., Vieira, C., & Magana, A. J. (2026) · IEEE Access · doi:10.1109/access.2026.3665047
“On the one hand, research has documented that students using LLMs can result in unintended negative outcomes, such as student over-reliance on the LLMs in critical learning tasks such as code generation and debugging [17].”
Jeon, J., Jang, J., & Jung, S. K. (2026) · IEEE Access · doi:10.1109/access.2026.3686194
“Research on large language models in programming education has expanded rapidly, but the evidence remains mixed [12], [13], [14], [15].”
Muepu, D. M., Watanobe, Y., Amin, M. F. I., & Mia, M. S. (2026) · IEEE Access · doi:10.1109/access.2026.3694715
“Jošt et al. analyse informal use of LLMs like ChatGPT and Copilot in an undergraduate React course [27].”
Methodology for exploiting AI as a tool for learning programming languages
Urbano, K., Bernardo, G., Auccahuasi, W., Meza, S., Castro-Mejia, P., Peña-Romero, M., Peña-Romero, L., & Auccahuasi, A. (2026) · Procedia Computer Science · doi:10.1016/j.procs.2026.06.375
Bendl, T., Krajňáková, L., Storrøsæter, H., Drápela, E., & Hanuš, M. (2026) · Journal of Information Technology Education Research · doi:10.28945/5834
“…converged around the erosion of CT, as frequent use of GenAI was associated with weaker independent performance, confirmation bias, and difficulty distinguishing students’ own ideas from AI - generated contributions (Gregory & Narang, 2024; Jošt et al., 2024; Kartal, 2024; Lijie et al., 2025).”
Yenidogan, M., Cömert, Z., & Çakır, D. (2026) · IEEE Access · doi:10.1109/access.2026.3737031
“Nevertheless, recent studies [13], [14], [19], [22] warn that excessive reasoning with GenAI verbosity can reduce clarity and impede learning transfer, reinforcing the need for adaptive explanation strategies.”
Damayanti, A. D., Gani, H., Feng, Z., Gani, H., Zuhriyah, S., Nurani, N., Djabir, N., & Wardani, N. I. (2025) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2025.0160436 · Cited by 2
“An experiment with 32 participants examined LLM use and its correlation with student performance, revealing a negative impact on grades when overused for essential tasks, emphasizing the need for balanced integration of LLM tools in education [31].”
Work in Progress: Learning by AI-Assisted Reverse Engineering (LARE) Using Aladdin
Anbuvanan, A., Wu, M.-Y., & Purzer, Ș. (2026) · 2026 ASEE Annual Conference & Exposition Proceedings · doi:10.18260/1-2--60915
Metadata
- Authors
- Jošt, G., Taneski, V., & Karakatič, S.
- Affiliations
- University of Maribor (SI)
- Corresponding author
- Jošt, Gregor
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2024-05-13
- Volume
- 14
- Issue
- 10
- Pages
- 4115
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W4396896417
- Cited by
- 143
- References
- 16
- Citations by year
- 2026: 66 · 2025: 69 · 2024: 8
- FWCI
- 53.95
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- dCOBISS.SI Digital Repository (publishedVersion, cc-by)
- University of Maribor digital library (University of Maribor) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Slovenian Research and Innovation Agency (ARIS) (P2-0057)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- large language models, programming education, learning outcomes, code generation, debugging, React applications, undergraduate students, critical thinking, problem-solving skills, ChatGPT, Copilot
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
