The Impact of Large Language Models on Programming Education and Student Learning Outcomes
Open accessAbstract
Recent advancements in Large Language Models (LLMs) like ChatGPT and Copilot have led to their integration into various educational domains, including software development education. Regular use of LLMs in the learning process is still not well-researched; thus, this paper intends to fill this gap. The paper explores the nuanced impact of informal LLM usage on undergraduate students’ learning outcomes in software development education, focusing on React applications. We carefully designed an experiment involving thirty-two participants over ten weeks where we examined unrestricted but not specifically encouraged LLM use and their correlation with student performance. Our results reveal a significant negative correlation between increased LLM reliance for critical thinking-intensive tasks such as code generation and debugging and lower final grades. Furthermore, a downward trend in final grades is observed with increased average LLM use across all tasks. However, the correlation between the use of LLMs for seeking additional explanations and final grades was not as strong, indicating that LLMs may serve better as a supplementary learning tool. These findings highlight the importance of balancing LLM integration with the cultivation of independent problem-solving skills in programming education.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Increased reliance on LLMs for critical thinking tasks such as code generation and debugging may negatively affect student performance, while their use for explanations appears more beneficial.
“Empirical evidence shows that increased reliance on LLMs, particularly for tasks requiring critical thinking such as code generation and debugging, may negatively affect student performance, while their use for explanations appears more beneficial, indicating a complex and non-trivial role in educational contexts [3].”
From An Integrated AHP–Fuzzy AHP Evaluation Framework for Large Language Models in Software Engineering Education (2026) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature
Lo, C. K. (2023) · Education Sciences · doi:10.3390/educsci13040410 · Cited by 2,024 · see citing works
Grassini, S. (2023) · Education Sciences · doi:10.3390/educsci13070692 · Cited by 1,048 · see citing works
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660 · see citing works
ChatGPT in education: Strategies for responsible implementation
Halaweh, M. (2023) · Contemporary Educational Technology · doi:10.30935/cedtech/13036 · Cited by 607 · see citing works
Impact of the Implementation of ChatGPT in Education: A Systematic Review
Montenegro-Rueda, M., Fernández-Cerero, J., Fernández-Batanero, J. M., & López-Meneses, E. (2023) · Computers · doi:10.3390/computers12080153 · Cited by 509 · see citing works
GitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417 · see citing works
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417 · see citing works
Fuchs, K. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1166682 · Cited by 353 · see citing works
Denny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287 · see citing works
Using GitHub Copilot to Solve Simple Programming Problems
Wermelinger, M. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569830 · Cited by 191 · see citing works
Improving Debugging Skills in the Classroom: The Effects of Teaching a Systematic Debugging Process
Michaeli, T., & Romeike, R. (2019) · Workshop in Primary and Secondary Computing Education · doi:10.1145/3361721.3361724 · Cited by 77 · see citing works
Github copilot in the classroom: learning to code with AI assistance
Puryear, B., & Sprint, G. (2022) · Journal of Computing Sciences in Colleges (JCSC; Formerly: Journal of Computing in Small Colleges) · doi:10.5555/3575618.3575622 · Cited by 76
Program Code Generation with Generative AIs
Idrisov, B., & Schlippe, T. (2024) · Algorithms · doi:10.3390/a17020062 · Cited by 70 · see citing works
Cambaz, D., & Zhang, X. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630958 · Cited by 62 · see citing works
The Potential Use of ChatGPT for Debugging and Bug Fixing
Haque, M. A., & Li, S. (2023) · EAI Endorsed Transactions on AI and Robotics · doi:10.4108/airo.v2i1.3276 · Cited by 44 · see citing works
From Copilot to Pilot: Towards AI Supported Software DevelopmentPreprint
Pudari, R., & Ernst, N. A. (2023) · arXiv.org · doi:10.48550/arxiv.2303.04142 · Cited by 37 · see citing works
Hliš, T., Četina, L., Beranič, T., & Pavlič, L. (2023) · Applied Sciences · doi:10.3390/app132413061 · Cited by 15 · see citing works
Tan, K., Pang, T., Fan, C., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.03433 · Cited by 15 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- [3]3. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, Ł.; Polosukhin, I. Attention Is All You Need. Adv. Neural Inf. Process. Syst. 2017 , 30 . [CrossRef]
- [17]17. Lazuardy, M.F.S.; Anggraini, D. Modern front end web architectures with react.js and next.js. Int. Res. J. Adv. Eng. Sci. 2022 , 7 , 132-141.
- [19]19. Carver, M.S.; Risinger, S.C. Improving children's debugging skills. In Empirical Studies of Programmers: Second Workshop ; Ablex Publishing Corp: New York, NY, USA, 1987; pp. 147-171.
Cited by
Sorted by citations
Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies
Deng, R., Jiang, M., Yu, X., Lu, Y., & Liu, S. (2025) · Computers & Education · doi:10.1016/j.compedu.2024.105224 · Cited by 531 · see citing works
Wang, J., & Fan, W. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04787-y · Cited by 322 · see citing works
“Therefore, studies with non-experimental or non-quasi-experimental designs were excluded, such as Jo š t et al. (2024).”
RetractedThe Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196 · see citing works
Prather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3689187.3709614 · Cited by 97 · see citing works
Large Language Models in Computer Science Education: A Systematic Literature Review
Raihan, M. N., Siddiq, M. L., Santos, J. C. S., & Zampieri, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701863 · Cited by 125 · see citing works
“From the perspective of the instructors, studies have indicated that CS in-structors found a negative correlation between the usage of LLMs and students’ grades [59].”
Wu, F., Dang, Y., & Li, M. (2025) · Behavioral Sciences · doi:10.3390/bs15040467 · Cited by 130 · see citing works
“Jošt et al. (2024) suggested that students who used LLMs more frequently for code generation received a lower final grade.”
Student-AI Interaction: A Case Study of CS1 students
Amoozadeh, M., Nam, D., Prol, D., Alfageeh, A., Prather, J. E., Hilton, M. L., Ragavan, S. S., & Alipour, M. A. (2024) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699567 · Cited by 41 · see citing works
Generative AI Tools in Higher Education: A Meta-Analysis of Cognitive Impact
Qu, X., Sherwood, J., Liu, P., & Aleisa, N. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719841 · Cited by 44 · see citing works
“Over-reliance on AI tools, as reported in studies like Jošt et al. (2024)[6], can hinder the development of critical thinking and problem-solving abilities.”
InfluentialGarcia, M. B. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13452-5 · Cited by 40 · see citing works
“This code generation capability, however, risks becoming a double-edged sword as students might increasingly turn to ChatGPT for swift solutions (Budhiraja et al., 2024; Jošt et al., 2024).”
Otto, S., Lavi, R., & Bertel, L. B. (2025) · Computers & Education · doi:10.1016/j.compedu.2025.105444 · Cited by 36 · see citing works
How Do Programming Students Use Generative AI?
Rahe, C., & Maalej, W. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3715762 · Cited by 45 · see citing works
“Furthermore, reliance on code generation can negatively impact code authoring skills [Jošt et al. 2024; Kazemitabaar et al. 2023].”
Humble, N. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00494-x · Cited by 40 · see citing works
“…al., 2024c; Fernandez et al., 2024; Finnie-Ansley et al., 2023; Groothuijsen et al., 2024; Gumina et al., 2023; Hellas et al., 2023; Husain, 2024; Jošt et al., 2024; Kuramitsu et al., 2024; Liao et al., 2024; Liffiton et al., 2023; Prasad & Sane, 2024; Prather et al., 2023; Rajala, 2023; Savelka…”
ChatGPT as a cognitive crutch: Evidence from a randomized controlled trial on knowledge retention
Barcaui, A. (2025) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2025.102287 · Cited by 51 · see citing works
Integrating Natural Language Prompting Tasks in Introductory Programming Courses
Kerslake, C., Denny, P. C., Smith, D. H., Prather, J. E., Leinonen, J., Luxton-Reilly, A., & MacNeil, S. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649165.3690125 · Cited by 13 · see citing works
Adejumo, A. A., Oyelere, S. S., Sanusi, I. T., & Suhonen, J. T. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100570 · Cited by 22 · see citing works
Underreporting of AI Use: The Role of Social Desirability Bias
Ling, Y., Kale, A., & Imas, A. (2026) · International Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791073 · Cited by 16 · see citing works
Dawson, M. G., Deer, R., & Boguslawski, S. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100724 · Cited by 14 · see citing works
Exploring the Impact of LLM Prompting on Students’ Learning
Mutanga, M. B., Msane, J., Mndaweni, T. N., Hlongwane, B. B., & Ngcobo, N. Z. (2025) · Trends in Higher Education · doi:10.3390/higheredu4030031 · Cited by 23 · see citing works
Tang, B., Liang, J., Hu, W., & Luo, H. (2025) · Systems · doi:10.3390/systems13070555 · Cited by 16 · see citing works
Enhancing EFL Speaking Skills with AI-Powered Word Guessing: A Comparison of Human and AI Partners
Pituxcoosuvarn, M., 谷村, 緑., Murakami, Y., & White, J. S. (2025) · Information · doi:10.3390/info16060427 · Cited by 10 · see citing works
Using large language models to facilitate academic work in the psychological sciences
Sohail, A., & Zhang, L. (2025) · Current Psychology · doi:10.1007/s12144-025-07438-2 · Cited by 18 · see citing works
“Indeed, when using LLM tools to complete a programming project, students demonstrate practical progress but report hindered learning (Tanay et al., 2024 ), and an increased reliance on LLMs for programming tasks subsequently lowers performance on critical thinking assessments (Jošt et al., 2024 ).”
Zhang, L., Jiang, Q., Xiong, W., & Zhao, W. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331251333874 · Cited by 15 · see citing works
Khan, M., Akbar, M. A., & Kasurinen, J. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3727967.3756831 · Cited by 7 · see citing works
Denny, P. C., Kumar, V., MacNeil, S., Prather, J. E., & Leinonen, J. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729093 · Cited by 8 · see citing works
Ten Natural Language Processing Tasks with Generative Artificial Intelligence
Golec, J., & Hachaj, T. (2025) · Applied Sciences · doi:10.3390/app15169057 · Cited by 13 · see citing works
Badger, M., López, N., & Nissen, C. F. R. (2026) · International Journal of Technology in Education · doi:10.46328/ijte.5111 · Cited by 9 · see citing works
“Learners were instructed about what specific tasks to complete with AI assistance, how to interact effectively with the tool, and how to utilize it as a learning facilitator (Hsu, 2023; Jost et al., 2024; Song & Song, 2023; Zhou & Kim, 2024).”
InfluentialAlpizar-Chacón, I., Keuning, H., de Jong, I., Lykourentzou, I., & Rings, S. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770008 · Cited by 6 · see citing works
Xiao, R., Hou, X., Ye, R., Kazemitabaar, M., Diana, N., Liut, M., & Stamper, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.19107 · Cited by 1 · see citing works
“The un-productive LLM-based help-seeking can be correlated with declining learning outcomes compared to students before LLM became prevalent (Ramirez Osorio, Zavaleta Bernuy, Simion and Liut, 2025) or students in controlled groups without access to LLMs (Jošt, Taneski and Karakatič, 2024).”
Learning, Behavior, and Pedagogy: A Systematic Review of Generative AI Use in Programming Education
Huang, T., & Tseng, H.-P. (2026) · International Journal of Information and Education Technology · doi:10.18178/ijiet.2026.16.1.2487 · Cited by 6 · see citing works
“ChatGPT 3.5 was the most commonly used version, while two studies used both ChatGPT and GitHub Copilot [28, 37].”
InfluentialMartin-Lopez, A., Tufano, R., Guglielmi, E., Sánchez, A. B., Sanz, A. L., Scalabrino, S., Oliveto, R., Segura, S., & Bavota, G. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3679627 · Cited by 4 · see citing works
“Several studies investigated the impact of AI support on students and novice developers [31], [32], [33], [34], [35], with some reporting neutral or positive changes in students’ learning outcomes [32], [33], and others reporting negative effects [31], [34], [35].”
Learning with large language models: beyond prompt engineering
Ding, L., O’Berry, R., Tallent, H., C., S. G. C. G., & Gapud, A. A. (2026) · Education and Information Technologies · doi:10.1007/s10639-026-13924-2 · Cited by 6 · see citing works
“Whereas another study by Jost et al. (2024) in programming education revealed that an increase in students’ reliance on LLMs for critical thinking-intensive tasks, such as code generation and debugging, correlated with lower grades amongst students.”
Evaluating lab assistant chatbot on student learning and behaviors in a programming short course
Noraset, T., Supratak, A., Ragkhitwetsagul, C., Worathong, N., & Tuarob, S. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100527 · Cited by 6 · see citing works
The Influence of AI-assisted Tools on Engineering Project Outcomes
Dinu, A. (2025) · Revista Romaneasca pentru Educatie Multidimensionala · doi:10.18662/rrem/17.3/1024 · Cited by 5 · see citing works
“These tools are often praised for their role in enhancing learning efficiency, aiding in academic writing, and supporting personalized feedback (Jošt et al., 2024; Sajja et al., 2024; Susnjak, 2023).”
Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education
Mak, J., Nakatumba-Nabende, J., Clear, T., Clear, A., Abluwi, I., Andrei, O., Angeli, L., MacNeil, S., Oyelere, S. S., Hale Rattigan, M., et al. (2025) · Proceedings of the 2025 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3760545.3783975 · Cited by 12 · see citing works
Rozputnia, B., Shevchenko, L., Umanets, V., Yashchuk, S., & Sabadosh, Y. (2025) · Environment Technology Resources Proceedings of the International Scientific and Practical Conference · doi:10.17770/etr2025vol3.8537 · Cited by 6 · see citing works
“LLMs necessitate a reevaluation of their educational role [10].”
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B., Kimmel, B., Wright, J., & Briggs, B. (2024) · arXiv · Cited by 196 · see citing works
Gambo, O. O., Onifade, O. J., Olanrewaju-Smart, W., Abuya, O. J., Salawu, R. E., Massenon, R., & Gambo, I. (2026) · Discover Education · doi:10.1007/s44217-026-01142-4 · Cited by 7 · see citing works
“Some studies, such as those by Jošt et al. [40] and Sok and Heng [89], highlighted that reliance on ChatGPT could lead to superficial learning, where students may obtain solutions without fully understanding the underlying concepts.”
A Generative AI Tool to Foster and Assess Authentic Learning: A Case Study in Teaching SQL
Sooriamurthi, R., Tu, X., & Pensky, A. C. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729022 · Cited by 9 · see citing works
Kamberovic, M., Delić, A., & Krivic, S. (2025) · User Modeling, Adaptation, and Personalization · doi:10.1145/3708319.3733692 · Cited by 5 · see citing works
W iki H int : A Human-Annotated Dataset for Hint Ranking and Generation
Mozafari, J., Gerhold, F., & Jatowt, A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730284 · Cited by 3 · see citing works
Reflexões sobre o Uso de LLMs no Ensino de Programação
Rosa, Y. S., Garcia, P., Constantino, K., & Figueiredo, E. (2025) · Anais do V Simpósio Brasileiro de Educação em Computação (EDUCOMP 2025) · doi:10.5753/educomp.2025.5366 · Cited by 4 · see citing works
ÖZGÜL, E. (2026) · Kastamonu Eğitim Dergisi · doi:10.24106/kefdergi.1878122 · Cited by 3 · see citing works
GenAI Voice Mode in Programming Education
Jacobs, S., & Kiesler, N. (2025) · European Conference on Modelling and Simulation · doi:10.1145/3769994.3770001 · Cited by 4 · see citing works
Benevides, K. D. G., Rufino, L. J. G. C., dos Santos, D. F. A., Benevides, M. P., Pimenta, C. D., de Oliveira, M. R., & Moura, R. A. D. (2025) · Aracê. · doi:10.56238/arev7n8-085 · Cited by 4 · see citing works
Hernandez, A. A., Albina, E. M., & Caballero, A. R. (2025) · International Symposium on Computer, Consumer and Control · doi:10.1109/is3c65361.2025.11131023 · Cited by 7 · see citing works
“Moreso, professors experience high workload and time constraints, which limits their capacity to provide aligned feedback to students on programming assessment tasks [9].”
Refuting LLM-generated Code with Reactive Task Comprehension
Hebbar, S. V., Sasmita Harini, S., & Kumar, V. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729100 · Cited by 2 · see citing works
Research trends and the impact of ChatGPT on educational environments
Firdaus, T., Mufidah, R., Hamida, R. N., Febrianti, R. I., & Guivara, A. E. R. (2025) · International Journal of Social Sciences and Education Research · doi:10.24289/ijsser.1609741 · Cited by 4 · see citing works
Bidry, M., Hanine, M., Ouaguid, A., & Obidallah, W. (2025) · IEEE Access · doi:10.1109/access.2025.3636891 · Cited by 4
“In addition to text-based tools, visual generative models such as DALL · E and Midjourney [60], [61] have gained popularity in educational environments that emphasize creativity and design [54].”
Generative AI in higher education: A systematic review of empirical uses, outcomes, and risks
Hawi, N. S., & Samaha, M. (2026) · International Journal of Educational Research Open · doi:10.1016/j.ijedro.2026.100584 · Cited by 5 · see citing works
Effects of Generative AI Tools On Students’ Critical Thinking: A Three-Level Meta-Analysis
Sun, X., Liu, C., Li, P., & Cai, Z. (2026) · Educational Psychology Review · doi:10.1007/s10648-026-10180-1 · Cited by 5 · see citing works
Mustapha, R., Ragmoun, W., & Hammou, S. B. (2026) · PLoS ONE · doi:10.1371/journal.pone.0347120 · Cited by 3 · see citing works
“Learning outcomes are defined statements that outline what learners should be able to consider, comprehend, and achieve following specific learning experiences [24,1].”
STAP: A Socratic Tutor for Adaptive Programming with Pedagogical Scaffolding
Xie, W., Shi, H., Li, Y.-X., & Liu, J.-W. (2025) · Proceedings of the 2025 2nd International Symposium on Artificial Intelligence for Education · doi:10.1145/3775073.3775165 · Cited by 3 · see citing works
Concepts for Teaching Software Development in the Age of AI-Tools
Böttcher, A., Thurner, V., & Zönnchen, B. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016505 · Cited by 2 · see citing works
“For instance, while [9] found no significant impact of ChatGPT on the students’ learning outcomes in introductory programming, [10] reported a negative correlation between increased usage of large language models on the one hand, and both students’ grades and critical thinking skills on the other…”
Coelho, R., Bjune, A. E., Ellingsen, S., Solheim, B. M., Thormodsæter, R. S., Wasson, B., & Cotner, S. H. (2025) · Journal of Science Education and Technology · doi:10.1007/s10956-025-10216-1 · Cited by 4 · see citing works
“Moreover, research in programming, for example, indicates that introducing GenAI tools before students develop foundational problem-solving skills can hinder learning (Jošt et al., 2024).”
Andriyana, Hidayat, A., Hanifah, I., & Ramadona, D. C. (2026) · Buletin Pengembangan Perangkat Pembelajaran · doi:10.23917/bppp.v8i1.16355 · Cited by 2
“Lebih luas lagi, visi pengembangan LLM dalam pendidikan menempatkan teknologi ini sebagai komponen strategis dalam ekosistem pembelajaran digital yang mampu mendukung personalisasi, akses pengetahuan, dan pengambilan keputusan pedagogis berbasis data [9].”
Janabi, I. A., & Bland, T. (2026) · AI · doi:10.3390/ai7060223 · Cited by 2 · see citing works
EFFICIENCY, ENGAGEMENT, AND COGNITIVE OFFLOADING: A REVIEW OF GENERATIVE AI IN EDUCATION
Khalid, N., Kasbun, R., & Rahman, N. F. A. (2026) · International Journal of Education Psychology and Counseling · doi:10.35631/ijepc.1162072 · Cited by 4 · see citing works
Lin, M. P.-C., Huang, J.-Y., Chang, D., Tembrevilla, G., Bowen, G. M., Poitras, E., Janarthanan, V., & Ryoo, J. (2026) · AI in Education · doi:10.3390/aieduc2010004 · Cited by 2 · see citing works
“When instructional tasks and assessments are not explicitly designed to accommodate or constrain AI use, the presence of LLMs may inadvertently undermine the very competencies being evaluated (Jošt et al., 2024; Zhai et al., 2024).”
Assessing the Software Security Comprehension of Large Language Models
Siddiq, M. L., Sekerak, N., Karam, A., Leal, M., Islam-Gomes, A., & Santos, J. C. S. (2025) · arXiv.org · doi:10.48550/arxiv.2512.21238 · Cited by 2
“Conversely, instructors express concerns about declining problem-solving skills and curriculum integrity [45, 46].”
Schotter, T., Kawas, S., Prather, J., Leinonen, J., Ippolito, J., & Nelson, G. L. (2025) · arXiv.org · doi:10.48550/arxiv.2505.18771 · Cited by 2
Riztha, F., Wickramarachchi, R., Asanka, D., & Disssanayke, M. (2024) · International Conference on Advancements in Computing (ICAC) · doi:10.1109/icac64487.2024.10850942 · Cited by 4 · see citing works
Takamatsu, N., Kamioka, E., Tran, C. M., & Phan, T. X. (2025) · IEEE Access · doi:10.1109/access.2025.3638804 · Cited by 2
“Although their educational utility has been explored in intelligent tutoring systems and programming support tools for beginners [16], [17], applications aimed at improving accessibility for VIs are still rare.”
Help or Hype? Exploring LLM-based Chatbots in Self-Regulated Learning
Tsang, R., & WOOD, S. (2025) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--56687 · Cited by 2 · see citing works
Jović, J. L., Domazet, D. S., Vesić, N. O., Ranđelović, B. M., & Simjanović, D. J. (2026) · Mathematics · doi:10.3390/math14101637 · Cited by 1 · see citing works
In-depth exploration of text similarity calculation based on improved VSM and DeepSeek
Wu, H., Guo, M., & 郭庆琳 (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200703 · Cited by 1 · see citing works
Tang, Z., Wei, W., Liang, K., & Lam, C. K. (2026) · Systems · doi:10.3390/systems14060685 · Cited by 1 · see citing works
Large language models in software development learning
Hernández, S. C., Cruz, M. P. T., Sánchez, J. M. E., Luna-Gómez, A. A., Orozco, A. M. P., & Medina, R. H. (2026) · Discover Education · doi:10.1007/s44217-026-01750-0
“On this topic, the impact of the informal use of LLMs for learning programming was measured with a group of thirty-two students, and it was concluded that there is a negative relationship between the use of these AIs and student performance [69].”
Characterization and Effects of CS2 Learning with GenAI, Visualization, and Human Support
Yong, Q., Estey, A., & Nacenta, M. A. (2026) · International Computing Education Research Workshop · doi:10.1145/3765964.3811663 · Cited by 1
Ouaazki, A., Shibani, A., Knight, S., & Holzer, A. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100608 · Cited by 1 · see citing works
What do students want from AI? Exploring expectations and preferences in programming education
Kamberovic, M., & Krivić, S. (2026) · Artificial Intelligence in Education · doi:10.1108/aiie-08-2025-0260
Adauto-Medina, W., Chamorro-Atalaya, O., Olivares-Zegarra, S., Arévalo-Tuesta, J. A., Arones, M., Aybar-Bellido, I., León-Velarde, C., Fernández-Flores, S., Quispe-Andía, A., & Auqui-Ramos, E. (2026) · Sustainability · doi:10.37766/inplasy2026.1.0066 · Cited by 1
Changes in Coding Behavior and Performance Since the Introduction of LLMs
Zhang, Y., Savelka, J., Goldstein, S., & Conway, M. (2026) · International Conference on Learning Analytics and Knowledge · doi:10.1145/3785022.3785075 · Cited by 1
Permatasari, P., Mentari, M., Kinari, S. A., Aung, S. T., Funabiki, N., Kyaw, H. H. S., & Wai, K. (2025) · Analytics · doi:10.3390/analytics5010002 · Cited by 1
“Recent research has shown that the frequent use of Large Language Models (LLM) such as ChatGPT [12] or Copilot [13] for code generation correlates with lower student performances, reinforcing the concern that this AI-assisted coding can lead to academic integrity issues including plagiarism [14].”
Redefining Undergraduate Computing Education with GenAI-Facilitated Authentic In-Term Assessment
Masood, A., Wijeratne, V., & Alam, A. S. (2025) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale66047.2025.11346650 · Cited by 1 · see citing works
Student-Perceived Cognitive Load of LLM-Generated Programming Exercises
Khan, M. F. A., Feri, L., Nguyen, H., & Karimi, H. (2025) · International Conference on Data Science and Advanced Analytics · doi:10.1109/dsaa65442.2025.11247983 · Cited by 1 · see citing works
Avaliando o Papel do GPT-4o na Aprendizagem de Engenharia de Software: Um Relato de Experiência
Branco, B. M. D. C. C., Neto, A. L. D. M., Avelino, G. A., & Machado, V. P. (2025) · Simpósio Brasileiro de Engenharia de Software (SBES) · doi:10.5753/sbes.2025.10716 · Cited by 1 · see citing works
Lin, T.-C., Chiu, C., Wang, P.-T., & Fang, L.-D. (2025) · Multimedia · doi:10.3390/multimedia1010003 · Cited by 4 · see citing works
“Jošt et al. [30] and Zhang et al. [31] quantified the impact of large language models on programming education, demonstrating a 34% improvement in skill acquisition and a 41% reduction in learning time through AI-assisted instruction.”
Can Large Language Models Help Students Prove Software Correctness? An Experimental Study with Dafny
Carreira, C., Silva, Á. F., Abreu, A., & Mendes, A. (2025) · IEEE International Conference on Software Engineering and Formal Methods · doi:10.48550/arxiv.2506.22370 · Cited by 1
“This disconnect raises important questions about dependency, as students may rely on LLMs as a substitute for their reasoning, consistent with prior work [6,18].”
Hyseni, L. N., & Dermaku, A. (2025) · International Journal of Computational and Experimental Science and Engineering · doi:10.22399/ijcesen.1846 · Cited by 1 · see citing works
Exploring the Role of Large Language Models as Artificial Tutors
Zönnchen, B., Hobelsberger, M., Socher, G., Thurner, V., & Ottinger, S. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016571 · Cited by 1 · see citing works
“Studies present mixed findings: some report improved learning outcomes with LLM integration in programming courses [12], while others find no significant effect [13] or even negative outcomes [14].”
Learning a First Programming Language with LLM Tutor
Vukojičić, M., & Korica, I. (2025) · SINTEZA · doi:10.15308/sinteza-2025-331-336 · Cited by 1 · see citing works
“Some of the results are not in favour of LLMs in the domain of higher education [4] in addition to the domain of lower secondary education [5].”
Exploring the Impact of Large Language Models on Building Construction Students: A Pilot Study
Leathem, T., Wetzel, E., Farrow, C. B., & Umer, M. (2025) · EPiC series in built environment · doi:10.29007/llkw · Cited by 1 · see citing works
“However, there are also concerns about their potential to promote plagiarism and hinder deep learning (Jošt et al., 2024; Kasneci et al., 2023).”
Zhu, S., & Liu, Z. (2026) · Proceedings of the 2026 International Conference on Big Data and Informatization Education · doi:10.1145/3806980.3806985
Song, J., Wei, Z., He, C., Liao, J., Xu, X., & Yip, D. K. M. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2745545
Abramskiy, M. M., Ozdemir, G., Khamitova, I., & Shapkina, I. (2026) · Education & Self Development · doi:10.26907/esd.21.3.07
Biczó, Z. (2026) · Sociedad y Economía · doi:10.14267/1588970x.2026.024
Burger, C., Trotter, C., Carlisle, J. D., & Walter, C. (2026) · doi:10.48550/arxiv.2609.26733
A Critical Review of Critical Thinking in HCI Research About AI
Inie, N., Kristensen, A. M., Misfeldt, M., & Dalsgaard, P. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2722522
Rashidov, A., & Rashidova, F. (2026) · 2026 11th International Conference on Computer Science and Engineering (UBMK) · doi:10.1109/ubmk71847.2026.11745325
“In engineering education, this is of particular significance, as learning activities often involve not only the acquisition of theoretical knowledge but also the development of practical skills, technical thinking and the ability to solve problems independently [4].”
Nephron Flow Lab: Vibe Coding a Visual Nephrology Learning App
Calderwood, I., & Bland, T. (2026) · Multimodal Technologies and Interaction · doi:10.3390/mti10090093
Gadot, R., & Tsybulsky, D. (2026) · Education Sciences · doi:10.3390/educsci16091402
Examining Student Dependence on Generative AI tools in Programming Education
Abioye, K. B. (2026) · Journal of Education Learning and Management · doi:10.69739/jelm.v3i2.2151
“Similar concerns were reported by Jošt et al. (2024), who found a significant negative relationship between LLM use and final grades, particularly in tasks such as code generation and debugging.”
Zuin, M., & Donadel, V. (2026) · European Journal of Investigation in Health Psychology and Education · doi:10.3390/ejihpe16090130
“…development, potentially creating conditions of overreliance and hindering a critical and informed approach to managing the information provided by AI systems (Danaher, 2018; Shanmugasundaram & Tamilarasu, 2023; Jošt et al., 2024; Zhai et al., 2024; Abubakar et al., 2025; Sarwar et al., 2026).”
The Governance Gap in Contemporary LLM-Based Agentic Systems: A Structural Diagnostic Review
Valdez-Cantú, C., Cantoral-Ceballos, J. A., & Alvarado-Uribe, J. (2026) · AI · doi:10.3390/ai7080322
How well do LLMs understand code?
Xu, J., Sun, R.-L., Ding, Z., Li, W., Han, K., Zhang, H., Wang, C.-Y., Wang, H.-X., Wang, W., Sun, Y.-Z., et al. (2026) · Communications AI & Computing · doi:10.1038/s44488-026-00014-y
Scaffolding Autocomplete: Improving Guidance for Learners using Generative Code Suggestions
Prather, J. E., MacNeil, S., Luxton-Reilly, A., Margulieux, L. E., Reeves, B. N., Denny, P. C., Leinonen, J., Homer, J., Nabid, R. A., & Rossetti, R. L. (2026) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3765964.3811654
Teaching Programming in the Age of AI: Pedagogical Strategies for Computational Sociology
Ramos, M. C., & Liang, F. (2026) · Teaching Sociology · doi:10.1177/0092055x261469125
“AI tools are generally more effective when used for comprehension and refinement after students have attempted to solve a problem on their own rather than for generating complete solutions (Güner and Er 2025; Jošt et al. 2024; Prather et al. 2025).”
Álvarez-Jiménez, M., Pomagualli, A. G. C., Gamboa, D., & Oscullo, D. S. P. (2026) · Computers · doi:10.3390/computers15070466
“Its benefits are more likely when use is structured and instructor-guided [23], whereas substituting core cognitive work with GenAI is associated with superficial or even negative learning outcomes [4,5].”
Ding, L., Rodenberg, R., Er, E., Nguyen, H., & Yoon, M. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2026.2702559
Wang, C., Petrie, C., Stouras, M., Ettlin, N., George, A., Mejia-Domenzain, P., Swamy, V., Käser, T., & Svensson, O. (2026) · ACM Conference on Learning @ Scale · doi:10.1145/3774398.3811623
- Authors
- Jošt, G., Taneski, V., & Karakatič, S.
- Affiliations
- University of Maribor (SI)
- Corresponding author
- Jošt, Gregor
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2024-05-13
- Volume
- 14
- Issue
- 10
- Pages
- 4115
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W4396896417
- Cited by
- 143
- References
- 16
- Citations by year
- 2026: 66 · 2025: 69 · 2024: 8
- FWCI
- 53.95
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- dCOBISS.SI Digital Repository (publishedVersion, cc-by)
- University of Maribor digital library (University of Maribor) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Slovenian Research and Innovation Agency (ARIS) (P2-0057)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- large language models, programming education, learning outcomes, code generation, debugging, React applications, undergraduate students, critical thinking, problem-solving skills, ChatGPT, Copilot
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
