The DOI matches its Crossref record; 18 of the 18 references it cites were found in a registry or an index.
The Impact of Large Language Models on Programming Education and Student Learning Outcomes
Open accessAbstract
Recent advancements in Large Language Models (LLMs) like ChatGPT and Copilot have led to their integration into various educational domains, including software development education. Regular use of LLMs in the learning process is still not well-researched; thus, this paper intends to fill this gap. The paper explores the nuanced impact of informal LLM usage on undergraduate students’ learning outcomes in software development education, focusing on React applications. We carefully designed an experiment involving thirty-two participants over ten weeks where we examined unrestricted but not specifically encouraged LLM use and their correlation with student performance. Our results reveal a significant negative correlation between increased LLM reliance for critical thinking-intensive tasks such as code generation and debugging and lower final grades. Furthermore, a downward trend in final grades is observed with increased average LLM use across all tasks. However, the correlation between the use of LLMs for seeking additional explanations and final grades was not as strong, indicating that LLMs may serve better as a supplementary learning tool. These findings highlight the importance of balancing LLM integration with the cultivation of independent problem-solving skills in programming education.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Empirical evidence shows that increased reliance on LLMs, particularly for tasks requiring critical thinking such as code generation and debugging, may negatively affect student performance, while their use for explanations appears more beneficial, indicating a complex and non-trivial role in educational contexts [3].”
From An Integrated AHP–Fuzzy AHP Evaluation Framework for Large Language Models in Software Engineering Education (2026) · cites this paper as [3]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
What Is the Impact of ChatGPT on Education? A Rapid Review of the Literature
Lo, C. K. (2023) · Education Sciences · doi:10.3390/educsci13040410 · Cited by 2,024
Grassini, S. (2023) · Education Sciences · doi:10.3390/educsci13070692 · Cited by 1,048
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
ChatGPT in education: Strategies for responsible implementation
Halaweh, M. (2023) · Contemporary Educational Technology · doi:10.30935/cedtech/13036 · Cited by 607
Impact of the Implementation of ChatGPT in Education: A Systematic Review
Montenegro-Rueda, M., Fernández-Cerero, J., Fernández-Batanero, J. M., & López-Meneses, E. (2023) · Computers · doi:10.3390/computers12080153 · Cited by 509
GitHub Copilot AI pair programmer: Asset or Liability?
Dakhel, A. M., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., Ming, Z., & Jiang (2022) · arXiv · doi:10.48550/arxiv.2206.15331 · Cited by 417
GitHub Copilot AI pair programmer: Asset or Liability?
Moradi Dakhel, A., Majdinasab, V., Nikanjam, A., Khomh, F., Desmarais, M. C., & Jiang, Z. M. (. (2023) · Journal of Systems and Software · doi:10.1016/j.jss.2023.111734 · Cited by 417
Fuchs, K. (2023) · Frontiers in Education · doi:10.3389/feduc.2023.1166682 · Cited by 353
Denny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287
Using GitHub Copilot to Solve Simple Programming Problems
Wermelinger, M. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569830 · Cited by 191
Research works printed but not linked: 3
- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., and Polosukhin, I. (2017). Attention Is All You Need. Adv. Neural Inf. Process. Syst., 30.
- Lazuardy, 2022, Modern front end web architectures with react.js and next.js, Int. Res. J. Adv. Eng. Sci., 7, 132
- Carver, M.S., and Risinger, S.C. (1987). Empirical Studies of Programmers: Second Workshop, Ablex Publishing Corp.
Cited by
Sorted by citations
Does ChatGPT enhance student learning? A systematic review and meta-analysis of experimental studies
Deng, R., Jiang, M., Yu, X., Lu, Y., & Liu, S. (2025) · Computers & Education · doi:10.1016/j.compedu.2024.105224 · Cited by 531
Wang, J., & Fan, W. (2025) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-04787-y · Cited by 322
“Therefore, studies with non-experimental or non-quasi-experimental designs were excluded, such as Jo š t et al. (2024).”
RetractedThe Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196
Prather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3689187.3709614 · Cited by 168
Large Language Models in Computer Science Education: A Systematic Literature Review
Raihan, M. N., Siddiq, M. L., Santos, J. C. S., & Zampieri, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701863 · Cited by 165
“From the perspective of the instructors, studies have indicated that CS in-structors found a negative correlation between the usage of LLMs and students’ grades [59].”
Wu, F., Dang, Y., & Li, M. (2025) · Behavioral Sciences · doi:10.3390/bs15040467 · Cited by 143
“Jošt et al. (2024) suggested that students who used LLMs more frequently for code generation received a lower final grade.”
Student-AI Interaction: A Case Study of CS1 students
Amoozadeh, M., Nam, D., Prol, D., Alfageeh, A., Prather, J. E., Hilton, M. L., Ragavan, S. S., & Alipour, M. A. (2024) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699567 · Cited by 68
Generative AI Tools in Higher Education: A Meta-Analysis of Cognitive Impact
Qu, X., Sherwood, J., Liu, P., & Aleisa, N. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3719841 · Cited by 49
“Over-reliance on AI tools, as reported in studies like Jošt et al. (2024)[6], can hinder the development of critical thinking and problem-solving abilities.”
InfluentialGarcia, M. B. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13452-5 · Cited by 54
“This code generation capability, however, risks becoming a double-edged sword as students might increasingly turn to ChatGPT for swift solutions (Budhiraja et al., 2024; Jošt et al., 2024).”
Otto, S., Lavi, R., & Bertel, L. B. (2025) · Computers & Education · doi:10.1016/j.compedu.2025.105444 · Cited by 39
How Do Programming Students Use Generative AI?
Rahe, C., & Maalej, W. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3715762 · Cited by 45
“Furthermore, reliance on code generation can negatively impact code authoring skills [Jošt et al. 2024; Kazemitabaar et al. 2023].”
Humble, N. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00494-x · Cited by 48
“…al., 2024c; Fernandez et al., 2024; Finnie-Ansley et al., 2023; Groothuijsen et al., 2024; Gumina et al., 2023; Hellas et al., 2023; Husain, 2024; Jošt et al., 2024; Kuramitsu et al., 2024; Liao et al., 2024; Liffiton et al., 2023; Prasad & Sane, 2024; Prather et al., 2023; Rajala, 2023; Savelka…”
ChatGPT as a cognitive crutch: Evidence from a randomized controlled trial on knowledge retention
Barcaui, A. (2025) · Social Sciences & Humanities Open · doi:10.1016/j.ssaho.2025.102287 · Cited by 51
Integrating Natural Language Prompting Tasks in Introductory Programming Courses
Kerslake, C. P., Denny, P. C., Smith, D. H., Prather, J. E., Leinonen, J., Luxton-Reilly, A., & MacNeil, S. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649165.3690125 · Cited by 26
Adejumo, A. A., Oyelere, S. S., Sanusi, I. T., & Suhonen, J. T. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100570 · Cited by 24
Underreporting of AI Use: The Role of Social Desirability Bias
Ling, Y., Kale, A. M., & Imas, A. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3791073 · Cited by 24
Dawson, M. G., Deer, R., & Boguslawski, S. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100724 · Cited by 24
Exploring the Impact of LLM Prompting on Students’ Learning
Mutanga, M. B., Msane, J., Mndaweni, T. N., Hlongwane, B. B., & Ngcobo, N. Z. (2025) · Trends in Higher Education · doi:10.3390/higheredu4030031 · Cited by 23
Tang, B., Liang, J., Hu, W., & Luo, H. (2025) · Systems · doi:10.3390/systems13070555 · Cited by 17
Enhancing EFL Speaking Skills with AI-Powered Word Guessing: A Comparison of Human and AI Partners
Pituxcoosuvarn, M., 谷村, 緑., Murakami, Y., & White, J. S. (2025) · Information · doi:10.3390/info16060427 · Cited by 29
Using large language models to facilitate academic work in the psychological sciences
Sohail, A., & Zhang, L. (2025) · Current Psychology · doi:10.1007/s12144-025-07438-2 · Cited by 23
“Indeed, when using LLM tools to complete a programming project, students demonstrate practical progress but report hindered learning (Tanay et al., 2024 ), and an increased reliance on LLMs for programming tasks subsequently lowers performance on critical thinking assessments (Jošt et al., 2024 ).”
Zhang, L., Jiang, Q., Xiong, W., & Zhao, W. (2025) · Journal of Educational Computing Research · doi:10.1177/07356331251333874 · Cited by 15
Khan, M., Akbar, M. A., & Kasurinen, J. (2025) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3727967.3756831 · Cited by 12
Denny, P. C., Kumar, V., MacNeil, S., Prather, J. E., & Leinonen, J. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729093 · Cited by 8
Ten Natural Language Processing Tasks with Generative Artificial Intelligence
Golec, J., & Hachaj, T. (2025) · Applied Sciences · doi:10.3390/app15169057 · Cited by 14
The Impact of GenAI Chatbots on Student Learning in Higher Education: A Literature Review
Badger, M., López, N., & Nissen, C. F. R. (2026) · International Journal of Technology in Education · doi:10.46328/ijte.5111 · Cited by 9
“Learners were instructed about what specific tasks to complete with AI assistance, how to interact effectively with the tool, and how to utilize it as a learning facilitator (Hsu, 2023; Jost et al., 2024; Song & Song, 2023; Zhou & Kim, 2024).”
InfluentialAlpizar-Chacón, I., Keuning, H., de Jong, I., Lykourentzou, I., & Rings, S. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770008 · Cited by 12
Xiao, R., Hou, X., Ye, R., Kazemitabaar, M., Diana, N., Liut, M., & Stamper, J. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2506.19107 · Cited by 1
“The un-productive LLM-based help-seeking can be correlated with declining learning outcomes compared to students before LLM became prevalent (Ramirez Osorio, Zavaleta Bernuy, Simion and Liut, 2025) or students in controlled groups without access to LLMs (Jošt, Taneski and Karakatič, 2024).”
Learning, Behavior, and Pedagogy: A Systematic Review of Generative AI Use in Programming Education
Huang, T., & Tseng, H.-P. (2026) · International Journal of Information and Education Technology · doi:10.18178/ijiet.2026.16.1.2487 · Cited by 7
“ChatGPT 3.5 was the most commonly used version, while two studies used both ChatGPT and GitHub Copilot [28, 37].”
InfluentialMartin-Lopez, A., Tufano, R., Guglielmi, E., Sánchez, A. B., Sanz, A. L., Scalabrino, S., Oliveto, R., Segura, S., & Bavota, G. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3679627 · Cited by 6
“Several studies investigated the impact of AI support on students and novice developers [31], [32], [33], [34], [35], with some reporting neutral or positive changes in students’ learning outcomes [32], [33], and others reporting negative effects [31], [34], [35].”
Learning with large language models: beyond prompt engineering
Ding, L., O’Berry, R., Tallent, H., C., S. G. C. G., & Gapud, A. A. (2026) · Education and Information Technologies · doi:10.1007/s10639-026-13924-2 · Cited by 6
“Whereas another study by Jost et al. (2024) in programming education revealed that an increase in students’ reliance on LLMs for critical thinking-intensive tasks, such as code generation and debugging, correlated with lower grades amongst students.”
Evaluating lab assistant chatbot on student learning and behaviors in a programming short course
Noraset, T., Supratak, A., Ragkhitwetsagul, C., Worathong, N., & Tuarob, S. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100527 · Cited by 6
The Influence of AI-assisted Tools on Engineering Project Outcomes
Dinu, A. (2025) · Revista Romaneasca pentru Educatie Multidimensionala · doi:10.18662/rrem/17.3/1024 · Cited by 5
“These tools are often praised for their role in enhancing learning efficiency, aiding in academic writing, and supporting personalized feedback (Jošt et al., 2024; Sajja et al., 2024; Susnjak, 2023).”
Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education
Mak, J., Nakatumba-Nabende, J., Clear, T., Clear, A., Abluwi, I., Andrei, O., Angeli, L., MacNeil, S., Oyelere, S. S., Hale Rattigan, M., et al. (2025) · Proceedings of the 2025 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3760545.3783975 · Cited by 12
Rozputnia, B., Shevchenko, L., Umanets, V., Yashchuk, S., & Sabadosh, Y. (2025) · Environment Technology Resources Proceedings of the International Scientific and Practical Conference · doi:10.17770/etr2025vol3.8537 · Cited by 6
“LLMs necessitate a reevaluation of their educational role [10].”
The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B., Kimmel, B., Wright, J., & Briggs, B. (2024) · arXiv · Cited by 196
Gambo, O. O., Onifade, O. J., Olanrewaju-Smart, W., Abuya, O. J., Salawu, R. E., Massenon, R., & Gambo, I. (2026) · Discover Education · doi:10.1007/s44217-026-01142-4 · Cited by 10
“Some studies, such as those by Jošt et al. [40] and Sok and Heng [89], highlighted that reliance on ChatGPT could lead to superficial learning, where students may obtain solutions without fully understanding the underlying concepts.”
A Generative AI Tool to Foster and Assess Authentic Learning: A Case Study in Teaching SQL
Sooriamurthi, R., Tu, X., & Pensky, A. C. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729022 · Cited by 9
Kamberovic, M., Delić, A., & Krivic, S. (2025) · User Modeling, Adaptation, and Personalization · doi:10.1145/3708319.3733692 · Cited by 5
W iki H int : A Human-Annotated Dataset for Hint Ranking and Generation
Mozafari, J., Gerhold, F., & Jatowt, A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730284 · Cited by 6
Reflexões sobre o Uso de LLMs no Ensino de Programação
Rosa, Y. S., Garcia, P., Constantino, K., & Figueiredo, E. (2025) · Anais do V Simpósio Brasileiro de Educação em Computação (EDUCOMP 2025) · doi:10.5753/educomp.2025.5366 · Cited by 4
ÖZGÜL, E. (2026) · Kastamonu Eğitim Dergisi · doi:10.24106/kefdergi.1878122 · Cited by 3
GenAI Voice Mode in Programming Education
Jacobs, S., & Kiesler, N. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770001 · Cited by 7
Benevides, K. D. G., Rufino, L. J. G. C., dos Santos, D. F. A., Benevides, M. P., Pimenta, C. D., de Oliveira, M. R., & Moura, R. A. D. (2025) · Aracê. · doi:10.56238/arev7n8-085 · Cited by 17
Hernandez, A. A., Albina, E. M., & Caballero, A. R. (2025) · International Symposium on Computer, Consumer and Control · doi:10.1109/is3c65361.2025.11131023 · Cited by 7
“Moreso, professors experience high workload and time constraints, which limits their capacity to provide aligned feedback to students on programming assessment tasks [9].”
Refuting LLM-generated Code with Reactive Task Comprehension
Hebbar, S. V., Sasmita Harini, S., & Kumar, V. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729100 · Cited by 4
Research trends and the impact of ChatGPT on educational environments
Firdaus, T., Mufidah, R., Hamida, R. N., Febrianti, R. I., & Guivara, A. E. R. (2025) · International Journal of Social Sciences and Education Research · doi:10.24289/ijsser.1609741 · Cited by 6
Bidry, M., Hanine, M., Ouaguid, A., & Obidallah, W. J. (2025) · IEEE Access · doi:10.1109/access.2025.3636891 · Cited by 12
“In addition to text-based tools, visual generative models such as DALL · E and Midjourney [60], [61] have gained popularity in educational environments that emphasize creativity and design [54].”
Generative AI in higher education: A systematic review of empirical uses, outcomes, and risks
Hawi, N. S., & Samaha, M. (2026) · International Journal of Educational Research Open · doi:10.1016/j.ijedro.2026.100584 · Cited by 5
Effects of Generative AI Tools On Students’ Critical Thinking: A Three-Level Meta-Analysis
Sun, X., Liu, C., Li, P., & Cai, Z. (2026) · Educational Psychology Review · doi:10.1007/s10648-026-10180-1 · Cited by 7
Razzouki, M., Ragmoun, W., & Hammou, S. B. (2026) · PLoS ONE · doi:10.1371/journal.pone.0347120 · Cited by 2
“Learning outcomes are defined statements that outline what learners should be able to consider, comprehend, and achieve following specific learning experiences [24,1].”
STAP: A Socratic Tutor for Adaptive Programming with Pedagogical Scaffolding
Xie, W., Shi, H., Li, Y.-X., & Liu, J.-W. (2025) · Proceedings of the 2025 2nd International Symposium on Artificial Intelligence for Education · doi:10.1145/3775073.3775165 · Cited by 3
Concepts for Teaching Software Development in the Age of AI-Tools
Böttcher, A., Thurner, V., & Zönnchen, B. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016505 · Cited by 3
“For instance, while [9] found no significant impact of ChatGPT on the students’ learning outcomes in introductory programming, [10] reported a negative correlation between increased usage of large language models on the one hand, and both students’ grades and critical thinking skills on the other…”
Coelho, R., Bjune, A. E., Ellingsen, S., Solheim, B. M., Thormodsæter, R. S., Wasson, B., & Cotner, S. H. (2025) · Journal of Science Education and Technology · doi:10.1007/s10956-025-10216-1 · Cited by 4
“Moreover, research in programming, for example, indicates that introducing GenAI tools before students develop foundational problem-solving skills can hinder learning (Jošt et al., 2024).”
Andriyana, A., Hidayat, A., Hanifah, I., & Ramadona, D. C. (2026) · Buletin Pengembangan Perangkat Pembelajaran · doi:10.23917/bppp.v8i1.16355 · Cited by 1
“Lebih luas lagi, visi pengembangan LLM dalam pendidikan menempatkan teknologi ini sebagai komponen strategis dalam ekosistem pembelajaran digital yang mampu mendukung personalisasi, akses pengetahuan, dan pengambilan keputusan pedagogis berbasis data [9].”
Janabi, I. A., & Bland, T. (2026) · AI · doi:10.3390/ai7060223 · Cited by 2
EFFICIENCY, ENGAGEMENT, AND COGNITIVE OFFLOADING: A REVIEW OF GENERATIVE AI IN EDUCATION
Khalid, N., Kasbun, R., & Rahman, N. F. A. (2026) · International Journal of Education Psychology and Counseling · doi:10.35631/ijepc.1162072 · Cited by 4
Lin, M. P.-C., Huang, J.-Y., Chang, D., Tembrevilla, G., Bowen, G. M., Poitras, E., Janarthanan, V., & Ryoo, J. (2026) · AI in Education · doi:10.3390/aieduc2010004 · Cited by 4
“When instructional tasks and assessments are not explicitly designed to accommodate or constrain AI use, the presence of LLMs may inadvertently undermine the very competencies being evaluated (Jošt et al., 2024; Zhai et al., 2024).”
Assessing the Software Security Comprehension of Large Language ModelsPreprint
Siddiq, M. L., Sekerak, N., Antonio, K., Maria, L., Islam-Gomes, A., & Santos, J. C. S. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2512.21238 · Cited by 5
“Conversely, instructors express concerns about declining problem-solving skills and curriculum integrity [45, 46].”
Schotter, T., Kawas, S., Prather, J., Leinonen, J., Ippolito, J., & Nelson, G. L. (2025) · arXiv.org · doi:10.48550/arxiv.2505.18771 · Cited by 2
Riztha, F., Wickramarachchi, R., Asanka, D., & Disssanayke, M. (2024) · International Conference on Advancements in Computing (ICAC) · doi:10.1109/icac64487.2024.10850942 · Cited by 4
Takamatsu, N., Kamioka, E., Tran, C. M., & Tan, P. X. (2025) · IEEE Access · doi:10.1109/access.2025.3638804 · Cited by 1
“Although their educational utility has been explored in intelligent tutoring systems and programming support tools for beginners [16], [17], applications aimed at improving accessibility for VIs are still rare.”
Help or Hype? Exploring LLM-based Chatbots in Self-Regulated Learning
Tsang, R., & WOOD, S. (2025) · ASEE Annual Conference & Exposition · doi:10.18260/1-2--56687 · Cited by 2
Jović, J. L., Domazet, D. S., Vesić, N. O., Ranđelović, B. M., & Simjanović, D. J. (2026) · Mathematics · doi:10.3390/math14101637 · Cited by 1
In-depth exploration of text similarity calculation based on improved VSM and DeepSeek
Wu, H., Guo, M., & 郭庆琳 (2026) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2026.200703 · Cited by 1
Tang, Z., Wei, W., Liang, K., & Lam, C. K. (2026) · Systems · doi:10.3390/systems14060685 · Cited by 1
Large language models in software development learning
Hernández, S. C., Cruz, M. P. T., Sánchez, J. M. E., Luna-Gómez, A. A., Orozco, A. M. P., & Medina, R. H. (2026) · Discover Education · doi:10.1007/s44217-026-01750-0
“On this topic, the impact of the informal use of LLMs for learning programming was measured with a group of thirty-two students, and it was concluded that there is a negative relationship between the use of these AIs and student performance [69].”
Characterization and Effects of CS2 Learning with GenAI, Visualization, and Human Support
Yong, Q., Nacenta, M. A., & Estey, A. (2026) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3765964.3811663 · Cited by 1
Ouaazki, A., Shibani, A., Knight, S., & Holzer, A. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100608 · Cited by 1
What do students want from AI? Exploring expectations and preferences in programming education
Kamberovic, M., & Krivić, S. (2026) · Artificial Intelligence in Education · doi:10.1108/aiie-08-2025-0260
Adauto-Medina, W., Chamorro-Atalaya, O., Olivares-Zegarra, S., Arévalo-Tuesta, J. A., Arones, M., Aybar-Bellido, I., León-Velarde, C., Fernández-Flores, S., Quispe-Andía, A., & Auqui-Ramos, E. (2026) · Sustainability · doi:10.37766/inplasy2026.1.0066 · Cited by 1
Changes in Coding Behavior and Performance Since the Introduction of LLMs
Zhang, Y., Savelka, J., Goldstein, S. C., & Conway, M. (2026) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3785022.3785075 · Cited by 1
Permatasari, P., Mentari, M., Kinari, S. A., Aung, S. T., Funabiki, N., Kyaw, H. H. S., & Wai, K. (2025) · Analytics · doi:10.3390/analytics5010002 · Cited by 1
“Recent research has shown that the frequent use of Large Language Models (LLM) such as ChatGPT [12] or Copilot [13] for code generation correlates with lower student performances, reinforcing the concern that this AI-assisted coding can lead to academic integrity issues including plagiarism [14].”
Redefining Undergraduate Computing Education with GenAI-Facilitated Authentic In-Term Assessment
Masood, A., Wijeratne, V., & Alam, A. S. (2025) · IEEE International Conference on Teaching, Assessment, and Learning for Engineering · doi:10.1109/tale66047.2025.11346650 · Cited by 1
Student-Perceived Cognitive Load of LLM-Generated Programming Exercises
Khan, M. F. A., Feri, L., Nguyen, H., & Karimi, H. (2025) · International Conference on Data Science and Advanced Analytics · doi:10.1109/dsaa65442.2025.11247983 · Cited by 1
Avaliando o Papel do GPT-4o na Aprendizagem de Engenharia de Software: Um Relato de Experiência
Branco, B. M. D. C. C., Neto, A. L. D. M., Avelino, G. A., & Machado, V. P. (2025) · Simpósio Brasileiro de Engenharia de Software (SBES) · doi:10.5753/sbes.2025.10716 · Cited by 1
Lin, T.-C., Chiu, C., Wang, P.-T., & Fang, L.-D. (2025) · Multimedia · doi:10.3390/multimedia1010003 · Cited by 4
“Jošt et al. [30] and Zhang et al. [31] quantified the impact of large language models on programming education, demonstrating a 34% improvement in skill acquisition and a 41% reduction in learning time through AI-assisted instruction.”
Can Large Language Models Help Students Prove Software Correctness? An Experimental Study with Dafny
Carreira, C., Silva, Á. F., Abreu, A., & Mendes, A. (2025) · IEEE International Conference on Software Engineering and Formal Methods · doi:10.48550/arxiv.2506.22370 · Cited by 1
“This disconnect raises important questions about dependency, as students may rely on LLMs as a substitute for their reasoning, consistent with prior work [6,18].”
Hyseni, L. N., & Dermaku, A. (2025) · International Journal of Computational and Experimental Science and Engineering · doi:10.22399/ijcesen.1846 · Cited by 2
Exploring the Role of Large Language Models as Artificial Tutors
Zönnchen, B., Hobelsberger, M., Socher, G., Thurner, V., & Ottinger, S. (2025) · IEEE Global Engineering Education Conference (EDUCON) · doi:10.1109/educon62633.2025.11016571 · Cited by 1
“Studies present mixed findings: some report improved learning outcomes with LLM integration in programming courses [12], while others find no significant effect [13] or even negative outcomes [14].”
Learning a First Programming Language with LLM Tutor
Vukojičić, M., & Korica, I. (2025) · SINTEZA · doi:10.15308/sinteza-2025-331-336 · Cited by 1
“Some of the results are not in favour of LLMs in the domain of higher education [4] in addition to the domain of lower secondary education [5].”
Exploring the Impact of Large Language Models on Building Construction Students: A Pilot Study
Leathem, T., Wetzel, E., Farrow, C. B., & Umer, M. (2025) · EPiC series in built environment · doi:10.29007/llkw · Cited by 1
“However, there are also concerns about their potential to promote plagiarism and hinder deep learning (Jošt et al., 2024; Kasneci et al., 2023).”
Zhu, S., & Liu, Z. (2026) · Proceedings of the 2026 International Conference on Big Data and Informatization Education · doi:10.1145/3806980.3806985
Song, J., Wei, Z., He, C., Liao, J., Xu, X., & Yip, D. K. M. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2745545
Abramskiy, M. M., Ozdemir, G., Khamitova, I., & Shapkina, I. (2026) · Education & Self Development · doi:10.26907/esd.21.3.07
Biczó, Z. (2026) · Society and Economy · doi:10.14267/1588970x.2026.024
Burger, C., Trotter, C., Carlisle, J. D., & Walter, C. (2026) · doi:10.48550/arxiv.2609.26733
A Critical Review of Critical Thinking in HCI Research About AI
Inie, N., Kristensen, A. M., Misfeldt, M., & Dalsgaard, P. (2026) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2026.2722522
Rashidov, A., & Rashidova, F. (2026) · 2026 11th International Conference on Computer Science and Engineering (UBMK) · doi:10.1109/ubmk71847.2026.11745325
“In engineering education, this is of particular significance, as learning activities often involve not only the acquisition of theoretical knowledge but also the development of practical skills, technical thinking and the ability to solve problems independently [4].”
Nephron Flow Lab: Vibe Coding a Visual Nephrology Learning App
Calderwood, I., & Bland, T. (2026) · Multimodal Technologies and Interaction · doi:10.3390/mti10090093
Gadot, R., & Tsybulsky, D. (2026) · Education Sciences · doi:10.3390/educsci16091402
Examining Student Dependence on Generative AI tools in Programming Education
Abioye, K. B. (2026) · Journal of Education Learning and Management · doi:10.69739/jelm.v3i2.2151
“Similar concerns were reported by Jošt et al. (2024), who found a significant negative relationship between LLM use and final grades, particularly in tasks such as code generation and debugging.”
Zuin, M., & Donadel, V. (2026) · European Journal of Investigation in Health Psychology and Education · doi:10.3390/ejihpe16090130
“…development, potentially creating conditions of overreliance and hindering a critical and informed approach to managing the information provided by AI systems (Danaher, 2018; Shanmugasundaram & Tamilarasu, 2023; Jošt et al., 2024; Zhai et al., 2024; Abubakar et al., 2025; Sarwar et al., 2026).”
The Governance Gap in Contemporary LLM-Based Agentic Systems: A Structural Diagnostic Review
Valdez-Cantú, C., Cantoral-Ceballos, J. A., & Alvarado-Uribe, J. (2026) · AI · doi:10.3390/ai7080322
How well do LLMs understand code?
Xu, J., Sun, R., Ding, Z., Li, W., Han, K., Zhang, H., Wang, C., Wang, H., Wang, W. W., Sun, Y., et al. (2026) · Communications AI & Computing · doi:10.1038/s44488-026-00014-y
Scaffolding Autocomplete: Improving Guidance for Learners using Generative Code Suggestions
Prather, J. E., MacNeil, S., Luxton-Reilly, A., Margulieux, L. E., Reeves, B. N., Denny, P. C., Leinonen, J., Homer, J., Nabid, R. A., & Rossetti, R. L. (2026) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3765964.3811654
Teaching Programming in the Age of AI: Pedagogical Strategies for Computational Sociology
Ramos, M. C., & Liang, F. (2026) · Teaching Sociology · doi:10.1177/0092055x261469125
“AI tools are generally more effective when used for comprehension and refinement after students have attempted to solve a problem on their own rather than for generating complete solutions (Güner and Er 2025; Jošt et al. 2024; Prather et al. 2025).”
Álvarez-Jiménez, M., Pomagualli, A. G. C., Gamboa, D., & Oscullo, D. S. P. (2026) · Computers · doi:10.3390/computers15070466 · Cited by 1
“Its benefits are more likely when use is structured and instructor-guided [23], whereas substituting core cognitive work with GenAI is associated with superficial or even negative learning outcomes [4,5].”
Ding, L., Rodenberg, R., Er, E., Nguyen, H., & Yoon, M. (2026) · Interactive Learning Environments · doi:10.1080/10494820.2026.2702559
Wang, C., Petrie, C., Stouras, M., Ettlin, N., George, A., Mejia-Domenzain, P., Swamy, V., Käser, T., & Svensson, O. (2026) · ACM Conference on Learning @ Scale · doi:10.1145/3774398.3811623
Metadata
- Authors
- Jošt, G., Taneski, V., & Karakatič, S.
- Affiliations
- University of Maribor (SI)
- Corresponding author
- Jošt, Gregor
- Venue
- Applied Sciences
- Publisher
- MDPI AG
- Published
- 2024-05-13
- Volume
- 14
- Issue
- 10
- Pages
- 4115
- ISSN
- 2076-3417 (ISSN-L)
- OpenAlex
- W4396896417
- Cited by
- 143
- References
- 16
- Citations by year
- 2026: 66 · 2025: 69 · 2024: 8
- FWCI
- 53.95
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Applied Sciences (publishedVersion, cc-by)
- dCOBISS.SI Digital Repository (publishedVersion, cc-by)
- University of Maribor digital library (University of Maribor) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Slovenian Research and Innovation Agency (ARIS) (P2-0057)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Social Sciences › Education › Artificial Intelligence in Education
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- large language models, programming education, learning outcomes, code generation, debugging, React applications, undergraduate students, critical thinking, problem-solving skills, ChatGPT, Copilot
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
