CoChat
Sign Up
Cited by 131Teaching and Learning ProgrammingArtificial Intelligence in EducationIntelligent Tutoring Systems and Adaptive Learning

Evaluating the Effectiveness of LLMs in Introductory Computer Science Education: A Semester-Long Field Study

Open access
Lyu, Wenhan, Wang, Yimeng, Chung, Tingting (Rachel), Sun, Yifan, Zhang, Yixuan·2024·Proceedings of the Eleventh ACM Conference on Learning @ Scale
DOI 10.1145/3657604.3662036
Open this paper in CoChat — free account

Abstract

The integration of AI assistants, especially through the development of Large Language Models (LLMs), into computer science education has sparked significant debate, highlighting both their potential to augment student learning and the risks associated with their misuse. An emerging body of work has looked into using LLMs in education, primarily focusing on evaluating the performance of existing models or conducting short-term human subject studies. However, very little work has examined the impacts of LLM-powered assistants on students in entry-level programming courses, particularly in real-world contexts and over extended periods. To address this research gap, we conducted a semester-long, between-subjects study with 50 students using CodeTutor, an LLM-powered assistant developed by our research team. Our study results show that students who used CodeTutor (the "CodeTutor group" as the experimental group) achieved statistically significant improvements in their final scores compared to peers who did not use the tool (the "control group"). Within the CodeTutor group, those without prior experience with LLM-powered tools demonstrated significantly greater performance gain than their counterparts. We also found that students expressed positive feedback regarding CodeTutor's capability to comprehend their queries and assist in learning programming language syntax. However, they had concerns about CodeTutor's limited role in developing critical thinking skills. Over the course of the semester, students' agreement with CodeTutor's suggestions decreased, with a growing preference for support from traditional human teaching assistants. Our findings also show that students turned to CodeTutor for different tasks, including programming task completion, syntax comprehension, and debugging, particularly seeking help for programming assignments. Our analysis further reveals that the quality of user prompts was significantly correlated with CodeTutor's response effectiveness. Building upon these results, we discuss the implications of our findings for the need to integrate Generative AI literacy into curricula to foster critical thinking skills, and turn to examining the temporal dynamics of user engagement with LLM-powered tools. We further discuss the discrepancy between the anticipated functions of tools and students' actual capabilities, which sheds light on the need for tailored strategies to improve educational outcomes.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3657604.3662036

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • This offers a process-level account of how performance and engagement metrics can improve while markers of conceptual integration or transfer may lag.

    “This offers a process-level account of how performance and engagement metrics can improve (Fan et al., 2025; Yilmaz & Yilmaz, 2023; Lyu et al., 2024; Qi et al., 2025) while markers of conceptual integration or transfer may lag (Jose et al., 2025; Bastani et al., 2024).”

    From Tool, tutor, or crutch?: A grounded theory of cognitive scaffolding and offloading in AI-assisted programming education (2026) · cites this paper as Lyu et al., 2024

    Section 4.1 (pp. 5–6) reports significant score improvements for students using CodeTutor, supporting the performance portion attributed to Lyu et al. Section 4.3 (pp. 7–8) documents engagement through conversations but does not show that engagement metrics improved. The sentence cites other papers for the separate claim about conceptual integration or transfer lagging. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Large-sample and field studies report robust performance and affect benefits with GenAI, and meta-analyses suggest positive effects on both lower- and higher-order outcomes as well as motivation/engagement, moderated by discipline, learning strategy, and context.

    “Large-sample and field studies report robust performance and affect benefits with GenAI (e.g., efficiency, accuracy, reduced anxiety) (Fan et al., 2025; Yilmaz & Yilmaz, 2023; Lyu et al., 2024; Shihab et al., 2025), and meta-analyses further suggest positive effects on both lower- and higher-order outcomes as well as motivation/engagement, moderated by discipline, learning strategy, and context (X”

    From Tool, tutor, or crutch?: A grounded theory of cognitive scaffolding and offloading in AI-assisted programming education (2026) · cites this paper as Lyu et al., 2024

    Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

  • AI assistance can improve short-term performance.

    “Empirical evidence suggests that AI assistance can improve short-term performance.”

    From Tool, tutor, or crutch?: A grounded theory of cognitive scaffolding and offloading in AI-assisted programming education (2026) · cites this paper as Lyu et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 41

  1. [1][1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023). https://doi.org/10.48550/arXiv.2303.08774 - · DOI · Paper page
  2. [3][3] John R Anderson, C Franklin Boyle, and Brian J Reiser. 1985. Intelligent tutoring systems. Science 228, 4698 (1985), 456-462. https://doi.org/10.1126/science.228. 4698.456 - · DOI
  3. [4][4] Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software 67, 1 (2015), 1-48. https://doi.org/10.18637/jss.v067.i01 - · DOI · Paper page
  4. [5][5] Peter Brusilovsky et al. 1998. Adaptive educational systems on the world-wideweb: A review of available technologies. In Proceedings of Workshop" WWW-Based Tutoring" at 4th International Conference on Intelligent Tutoring Systems (ITS'98), San Antonio, TX . -
  5. [6][6] Peter Brusilovsky, Elmar Schwarz, and Gerhard Weber. 1996. ELM-ART: An intelligent tutoring system on World Wide Web. In Intelligent Tutoring Systems: Third International Conference, ITS'96 Montréal, Canada, June 12-14, 1996 Proceedings 3 . Springer, 261-269. https://doi.org/10.1007/3-540-61327-7\_123 - · DOI
  6. [7][7] Cory J Butz, Shan Hua, and R Brien Maguire. 2006. A web-based bayesian intelligent tutoring system for computer programming. Web Intelligence and Agent Systems: An International Journal 4, 1 (2006), 77-97. -
  7. [8][8] Jan Clusmann, Fiona R Kolbinger, Hannah Sophie Muti, Zunamys I Carrero, Jan-Niklas Eckardt, Narmin Ghaffari Laleh, Chiara Maria Lavinia Löffler, SophieCaroline Schwarzkopf, Michaela Unger, Gregory P Veldhuizen, et al. 2023. The future landscape of large language models in medicine. Communications Medicine 3, 1 (2023), 141. https://doi.org/10.1038/s43856-023-00370-1 - · DOI · Paper page
  8. [9][9] Albert T Corbett, Kenneth R Koedinger, and John R Anderson. 1997. Intelligent tutoring systems. In Handbook of human-computer interaction . Elsevier, 849-874. https://doi.org/10.1016/B978-044481862-1.50103-5 - · DOI
  9. [10][10] Dorottya Demszky and Jing Liu. 2023. M-Powering Teachers: Natural Language Processing Powered Feedback Improves 1: 1 Instruction and Student Outcomes. (2023). https://doi.org/10.1145/3573051.3593379 - · DOI
  10. [11][11] Paul Denny, Sami Sarsa, Arto Hellas, and Juho Leinonen. 2022. Robosourcing Educational Resources-Leveraging Large Language Models for Learnersourcing. arXiv preprint arXiv:2211.04715 (2022). https://doi.org/10.1145/3501385.3543957 - · DOI · Paper page
  11. [12][12] Felix Dobslaw and Peter Bergh. 2023. Experiences with Remote Examination Formats in Light of GPT-4. arXiv preprint arXiv:2305.02198 (2023). https://doi. org/10.48550/arXiv.2305.02198 - · DOI
  12. [13][13] Gilan M El Saadawi, Eugene Tseytlin, Elizabeth Legowski, Drazen Jukic, Melissa Castine, Jeffrey Fine, Robert Gormley, and Rebecca S Crowley. 2008. A natural language intelligent tutoring system for training pathologists: Implementation and evaluation. Advances in health sciences education 13 (2008), 709-722. https: //doi.org/10.1007/s10459-007-9081-3 - · DOI · Paper page
  13. [14][14] Mark Elsom-Cook. 1984. Design considerations of an intelligent tutoring system for programming languages . Ph. D. Dissertation. University of Warwick. -
  14. [16][16] Arthur C Graesser, Xiangen Hu, and Robert Sottilare. 2018. Intelligent tutoring systems. In International handbook of the learning sciences . Routledge, 246-255. -
  15. [17][17] Morgan Gustafson. 2022. The Effect of Homework Completion on Students' Academic Performance. Dissertations, Theses, and Projects. https://red.mnstate. edu/thesis/662 662. -
  16. [18][18] Yann Hicke, Anmol Agarwal, Qianou Ma, and Paul Denny. 2023. ChaTA: Towards an Intelligent Question-Answer Teaching Assistant using Open-Source LLMs. arXiv preprint arXiv:2311.02775 (2023). https://doi.org/10.48550/arXiv.2311.02775 - · DOI
  17. [19][19] Danial Hooshyar, Rodina Binti Ahmad, Moslem Yousefi, Farrah Dina Yusop, and S-J Horng. 2015. A flowchart-based intelligent tutoring system for improving problem-solving skills of novice programmers. Journal of computer assisted learning 31, 4 (2015), 345-361. https://doi.org/10.1111/jcal.12099 - · DOI
  18. [23][23] Harsh Kumar, Ilya Musabirov, Mohi Reza, Jiakai Shi, Anastasia Kuzminykh, Joseph Jay Williams, and Michael Liut. 2023. Impact of Guidance and Interaction 24. Strategies for LLM Use on Learner Performance and Perception. arXiv preprint arXiv:2310.13712 (2023). https://doi.org/10.48550/arXiv.2310.13712 - · DOI
  19. [24][24] Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing code explanations created by students and large language models. arXiv preprint arXiv:2304.03938 (2023). https://doi.org/10.48550/arXiv.2304.03938 - · DOI
  20. [25][25] Juho Leinonen, Arto Hellas, Sami Sarsa, Brent Reeves, Paul Denny, James Prather, and Brett A Becker. 2023. Using large language models to enhance programming error messages. In Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 . 563-569. https://doi.org/10.1145/3545945.3569770 - · DOI · Paper page
  21. [26][26] Mark Liffiton, Brad E Sheese, Jaromir Savelka, and Paul Denny. [n. d.]. Codehelp: Using large language models with guardrails for scalable support in programming classes. ([n. d.]), 1-11. https://doi.org/10.1145/3631802.3631830 - · DOI · Paper page
  22. [27][27] Atharva Mehta, Nipun Gupta, Dhruv Kumar, Pankaj Jalote, et al. 2023. Can ChatGPT Play the Role of a Teaching Assistant in an Introductory Programming Course? arXiv preprint arXiv:2312.07343 (2023). https://doi.org/10.48550/arXiv. 2312.07343 - · DOI
  23. [28][28] Jesse G Meyer, Ryan J Urbanowicz, Patrick CN Martin, Karen O'Connor, Ruowang Li, Pei-Chen Peng, Tiffani J Bright, Nicholas Tatonetti, Kyoung Jae Won, Graciela Gonzalez-Hernandez, et al. 2023. ChatGPT and large language models in academia: opportunities and challenges. BioData Mining 16, 1 (2023), 20. https://doi.org/10.1186/s13040-023-00339-9 - · DOI · Paper page
  24. [29][29] Hyacinth S Nwana. 1990. Intelligent tutoring systems: an overview. Artificial Intelligence Review 4, 4 (1990), 251-277. https://doi.org/10.1007/BF00168958 - · DOI · Paper page
  25. [35][35] Maciej Pankiewicz and Ryan S Baker. 2023. Large Language Models (GPT) for automating feedback on programming assignments. arXiv preprint arXiv:2307.00150 (2023). https://doi.org/10.48550/arXiv.2307.00150 - · DOI
  26. [36][36] Mike Perkins, Jasper Roe, Darius Postma, James McGaughran, and Don Hickerson. 2023. Detection of GPT-4 generated text in higher education: Combining academic judgement and software to identify generative AI tool misuse. Journal of Academic Ethics (2023), 1-25. https://doi.org/10.1007/s10805-023-09492-6 - · DOI · Paper page
  27. [37][37] Tung Phung, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023. Generating High-Precision Feedback for Programming Syntax Errors using Large Language Models. arXiv preprint arXiv:2302.04662 (2023). https://doi.org/10.48550/arXiv.2302.04662 - · DOI · Paper page
  28. [38][38] Tung Phung, Victor-Alexandru Pădurean, José Cambronero, Sumit Gulwani, Tobias Kohn, Rupak Majumdar, Adish Singla, and Gustavo Soares. 2023. Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4, and Human Tutors. International Journal of Management 21, 2 (2023), 100790. https://doi.org/10.48550/arXiv.2306.17156 - · DOI
  29. [39][39] Russell A Poldrack, Thomas Lu, and Gašper Beguš. 2023. AI-assisted coding: Experiments with GPT-4. arXiv preprint arXiv:2304.13187 (2023). https://doi.org/ 10.48550/arXiv.2304.13187 - · DOI
  30. [42][42] Steven Ritter, John R Anderson, Kenneth R Koedinger, and Albert Corbett. 2007. Cognitive Tutor: Applied research in mathematics education. Psychonomic bulletin & review 14 (2007), 249-255. https://doi.org/10.3758/BF03194060 - · DOI
  31. [43][43] Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. 2022. Automatic generation of programming exercises and code explanations using large language models. In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1 . 27-43. -
  32. [45][45] Brad Sheese, Mark Liffiton, Jaromir Savelka, and Paul Denny. 2023. Patterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant. arXiv preprint arXiv:2310.16984 (2023). https://doi.org/10.1145/ 3636243.3636249 - · arXiv
  33. [46][46] Derek Sleeman and John Seely Brown. 1982. Intelligent tutoring systems . London: Academic Press. -
  34. [48][48] Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, et al. 2024. Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561 (2024). https://doi.org/10.48550/arXiv.2401.05561 - · DOI · Paper page
  35. [49][49] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. 2023. Large language models in medicine. Nature medicine 29, 8 (2023), 1930-1940. https://doi.org/10.1038/ s41591-023-02448-8 -
  36. [53][53] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022). https://doi.org/10.48550/arXiv.2206.07682 - · DOI · Paper page
  37. [54][54] Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Derek F Wong, and Lidia S Chao. 2023. A survey on llm-gernerated text detection: Necessity, methods, and future directions. arXiv preprint arXiv:2310.14724 (2023). https://doi.org/10. [48550/arXiv.2310.14724](https://doi.org/10.48550/arXiv.2310.14724) - · DOI · Paper page
  38. [55][55] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. 2023. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564 (2023). https://doi.org/10.48550/arXiv.2303.17564 - · DOI · Paper page
  39. [57][57] Yixuan Zhang, Yimeng Wang, Nutchanon Yongsatianchot, Joseph D Gaggiano, Nurul M Suhaimi, Anne Okrah, Miso Kim, Jacqueline Griffin, and Andrea G Parker. 2024. Profiling the Dynamics of Trust & Distrust in Social Media: A Survey Study. (2024). https://doi.org/10.1145/3613904.3642927 - · DOI · Paper page
  40. [59][59] Kyrie Zhixuan Zhou, Zachary Kilhoffer, Madelyn Rose Sanfilippo, Ted Underwood, Ece Gumusel, Mengyi Wei, Abhinav Choudhry, and Jinjun Xiong. 2024. "The teachers are confused as well": A Multiple-Stakeholder Ethics Discussion on Large Language Models in Computing Education. arXiv preprint arXiv:2401.12453 (2024). https://doi.org/10.48550/arXiv.2401.12453 - · DOI
  41. [60][60] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910 (2022). https://doi.org/10. 48550/arXiv.2211.01910 · arXiv

Cited by

Sorted by citations

Authors
Lyu, W., Wang, Y., Chung, T. (., Sun, Y., & Zhang, Y.
Affiliations
William & Mary (US)
ORCID
Venue
Proceedings of the Eleventh ACM Conference on Learning @ Scale
Publisher
ACM
Published
2024-07-09
Pages
63-74
DOI
10.1145/3657604.3662036
OpenAlex
W4395065007
Cited by
131
References
11
Citations by year
2026: 66 · 2025: 61 · 2024: 4
FWCI
57.42
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Conference on Learning @ Scale (L@S) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords
large language models, introductory computer science, programming education, generative AI literacy, student perceptions, critical thinking skills, prompt quality, long-term field experiment

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review