Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Advances in natural language processing have resulted in large language models (LLMs) that can generate code and code explanations. In this paper, we report on our experiences generating multiple code explanation types using LLMs and integrating them into an interactive e-book on web software development. Three different types of explanations -- a line-by-line explanation, a list of important concepts, and a high-level summary of the code -- were created. Students could view explanations by clicking a button next to code snippets, which showed the explanation and asked about its utility. Our results show that all explanation types were viewed by students and that the majority of students perceived the code explanations as helpful to them. However, student engagement varied by code snippet complexity, explanation type, and code snippet length. Drawing on our experiences, we discuss future directions for integrating explanations generated by LLMs into CS classrooms.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This has been inspired by many other instances where LLMs have already demonstrated remarkable effectiveness in generating novel language artifacts. These include course readings [22], code explanations [23, 28, 29, 41], model solutions [9, 35, 42], feedback [34], or responses to help requests [18, 26, 46, 47].”
From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [28]
In the citing paper
“On the other hand, recent work has shown that AI tools can be used to produce high-quality learning resources, including programming exercises [32], explanations of code [24,28], and even on-demand feedback to students [27].”
From Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study (2024) · cites this paper as [28]
In the citing paper
“MacNeil et al. found that students generally found code explanations generated by LLMs useful for learning, but the perceived utility of the explanations and students' engagement with them varied by explanation type [137].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [137]
In the citing paper
“However, non-code models such as GPT-3 have been found to be more apt at creating higher level code explanations [137].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [137]
In the citing paper
“Students rated these explanations on a 5-point Likert scale, confirming that explanations matched the code and were useful for learning in general, whereas line-by-line explanations were rated as least helpful.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [137]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Review of recent systems for automatic assessment of programming assignments
Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O. (2010) · Proceedings of the 10th Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930480 · Cited by 554 · cited 1×
“through the use of automated grading systems [1, 18, 30].”
MethodAutomatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · cited 8×
“Explanations of code can also be generated by LLMs [24, 37].”
BackgroundMethodInfluentialSelf-citationOnline python tutor: embeddable web-based program visualization for cs education
Guo, P. J. (2013) · Proceeding of the 44th ACM technical symposium on Computer science education · doi:10.1145/2445196.2445368 · Cited by 396
Designing the whyline: a debugging interface for asking questions about program behavior
Ko, A. J., & Myers, B. A. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985712 · Cited by 361
PeerWise: students sharing their multiple choice questions
Denny, P., Hamer, J., Luxton-Reilly, A., & Purchase, H. (2008) · Proceedings of the Fourth international Workshop on Computing Education Research · doi:10.1145/1404520.1404526 · Cited by 211
Self-citationPair programming in education: a literature review
Hanks, B., Fitzgerald, S., McCauley, R., Murphy, L., & Zander, C. (2011) · Computer Science Education · doi:10.1080/08993408.2011.579808 · Cited by 205 · cited 1×
“For example in pair programming, students work together which requires them to often explain their code and their thinking process to their partner [16].”
BackgroundGenerating Diverse Code Explanations using the GPT-3 Large Language Model
MacNeil, S., Tran, A., Mogil, D., Bernstein, S., Ross, E., & Huang, Z. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 2 · doi:10.1145/3501709.3544280 · Cited by 200 · cited 6×
“In computer science classrooms, LLMs are beginning to be used to generate assignments [36], help students to write their code [3, 40], and to generate explanations of code snippets to facilitate learning [23].”
BackgroundMethodInfluentialSelf-citationWheel-Spinning: Students Who Fail to Master a Skill
Beck, J. E., & Gong, Y. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-39112-5_44 · Cited by 161 · cited 1×
“However, these grading systems have become increasingly focused on providing formative feedback such as hints [31, 34] error-specific feedback [24, 34] to help students avoid getting stuck in counterproductive behaviors when their answers are incorrect [4].”
BackgroundiSnap: Towards Intelligent Tutoring in Novice Programming Environments
Price, T. W., Dong, Y., & Lipovac, D. (2017) · Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education · doi:10.1145/3017680.3017762 · Cited by 140
Research works printed but not linked: 12
- Kirsti M Ala-Mutka. 2005. A survey of automated assessment approaches for programming assignments. Computer science education, Vol. 15, 2 (2005), 83--102.
- Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le and Charles Sutton. 2021. Program Synthesis with Large Language Models. arxiv: 2108.07732 [cs.PL] · arXiv
- Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in neural information processing systems. 1877--1901.
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). · arXiv
- James Finnie-Ansley, Paul Denny, Brett A Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conf. 10--19.
- Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022. Sparks: Inspiration for Science Writing Using Language Models. In Designing Interactive Systems Conf. (Virtual Event, Australia) (DIS '22). ACM, New York, NY, USA, 1002--1019.
- Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586 (2021). · arXiv
- José Carlos Paiva, José Paulo Leal, and Álvaro Figueira. 2022. Automated Assessment in Computer Science Education: A State-of-the-Art Review. ACM Transactions on Computing Education (TOCE) (2022).
- Jaanus Pöial. 2020. Challenges of Teaching Programming in StackOverflow Era. In International Conf. on Interactive Collaborative Learning. Springer, 703--710.
- Wengran Wang, Yudong Rao, Rui Zhi, Samiha Marwan, Ge Gao, and Thomas W. Price. 2020. Step Tutor: Supporting Students through Step-by-Step Example-Based Feedback. In Proceedings of the 2020 ACM Conf. on Innovation and Technology in Computer Science Education. ACM, New York, NY, USA, 391--397.
- Jacqueline L. Whalley, Raymond Lister, Errol Thompson, Tony Clear, Phil Robbins, P. K. Ajith Kumar, and Christine Prasad. 2006. An Australasian Study of Reading and Comprehension Skills in Novice Programmers, Using the Bloom and SOLO Taxonomies. In Proc. of the 8th Australasian Conf. on Computing Education - Volume 52. Australian Computer Society, Inc., AUS, 243--252.
- Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft: Story Writing With Large Language Models. In 27th International Conf. on Intelligent User Interfaces. 841--852.
Cited by
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“Initial evaluation by MacNeil et al. [80] on their interactive Web development e-book, suggested that a majority of students perceived LLM-generated code explanations to be helpful.”
BackgroundMethodThe promise and challenges of generative AI in education
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2394886 · Cited by 543
“In particular, following early promising results, several teams attempted to leverage LLMs’ code explanation ability to produce working code examples, that is, examples of programming problem solutions augmented with code explanations (Hassany et al. 2023; Jury et al. 2024; MacNeil et al. 2023).”
Large Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
“Initial evaluation by MacNeil et al. [80] on their interactive Web development e-book, suggested that a majority of students perceived LLM-generated code explanations to be helpful.”
BackgroundMethodUsing an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
Self-citationComputing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316
“This follows a similar evaluation approach employed by the Codex developers [5].”
MethodSelf-citationKazemitabaar, M., Ye, R., Wang, X., Henley, A. Z., Denny, P., Craig, M., & Grossman, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642773 · Cited by 290
Self-citationPrather, J., Reeves, B. N., Denny, P., Becker, B. A., Leinonen, J., Luxton-Reilly, A., Powell, G., Finnie-Ansley, J., & Santos, E. A. (2024) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3617367 · Cited by 256
“Large language models have also been found to be capable of producing pedagogical code explanations [58, 67, 94].”
BackgroundSelf-citationAutomated Unit Test Improvement using Large Language Models at Meta
Alshahwan, N., Chheda, J., Finogenova, A., Gokkaya, B., Harman, M., Harper, I., Marginean, A., Sengupta, S., & Wang, E. (2024) · Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering · doi:10.1145/3663529.3663839 · Cited by 144
“A debate also continues concerning whether LLMs are beneficial or harmful to education, for example, with arguments against [33], and broadly in favor [20] of their use as educational tools, or as ways to support existing human-led education efforts [30, 38, 42].”
BackgroundLau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241
“used these tools to generate explanations for CS1-level code [68] and then embedded them within an interactive e-book to see how students engage with them [66].”
MethodLarge Language Models Meet NL2Code: A Survey
Zan, D., Chen, B., Zhang, F., Lu, D., Wu, B., Guan, B., Yongji, W., & Lou, J.-G. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.411 · Cited by 114
Towards an understanding of large language models in software engineering tasks
Zheng, Z., Ning, K., Zhong, Q., Chen, J., Chen, W., Guo, L., Wang, W., & Wang, Y. (2025) · Empirical Software Engineering · doi:10.1007/s10664-024-10602-0 · Cited by 116
Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · arXiv · doi:10.48550/arxiv.2306.05715 · Cited by 211
Self-citationComparing Code Explanations Created by Students and Large Language Models
Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., & Hellas, A. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588785 · Cited by 210
“MacNeil et al. used the GPT-3 model to generate explanations of short code fragments which then were presented to students in an online e-book alongside the corresponding code [24].”
BackgroundMethodSelf-citationThe Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers
Prather, J., Reeves, B. N., Leinonen, J., MacNeil, S., Randrianasolo, A. S., Becker, B. A., Kimmel, B., Wright, J., & Briggs, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671116 · Cited by 196
Self-citationPrompt Problems: A New Programming Exercise for the Generative AI Era
Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630909 · Cited by 187
Self-citationPrather, J., Leinonen, J., Kiesler, N., Gorson Benario, J., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · 2024 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3689187.3709614 · Cited by 96
“In-deed, GenAI can provide other kinds of advanced and customized help and qualitative feedback to students working on computing assignments [10, 11, 67, 81, 89, 94].”
Self-citationLarge Language Models in Computer Science Education: A Systematic Literature Review
Raihan, M. N., Siddiq, M. L., Santos, J. C. S., & Zampieri, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701863 · Cited by 219
Prompt Engineering or Fine-Tuning: An Empirical Assessment of LLMs for Code
Shin, J., Tang, C. J., Mohati, T., Nayebi, M., Wang, S., & Hemmati, H. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00082 · Cited by 181
“Previous literature also points to the verbosity of LLMs’ explanations of code [40], [87].”
Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145
Self-citationSavelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600142 · Cited by 141
“integrated LLM-generated code explanations into an interactive e-book and compared several different explanation types, such as line-by-line or summarization-oriented explanations, reporting students using line-by-line explanations the most despite them being considered the least useful according to the students [29].”
Desirable Characteristics for AI Teaching Assistants in Programming Education
Denny, P. C., MacNeil, S., Šavelka, J., Porter, L., & Luxton-Reilly, A. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653574 · Cited by 153
Self-citationGenerative AI in Computing Education: Perspectives of Students and Instructors
Zastudil, C., Rogalska, M., Kapp, C., Vaughn, J., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343467 · Cited by 136
“Researchers are already demonstrating the capabilities of these models to help students generate code [2], [3], [25], explain code to students [5], [6], [9], and even produce programming assignments [1], [4].”
BackgroundSelf-citationA Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Doughty, J., Wan, Z., Bompelli, A., Qayum, J., Wang, T., Zhang, J., Zheng, Y., Doyle, A., Sridhar, P., Agarwal, A., et al. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636256 · Cited by 132
Evaluating LLM-generated Worked Examples in an Introductory Programming Course
Jury, B., Lorusso, A., Leinonen, J., Denny, P., & Luxton-Reilly, A. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636252 · Cited by 95
“Due to the relatively recent emergence of LLMs, although they are beginning to be evaluated and integrated into a range of learning tools [16, 20, 33], to our knowledge, they have not been applied to the problem of automatically generating worked examples.”
BackgroundResultSelf-citationPatterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant
Sheese, B., Liffiton, M., Savelka, J., & Denny, P. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636249 · Cited by 74
“Thebreadthofthisworkdemonstratesboththeflexibility andthecapabilitiesofLLMs.”
Self-citationHow Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering
Choudhuri, R., Liu, D., Steinmacher, I., Gerosa, M., & Sarma, A. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639201 · Cited by 72
Next-Step Hint Generation for Introductory Programming Using Large Language Models
Roest, L., Keuning, H., & Jeuring, J. T. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636259 · Cited by 123
“Section 4 discusses this process in detail.”
BackgroundAdvancing building energy modeling with large language models: Exploration and case studies
Zhang, L., Chen, Z., & Ford, V. (2024) · Energy and Buildings · doi:10.1016/j.enbuild.2024.114788 · Cited by 102
“MacNeil et al. [8] reported on their experiences generating multiple code explanation types using LLMs and integrating them into an interactive e-book on web software development.”
MethodLLM for Test Script Generation and Migration: Challenges, Capabilities, and Opportunities
Yu, S., Fang, C., Ling, Y., Wu, C., & Chen, Z. (2023) · 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS) · doi:10.1109/qrs60937.2023.00029 · Cited by 64
Piccolo, S., Denny, P. C., Luxton-Reilly, A., Payne, S., & Ridge, P. G. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1011511 · Cited by 104
Self-citationThe Effects of Generative AI on Computing Students’ Help-Seeking Preferences
Hou, I., Mettille, S., Man, O., Li, Z., Zastudil, C., & MacNeil, S. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636248 · Cited by 97
“…[33, 51, 70] with researchers already demonstrating many remarkable capabilities of generative AI including generating code explanations [34, 39, 42], enhancing error messages [35], identifying bugs [38], solving multiple-choice programming problems [59, 61], solving Parsons problems [22],…”
BackgroundSelf-citationPhung, T., Pădurean, V.-A., Singh, A., Brooks, C., Cambronero, J., Gulwani, S., Singla, A., & Soares, G. (2024) · Proceedings of the 14th Learning Analytics and Knowledge Conference · doi:10.1145/3636555.3636846 · Cited by 63
Feedback-Generation for Programming Exercises With GPT-4
Azaiz, I., Kiesler, N., & Strickroth, S. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653594 · Cited by 115
Generative AI in Education: A Study of Educators' Awareness, Sentiments, and Influencing Factors
Ghimire, A., Pather, J., & Edwards, J. (2024) · 2024 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10892891 · Cited by 88
“They can automatically generate solutions, explanations, and examples to scaffold learning and reduce instructor effort [1], [15]–[18].”
Evaluating Contextually Personalized Programming Exercises Created with Generative AI
Logacheva, E., Hellas, A., Prather, J. E., Sarsa, S., & Leinonen, J. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671103 · Cited by 105
Self-citationPromptly: Using Prompt Problems to Teach Learners How to Effectively Utilize AI Code GeneratorsPreprint
Denny, P. C., Leinonen, J., Prather, J. E., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.16364 · Cited by 96
“used LLM-generated code explanations successfully in a web software development ebook [22], and Zingaro and Porter are completing a textbook for teaching introductory programming with Copilot and ChatGPT from day one [26].”
MethodSelf-citationWang, T., Díaz, D. V., Brown, C., & Chen, Y. (2023) · 2023 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc57772.2023.00018 · Cited by 59
“The challenge, as indicated in previous studies on trustworthy AI [50], is to strike a balance between trust and effective decision-making when interacting with AI assistants.”
BackgroundHarnessing LLMs in Curricular Design: Using GPT-4 to Support Authoring of Learning Objectives
Sridhar, P., Doyle, A., Agarwal, A., Bogart, C., Savelka, J., & Sakr, M. (2023) · arXiv · doi:10.48550/arxiv.2306.17459
“demonstrated that explanations of generated code can be offered at multiple different levels of abstraction [19, 20].”
BackgroundDetecting ChatGPT-Generated Code Submissions in a CS1 Course Using Machine Learning Models
Hoq, M., Shi, Y., Leinonen, J., Babalola, D., Lynch, C. F., Price, T., & Akram, B. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630826 · Cited by 96
“…to substantially support students during their programming process through offering programming support such as code explanation, worked examples, and feedback [19, 34– 36, 48 ], over-reliance over-reliance on such tools can Several studies have proposed methods to detect LLM-generated text.”
BackgroundMethodSelf-citationChatGPT in the Classroom. Exploring Its Potential and Limitations in a Functional Programming Course
Popovici, M.-D. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2269006 · Cited by 70
“MacNeil et al. (2023) concludes that explanations were beneficial for students, however, as far as we could observe, there is no reporting of the accuracy or correctness of the LLM-generated explanations.”
InfluentialMany bioinformatics programming tasks can be automated with ChatGPT
Piccolo, S. R., Denny, P., Luxton-Reilly, A., Payne, S., & Ridge, P. G. (2023) · arXiv · doi:10.48550/arxiv.2303.13528 · Cited by 69
Self-citationAutomatically Generating CS Learning Materials with Large Language Models
MacNeil, S., Tran, A., Leinonen, J., Denny, P., Kim, J., Hellas, A., Bernstein, S., & Sarsa, S. (2022) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 2 · doi:10.1145/3545947.3569630 · Cited by 44
“Our research team has developed code-explanations with the LLMs Codex [19] and GPT-3 [14, 15] which resulted in the development of a design space for LLM-generated code explanations [15].”
BackgroundMethodSelf-citationSoftware engineering education in the era of conversational AI: current trends and future directions
Sengul, C., Neykova, R., & Destefanis, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1436350 · Cited by 83
Prasad, P., & Sane, A. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630828 · Cited by 50
“LLMs generate code explanations [24] that are easy to understand [18] and helpful [23].”
BackgroundInvestigating the Potential of GPT-3 in Providing Feedback for Programming Assessments
Balse, R., Valaboju, B., Singhal, S., Warriem, J. M., & Prasad, P. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588852 · Cited by 63
“MacNeil et al. used Codex in a software development course to generate explanations for code examples in an online e-book [16].”
MethodComputing education using generative artificial intelligence tools: A systematic literature review
Agbo, F. J., Olivia, C., Oguibe, G. F., Sanusi, I. T., & Sani, G. (2025) · Computers and Education Open · doi:10.1016/j.caeo.2025.100266 · Cited by 75
Generating Multiple Choice Questions for Computing Courses Using Large Language Models
Tran, A., Angelikas, K., Rama, E., Okechukwu, C., Smith, D. H., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10342898 · Cited by 60
“For example, these models perform well on a variety of computing education pedagogical tasks, including generating code [36], explanations of code [12], [38], [41], coding concepts [42], and programming assignments [13].”
BackgroundSelf-citationDebugging with an AI Tutor: Investigating Novice Help-seeking Behaviors and Perceived Learning
Yang, S., Zhao, H., Xu, Y., Brennan, K. A., & Schneider, B. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671092 · Cited by 68
“LLMs can support key debugging tasks such as explaining code [26, 32], clarifying error messages [27], designing test cases [10], and identifying bugs [31].”
AI-Enhanced Auto-Correction of Programming Exercises: How Effective is GPT-3.5?
Azaiz, I., Deckarm, O., & Strickroth, S. (2023) · International Journal of Engineering Pedagogy (iJEP) · doi:10.3991/ijep.v13i8.45621 · Cited by 60
“Furthermore, LLMs have been used to generate good code explanations [18] and a study by Macneil et al. [21] showed that automatic generated code explanations by LLMs in an e-book were appreciated by students .”
MethodReeves, B., Sarsa, S., Prather, J., Denny, P., Becker, B. A., Hellas, A., Kimmel, B., Powell, G., & Leinonen, J. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588805 · Cited by 56
“for natural language) [29, 31] for producing code explanations.”
BackgroundSelf-citationZamfirescu-Pereira, J. D., Qi, L., Hartmann, B., DeNero, J., & Norouzi, N. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701864 · Cited by 59
Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study
Sheard, J., Denny, P., Hellas, A., Leinonen, J., Malmi, L., & Simon (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630880 · Cited by 54
“Another stream of research has explored the use of LLMs to create and enhance educational resources such as programming exercises [32], code explanations [24, 28, 32], and programming error messages [25, 29].”
BackgroundSelf-citationChugh, R., Turnbull, D., Morshed, A., Sabrina, F., Azad, S., Rashid, M. M., Kaisar, S., & Subramani, S. (2025) · Computer Applications in Engineering Education · doi:10.1002/cae.70002 · Cited by 57
Budhiraja, R., Joshi, I., Challa, J. S., Akolekar, H. D., & Kumar, D. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636257 · Cited by 52
Lohr, D., Keuning, H., & Kiesler, N. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13107 · Cited by 63
Exploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices
Xiao, R., Hou, X., & Stamper, J. C. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650937 · Cited by 60
Student Perspectives on Using a Large Language Model (LLM) for an Assignment on Professional Ethics
Grande, V., Kiesler, N., & Rodríguez, M. A. F. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653624 · Cited by 61
Abolnejadian, M., Alipour, S., & Taeb, K. (2024) · Extended Abstracts of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613905.3637148 · Cited by 33
“Researchers have shown how these models can assist students in various tasks, such as generating code [23, 27], explaining code [14, 16, 18], and creating programming assignments [8, 24].”
Decoding Logic Errors: A Comparative Study on Bug Detection by Students and Large Language Models
MacNeil, S., Denny, P. C., Tran, A., Leinonen, J., Bernstein, S., Hellas, A., Sarsa, S., & Kim, J. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636245 · Cited by 56
Self-citationHumble, N. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-024-00494-x · Cited by 57
“…al., 2023; Cambaz & Zhang, 2024; Canedo et al., 2024; Doughty et al., 2024; Du et al., 2024; Grévisse et al., 2024; Hang et al., 2024; Hou et al., 2024a; Husain, 2024; Jordan et al., 2024; Joshi et al., 2024; Kuramitsu et al., 2024; Ling & Afzaal, 2024; MacNeil et al., 2023; Nguyen & Allan, 2024).”
Mahon, J., Namee, B. M., & Becker, B. A. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653602 · Cited by 57
Hou, I., Man, O., Mettille, S., Gutierrez, S., Angelikas, K., & MacNeil, S. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636247 · Cited by 47
Self-citationCan Language Models Employ the Socratic Method? Experiments with Code Debugging
Al-Hossami, E., Bunescu, R., Smith, J., & Teehan, R. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630799 · Cited by 60
How Do Programming Students Use Generative AI?
Rahe, C., & Maalej, W. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3715762 · Cited by 45
“In a subsequent work, MacNeil et al. [2023] found that the explanations were mostly rated useful by students, both for their own understanding and as a general learning tool [MacNeil et al. 2023].”
Analyzing LLM Usage in an Advanced Computing Class in India
Arora, U., Garg, A., Gupta, A., Jain, S., Mehta, R. L., Oberoi, R., Prachi, Raina, A., Saini, M., Sharma, S., et al. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716657 · Cited by 52
“There have been numerous studies that have explored the impact, challenges, and also opportunities that come with these new technologies [3, 4, 6, 8, 9, 11, 12, 19, 20, 22, 23, 25, 27–29 , 33].”
Nathaniel, J., Oyelere, S. S., Suhonen, J. T., & Tedre, M. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100460 · Cited by 58
“These tools offer instant support for tasks such as coding, debugging, algorithm design, and even content generation, making programming more accessible than was previously the case (MacNeil et al., 2023; Yilmaz & Yilmaz, 2023).”
DBox: Scaffolding Algorithmic Programming Learning through Learner-LLM Co-Decomposition
Ma, S., Wang, J., Zhang, Y., Ma, X., & Wang, A. Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713748 · Cited by 63
“• A user study reported DBox’s effectiveness on algorithmic programming learning, including its effects on learners’ learning outcomes, perceptions, user experience, and usage patterns.”
Evaluating the Quality of LLM-Generated Explanations for Logical Errors in CS1 Student Programs
Balse, R., Kumar, V., Prasad, P., & Warriem, J. M. (2023) · Bangalore Annual Compute Conference · doi:10.1145/3627217.3627233 · Cited by 55
Evaluating large language models for software testing
Li, Y., Liu, P., Wang, H., Chu, J., & Wong, W. E. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2024.103942 · Cited by 42
Pirzado, F. A., Ahmed, A., Mendoza-Urdiales, R. A., & Terashima‐Marín, H. (2024) · IEEE Access · doi:10.1109/access.2024.3443621 · Cited by 65
“In some studies, the students reported that LLM-generated code explanations can help them understand complex code, positively impacting their learning outcomes [12].”
BackgroundImplications of ChatGPT for Data Science Education
Shen, Y., Ai, X., Raj, A. G. S., John, R. J. L., & Syamkumar, M. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630874 · Cited by 50
Chat Overflow: Artificially Intelligent Models for Computing Education - renAIssance or apocAIypse?
Denny, P., Becker, B. A., Leinonen, J., & Prather, J. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588773 · Cited by 32
“He is not sure if he is surprised or not that LLMs have offered yet another parallel between programming and natural languages, in that LLMs have demonstrated similar capabilities in both domains through very similar mechanisms.”
BackgroundSelf-citationIntegrating AI Tutors in a Programming Course
Ma, I., Krone-Martins, A., & Lopes, C. V. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649165.3690094 · Cited by 55
“LLMs have proven successful in solving a variety of Natural Language Processing (NLP) tasks [17, 24, 25, 28] and passing knowledge examinations crossing various fields [4, 13, 32].”
Enhancing Programming Error Messages in Real Time with Generative AI
Kimmel, B., Geisert, A. L., Yaro, L., Gipson, B., Hotchkiss, R., Osae-Asante, S. K., Vaught, H., Wininger, G., & Yamaguchi, C. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3647967 · Cited by 44
“For instructors, these models can automatically generate programming exercises [25] and code explanations [15].”
Bernstein, S., Denny, P. C., Leinonen, J., Kan, L., Hellas, A., Littlefield, M., Sarsa, S., & MacNeil, S. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653533 · Cited by 44
“Large language models (LLMs) have recently been shown to generate high-quality explanations [25, 28].”
InfluentialSelf-citationPrather, J. E., Reeves, B. N., Denny, P. C., Leinonen, J., MacNeil, S., Luxton-Reilly, A., Orvalho, J. G., Alipour, M. A., Alfageeh, A., Amarouche, T., et al. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716649 · Cited by 44
“MacNeil et al. reported that students found LLM-generated code explanations useful and helpful for their learning [44].”
Self-citationHou, I., Man, O., Hamilton, K., Muthusekaran, S., Johnykutty, J., Zadeh, L., & MacNeil, S. (2025) · Proceedings of the 30th ACM Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3724363.3729024 · Cited by 37
“This growth may beexplained bythe many associatedbenefits, suchas personalizedexplanations [5, 30, 34], intelligent teaching assistants [12, 24, 25, 31], and support for identifying bugs and debugging code [33, 57].”
Self-citationScholl, A., & Kiesler, N. (2024) · 2024 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893442 · Cited by 35
“In the context of introductory programming education, GenAI tools have been subject to research w.r.t. their application by educators and students [11], with a focus on students’ potential use [10], [12], [15], [17], [34]–[36].”
InfluentialFeng, H., Luxton-Reilly, A., Wuensche, B. C., & Denny, P. C. (2025) · Australasian Computing Education Conference · doi:10.1145/3716640.3716658 · Cited by 46
“Integrating GenAI into classroom activities and into the curriculum is essential for equipping students with the skills to leverage AI tools in the future.”
Self-citationBernstein, S., Rahman, A., Sharifi, N., Terbish, A., & MacNeil, S. (2025) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3769994.3770036 · Cited by 37
“For example, GenAI can explain code to students better than their peers [122, 138], can provide hints to help students get unstuck [115, 204], and can make learning materials more personally or culturally relevant [30].”
Self-citationWill Code Remain a Relevant User Interface for End-User Programming with Generative AI Models?
Sarkar, A. (2023) · Proceedings of the 2023 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software · doi:10.1145/3622758.3622882 · Cited by 23
“Learning.”
Joshi, I., Budhiraja, R., Tanna, P. D., Jain, L., Deshpande, M., Srivastava, A., Rallapalli, S., Akolekar, H. D., Challa, J. S., & Kumar, D. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.10694 · Cited by 38
“There is a growing body of research testing the integration of LLM-based tools like ChatGPT into various educational contexts[29, 33, 58, 73].”
BackgroundExperiences from Integrating Large Language Model Chatbots into the Classroom
Hellas, A., Leinonen, J., & Leppänen, L. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649165.3690101 · Cited by 37
“…of large language models, highlighting their potential in writing code and solving and creating programming assignments [8, 33, 35], explaining code [18, 22], identifying programming concepts [36], improving programming error messages [19, 34], and responding to students’ help requests [9].”
Self-citationPrather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Caspersen, M. E., Craig, M., Keuning, H., Kiesler, N., Kohn, T., et al. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 2 · doi:10.1145/3587103.3594206 · Cited by 26
“creating educational resources such as novel, personalized programming exercises [2, 10], code explanations that could help support students when they are working on exercises [6, 8], and enhanced programming error messages that might be easier to understand for novice programmers [7].”
BackgroundSelf-citationAnalyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPTPreprint
Scholl, A., Schiffner, D., & Kiesler, N. (2024) · arXiv (Cornell University) · doi:10.18420/delfi2024_05 · Cited by 32
Decoding Debugging Instruction: A Systematic Literature Review of Debugging Interventions
Yang, S., Baird, M., O’Rourke, E., Brennan, K., & Schneider, B. (2024) · ACM Transactions on Computing Education · doi:10.1145/3690652 · Cited by 29
“Prior research has demonstrated that LLMs can execute each step of the debugging process, including explaining code (Step 1) [98], rewriting error messages [87] (Step 2), diagnosing bugs in the code (Step 3) [50], and fixing them (Step 4) [50].”
How Instructors Incorporate Generative AI into Teaching Computing
Prather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649405.3659534 · Cited by 32
Self-citationLLM-aided explanations of EDA synthesis errors
Qiu, S., Tan, B., & Pearce, H. A. (2024) · IEEE LLM Aided Design Workshop (LAD) · doi:10.1109/lad62341.2024.10691721 · Cited by 36
A comparative study of AI ‐generated and human‐crafted learning objectives in computing education
Doyle, A., Sridhar, P., Agarwal, A., Šavelka, J., & Sakr, M. F. (2025) · Journal of Computer Assisted Learning · doi:10.1111/jcal.13092 · Cited by 30
Lyu, W., Wang, Y., Sun, Y., & Zhang, Y. (2025) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3698205.3729544 · Cited by 35
“Recent studies have explored various applications of LLM-based tools in computer science learning contexts, such as automated code explanations [34], enhanced debugging support [24, 27, 44], and generating exercises to help students practice [25, 40, 44, 46], as well as examining how students and…”
Can ChatGPT Play the Role of a Teaching Assistant in an Introductory Programming Course?Preprint
Anishka, Mehta, A., Gupta, N., Balachandran, A., Kumar, D., & Jalote, P. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2312.07343 · Cited by 33
“Notably, certain investigations have delved into the utility of ChatGPT for code de-bugging, providing suggestive fixes to code errors [19, 22].”
BackgroundPădurean, V.-A., Denny, P. C., Gotovos, A., & Singla, A. K. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3724363.3729094 · Cited by 25
Self-citationInsights from the Frontline: GenAI Utilization Among Software Engineering Students
Choudhuri, R., Ramakrishnan, A., Chatterjee, A., Trinkenreich, B., Steinmacher, I., Gerosa, M. A., & Sarma, A. (2025) · IEEE International Conference on Software Engineering Education and Training (CSEE&T) · doi:10.1109/cseet66350.2025.00007 · Cited by 38
“Beyond theoretical discussion, practical research efforts have aimed at enhancing student assistance in code generation on introductory programming problems [45], providing learning resources [46, 47, 48], explanations [49, 50, 47, 51], and finding issues in problematic code [52].”
Let's Ask AI About Their Programs: Exploring ChatGPT's Answers To Program Comprehension Questions
Lehtinen, T., Koutcheme, C., & Hellas, A. (2024) · IEEE/ACM International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET) · doi:10.1145/3639474.3640058 · Cited by 32
“…are utilized for solving programming tasks [1, 3, 40], which could allow focusing on e.g. algorithmic concepts earlier [1, 3], as well as highlighted the possibilities of embedding LLMs into e.g. student’s browsers to help students understand code in external resources such as StackOverflow [29].”
BackgroundMethodSelf-citationLee, C., Myung, J., Han, J., Jin, J., & Oh, A. H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2309.10419 · Cited by 25
Ali, M., Rao, P. V., Mai, Y., & Xie, B. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671097 · Cited by 33
Understanding the Human-LLM Dynamic: A Literature Survey of LLM Use in Programming TasksPreprint
Etsenake, D., & Nagappan, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.01026 · Cited by 30
“(2) Length of LLM response: The length of the LLM response does play a strong role in whether users accepted it upfront [30, 54, 56, 68, 83].”
Black, G., Rimal, B. P., & Vaidyan, V. M. (2024) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2024.3446695 · Cited by 26
Metadata
- Authors
- MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J.
- Affiliations
- Temple University (US)
- Aalto University (FI)
- University of Auckland (NZ)
- ORCID
- Venue
- Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1
- Publisher
- ACM
- Published
- 2023-03-02
- Pages
- 931-937
- OpenAlex
- W4323033814
- Cited by
- 195
- References
- 30
- Citations by year
- 2026: 27 · 2025: 59 · 2024: 79 · 2023: 29
- FWCI
- 58.66
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Aaltodoc (Aalto University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- code explanation, large language models, interactive e-books, computer science education, student engagement, explanation types, code generation, code complexity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.