The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book
Open accessAbstract
Advances in natural language processing have resulted in large language models (LLMs) that can generate code and code explanations. In this paper, we report on our experiences generating multiple code explanation types using LLMs and integrating them into an interactive e-book on web software development. Three different types of explanations -- a line-by-line explanation, a list of important concepts, and a high-level summary of the code -- were created. Students could view explanations by clicking a button next to code snippets, which showed the explanation and asked about its utility. Our results show that all explanation types were viewed by students and that the majority of students perceived the code explanations as helpful to them. However, student engagement varied by code snippet complexity, explanation type, and code snippet length. Drawing on our experiences, we discuss future directions for integrating explanations generated by LLMs into CS classrooms.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“This has been inspired by many other instances where LLMs have already demonstrated remarkable effectiveness in generating novel language artifacts. These include course readings [22], code explanations [23, 28, 29, 41], model solutions [9, 35, 42], feedback [34], or responses to help requests [18, 26, 46, 47].”
From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [28]
In the citing paper
“On the other hand, recent work has shown that AI tools can be used to produce high-quality learning resources, including programming exercises [32], explanations of code [24,28], and even on-demand feedback to students [27].”
From Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study (2024) · cites this paper as [28]
In the citing paper
“MacNeil et al. found that students generally found code explanations generated by LLMs useful for learning, but the perceived utility of the explanations and students' engagement with them varied by explanation type [137].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [137]
In the citing paper
“However, non-code models such as GPT-3 have been found to be more apt at creating higher level code explanations [137].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [137]
In the citing paper
“Students rated these explanations on a 5-point Likert scale, confirming that explanations matched the code and were useful for learning in general, whereas line-by-line explanations were rated as least helpful.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [137]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Review of recent systems for automatic assessment of programming assignments
Ihantola, P., Ahoniemi, T., Karavirta, V., & Seppälä, O. (2010) · Proceedings of the 10th Koli Calling International Conference on Computing Education Research · doi:10.1145/1930464.1930480 · Cited by 554
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497
Self-citationOnline python tutor: embeddable web-based program visualization for cs education
Guo, P. J. (2013) · Proceeding of the 44th ACM technical symposium on Computer science education · doi:10.1145/2445196.2445368 · Cited by 396
Designing the whyline: a debugging interface for asking questions about program behavior
Ko, A. J., & Myers, B. A. (2004) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/985692.985712 · Cited by 361
PeerWise: students sharing their multiple choice questions
Denny, P., Hamer, J., Luxton-Reilly, A., & Purchase, H. (2008) · Proceedings of the Fourth international Workshop on Computing Education Research · doi:10.1145/1404520.1404526 · Cited by 211
Self-citationPair programming in education: a literature review
Hanks, B., Fitzgerald, S., McCauley, R., Murphy, L., & Zander, C. (2011) · Computer Science Education · doi:10.1080/08993408.2011.579808 · Cited by 205
Generating Diverse Code Explanations using the GPT-3 Large Language Model
MacNeil, S., Tran, A., Mogil, D., Bernstein, S., Ross, E., & Huang, Z. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 2 · doi:10.1145/3501709.3544280 · Cited by 200
Self-citationMarwan, S., Gao, G., Fisk, S. R., Price, T., & Barnes, T. M. (2020) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3372782.3406264 · Cited by 164
Wheel-Spinning: Students Who Fail to Master a Skill
Beck, J. E., & Gong, Y. (2013) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-39112-5_44 · Cited by 161
Research works printed but not linked: 12
- Kirsti M Ala-Mutka. 2005. A survey of automated assessment approaches for programming assignments. Computer science education, Vol. 15, 2 (2005), 83--102.
- Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le and Charles Sutton. 2021. Program Synthesis with Large Language Models. arxiv: 2108.07732 [cs.PL] · arXiv
- Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan Prafulla Dhariwal Arvind Neelakantan Pranav Shyam Girish Sastry Amanda Askell Sandhini Agarwal Ariel Herbert-Voss Gretchen Krueger Tom Henighan Rewon Child Aditya Ramesh Daniel M. Ziegler Jeffrey Wu Clemens Winter Christopher Hesse Mark Chen Eric Sigler Mateusz Litwin Scott Gray Benjamin Chess Jack Clark Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in neural information processing systems. 1877--1901.
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). · arXiv
- James Finnie-Ansley, Paul Denny, Brett A Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conf. 10--19.
- Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022. Sparks: Inspiration for Science Writing Using Language Models. In Designing Interactive Systems Conf. (Virtual Event, Australia) (DIS '22). ACM, New York, NY, USA, 1002--1019.
- Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586 (2021). · arXiv
- José Carlos Paiva, José Paulo Leal, and Álvaro Figueira. 2022. Automated Assessment in Computer Science Education: A State-of-the-Art Review. ACM Transactions on Computing Education (TOCE) (2022).
- Jaanus Pöial. 2020. Challenges of Teaching Programming in StackOverflow Era. In International Conf. on Interactive Collaborative Learning. Springer, 703--710.
- Wengran Wang, Yudong Rao, Rui Zhi, Samiha Marwan, Ge Gao, and Thomas W. Price. 2020. Step Tutor: Supporting Students through Step-by-Step Example-Based Feedback. In Proceedings of the 2020 ACM Conf. on Innovation and Technology in Computer Science Education. ACM, New York, NY, USA, 391--397.
- Jacqueline L. Whalley, Raymond Lister, Errol Thompson, Tony Clear, Phil Robbins, P. K. Ajith Kumar, and Christine Prasad. 2006. An Australasian Study of Reading and Comprehension Skills in Novice Programmers, Using the Bloom and SOLO Taxonomies. In Proc. of the 8th Australasian Conf. on Computing Education - Volume 52. Australian Computer Society, Inc., AUS, 243--252.
- Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft: Story Writing With Large Language Models. In 27th International Conf. on Intelligent User Interfaces. 841--852.
Cited by
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
“Initial evaluation by MacNeil et al. [80] on their interactive Web development e-book, suggested that a majority of students perceived LLM-generated code explanations to be helpful.”
BackgroundMethodThe promise and challenges of generative AI in education
Giannakos, M., Azevedo, R., Brusilovsky, P., Cukurova, M., Dimitriadis, Y., Hernandez-Leo, D., Järvelä, S., Mavrikis, M., & Rienties, B. (2025) · Behaviour & Information Technology · doi:10.1080/0144929x.2024.2394886 · Cited by 543
Large Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
Self-citation
Metadata
- Authors
- MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J.
- Affiliations
- Temple University (US)
- Aalto University (FI)
- University of Auckland (NZ)
- ORCID
- Venue
- Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1
- Publisher
- ACM
- Published
- 2023-03-02
- Pages
- 931-937
- OpenAlex
- W4323033814
- Cited by
- 195
- References
- 30
- Citations by year
- 2026: 27 · 2025: 59 · 2024: 79 · 2023: 29
- FWCI
- 58.66
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Aaltodoc (Aalto University) (submittedVersion, other-oa)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- code explanation, large language models, interactive e-books, computer science education, student engagement, explanation types, code generation, code complexity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book. https://app.cochat.ai/papers/10.1145/3545945.3569785
- The paper
- MacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2023). Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book. Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1. https://doi.org/10.1145/3545945.3569785
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.