Generating Diverse Code Explanations using the GPT-3 Large Language Model
The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Good explanations are essential to efficiently learning introductory programming concepts [10]. To provide high-quality explanations at scale, numerous systems automate the process by tracing the execution of code [8, 12], defining terms [9], giving hints [16], and providing error-specific feedback [10, 16]. However, these approaches often require manual effort to configure and only explain a single aspect of a given code segment. Large language models (LLMs) are also changing how students interact with code [7]. For example, Github's Copilot can generate code for programmers [4], leading researchers to raise concerns about cheating [7]. Instead, our work focuses on LLMs' potential to support learning by explaining numerous aspects of a given code snippet. This poster features a systematic analysis of the diverse natural language explanations that GPT-3 can generate automatically for a given code snippet. We present a subset of three use cases from our evolving design space of AI Explanations of Code.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Similar results concerning teachers attitudes towards chatbots in education were reported in [34]: perceiving the AI chatbot as easy-touse and useful leads to greater acceptance of the chatbot.”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [34]
In the citing paper
“As for the chatbots' features, formal language by a chatbot leads to a higher intention of using it.”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [34]
In the citing paper
“This has been inspired by many other instances where LLMs have already demonstrated remarkable effectiveness in generating novel language artifacts. These include course readings [22], code explanations [23, 28, 29, 41], model solutions [9, 35, 42], feedback [34], or responses to help requests [18, 26, 46, 47].”
From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [29]
In the citing paper
“Tools like Codex are able to generate new programming assignments [36] and code explanations [29] when provided examples.”
From Using Large Language Models to Enhance Programming Error Messages (2023) · cites this paper as [29]
In the citing paper
“In the field of computer science, AI tools demonstrate tremendous potential in coding education, as evidenced by GPT-3's ability to generate diverse code explanations [11].”
From Transforming Education: A Comprehensive Review of Generative Artificial Intelligence in Educational Settings through Bibliometric and Content Analysis (2023) · cites this paper as [11]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
37 Million Compilations: Investigating Novice Programming Mistakes in Large-Scale Student Data
Altadmri, A., & Brown, N. C. C. (2015) · Proceedings of the 46th ACM Technical Symposium on Computer Science Education · doi:10.1145/2676723.2677258 · Cited by 261
What do teachers teach in introductory programming?
Schulte, C., & Bennedsen, J. (2006) · Proceedings of the second international workshop on Computing education research · doi:10.1145/1151588.1151593 · Cited by 181
Proceeding of the 44th ACM technical symposium on Computer science education
(2013) · doi:10.1145/2445196 · Cited by 166
iSnap: Towards Intelligent Tutoring in Novice Programming Environments
Price, T. W., Dong, Y., & Lipovac, D. (2017) · Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education · doi:10.1145/3017680.3017762 · Cited by 140
Autonomously Generating Hints by Inferring Problem Solving Policies
Piech, C., Sahami, M., Huang, J., & Guibas, L. (2015) · Proceedings of the Second (2015) ACM Conference on Learning @ Scale · doi:10.1145/2724660.2724668 · Cited by 138
Comprehension First: Evaluating a Novel Pedagogy and Tutoring System for Program Tracing in CS1
Nelson, G. L., Xie, B., & Ko, A. J. (2017) · Proceedings of the 2017 ACM Conference on International Computing Education Research · doi:10.1145/3105726.3106178 · Cited by 128
Marwan, S., Gao, G., Fisk, S., Price, T. W., & Barnes, T. (2020) · Proceedings of the 2020 ACM Conference on International Computing Education Research · doi:10.1145/3372782.3406264 · Cited by 114
Meaningful categorisation of novice programmer errors
McCall, D., & Kolling, M. (2014) · 2014 IEEE Frontiers in Education Conference (FIE) Proceedings · doi:10.1109/fie.2014.7044420 · Cited by 102
Research works printed but not linked: 2
- Mark Chen , Jerry Tworek , Heewoo Jun , Qiming Yuan , Henrique Ponde de Oliveira Pinto , Jared Kaplan, Harri Edwards, Yuri Burda , Nicholas Joseph, Greg Brockman , 2021 . Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021). Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021). · arXiv
- James Finnie-Ansley , Paul Denny , Brett A. Becker , Andrew Luxton-Reilly , and James Prather . 2022 . The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference ( Virtual Event, Australia) (ACE ’22). ACM, New York, NY, USA, 10–19. https://doi.org/10.1145/351 1861.3511863 10.1145/3511861.3511863 James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference(Virtual Event, Australia) (ACE ’22). ACM, New York, NY, USA, 10–19. https://doi.org/10.1145/3511861.3511863 · doi:10.1145/351
Cited by
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326
Chatting about ChatGPT: how may AI and GPT impact academia and libraries?
Lund, B. D., & Wang, T. (2023) · Library Hi Tech News · doi:10.1108/lhtn-01-2023-0009 · Cited by 1,355
Bahroun, Z., Anane, C., Ahmed, V., & Zacca, A. (2023) · Sustainability · doi:10.3390/su151712983 · Cited by 784
“[11] Explores the potential of large language models (LLMs) in providing explanations for programming concepts, contributing to the understanding of LLMs’ role in enhancing learning experiences.”
BackgroundChatGPT and large language models in academia: opportunities and challenges
Meyer, J. G., Urbanowicz, R. J., Martin, P. C. N., O’Connor, K., Li, R., Peng, P.-C., Bright, T. J., Tatonetti, N., Won, K. J., Gonzalez-Hernandez, G., et al. (2023) · BioData Mining · doi:10.1186/s13040-023-00339-9 · Cited by 569
“LLMs provide benefits and opportunities in education for students by assisting with research and academic writing [15] or using them as an interactive study guide which may include generating practice exams and being provided immediate feedback [16, 17].”
BackgroundThe role of ChatGPT in higher education: Benefits, challenges, and future research directions
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.29 · Cited by 561
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569759 · Cited by 401
Hong, W. C. H. (2023) · Journal of Educational Technology and Innovation · doi:10.61414/jeti.v5i1.103 · Cited by 365
Generative AI for Customizable Learning Experiences
Pesovski, I., Santos, R., Henriques, R., & Trajkovik, V. (2024) · Sustainability · doi:10.3390/su16073034 · Cited by 274
“The ability of ChatGPT to fluently explain code was also verified by MacNeil et al. [55].”
BackgroundLau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241
Using Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235
Exploring the Responses of Large Language Models to Beginner Programmers' Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · arXiv · doi:10.48550/arxiv.2306.05715 · Cited by 211
Comparing Code Explanations Created by Students and Large Language Models
Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., & Hellas, A. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588785 · Cited by 210
Self-citationMacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569785 · Cited by 195
“In computer science classrooms, LLMs are beginning to be used to generate assignments [36], help students to write their code [3, 40], and to generate explanations of code snippets to facilitate learning [23].”
BackgroundMethodInfluentialSelf-citationMacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2022) · arXiv · doi:10.48550/arxiv.2211.02265 · Cited by 195
Self-citationIbrahim, H., Liu, F., Asim, R., Battu, B., Benabderrahmane, S., Alhafni, B., Adnan, W., Alhanai, T., AlShebli, B., Baghdadi, R., et al. (2023) · Scientific Reports · doi:10.1038/s41598-023-38964-3 · Cited by 154
Generative AI: Is it a paradigm shift for higher education
O’Dea, X. (2024) · Studies in Higher Education · doi:10.1080/03075079.2024.2332944 · Cited by 151
Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600139 · Cited by 145
Generative AI in Computing Education: Perspectives of Students and Instructors
Zastudil, C., Rogalska, M., Kapp, C., Vaughn, J., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343467 · Cited by 136
“Researchers are already demonstrating the capabilities of these models to help students generate code [2], [3], [25], explain code to students [5], [6], [9], and even produce programming assignments [1], [4].”
BackgroundSelf-citationA Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Doughty, J., Wan, Z., Bompelli, A., Qayum, J., Wang, T., Zhang, J., Zheng, Y., Doyle, A., Sridhar, P., Agarwal, A., et al. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636256 · Cited by 132
Liu, M. X., Sarkar, A., Negreanu, C., Zorn, B., Williams, J., Toronto, N., & Gordon, A. D. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580817 · Cited by 119
Kosar, T., Ostojić, D., Liu, Y. D., & Mernik, M. (2024) · Mathematics · doi:10.3390/math12050629 · Cited by 107
Pardos, Z. A., & Bhandari, S. (2024) · PLOS ONE · doi:10.1371/journal.pone.0304013 · Cited by 97
The Effects of Generative AI on Computing Students’ Help-Seeking Preferences
Hou, I., Mettille, S., Man, O., Li, Z., Zastudil, C., & MacNeil, S. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636248 · Cited by 97
Self-citationDoes ChatGPT Help With Introductory Programming?An Experiment of Students Using ChatGPT in CS1
Xue, Y., Chen, H., Bai, G. R., Tairas, R., & Huang, Y. (2024) · Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training · doi:10.1145/3639474.3640076 · Cited by 96
Generative AI in Education: A Study of Educators' Awareness, Sentiments, and Influencing Factors
Ghimire, A., Pather, J., & Edwards, J. (2024) · 2024 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10892891 · Cited by 88
Bewersdorff, A., Seßler, K., Baur, A., Kasneci, E., & Nerdel, C. (2023) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2023.100177 · Cited by 76
How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering
Choudhuri, R., Liu, D., Steinmacher, I., Gerosa, M., & Sarma, A. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639201 · Cited by 72
O’Dea, X., Tsz Kit Ng, D., O’Dea, M., & Shkuratskyy, V. (2026) · Policy Futures in Education · doi:10.1177/14782103241287401 · Cited by 69
Ibrahim, K. (2023) · Language Testing in Asia · doi:10.1186/s40468-023-00260-2 · Cited by 66
“1 As a generative language model (Pavlik, 2023), GPT is a machine-learning neural network trained to analyze and interpret natural language texts and model the linguistic system of a language (MacNeil et al., 2022).”
BackgroundInvestigating the Potential of GPT-3 in Providing Feedback for Programming Assessments
Balse, R., Valaboju, B., Singhal, S., Warriem, J. M., & Prasad, P. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588852 · Cited by 63
Generating Multiple Choice Questions for Computing Courses Using Large Language Models
Tran, A., Angelikas, K., Rama, E., Okechukwu, C., Smith, D. H., & MacNeil, S. (2023) · 2023 IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10342898 · Cited by 60
“In the first study, we evaluated, GPT-3, a popular LLM that has been used across many recent computing education studies [11], [13], [16], [17], [41].”
BackgroundMethodInfluentialSelf-citationReeves, B., Sarsa, S., Prather, J., Denny, P., Becker, B. A., Hellas, A., Kimmel, B., Powell, G., & Leinonen, J. (2023) · Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3587102.3588805 · Cited by 56
“Prior work has used both Codex (which is optimized for source code) [36] and GPT-3 (which is optimized for natural language) [29, 31] for producing code explanations.”
MethodChatGPT in higher education—a student's perspective
Shuhaiber, A., Kuhail, M. A., & Salman, S. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2024.100565 · Cited by 55
Students’ Perceptions and Preferences of Generative Artificial Intelligence Feedback for Programming
Zhang, Z., Dong, Z., Shi, Y., Price, T., Matsuda, N., & Xu, D. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i21.30372 · Cited by 54
Tan, M., & Subramonyam, H. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642592 · Cited by 40
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2022) · arXiv · doi:10.48550/arxiv.2212.01020 · Cited by 36
How Do Programming Students Use Generative AI?
Rahe, C., & Maalej, W. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3715762 · Cited by 45
Potentials and implications of ChatGPT for ESL writing instruction
Ibrahim, K., & Kirkpatrick, R. (2024) · The International Review of Research in Open and Distributed Learning · doi:10.19173/irrodl.v25i3.7820 · Cited by 28
Soto, D., Higashida, M., Shirai, S., Ueda, M., & Uranishi, Y. (2024) · Procedia Computer Science · doi:10.1016/j.procs.2024.09.198 · Cited by 10
Mapping the Design Space of Interactions in Human-AI Text Co-creation Tasks
Ding, Z., & Chan, J. (2023) · arXiv · doi:10.48550/arxiv.2303.06430 · Cited by 9
“Previous research has indicated that Large language models (LLMs) can effectively handle well-defined, content curation tasks such as text summarization [9, 15], content refinement [17], and code explanation [18, 19].”
BackgroundAttribute Structuring Improves LLM-Based Evaluation of Clinical Text Summaries
Gero, Z., Singh, C., Xie, Y., Zhang, S., Subramanian, P., Vozila, P., Naumann, T., Gao, J., & Poon, H. (2024) · arXiv · doi:10.48550/arxiv.2403.01002 · Cited by 5
Opportunities and considerations for using artificial intelligence in bioinformatics education
Piccolo, S. R., Nathan, A., Brazas, M. D., Kandpal, M., Miró-Herrans, A. T., Kleinschmit, A. J., McClatchy, S., Mutheiwana, P., Nikolic, D., Gallo, L. I., et al. (2024) · Bioinformatics Advances · doi:10.1093/bioadv/vbaf169 · Cited by 2
“As another example, an instructor could refine existing code snippets by generating line-by-line explanations or characterizing common coding mistakes (MacNeil et al. 2022).”
Metadata
- Authors
- MacNeil, S., Tran, A., Mogil, D., Bernstein, S., Ross, E., & Huang, Z.
- Affiliations
- Temple University (US)
- University of California San Diego (US)
- Venue
- Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 2
- Publisher
- ACM
- Published
- 2022-08-07
- Pages
- 37-39
- OpenAlex
- W4291476001
- Cited by
- 200
- References
- 16
- Citations by year
- 2026: 13 · 2025: 42 · 2024: 90 · 2023: 52 · 2022: 3
- FWCI
- 60.88
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on International Computing Education Research (ICER) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- code explanation, GPT-3, large language models, introductory programming, programming education, natural language explanations, code generation, explanation diversity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.