Generating Diverse Code Explanations using the GPT-3 Large Language Model
The DOI matches its Crossref record; 16 of the 16 references it cites were found in a registry or an index.
Abstract
Good explanations are essential to efficiently learning introductory programming concepts [10]. To provide high-quality explanations at scale, numerous systems automate the process by tracing the execution of code [8, 12], defining terms [9], giving hints [16], and providing error-specific feedback [10, 16]. However, these approaches often require manual effort to configure and only explain a single aspect of a given code segment. Large language models (LLMs) are also changing how students interact with code [7]. For example, Github's Copilot can generate code for programmers [4], leading researchers to raise concerns about cheating [7]. Instead, our work focuses on LLMs' potential to support learning by explaining numerous aspects of a given code snippet. This poster features a systematic analysis of the diverse natural language explanations that GPT-3 can generate automatically for a given code snippet. We present a subset of three use cases from our evolving design space of AI Explanations of Code.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Similar results concerning teachers attitudes towards chatbots in education were reported in [34]: perceiving the AI chatbot as easy-touse and useful leads to greater acceptance of the chatbot.”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [34]
In the citing paper
“As for the chatbots' features, formal language by a chatbot leads to a higher intention of using it.”
From ChatGPT for good? On opportunities and challenges of large language models for education (2023) · cites this paper as [34]
In the citing paper
“This has been inspired by many other instances where LLMs have already demonstrated remarkable effectiveness in generating novel language artifacts. These include course readings [22], code explanations [23, 28, 29, 41], model solutions [9, 35, 42], feedback [34], or responses to help requests [18, 26, 46, 47].”
From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [29]
In the citing paper
“Tools like Codex are able to generate new programming assignments [36] and code explanations [29] when provided examples.”
From Using Large Language Models to Enhance Programming Error Messages (2023) · cites this paper as [29]
In the citing paper
“In the field of computer science, AI tools demonstrate tremendous potential in coding education, as evidenced by GPT-3's ability to generate diverse code explanations [11].”
From Transforming Education: A Comprehensive Review of Generative Artificial Intelligence in Educational Settings through Bibliometric and Content Analysis (2023) · cites this paper as [11]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
37 Million Compilations: Investigating Novice Programming Mistakes in Large-Scale Student Data
Altadmri, A., & Brown, N. C. C. (2015) · Proceedings of the 46th ACM Technical Symposium on Computer Science Education · doi:10.1145/2676723.2677258 · Cited by 261
What do teachers teach in introductory programming?
Schulte, C., & Bennedsen, J. (2006) · Proceedings of the second international workshop on Computing education research · doi:10.1145/1151588.1151593 · Cited by 181
Proceeding of the 44th ACM technical symposium on Computer science education
(2013) · doi:10.1145/2445196 · Cited by 166
Marwan, S., Gao, G., Fisk, S. R., Price, T., & Barnes, T. M. (2020) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3372782.3406264 · Cited by 164
Comprehension First: Evaluating a Novel Pedagogy and Tutoring System for Program Tracing in CS1
Nelson, G. L., Xie, B., & Ko, A. J. (2017) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3105726.3106178 · Cited by 159
iSnap: Towards Intelligent Tutoring in Novice Programming Environments
Price, T. W., Dong, Y., & Lipovac, D. (2017) · Proceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education · doi:10.1145/3017680.3017762 · Cited by 140
Autonomously Generating Hints by Inferring Problem Solving Policies
Piech, C., Sahami, M., Huang, J., & Guibas, L. J. (2015) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/2724660.2724668 · Cited by 139
Meaningful categorisation of novice programmer errors
McCall, D., & Kölling, M. (2014) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2014.7044420 · Cited by 103
Tutorons: Generating context-relevant, on-demand explanations and demonstrations of online code
Head, A., Appachu, C., Hearst, M. A., & Hartmann, B. (2015) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2015.7356972 · Cited by 52
“Unmuddying” course content using muddiest point reflections
Carberry, A. R., Krause, S. J., Ankeny, C. J., & Waters, C. (2013) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie.2013.6684966 · Cited by 23
Bringing "High-level" Down to Earth: Gaining Clarity in Conversational Programmer Learning Goals
Cunningham, K. I., Qiao, Y., Feng, A., & O'Rourke, E. (2022) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3478431.3499370 · Cited by 17
Complexitor: An Educational Tool for Learning Algorithm Time Complexity in Practical Manner
Elvina, E., & Karnalim, O. (2017) · ComTech Computer Mathematics and Engineering Applications · doi:10.21512/comtech.v8i1.3783 · Cited by 16
What makes big-O analysis difficult
ParkerMiranda, & LewisColleen (2014) · Journal of computing sciences in colleges · doi:10.5555/2591468.2591495 · Cited by 7
Perez, D., Zahedi, L., Ross, M. S., Jia, Z., Vinci-Cannava, T., Kramer, L., & Charters, M. (2020) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie44824.2020.9273932 · Cited by 3
Research works printed but not linked: 2
- Mark Chen , Jerry Tworek , Heewoo Jun , Qiming Yuan , Henrique Ponde de Oliveira Pinto , Jared Kaplan, Harri Edwards, Yuri Burda , Nicholas Joseph, Greg Brockman , 2021 . Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021). Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374(2021). · arXiv
- James Finnie-Ansley , Paul Denny , Brett A. Becker , Andrew Luxton-Reilly , and James Prather . 2022 . The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference ( Virtual Event, Australia) (ACE ’22). ACM, New York, NY, USA, 10–19. https://doi.org/10.1145/351 1861.3511863 10.1145/3511861.3511863 James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference(Virtual Event, Australia) (ACE ’22). ACM, New York, NY, USA, 10–19. https://doi.org/10.1145/3511861.3511863 · doi:10.1145/351
Cited by
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326
Chatting about ChatGPT: how may AI and GPT impact academia and libraries?
Lund, B. D., & Wang, T. (2023) · Library Hi Tech News · doi:10.1108/lhtn-01-2023-0009 · Cited by 1,355
Bahroun, Z., Anane, C., Ahmed, V., & Zacca, A. (2023) · Sustainability · doi:10.3390/su151712983 · Cited by 784
“[11] Explores the potential of large language models (LLMs) in providing explanations for programming concepts, contributing to the understanding of LLMs’ role in enhancing learning experiences.”
BackgroundChatGPT and large language models in academia: opportunities and challenges
Meyer, J. G., Urbanowicz, R. J., Martin, P. C. N., O’Connor, K., Li, R., Peng, P.-C., Bright, T. J., Tatonetti, N., Won, K. J., Gonzalez-Hernandez, G., et al. (2023) · BioData Mining · doi:10.1186/s13040-023-00339-9 · Cited by 569
“LLMs provide benefits and opportunities in education for students by assisting with research and academic writing [15] or using them as an interactive study guide which may include generating practice exams and being provided immediate feedback [16, 17].”
BackgroundThe role of ChatGPT in higher education: Benefits, challenges, and future research directions
(2023) · Journal of Applied Learning & Teaching · doi:10.37074/jalt.2023.6.1.29 · Cited by 561
Metadata
- Authors
- MacNeil, S., Tran, A., Mogil, D., Bernstein, S., Ross, E., & Huang, Z.
- Affiliations
- Temple University (US)
- University of California San Diego (US)
- Venue
- Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 2
- Publisher
- ACM
- Published
- 2022-08-07
- Pages
- 37-39
- OpenAlex
- W4291476001
- Cited by
- 200
- References
- 16
- Citations by year
- 2026: 13 · 2025: 42 · 2024: 90 · 2023: 52 · 2022: 3
- FWCI
- 60.88
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on International Computing Education Research (ICER) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- code explanation, GPT-3, large language models, introductory programming, programming education, natural language explanations, code generation, explanation diversity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.