Comparing Code Explanations Created by Students and Large Language Models
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Reasoning about code and explaining its purpose are fundamental skills for computer scientists. There has been extensive research in the field of computing education on the relationship between a student's ability to explain code and other skills such as writing and tracing code. In particular, the ability to describe at a high-level of abstraction how code will behave over all possible inputs correlates strongly with code writing skills. However, developing the expertise to comprehend and explain code accurately and succinctly is a challenge for many students. Existing pedagogical approaches that scaffold the ability to explain code, such as producing exemplar code explanations on demand, do not currently scale well to large classrooms. The recent emergence of powerful large language models (LLMs) may offer a solution. In this paper, we explore the potential of LLMs in generating explanations that can serve as examples to scaffold students' ability to understand and explain code. To evaluate LLM-created explanations, we compare them with explanations created by students in a large course (n ≈ 1000) with respect to accuracy, understandability and length. We find that LLM-created explanations, which can be produced automatically on demand, are rated as being significantly easier to understand and more accurate summaries of code than student-created explanations. We discuss the significance of this finding, and suggest how such models can be incorporated into introductory programming education.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“although LLMs are able to provide better explanations of code than students [123]”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [123]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“As introductory students realise that generative AI can outperform them on most tasks, they may lose motivation to learn the material, and become demoralised about the future of computing [123].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [123]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“This has been inspired by many other instances where LLMs have already demonstrated remarkable effectiveness in generating novel language artifacts. These include course readings [22], code explanations [23, 28, 29, 41], model solutions [9, 35, 42], feedback [34], or responses to help requests [18, 26, 46, 47].”
From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [23]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“LLMs can also generate good explanations of code [94,123,137,174].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [123]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“The most common concern expressed by authors about student learning was the potential for students to become over-reliant on generative AI tools to solve problems [33,85,123,126,169] and assist in debugging code [125,150,199].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [123]
Experimental — accuracy not yet measured
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 1×
“Some of these code explanations were selected from the explanations generated by students during Lab A, and some were generated by the large language model GPT-31 [4].”
BackgroundOn a Test of Whether one of Two Random Variables is Stochastically Larger than the Other
Mann, H. B., & Whitney, D. R. (1947) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177730491 · Cited by 14,320
The ASA Statement on p -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306 · cited 1×
“Following the guidelines of [45] and the broader discussion in [35], we use 𝑝 values as only one source of evidence and outline supporting statistics including two effect sizes – Rank-Biserial (RBC) Correlation [19] and Common-Language Effect Size (CLES) [28] – when presenting the results of the…”
MethodThe simple difference formula: An approach to teaching nonparametric correlation
Kerby, D. S. (2014) · Comprehensive Psychology · doi:10.2466/11.it.3.1 · Cited by 806 · cited 2×
“Following the guidelines of [46] and the broader discussion in [37], we use 𝑝 values as only one source of evidence and outline supporting statistics including two effect sizes – Rank-Biserial (RBC) Correlation [19] and Common-Language Effect Size (CLES) [30] – when presenting the results of the study.”
MethodProgram comprehension during software maintenance and evolution
Von Mayrhauser, A., & Vans, A. M. (1995) · Computer · doi:10.1109/2.402076 · Cited by 575
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · cited 3×
“This is also the type of explanation that LLMs seem to be best at creating [36].”
BackgroundMethodInfluentialSelf-citationBecker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569759 · Cited by 401
Self-citationOnline python tutor: embeddable web-based program visualization for cs education
Guo, P. J. (2013) · Proceeding of the 44th ACM technical symposium on Computer science education · doi:10.1145/2445196.2445368 · Cited by 396
Denny, P., Kumar, V., & Giacaman, N. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569823 · Cited by 287
Self-citationUsing Large Language Models to Enhance Programming Error Messages
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569770 · Cited by 235
Self-citationPair programming in education: a literature review
Hanks, B., Fitzgerald, S., McCauley, R., Murphy, L., & Zander, C. (2011) · Computer Science Education · doi:10.1080/08993408.2011.579808 · Cited by 205
Generating Diverse Code Explanations using the GPT-3 Large Language Model
MacNeil, S., Tran, A., Mogil, D., Bernstein, S., Ross, E., & Huang, Z. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 2 · doi:10.1145/3501709.3544280 · Cited by 200
Self-citationMacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2023) · Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3545945.3569785 · Cited by 195 · cited 2×
“MacNeil et al. used the GPT-3 model to generate explanations of short code fragments which then were presented to students in an online e-book alongside the corresponding code [24].”
BackgroundMethodSelf-citationMacNeil, S., Tran, A., Hellas, A., Kim, J., Sarsa, S., Denny, P., Bernstein, S., & Leinonen, J. (2022) · arXiv · doi:10.48550/arxiv.2211.02265 · Cited by 195
Self-citationAll syntax errors are not equal
Denny, P., Luxton-Reilly, A., & Tempero, E. (2012) · Proceedings of the 17th ACM annual conference on Innovation and technology in computer science education · doi:10.1145/2325296.2325318 · Cited by 144 · cited 1×
“Regardless, students may face roadblocks, including logical errors [12] and syntactical errors [10] when trying to understand code.”
BackgroundSelf-citationA Controlled Experiment for Program Comprehension through Trace Visualization
Cornelissen, B., Zaidman, A., & Arie Van Deursen, A. (2010) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2010.47 · Cited by 139
Lister, R., Fidge, C., & Teague, D. (2009) · Proceedings of the 14th annual ACM SIGCSE conference on Innovation and technology in computer science education · doi:10.1145/1562877.1562930 · Cited by 118
Learning programming via worked-examples: Relation of learning styles to cognitive load
Abdul-Rahman, S.-S., & du Boulay, B. (2014) · Computers in Human Behavior · doi:10.1016/j.chb.2013.09.007 · Cited by 92
Ullah, Z., Lajis, A., Jamjoom, M., Altalhi, A., Al‐Ghamdi, A., & Saleem, F. (2018) · Computer Applications in Engineering Education · doi:10.1002/cae.21974 · Cited by 79
IntroAssist: A Tool to Support Writing Introductory Help Requests
Hui, J. S., Gergle, D., & Gerber, E. M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173596 · Cited by 76
Exploring the Impact of Worked Examples in a Novice Programming Environment
Zhi, R., Price, T. W., Marwan, S., Milliken, A., Barnes, T., & Chi, M. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287385 · Cited by 75 · cited 1×
“Learning by example is an effective pedagogical technique, often employed in programming education [2, 48].”
BackgroundCommon logic errors made by novice programmers
Ettles, A., Luxton-Reilly, A., & Denny, P. (2018) · Proceedings of the 20th Australasian Computing Education Conference · doi:10.1145/3160489.3160493 · Cited by 70
Self-citationAbility to 'explain in plain english' linked to proficiency in computer-based programming
Murphy, L., Fitzgerald, S., Lister, R., & McCauley, R. (2012) · Proceedings of the ninth annual international conference on International computing education research · doi:10.1145/2361276.2361299 · Cited by 57
(2009) · doi:10.5555/1862712.1862724 · Cited by 49
Self-citationLearning by Taking Apart: Deconstructing Code by Reading, Tracing, and Debugging
Griffin, J. M. (2016) · Conference on Information Technology Education · doi:10.1145/2978192.2978231 · Cited by 45
Inferential Statistics in Computing Education Research: A Methodological Review
Sanders, K., Sheard, J., Becker, B. A., Eckerdal, A., Hamouda, S., & Simon (2019) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3291279.3339408 · Cited by 40
Evaluating the Quality of Learning Resources: A Learnersourcing Approach
Abdi, S., Khosravi, H., Sadiq, S., & Demartini, G. (2021) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2021.3058644 · Cited by 40
The Impact of Adding Textual Explanations to Next-step Hints in a Novice Programming Environment
Marwan, S., Lytle, N., Williams, J. J., & Price, T. (2019) · Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3304221.3319759 · Cited by 38
Becker, B. A., Denny, P., Finnie-Ansley, J., Luxton-Reilly, A., Prather, J., & Santos, E. A. (2022) · arXiv · doi:10.48550/arxiv.2212.01020 · Cited by 36
Self-citationCrowdsourcing programming assignments with CrowdSorcerer
Pirttinen, N., Kangas, V., Nikkarinen, I., Nygren, H., Leinonen, J., & Hellas, A. (2018) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3197091.3197117 · Cited by 35
Self-citationA top-down approach to teaching programming
Reek, M. M. (1995) · ACM SIGCSE Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/199688.199696 · Cited by 34
Creating Guided Code Explanations with chat.codes
Oney, S., Brooks, C. A., & Resnick, P. (2018) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3274400 · Cited by 33
Code comprehension problems as learning events
DeLyser, L. A., Stehlik, M., & Carver, S. M. (2012) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/2325296.2325319 · Cited by 31
Benefits of Self-explanation in Introductory Programming
Vihavainen, A., Miller, C. S., & Settle, A. (2015) · Proceedings of the 46th ACM Technical Symposium on Computer Science Education · doi:10.1145/2676723.2677260 · Cited by 30
Students Struggle to Explain Their Own Program Code
Lehtinen, T., Lukkarinen, A., & Haaranen, L. (2021) · Proceedings of the 26th ACM Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3430665.3456322 · Cited by 26
Lehtinen, T., Haaranen, L., & Leinonen, J. (2023) · Proceedings of the 25th Australasian Computing Education Conference · doi:10.1145/3576123.3576129 · Cited by 17
Self-citationCrowdsourcing Content Creation for SQL Practice
Leinonen, J., Pirttinen, N., & Hellas, A. (2020) · Proceedings of the 2020 ACM Conference on Innovation and Technology in Computer Science Education · doi:10.1145/3341525.3387385 · Cited by 16
Self-citationBringing "High-level" Down to Earth: Gaining Clarity in Conversational Programmer Learning Goals
Cunningham, K., Qiao, Y., Feng, A., & O'Rourke, E. (2022) · Proceedings of the 53rd ACM Technical Symposium on Computer Science Education · doi:10.1145/3478431.3499370 · Cited by 15
How do Students Experience and Judge Software Comprehension Techniques?
Hebig, R., Ho-Quang, T., Jolak, R., Schröder, J., Linero, H., Ågren, M., & Maro, S. (2020) · International Conference on Program Comprehension · doi:10.1145/3387904.3389283 · Cited by 8
Experimenting with Model Solutions as a Support Mechanism
Nygren, H., Leinonen, J., Pirttinen, N., Leinonen, A., & Hellas, A. (2019) · Proceedings of the 1st UK & Ireland Computing Education Research Conference on - UKICER · doi:10.1145/3351287.3351288 · Cited by 4
Self-citation
Research works printed but not linked: 10
- Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021). · arXiv
- Paul Denny, Sami Sarsa, Arto Hellas, and Juho Leinonen. 2022. Robosourcing Educational Resources--Leveraging Large Language Models for Learnersourcing. arXiv preprint arXiv:2211.04715 (2022). · arXiv
- James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conf. ACM, 10--19.
- Stephen MacNeil, Zijian Ding, Kexin Quan, Thomas j Parashos, Yajie Sun, and Steven P Dow. 2021. Framing Creative Work: Helping Novices Frame Better Problems through Interactive Scaffolding. In Creativity and Cognition. 1--10.
- Kenneth O McGraw and Seok P Wong. 1992. A common language effect size statistic. Psychological bulletin, Vol. 111, 2 (1992), 361.
- Nea Pirttinen and Juho Leinonen. 2022. Can Students Review Their Peers? Comparison of Peer and Instructor Reviews. In Proc. of the 27th ACM Conf. on Innovation and Technology in Computer Science Education Vol 1.
- Judy Sheard, Angela Carbone, Raymond Lister, Beth Simon, Errol Thompson, and Jacqueline L. Whalley. 2008. Going SOLO to Assess Novice Programmers. In Proc. of the 13th Annual Conf. on Innovation and Technology in Computer Science Education. ACM, 209--213.
- Simon and Susan Snowdon. 2011. Explaining Program Code: Giving Students the Answer Helps - but Only Just. In Proc. of the Seventh Int. Workshop on Computing Education Research. ACM, 93--100.
- Ron Sun, Edward Merrill, and Todd Peterson. 2000. Knowledge Acquisition Via Bottom-up Learning. Knowledge-Based Systems (2000), 249--291.
- Jacqueline L. Whalley, Raymond Lister, Errol Thompson, Tony Clear, Phil Robbins, P. K. Ajith Kumar, and Christine Prasad. 2006. An Australasian Study of Reading and Comprehension Skills in Novice Programmers, Using the Bloom and SOLO Taxonomies. In Proc. of the 8th Australasian Conf. on Computing Education - Volume 52. Australian Computer Society, Inc., AUS, 243--252.
Cited by
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
The Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
“The most common concern expressed by authors about student learning was the potential for students to become over-reliant on generative AI tools to solve problems [34, 82, 115, 118, 158] and assist in debugging code [116, 140, 187].”
BackgroundInfluentialSelf-citationA Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4
Kalyan, K. S. (2024) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100048 · Cited by 335
Computing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316
“Recent work has also found that LLM-created code explanations are rated more highly on average by students than code explanations created by their peers [20].”
BackgroundSelf-citationKazemitabaar, M., Ye, R., Wang, X., Henley, A. Z., Denny, P., Craig, M., & Grossman, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642773 · Cited by 290
Self-citation
Metadata
- Authors
- Leinonen, J., Denny, P., MacNeil, S., Sarsa, S., Bernstein, S., Kim, J., Tran, A., & Hellas, A.
- Affiliations
- University of Auckland (NZ)
- Temple University (US)
- Aalto University (FI)
- ORCID
- Venue
- Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1
- Publisher
- ACM
- Published
- 2023-06-29
- Pages
- 124-130
- OpenAlex
- W4382654294
- Cited by
- 210
- References
- 40
- Citations by year
- 2026: 37 · 2025: 76 · 2024: 78 · 2023: 19
- FWCI
- 63.20
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Aaltodoc (Aalto University) (submittedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- Ulla Tuominen Foundation
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
- Keywords
- code explanation, large language models, program comprehension, scaffolding, introductory programming education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.