CoChat
Sign Up
Cited by 235Teaching and Learning ProgrammingSoftware Engineering ResearchIntelligent Tutoring Systems and Adaptive Learning

Using Large Language Models to Enhance Programming Error Messages

Open access
Leinonen, Juho, Hellas, Arto, Sarsa, Sami, Reeves, Brent, Denny, Paul, Prather, James, Becker, Brett A.·2023·Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1
DOI 10.1145/3545945.3569770
Open this paper in CoChat — free accountOpen PDF

Abstract

A key part of learning to program is learning to understand programming error messages. They can be hard to interpret and identifying the cause of errors can be time-consuming. One factor in this challenge is that the messages are typically intended for an audience that already knows how to program, or even for programming environments that then use the information to highlight areas in code. Researchers have been working on making these errors more novice friendly since the 1960s, however progress has been slow. The present work contributes to this stream of research by using large language models to enhance programming error messages with explanations of the errors and suggestions on how to fix them. Large language models can be used to create useful and novice-friendly enhancements to programming error messages that sometimes surpass the original programming error messages in interpretability and actionability. These results provide further evidence of the benefits of large language models for computing educators, highlighting their use in areas known to be challenging for students. We further discuss the benefits and downsides of large language models and highlight future streams of research for enhancing programming error messages.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper explores whether Codex can enhance Python programming error messages with novice-friendly explanations and fixes. It finds generated explanations are often comprehensible and sometimes improve on original messages, but suggested fixes are frequently incorrect and not ready for production classroom use.

Key findings

  • Codex outputs were comprehensible in 88% of cases, included an explanation in 84% of cases, and those explanations were correct in 57% of cases with an explanation (48% of all inputs).
  • Although 70% of outputs proposed a fix, only 47% of those fixes were correct (33% of all inputs).
  • Overall, Codex-generated content was judged an improvement over the original error message in 54% of cases; with temperature 0, over 70% were improvements, while temperature 0.7 reached only up to 50%.
  • Performance varied by error message: correct explanations ranged from 11% for “unexpected EOF while parsing” to 83% for “can’t assign to function call,” and improvement ranged from 22% for “unexpected EOF while parsing” to 78% for “can’t assign to function call” and “invalid token.”

Methods

  • The authors selected nine Python error messages previously reported as most unreadable, then wrote three example programs per message: simple, function-with-strings, and library-based.
  • Using Codex model `code-davinci-002`, they tested five prompts and chose the one with fewest empty responses (4/81), then generated three explanations per error-message/program combination: one at temperature 0 and two at temperature 0.7, yielding 81 unique combinations.
  • Two experienced instructors independently rated seven aspects (e.g., comprehensible, has explanation, explanation correct, has fix, fix correct, improvement over original) with Cohen’s kappa 0.83; percentages were calculated over 162 ratings.

Limitations

  • The paper states that it used Python 3.6, author-created rather than student code, mostly short programs, and only singular errors; newer Python versions have improved some analyzed messages.
  • The paper also states that explanation and fix were requested in a single prompt and that it used zero-shot learning without examples, which may have limited performance.
  • Evident from the text: evaluation used only two raters with binary yes/no judgments, no student or classroom deployment was tested, and results are tied to one Codex model, so they serve as a baseline rather than evidence of classroom effectiveness.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3545945.3569770

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 38 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    The number of empty responses for the five prompts was 4, 6, 7, 16, and 27 out of 81 generated explanations, respectively.

    “The number of empty responses was 4, 6, 7, 16 and 27 out of 81 generated explanations respectively, for prompts 1 to 5 above.”
    Quote in the paperSubstantial claim
  • Findingown data

    In most cases, Codex successfully created an explanation of the error message, with the percentage ranging from 67% to 100% depending on error message.

    “In most cases, Codex successfully created an explanation of the error message (67% to 100% of the time depending on error message)”
    Quote in the paperSubstantial claim
  • Findingown data

    The first prompt generated the fewest empty responses.

    “We chose the first for the analysis as it generated the fewest empty responses.”
    Quote in the paperSubstantial claim
  • Findingown data

    In the majority of cases, Codex provided a fix in the generated explanation, with the percentage ranging from 44% to 89%.

    “we found that in the majority of cases, Codex provided a fix in the generated explanation (44% to 89% of cases).”
    Quote in the paperSubstantial claim
  • Findingown data

    There were certain cases where Codex was more likely to provide faulty explanations and suggest fixes that are incorrect.

    “we found that there were certain cases where Codex was more likely to provide faulty explanations and suggest fixes that are incorrect”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 27
  • May combine two claims. 18
  • The quote only partly states this claim. 4

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Codex suggests that the issue is related to indentation.

    “Codex suggests that the issue is related to indentation.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Although 70% of the created content included a fix, when a fix was included it was correct only under half of the time (47%).

    “Although 70% of the created content included a fix, when a fix was included it was correct only under half of the time (47%).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing), S4 (based). May combine two claims. The quote only partly states this claim.

    Findingown data

    The output was considered an improvement in over 70% of the cases with a temperature value of 0, while only up to 50% of the cases with a temperature value of 0.7.

    “If we only consider the results from using a temperature value of 0, over 70% of the Codex outputs were considered an improvement over the original programming error message.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingreasoning
    § Add more spaces or tabs to the start of each line of code to be able to run it.

    One researcher at times considered the error message as an improvement if it pointed the students in the correct direction, even if the explanation by itself would be faulty.

    “one of the researchers at times considered the error message as an improvement if it pointed the students in the correct direction, even if the explanation by itself would be faulty.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Fixes created with the temperature value of 0 were better on average compared to those created with a value of 0.7, but they were still correct in only around half of the cases: 42%, 56%, and 61% for functions with strings, simple programs, and library related programs respectively.

    “While the fixes created with the temperature value of 0 were better on average compared to those created with a value of 0.7, they were still correct in only around half of the cases -42%, 56%, and 61% of the cases for functions with strings, simple programs, and library related programs respectively.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

    Findingown data

    In one case, the generated output included a correct explanation of the problem but also stated that 'this is not a correct explanation'.

    “the generated output actually does include a correct explanation of the problem -”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    Two examples of outputs where both the explanation and suggested fix generated by Codex were incorrect are shown in Codex Examples 3 and 4.

    “Two examples of outputs where both the explanation and suggested fix generated by Codex were incorrect are shown in Codex Examples 3 and 4.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    For explaining programming error messages and creating suggestions for fixes to the source code that produced those errors, using a temperature value of 0 resulted in considerably better outputs, which holds for all three program categories.

    “From the table, it is evident that for the task of explaining PEMs and creating suggestions for fixes to the source code that produced those errors, using a temperature value of 0 resulted in considerably better outputs, which holds for all three program categories.”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data

    The program category 'function with strings' had the lowest scores overall.

    “the program category”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In one case, Codex focused too much on the 'Plain English' portion of the input and started generating irrelevant content related to 'looking for a plain English explanation'.

    “In one case, Codex seems to have focused too much on the”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingnone given

    Some outputs, specifically two examples, were considered good enough to be shown to students without modifications.

    “that we consider good enough to be shown to students without modifications.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Early work by Finnie-Ansley et al. found that Codex outperformed most students and was capable of generating a variety of correct solutions to given problems.

    “They found that it outperformed most students, and was capable of generating a variety of correct solutions to given problems [19].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [19]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Artificial intelligence and machine learning approaches have been used for finding and repairing errors in programs, but only very fundamental approaches have been applied to researching programming error messages.

    “Artificial intelligence and machine learning approaches have been used for finding and repairing errors in programs [1,20,21] but only very fundamental approaches have been applied to researching PEMs [11].”
    Quote in the paperSubstantial claim0 of 4 sources checked
    • cites [1]

      Not checked in the free run.

    • cites [20] source

      Not checked in the free run.

    • cites [21] source

      Not checked in the free run.

    • cites [11]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Codex could solve introductory programming problems and ranked in the top quartile compared to a cohort of actual students in a large introductory programming course.

    “One study found that Codex, a text-to-code model built on top of GPT-3, a text-to-text model (see Section 2.2) could solve introductory programming problems, and ranked Codex in the top quartile when compared to a cohort of actual students in a large introductory programming course [19].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [19]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Although some studies have shown positive effects of rewording messages for novices, in general the evidence for the effectiveness of enhanced compiler error messages is not overwhelming.

    “However, although some studies have shown positive effects of rewording messages for novices [9,16], in general the evidence for the effectiveness of enhanced compiler error messages is not overwhelming [7].”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites [9] source

      Not checked in the free run.

    • cites [16] source

      Not checked in the free run.

    • cites [7] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    One limitation of prior work in this area is the manual effort required to generate enhanced programming error messages and a lack of clear guidance for addressing core issues such as readability.

    “One of the limitations of prior work in this area is the manual effort required to generate enhanced PEMs and a lack of clear guidance for addressing core issues such as readability [17].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [17]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Large language models, particularly pre-trained transformer models, have rapidly become a core technology in natural language processing.

    “Large Language Models (LLMs), particularly pre-trained transformer models, have rapidly become a core technology in natural language processing [26].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [26] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    For over six decades, researchers have attempted to improve programming error messages, and there is still a call for more work on the topic.

    “For over six decades, researchers have attempted to improve these messages, and still there is a call for more work on the topic [7].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [7] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Codex is essentially a GPT-3 model fine-tuned with more than 50 million repositories representing the majority of Python code available on GitHub, totalling 159 GB of source code.

    “Codex which is essentially a GPT-3 model that has been fine-tuned with more than 50 million repositories representing the majority of Python code available on GitHub totalling 159 GB of source code [13].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [13]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Researchers and instructors have reported programming error messages to be difficult for students since at least 1965.

    “Researchers and instructors have reported PEMs to be difficult for students since at least 1965 [35].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [35] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Recent attempts have been made to put error messages into more natural language by focusing on an increase in readability.

    “Some recent attempts have been made to put error messages into more natural language by focusing on an increase in readability [8,17].”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [8]

      Not checked in the free run.

    • cites [17]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Programming error messages may contribute to the perception that programming is overly challenging.

    “possibly to the extent that they contribute to the perception that programming is overly challenging [6].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [6] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Sarsa et al. found that most problems generated by Codex were novel and sensible, and that the generated code explanations were generally correct and thorough.

    “They found that most of the problems generated by Codex were novel and sensible, and that the generated code explanations were generally correct and thorough.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [36] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Tools like Codex can generate new programming assignments and code explanations when provided examples.

    “Tools like Codex are able to generate new programming assignments [36] and code explanations [29] when provided examples.”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [36] source

      Not checked in the free run.

    • cites [29] source

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Increasing the readability of programming error messages improves student understanding of error messages and the number of successful code corrections.

    “This has been shown to improve student understanding of error messages and the number of successful code corrections [16].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [16] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The evaluation was performed by two researchers, both of whom have experience teaching introductory programming.

    “The evaluation was performed by two researchers, both of whom have experience teaching introductory programming.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    These results provide further evidence of the benefits of large language models for computing educators, highlighting their use in areas known to be challenging for students.

    “These results provide further evidence of the benefits of large language models for computing educators, highlighting their use in areas known to be challenging for students.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § Add more spaces or tabs to the start of each line of code to be able to run it.

    The utility of these explanations depends on if students understand the implications of the suggestions.

    “The utility of these explanations also depends on if students understand the implications of the suggestions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    Incorrect suggestions about indentation could exacerbate novices' struggles by potentially misleading students and introducing misconceptions related to correct indentation.

    “these incorrect suggestions could exacerbate this by potentially misleading students and even introduce misconceptions related to correct indentation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given
    § Add more spaces or tabs to the start of each line of code to be able to run it.

    The original error messages might be more useful for more experienced students who have learned to interpret them.

    “the original error messages might be more useful for more experienced students who have learned to interpret them.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Few-shot learning can drastically improve the performance of large language models.

    “Prior work has found that giving even just a few examples (i.e.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [12]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Add more spaces or tabs to the start of each line of code to be able to run it.

    Prior research has shown that LLMs used to facilitate source code creation may lead students down debugging rabbitholes.

    “Prior research investigating LLMs has shown that when used to facilitate the creation of source code, they may lead students down debugging rabbitholes [38]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [38] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Novices often struggle with indentation.

    “As novices often struggle with indentation [24,27]”
    Quote in the paperNot a substantial claim0 of 2 sources checked
    • cites [24] source

      Not checked in the free run.

    • cites [27] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Comparing the incorrect outputs with the correct outputs, the messages seem similarly confident in tone, which could potentially mislead students.

    “Comparing these incorrect outputs with the correct outputs in Codex Examples 1 and 2, we observe that the messages seem similarly confident in their tone, which could potentially mislead students.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    The present results create a baseline that future model performance can be compared to.

    “these results create a baseline that future model performance can be compared to.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § Add more spaces or tabs to the start of each line of code to be able to run it.

    Prior research has shown that LLMs used to facilitate source code creation may introduce security flaws.

    “or even introduce security flaws [30].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [30] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Using large language models to explain programming error messages is feasible and shows promise.

    “Our results suggest that using large language models to explain programming error messages (PEMs) is feasible and shows promise.”
    Quote in the paperSubstantial claim
37 of 37

  • The paper relies on this without support.

    No prior work has explored the use of large language models for improving programming error messages.

    “To our knowledge, no prior work has explored the use of large language models for improving PEMs.”

  • The paper relies on this without support.

    Codex's performance is representative of large language models generally.

    “We find that PEM explanations and suggested fixes generated by LLMs are not yet ready for production use in introductory programming classes”

  • The paper relies on this without support.

    Students may interpret faulty LLM outputs as coming from an authority.

    “there are risks that students may interpret potentially faulty LLM outputs as coming from an authority”

  • The paper relies on this without support.

    Students may attempt to fix their programs in ways that do not actually help.

    “and attempt to fix their programs in ways that do not actually help.”

  • The paper relies on this without support.

    The two researchers' independent evaluation is a valid measure of explanation quality.

    “the two researchers who independently evaluated the error messages”
5 of 5

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI has already been used to improve error messages.

    “At a simpler level, AI has already been used, for example, to improve error messages [25].”

    From Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study (2024) · cites this paper as [25]

  • Leinonen et al. found that Codex could enhance programming error messages approximately 54% of the time on average, noting that this performance is not good enough for using the model directly with students.

    “Leinonen et al. found that Codex could enhance programming error messages-which are notoriously hard for students to understand-approximately 54% of the time on average, noting that this performance is not good enough for using the model directly with students [125].”

    From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [125]

  • Leinonen et al. suggest that LLMs could help students understand terse error messages which have traditionally been a source of difficulty for novice learners.

    “Leinonen et al. suggest that LLMs could help students understand terse error messages [125] which have traditionally been a source of difficulty for novice learners [36].”

    From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [125]

  • The most common concern expressed by authors about student learning was the potential for students to become over-reliant on generative AI tools to solve problems and assist in debugging code.

    “The most common concern expressed by authors about student learning was the potential for students to become over-reliant on generative AI tools to solve problems [33,85,123,126,169] and assist in debugging code [125,150,199].”

    From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [125]

  • Temperature 0 should yield the most deterministic responses and has been found to work well for feedback in prior work.

    “For temperature, a parameter that controls randomness in LLM responses, we used 0, which should yield the most deterministic responses and has been found to work well for feedback in prior work [45].”

    From Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests (2023) · cites this paper as [45]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • 81 outputs are not 81 unique temperature-value combinations

    Minor

    For each error message and program, the study generated three outputs but used only two temperature values: one output at 0 and two at 0.7. The two outputs at 0.7 therefore share the same error-message, program-category, and temperature-value combination, so the stated 81 combinations are not all unique.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 30

  1. Toufique Ahmed, Noah Rose Ledesma, and Premkumar Devanbu. 2021. SYNFIX: Automatically Fixing Syntax Errors using Compiler Diagnostics. arXiv preprint arXiv:2104.14671 (2021). · arXiv
  2. 10.1145/3183377.3183383 · DOI
  3. Titus Barik. 2018. Error Messages as Rational Reconstructions. Ph.,D. Dissertation. North Carolina State University.
  4. 10.1145/2839509.2844584 · DOI
  5. 10.1145/3469115 · DOI · Paper page
  6. 10.1145/3344429.3372508 · DOI · Paper page
  7. Brett A. Becker, Paul Denny, James Prather, Raymond Pettit, Robert Nix, and Catherine Mooney. 2021. Towards Assessing the Readability of Programming Error Messages. In Australasian Computing Education Conference. ACM, 181--188.
  8. 10.1080/08993408.2016.1225464 · DOI · Paper page
  9. 10.1145/3159450.3159461 · DOI
  10. Brett A Becker and Catherine Mooney. 2016. Categorizing Compiler Error Messages with Principal Component Analysis. In 12th China-Europe International Symposium on Software Engineering Education (CEISEE 2016).
  11. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language Models are Few-shot Learners. Advances in Neural Information Processing Systems, Vol. 33 (2020), 1877--1901.
  12. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021). · arXiv
  13. 10.1145/2591708.2591748 · DOI
  14. 10.1145/1999747.1999807 · DOI · Paper page
  15. 10.1145/3341525.3387384 · DOI · Paper page
  16. Paul Denny, James Prather, Brett A. Becker, Catherine Mooney, John Homer, Zachary C Albrecht, and Garrett B. Powell. 2021. On Designing Programming Error Messages for Novices: Readability and Its Constituent Factors. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM.
  17. Paul Denny, Sami Sarsa, Arto Hellas, and Juho Leinonen. 2022. Robosourcing Educational Resources--Leveraging Large Language Models for Learnersourcing. arXiv preprint arXiv:2211.04715 (2022). · arXiv
  18. James Finnie-Ansley, Paul Denny, Brett A Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conference. 10--19.
  19. 10.1609/aaai.v31i1.10742 · DOI · Paper page
  20. Ioannis Karvelas, Annie Li, and Brett A. Becker. 2020. The Effects of Compilation Mechanisms and Error Message Presentation on Novice Programmer Behavior. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. ACM, 759--765.
  21. 10.1145/3287324.3287381 · DOI · Paper page
  22. J Richard Landis and Gary G Koch. 1977. The Measurement of Observer Agreement for Categorical Data. biometrics (1977), 159--174.
  23. 10.1145/3490443 · DOI · Paper page
  24. 10.1145/3287324.3287503 · DOI · Paper page
  25. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. arXiv preprint arXiv:2107.13586 (2021). · arXiv
  26. 10.1145/3501709.3544280 · DOI · Paper page
  27. 10.1145/3230977.3230981 · DOI · Paper page
  28. 10.1145/3105726.3106169 · DOI · Paper page
  29. 10.1145/3291279.3339423 · DOI · Paper page
  30. 10.1145/3501385.3543957 · DOI · Paper page

Cited by

Sorted by citations

Authors
Leinonen, J., Hellas, A., Sarsa, S., Reeves, B., Denny, P., Prather, J., & Becker, B. A.
Affiliations
  • Aalto University (FI)
  • Abilene Christian University (US)
  • University of Auckland (NZ)
  • University College Dublin (IE)
ORCID
Venue
Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1
Publisher
ACM
Published
2023-03-02
Pages
563-569
DOI
10.1145/3545945.3569770
OpenAlex
W4323033692
Semantic Scholar
668faca09fcefd18a46ab5ce4eab765c065e1d5e
Cited by
235
References
8
Citations by year
2026: 39 · 2025: 64 · 2024: 87 · 2023: 45
FWCI
70.95
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Technical Symposium on Computer Science Education (SIGCSE) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Aaltodoc (Aalto University) (submittedVersion, cc-by)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
Keywords
large language models, error explanation, programming education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review