CoChat
Sign Up
Cited by 145Teaching and Learning ProgrammingArtificial Intelligence in EducationSoftware Engineering Research

Exploring the Responses of Large Language Models to Beginner Programmers’ Help Requests

Open access
Hellas, Arto, Leinonen, Juho, Sarsa, Sami, Koutcheme, Charles, Kujanpää, Lilja, Sorva, Juha·2023·Proceedings of the 2023 ACM Conference on International Computing Education Research V.1
DOI 10.1145/3568813.3600139
Open this paper in CoChat — free accountOpen PDF

Abstract

Background and Context: Over the past year, large language models (LLMs) have taken the world by storm. In computing education, like in other walks of life, many opportunities and threats have emerged as a consequence.

AI snapshot

Written from the paper’s full text.

TL;DR

This paper studies how OpenAI Codex and GPT-3.5 respond to beginner programmers’ help requests from an online Dart course, finding GPT-3.5 generally outperforms Codex but both are unreliable. LLMs often identify at least one real issue and give sensible advice, yet frequently miss issues, report false positives, and provide model solutions despite being asked not to.

Key findings

  • Across 150 help requests, the authors identified 275 issues (average 1.9 per request): logic errors in 72%, input/output problems in 34%, syntax errors in 8.0%, very incomplete solutions in 5.3%, semicolon issues in 2.7%, and hidden automated-test requirements in 2.0%.
  • GPT-3.5 identified at least one actual issue in 90% of cases in both English and Finnish, versus Codex at 70% in English and 33% in Finnish; GPT-3.5 identified all issues in 57% (English) and 53% (Finnish), versus Codex at 13% and 17%. False positives were common: 40% for GPT-3.5 English, 23% for GPT-3.5 Finnish, 40% for Codex English, and 43% for Codex Finnish.
  • In the extended GPT-3.5 English analysis of all 150 requests, it identified at least one actual issue in 82%, all actual issues in 55%, and non-existent issues in 48%; 99% of responses included code, 65% included a model solution, and 0% included automated tests.
  • GPT-3.5 performed better on some logic issues than output formatting: it addressed all issues in 35% of conditional cases, 73% of iteration cases, and 57% of arithmetic cases, compared with 44% for formatting, 54% for unwanted outputs, and 50% for missing outputs.

Methods

  • The data came from a 2022 open online introductory programming course at Aalto University taught in Finnish with Dart; the course had 4,247 distinct students, 120,583 submissions, and 831 help requests. The authors selected the 15 exercises with the most help requests and randomly sampled 10 help requests per exercise, yielding 150 help requests.
  • They prompted two LLMs—OpenAI Codex (`code-davinci-002`) and GPT-3.5 (`gpt-3.5-turbo`)—with English and Finnish versions of a prompt template, using temperature 0, generating 600 responses total (150 help requests × 2 languages × 2 models).
  • Student code was qualitatively annotated for issues by one experienced course teacher; LLM responses were manually categorized on seven aspects, first comparing 120 responses across models/languages, then analyzing all 150 GPT-3.5 English responses and thematically analyzing researcher notes.

Limitations

  • The paper states that findings are limited by the specific context: Dart is relatively new for LLMs, the course is fully online with an atypical scope and student cohort, and only a minority of submissions had associated help requests.
  • The authors state that newer LLMs already existed (GPT-4 was not programmatically available), that qualitative analysis used a single coder, and that they relied on a single request rather than an ongoing conversational interaction.
  • Evident from the text: Codex API discontinuation makes replication of those results nearly impossible, and the prompt engineering plus manual cleanup of Codex responses may affect comparability.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3568813.3600139

Cites

65 of 65 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

0 of 8 cited sources checked · 0 supported · 0 flagged

  • Findingown data

    There was a clear difference in performance between GPT-3.5 and Codex.

    “The table shows a clear difference in performance between GPT-3.5 and Codex.”
    Quote in the paperSubstantial claim
  • Findingnone given

    Model solutions are frequently provided even when the LLM is prompted not to.

    “Model solutions are frequently provided even when the LLM is prompted not to.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The teacher's annotation of issues in student code is a valid ground truth for what issues are actually present.
    • Rests on an assumption: The 15 exercises with the most help requests and 10 randomly sampled help requests per exercise are representative of help requests in the course.
  • Findingown data

    Logic errors were the most common theme, present in 108 help requests (72%).

    “From most to least common, they are: (1) logic errors, present in 108 help requests, 72%;”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The teacher's annotation of issues in student code is a valid ground truth for what issues are actually present.
    • Rests on an assumption: The 15 exercises with the most help requests and 10 randomly sampled help requests per exercise are representative of help requests in the course.
  • Findingown data

    All of the LLM responses were phrased as actual attempts to help.

    “All of the LLM responses were phrased as actual attempts to help.”
    Quote in the paperNot a substantial claim
  • Findingown data

    Arithmetic was a logic error sub-theme in 23 requests.

    “• Arithmetic (23).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The teacher's annotation of issues in student code is a valid ground truth for what issues are actually present.
    • Rests on an assumption: The 15 exercises with the most help requests and 10 randomly sampled help requests per exercise are representative of help requests in the course.
  • May combine two claims. 31
  • Classified as probably not a substantial claim. 33
  • The quote only partly states this claim. 4

  • May combine two claims.

    Findingown data

    Including the model solution in the prompt often led to a response explaining that solution and increased the chance of the solution being echoed in the response.

    “Another observation was that including the model solution in the prompt often led to a response explaining that solution and increased the chance of the solution being echoed in the response.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In many cases, a help request had more than one issue (1.9 on average).

    “In many cases, a help request had more than one issue (1.9 on average)”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Findingown data

    The help request texts were unnecessary, as they were generally uninformative beyond indicating that the student was struggling.

    “During prompt engineering, we observed that the help request texts were unnecessary, as they were generally uninformative beyond indicating that the student was struggling.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In 9 responses, the LLM itself was given agency.

    “In nine responses, the LLM itself was given agency”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Both LLMs frequently find at least one actual issue in each student program; GPT-3.5 does so in 90% of cases.

    “Both LLMs frequently find at least one actual issue in each student program (GPT-3.5 in 90% of the cases).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The teacher's annotation of issues in student code is a valid ground truth for what issues are actually present.
    • Rests on an assumption: The 15 exercises with the most help requests and 10 randomly sampled help requests per exercise are representative of help requests in the course.

  • The quote only partly states this claim.

    Findingown data

    The vast majority of logic errors fell under one of three sub-themes: conditionals, iteration, and arithmetic.

    “The vast majority of logic errors fell under one of three sub-themes:”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The teacher's annotation of issues in student code is a valid ground truth for what issues are actually present.
    • Rests on an assumption: The 15 exercises with the most help requests and 10 randomly sampled help requests per exercise are representative of help requests in the course.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    For input/output errors, missing printouts was a dominant sub-theme, with 10 requests.

    “Missing printouts (10).”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    For 82 of those 123 help requests, GPT-3.5 identified and mentioned all actual issues (55%).

    “for 82 of those, it identified and mentioned all actual issues.”
    • Also stated in: § 5.1 LLM Performance on Help Requests · plain
      “all were identified and mentioned in 55% of the cases.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    109 of the 150 GPT-3.5 English responses received at least one annotation.

    “For further insight, we annotated the LLM responses with freeform notes, noting any phenomena that appeared potentially interesting from a pedagogical point of view; 109 of the 150 responses received at least one annotation.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The teacher's annotation of issues in student code is a valid ground truth for what issues are actually present.
    • Rests on an assumption: The 15 exercises with the most help requests and 10 randomly sampled help requests per exercise are representative of help requests in the course.

  • May combine two claims.

    Findingown data

    Neither LLM excels at finding all issues; GPT-3.5 finds all issues 57% of the time.

    “Neither LLM excels at finding all the issues (GPT-3.5 finding them 57% of the time).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The teacher's annotation of issues in student code is a valid ground truth for what issues are actually present.
    • Rests on an assumption: The 15 exercises with the most help requests and 10 randomly sampled help requests per exercise are representative of help requests in the course.

  • May combine two claims.

    Findingown data

    False positives are common; GPT-3.5 has a 40% chance of identifying a non-existent issue.

    “False positives are common (40% chance for GPT-3.5).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The teacher's annotation of issues in student code is a valid ground truth for what issues are actually present.
    • Rests on an assumption: The 15 exercises with the most help requests and 10 randomly sampled help requests per exercise are representative of help requests in the course.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Failing to meet hidden requirements in automated tests was present in 3 help requests (2.0%).

    “(6) failing to meet hidden requirements in automated tests, 3, 2.0%.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The teacher's annotation of issues in student code is a valid ground truth for what issues are actually present.
    • Rests on an assumption: The 15 exercises with the most help requests and 10 randomly sampled help requests per exercise are representative of help requests in the course.

  • The quote only partly states this claim.

    Findingown data

    For input/output errors, unwanted printouts was a dominant sub-theme, with 24 requests.

    “Unwanted printouts (24).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    There was considerable variation between exercises in the issues that arose.

    “As expected, there was considerable variation between exercises.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    During 2022, the course had 4,247 distinct students, who made 120,583 submissions to programming exercises, and 831 help requests were submitted.

    “During the year, there were 4,247 distinct students in the course, who collectively made 120,583 submissions to programming exercises.831 help requests were submitted.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    All programs associated with a help request had at least one issue; the maximum was six.

    “All programs associated with a help request had at least one issue; the maximum was six.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The teacher's annotation of issues in student code is a valid ground truth for what issues are actually present.
    • Rests on an assumption: The 15 exercises with the most help requests and 10 randomly sampled help requests per exercise are representative of help requests in the course.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Problems with semicolons were present in 4 help requests (2.7%).

    “(5) problems with semicolonsfoot_7 , 4, 2.7%;”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The teacher's annotation of issues in student code is a valid ground truth for what issues are actually present.
    • Rests on an assumption: The 15 exercises with the most help requests and 10 randomly sampled help requests per exercise are representative of help requests in the course.

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Findingnone given

    It appeared unnecessary to include trivial starter code (an empty function).

    “Moreover, it appeared unnecessary to include trivial starter code (an empty function).”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    GPT-3.5 identified non-existing issues in 72 help requests (48%).

    “The LLM identified non-existing issues in 72 help requests.”
    • Also stated in: § 5.1 LLM Performance on Help Requests · plain
      “48% of them reported on issues that did not actually exist in the student's code.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Typically, a single subtheme was prevalent in a particular exercise, but some exercises had a varied mix of issues.

    “Typically, a single subtheme was prevalent in a particular exercise (e.g., conditionals in the Verification of input exercise; formatting issues in First and last name), but there were some exercises with a varied mix of issues.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Codex identified and mentioned at least one actual issue in 70% of English cases and 33% of Finnish cases.

    “Codex succeeded 70% of the time in English, with Finnish performance far behind at 33%.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    There are exercise-specific differences in the extent to which GPT-3.5 responses address the issues, with no obvious pattern.

    “there are exercise-specific differences in the extent to which the responses address the issues; there is no obvious pattern, however.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In 150 help requests, there were a total of 275 issues, an average of 1.9 issues per help request.

    “In 150 help requests, we identified a total of 275 issues, for an average of 1.9 issues per help request.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The teacher's annotation of issues in student code is a valid ground truth for what issues are actually present.
    • Rests on an assumption: The 15 exercises with the most help requests and 10 randomly sampled help requests per exercise are representative of help requests in the course.

  • May combine two claims.

    Findingown data

    Other, less common logic errors included misusing function parameters, printing in a function when expected to return a value, misplacing logic, and placing variables outside of functions.

    “Other, less common logic errors included misusing function parameters, printing in a function when expected to return a value, misplacing logic, and placing variables outside of functions (leading, e.g., to a sum variable getting incremented over multiple function calls).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The teacher's annotation of issues in student code is a valid ground truth for what issues are actually present.
    • Rests on an assumption: The 15 exercises with the most help requests and 10 randomly sampled help requests per exercise are representative of help requests in the course.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Anecdotally, our dataset had several instances of the dreaded semicolon-in-conditional: if (foo); { ... }.

    “Anecdotally, our dataset had several instances of the dreaded semicolon-in-conditional: if (foo); { ... }.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    During 2022, there were 120,583 submissions but only 831 (0.7%) of them had a help request.

    “During 2022, we got 120,583 submissions but only 831 (0.7%) of them had a help request.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In comparison with some prior studies that have reported data on syntax errors, our sample had relatively few: syntax errors featured in only 8.0% of the help requests.

    “our sample had relatively few: syntax errors featured in only 8.0% of the help requests.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    These syntax errors were often jointly present with other types of issues.

    “These syntax errors were often jointly present with other types of issues.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Findingown data

    The Finnish in the LLM's responses was in general understandable and could have been shown to students, as far as language quality was concerned.

    “The Finnish in the LLM's responses was also in general understandable and could have been shown to students, as far as language quality was concerned.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingreasoning

    Such struggles with I/O reduced later on in the course, perhaps suggesting that students learned to pay closer attention to the handouts.

    “Such struggles with I/O reduced later on in the course, perhaps suggesting that students learned to pay closer attention to the handouts.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Although the LLM responses sought to address the technical issues and at times provided positive feedback, we saw little indication of the feedback being adjusted to the beginner level of the programming exercises being solved or to the context.

    “we saw little indication of the feedback being adjusted to the (beginner) level of the programming exercises being solved or to the context (the introductory course that we mentioned in the prompt).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    The research was conducted in compliance with local ethical principles and guidelines.

    “The research was conducted in compliance with the local ethical principles and guidelines.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Temperature 0 should yield the most deterministic responses and has been found to work well for feedback in prior work.

    “For temperature, a parameter that controls randomness in LLM responses, we used 0, which should yield the most deterministic responses and has been found to work well for feedback in prior work [45].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [45] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationreasoning

    LLMs make some of the same mistakes that students do, perhaps especially when formatting output as required by automated assessment systems.

    “the results highlight the unreliability of LLMs: LLMs make some of the same mistakes that students do, perhaps especially when formatting output as required by automated assessment systems.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The simulated scenario of feeding actual help requests to LLMs outside the course is representative of how LLMs would respond to student help requests in actual programming courses.
    • Rests on an assumption: The teacher's annotation of issues in student code is a valid ground truth for what issues are actually present.
    • Rests on an assumption: The 15 exercises with the most help requests and 10 randomly sampled help requests per exercise are representative of help requests in the course.

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    The source code from the 150 help requests was annotated with issues that a teacher would provide feedback on by one researcher who is the teacher responsible for the course, has more than a decade of experience in teaching introductory programming, and has specific experience of answering help requests in this course.

    “We annotated the source code from the 150 help requests with issues that a teacher would provide feedback on.This was carried out by one of the researchers, who is the teacher responsible for the course, has more than a decade of experience in teaching introductory programming, and has specific experience of answering help requests in this course.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The help requests were annotated again instead of using existing answers to ensure a consistent baseline for the present analysis.

    “We chose to annotate the help requests again instead of using existing answers to these help requests, as the help requests had been previously answered by a pool of teachers and teaching assistants, and we wanted a consistent baseline for the present analysis.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Interpretationown data

    The results highlight the utility of LLMs in programming education.

    “Our results continue to highlight the utility of LLMs in programming education.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The simulated scenario of feeding actual help requests to LLMs outside the course is representative of how LLMs would respond to student help requests in actual programming courses.
    • Rests on an assumption: The teacher's annotation of issues in student code is a valid ground truth for what issues are actually present.
    • Rests on an assumption: The 15 exercises with the most help requests and 10 randomly sampled help requests per exercise are representative of help requests in the course.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    To avoid leaking personal information to third-party services, the inputs fed to the LLMs were manually vetted during prompt engineering and final generation.

    “To avoid leaking any personal information to third-party services, we manually vetted the inputs that we fed to the LLMs, both during prompt engineering and during the final generation of the responses.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    Instructional guidance should attend to the student's current level of domain knowledge; a mismatch will result in poorer learning outcomes.

    “Instructional guidance should attend to the student's current level of domain knowledge; a mismatch will result in poorer learning outcomes [37].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [37]

      Not checked in the free run.

  • May combine two claims.

    Premisecitation

    Praising on easy tasks may lead students simply to dismiss the feedback; at worst, it may implicitly suggest that the student lacks ability and demotivate the student.

    “Praising on easy tasks may lead students simply to dismiss the feedback; at worst, it may implicitly suggest that the student lacks ability [8] and demotivate the student.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [8]

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    The I/O issues might stem from worked examples in the course materials.

    “the I/O issues might stem from worked examples in the course materials”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given

    These findings are specific to the studied context and their generalizability remains to be determined.

    “These findings, too, are specific to the studied context and their generalizability remains to be determined.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    There are cases where model solutions are useful: model solutions have been deliberately provided to students in some prior research, and they are also often provided in automated learning environments that are not focused on grading.

    “model solutions have been deliberately provided to students in some prior research [62,63], and they are also often provided in automated learning environments that are not focused on grading [34].”
    Quote in the paperNot a substantial claim0 of 3 sources checked
    • cites [62] source

      Not checked in the free run.

    • cites [63] source

      Not checked in the free run.

    • cites [34] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    GPT-3.5 might be described as quite effective at issue-hunting, but it is far from reliable in terms of finding all the issues, and false positives are common.

    “Overall, GPT-3.5 might be described as quite effective at issue-hunting, but it is far from reliable in terms of finding all the issues, and false positives are common as well.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning

    The issues that students request help with might thus differ from other courses.

    “the issues that students request help with might thus differ from other courses.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The course uses a relatively recent programming language (Dart), which the LLMs will not have 'seen' as much as some other programming languages.
    • Rests on an assumption: The course is fully online, and its scope and student cohort are different from many typical introductory programming courses at the university level.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The vast majority of prior research has not explicitly collected information on whether students want help, so some earlier findings about student 'failure' may in fact be related to students employing the submission system as a feedback mechanism, not necessarily needing help but simply checking whether they are on the right path.

    “the vast majority of prior research has not explicitly collected information on whether students want help, so some earlier findings about student 'failure' may in fact be related to students employing the submission system as a feedback mechanism, not necessarily needing help but simply checking whether they are on the right path.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    While working on this article, the authors had no access to the GPT-4 API.

    “While working on this article, we had no access to the GPT-4 API.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Prior research such as the reported mismatch between educators' beliefs about students' mistakes and logged data about mistakes might be explained in part by students asking for help from educators only when they really need help, which might differ from how they employ automated systems.

    “prior research such as the reported mismatch between educators' beliefs about students' mistakes and logged data about mistakes [9] might be explained in part by students asking for help from educators only when they really need help, which might differ from how they employ automated systems.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [9] source

      Not checked in the free run.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Prior research on a help request platform for programming noted that only one third of the students who open up a help request dialog end up writing a help request, and that even the prompts used in the help request dialog can influence whether a help request gets sent.

    “Prior research on a help request platform for programming noted that only one third of the students who open up a help request dialog end up writing a help request, and that even the prompts used in the help request dialog can influence whether a help request gets sent [77].”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [77] source

      Not checked in the free run.

  • May combine two claims.

    Interpretationown data

    Both large language models were able to identify some issues in the help requests, but GPT-3.5 was considerably more accurate than Codex.

    “Both large language models were able to identify some issues in the help requests, but GPT-3.5 was considerably more accurate than Codex.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    In the present study, we relied on a single request to the model.

    “In the present study, we relied on a single request to the model.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

    Interpretationreasoning

    LLMs are a potentially excellent supplement for programming teachers and teaching assistants, available at scale to serve the ever-increasing numbers of programming students.

    “We see LLMs as a potentially excellent supplement for programming teachers and teaching assistants, available at scale to serve the ever-increasing numbers of programming students.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    LLMs are not a replacement for teachers.

    “Not as a replacement for teachers, however.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    Our qualitative analysis employed a single coder, which is a threat to reliability.

    “Our qualitative analysis employed a single coder, which is a threat to reliability.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Overall, the LLMs' responses are usually sensible and potentially helpful.

    “Overall, we find that the LLMs' responses are usually sensible and potentially helpful (RQ2).”
    Quote in the paperSubstantial claim
55 of 55

  • The paper relies on this without support.

    The simulated scenario of feeding actual help requests to LLMs outside the course is representative of how LLMs would respond to student help requests in actual programming courses.

    “we study a simulacrum of such a scenario: we take actual help requests collected during a programming course (and answered then by humans) and feed the requests as input to LLMs so that we the researchers may explore the responses.”

  • The paper relies on this without support.

    The teacher's annotation of issues in student code is a valid ground truth for what issues are actually present.

    “We annotated the source code from the 150 help requests with issues that a teacher would provide feedback on.This was carried out by one of the researchers, who is the teacher responsible for the course, has more than a decade of experience in teaching introductory programming, and has specific experience of answering help requests in this course.”

  • The paper relies on this without support.

    The 15 exercises with the most help requests and 10 randomly sampled help requests per exercise are representative of help requests in the course.

    “we focus on the fifteen programming exercises with the most help requests (out of 64 exercises in total).”
    “For each of the 15 exercises with the most help requests, we randomly sampled ten, which yielded a body of 150 help requests in total.”

  • The paper relies on this without support.

    The course uses a relatively recent programming language (Dart), which the LLMs will not have 'seen' as much as some other programming languages.

    “The course uses a relatively recent programming language (Dart), which the LLMs will not have 'seen' as much as some other programming languages.”

  • The paper relies on this without support.

    The course is fully online, and its scope and student cohort are different from many typical introductory programming courses at the university level.

    “the course is fully online, and its scope and student cohort are different from many typical introductory programming courses at the university level”

  • The paper relies on this without support.

    The proliferation of LLMs and their inevitable use by both novices and professional programmers.

    “The proliferation of LLMs and their inevitable use by both novices and professional programmers”

  • The paper relies on this without support.

    Modern LLMs perform fairly well in various languages but best in English.

    “modern LLMs perform fairly well in various languages but best in English”

  • The paper relies on this without support.

    Large language models are already a part of programming education today and will be in the future.

    “they will be a part of programming education in the futurethey are already a part of it today”
8 of 8

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Unless explicitly instructed otherwise, GPT-3.5 has a tendency to be verbose and confident.

    “Unless explicitly instructed otherwise, GPT-3.5 has a tendency to be verbose [Kabir et al. 2024] and confident [Hellas et al. 2023].”

    From How Do Programming Students Use Generative AI? (2025) · cites this paper as Hellas et al. 2023

  • LLMs tended to hallucinate issues in student code, which could cause students to focus on non-existent issues instead of actual issues in their code.

    “Hellas et al. [94] found that LLMs tended to hallucinate issues in student code which could cause students to focus on these non-existent issues instead of the actual issues in their code.”

    From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [94]

  • AI-generated content is not curated for specific courses, so generated learning material could include syntax, programming constructs, or other content inappropriate for students in a given course.

    “AI generated content is not curated for specific courses, so learning material generated could potentially include syntax, programming constructs, or other content that is inappropriate for students in a given course [33,94].”

    From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [94]

  • LLMs can generate good explanations of code.

    “LLMs can also generate good explanations of code [94,123,137,174].”

    From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [94]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Average issues per help request is misstated

    Minor

    Dividing 275 issues by 150 help requests gives about 1.83, which rounds to 1.8 rather than 1.9 issues per request.

    Reported 1.9 · recomputed 1.833333

  • Number of very incomplete submissions changes from eight to six

    Minor

    The issue classification reports eight requests with very incomplete solutions, but the later discussion says there were six requests in that classification.

2 of 2

Cites

Flagged first, then by citations

Research works printed but not linked: 23

  1. Kirsti  M Ala-Mutka . 2005. A survey of automated assessment approaches for programming assignments. Computer science education 15, 2 ( 2005 ), 83–102. Kirsti M Ala-Mutka. 2005. A survey of automated assessment approaches for programming assignments. Computer science education 15, 2 (2005), 83–102.
  2. Tom Brown , Benjamin Mann , Nick Ryder , Melanie Subbiah , Jared  D Kaplan , Prafulla Dhariwal , Arvind Neelakantan , Pranav Shyam , Girish Sastry , Amanda Askell , 2020. Language models are few-shot learners. Advances in neural information processing systems 33 ( 2020 ), 1877–1901. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  3. Ruth Butler . 1998. Determinants of help seeking: Relations between perceived reasons for classroom help-avoidance and help-seeking behaviors in an experimental context.J. of Educ. Psychology 90, 4 ( 1998 ). Ruth Butler. 1998. Determinants of help seeking: Relations between perceived reasons for classroom help-avoidance and help-seeking behaviors in an experimental context.J. of Educ. Psychology 90, 4 (1998).
  4. James Finnie-Ansley , Paul Denny , Brett  A. Becker , Andrew Luxton-Reilly , and James Prather . 2022 . The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conf.ACM, 10–19 . James Finnie-Ansley, Paul Denny, Brett A. Becker, Andrew Luxton-Reilly, and James Prather. 2022. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. In Australasian Computing Education Conf.ACM, 10–19.
  5. Sepp Hochreiter and Jürgen Schmidhuber . 1997. Long short-term memory. Neural computation 9, 8 ( 1997 ), 1735–1780. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780.
  6. Ziwei Ji , Nayeon Lee , Rita Frieske , Tiezheng Yu , Dan Su , Yan Xu , Etsuko Ishii , Yejin Bang , Andrea Madotto , and Pascale Fung . 2022. Survey of hallucination in natural language generation. Comput. Surveys ( 2022 ). Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022. Survey of hallucination in natural language generation. Comput. Surveys (2022).
  7. W.  Lewis Johnson , Elliot Soloway , Benjamin Cutler , and Steven Draper . 1983 . Bug Catalogue: I. Technical Report . Yale University , YaleU/CSD/ RR #286. W. Lewis Johnson, Elliot Soloway, Benjamin Cutler, and Steven Draper. 1983. Bug Catalogue: I. Technical Report. Yale University, YaleU/CSD/RR #286.
  8. Slava Kalyuga . 2007. Expertise reversal effect and its implications for learner-tailored instruction. Educational psychology review 19 ( 2007 ), 509–539. Slava Kalyuga. 2007. Expertise reversal effect and its implications for learner-tailored instruction. Educational psychology review 19 (2007), 509–539.
  9. Stuart  A Karabenick . 2004. Perceived achievement goal structure and college student help seeking.J of educational psychology 96, 3 ( 2004 ). Stuart A Karabenick. 2004. Perceived achievement goal structure and college student help seeking.J of educational psychology 96, 3 (2004).
  10. Majeed Kazemitabaar , Justin Chow , Carl Ka To Ma , Barbara J Ericson, David Weintrop, and Tovi Grossman. 2023 . Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming . arXiv preprint arXiv:2302.07427 (2023). Majeed Kazemitabaar, Justin Chow, Carl Ka To Ma, Barbara J Ericson, David Weintrop, and Tovi Grossman. 2023. Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming. arXiv preprint arXiv:2302.07427 (2023). · arXiv
  11. Teemu Koivisto and Arto Hellas . 2022 . Evaluating CodeClusters for Effectively Providing Feedback on Code Submissions. In 2022 IEEE Frontiers in Education Conference (FIE). IEEE, 1–9. Teemu Koivisto and Arto Hellas. 2022. Evaluating CodeClusters for Effectively Providing Feedback on Code Submissions. In 2022 IEEE Frontiers in Education Conference (FIE). IEEE, 1–9.
  12. Juho Leinonen , Paul Denny , Stephen MacNeil , Sami Sarsa , Seth Bernstein , Joanne Kim , Andrew Tran , and Arto Hellas . 2023. Comparing Code Explanations Created by Students and Large Language Models. arXiv preprint arXiv:2304.03938 ( 2023 ). Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing Code Explanations Created by Students and Large Language Models. arXiv preprint arXiv:2304.03938 (2023). · arXiv
  13. Leo Leppänen , Arto Hellas , and Juho Leinonen . 2022 . Piloting Natural Language Generation for Personalized Progress Feedback. In 2022 IEEE Frontiers in Education Conference (FIE). IEEE, 1–8. Leo Leppänen, Arto Hellas, and Juho Leinonen. 2022. Piloting Natural Language Generation for Personalized Progress Feedback. In 2022 IEEE Frontiers in Education Conference (FIE). IEEE, 1–8.
  14. Ye Mao . 2019 . One minute is enough: Early prediction of student success and event-level difficulty during novice programming tasks . In In: Proceedings of the 12th International Conference on Educational Data Mining (EDM 2019). Ye Mao. 2019. One minute is enough: Early prediction of student success and event-level difficulty during novice programming tasks. In In: Proceedings of the 12th International Conference on Educational Data Mining (EDM 2019).
  15. Moin Nadeem , Anna Bethke , and Siva Reddy . 2020. StereoSet: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456 ( 2020 ). Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. StereoSet: Measuring stereotypical bias in pretrained language models. arXiv preprint arXiv:2004.09456 (2020). · arXiv
  16. Daniel  FO Onah , Jane Sinclair , and Russell Boyatt . 2014 . Dropout rates of massive open online courses: behavioural patterns . EDULEARN14 proceedings (2014), 5825–5834. Daniel FO Onah, Jane Sinclair, and Russell Boyatt. 2014. Dropout rates of massive open online courses: behavioural patterns. EDULEARN14 proceedings (2014), 5825–5834.
  17. José Carlos Paiva , José Paulo Leal , and Álvaro Figueira . 2022. Automated Assessment in Computer Science Education: A State-of-the-Art Review. ACM Transactions on Computing Education (TOCE) ( 2022 ). José Carlos Paiva, José Paulo Leal, and Álvaro Figueira. 2022. Automated Assessment in Computer Science Education: A State-of-the-Art Review. ACM Transactions on Computing Education (TOCE) (2022).
  18. Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , Ilya Sutskever , 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 ( 2019 ), 9. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
  19. Allison  M Ryan , Margaret  H Gheen , and Carol Midgley . 1998. Why do some students avoid asking for help? An examination of the interplay among students ’ academic efficacy, teachers’ social–emotional role, and the classroom goal structure.J. of educational psychology 90, 3 ( 1998 ). Allison M Ryan, Margaret H Gheen, and Carol Midgley. 1998. Why do some students avoid asking for help? An examination of the interplay among students’ academic efficacy, teachers’ social–emotional role, and the classroom goal structure.J. of educational psychology 90, 3 (1998).
  20. Allison  M Ryan , Paul  R Pintrich , and Carol Midgley . 2001. Avoiding seeking help in the classroom: Who and why?Educational Psychology Review 13, 2 ( 2001 ), 93–114. Allison M Ryan, Paul R Pintrich, and Carol Midgley. 2001. Avoiding seeking help in the classroom: Who and why?Educational Psychology Review 13, 2 (2001), 93–114.
  21. Sami Sarsa , Jesper Pettersson , and Arto Hellas . 2022 . How to Help to Ask for Help? Help Request Prompt Structure Influence on Help Request Quantity and Course Retention. In 2022 IEEE Frontiers in Education Conference (FIE). IEEE, 1–9. Sami Sarsa, Jesper Pettersson, and Arto Hellas. 2022. How to Help to Ask for Help? Help Request Prompt Structure Influence on Help Request Quantity and Course Retention. In 2022 IEEE Frontiers in Education Conference (FIE). IEEE, 1–9.
  22. Elliot Soloway , Kate Ehrlich , Jeffrey  G. Bonar , and Judith Greenspan . 1982. What do novices know about programming? In Directions in Human–Computer Interactions, Albert Badre and Ben Shneiderman (Eds.). Vol. 6 . Ablex Publishing , 27–54. Elliot Soloway, Kate Ehrlich, Jeffrey G. Bonar, and Judith Greenspan. 1982. What do novices know about programming? In Directions in Human–Computer Interactions, Albert Badre and Ben Shneiderman (Eds.). Vol. 6. Ablex Publishing, 27–54.
  23. Ashish Vaswani , Noam Shazeer , Niki Parmar , Jakob Uszkoreit , Llion Jones , Aidan  N Gomez , Łukasz Kaiser , and Illia Polosukhin . 2017. Attention is all you need. Advances in neural information processing systems 30 ( 2017 ). Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).

Cited by

Sorted by citations

Authors
Hellas, A., Leinonen, J., Sarsa, S., Koutcheme, C., Kujanpää, L., & Sorva, J.
Affiliations
  • Aalto University (FI)
  • University of Auckland (NZ)
ORCID
Venue
Proceedings of the 2023 ACM Conference on International Computing Education Research V.1
Publisher
ACM
Published
2023-08-07
Pages
93-105
DOI
10.1145/3568813.3600139
OpenAlex
W4380353816
Semantic Scholar
87bd28003ca39c7ce5a2d7ca6a03446b2e482a4f
Cited by
145
References
65
Citations by year
2026: 28 · 2025: 57 · 2024: 50 · 2023: 10
FWCI
43.85
Type
conference paper
Language
en
Open access
Licence
Locations
  • ACM Conference on International Computing Education Research (ICER) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • Aaltodoc (Aalto University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
Ulla Tuominen Foundation
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
large language models, novice programmers, help requests, programming education, AI-assisted programming

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review