CoChat
Sign Up
Cited by 39Teaching and Learning ProgrammingArtificial Intelligence in EducationSoftware Engineering Research

Integrating Generative AI into Programming Education: Student Perceptions and the Challenge of Correcting AI Errors

Open access
Kohen-Vacs, Dan, Usher, Maya, Jansen, Marc·2025·International Journal of Artificial Intelligence in Education
DOI 10.1007/s40593-025-00496-4
Open this paper in CoChat — free accountOpen PDF

Abstract

This paper presents two complementary quantitative studies examining the integration of generative AI (GenAI) tools into programming courses in higher education. Study 1 investigated undergraduate students’ perceptions of GenAI tools, focusing on usefulness in coursework, creativity enhancement, behavioral intention to use, and concerns and critiques. Study 2 evaluated undergraduate students’ performance in correcting code generated by large language models (LLMs) during programming exams, comparing this performance to their results on instructor-designed programming tasks. Findings from Study 1 revealed generally favorable student perceptions of GenAI. Participants reported medium-to-high levels of perceived usefulness, particularly emphasizing GenAI’s potential to enhance learning efficiency and support creative problem-solving. Students also expressed strong intentions to continue using GenAI tools in their studies, while reported concerns were relatively low and centered primarily on the risk of over-reliance. Findings from Study 2 showed that students encountered significantly greater difficulty when correcting LLM-generated code compared to traditional exam tasks, highlighting the unique challenges posed by AI-generated outputs in assessment contexts. Taken together, the results suggest that while students recognize the value of GenAI tools in supporting learning and creative exploration, programming education must also focus on developing students’ skills in critically evaluating and correcting AI-generated content to ensure effective and responsible integration.

AI snapshot

Written from the paper’s full text.

TL;DR

Two quantitative studies found undergraduates generally viewed generative AI favorably for programming learning, especially for efficiency and creativity, but students performed significantly worse when correcting LLM-generated code than on instructor-designed exam tasks. The authors argue programming education must scaffold critical evaluation and correction of AI-generated content.

Key findings

  • Study 1 (n = 64, Israel): students reported favorable perceptions of GenAI. Usefulness in coursework averaged 3.63 (SD = 0.87), significantly above the neutral midpoint, t(63) = 5.76, p < .001, d = 0.87; creativity enhancement averaged 3.66 (SD = 0.98), t(63) = 5.38, p < .001, d = 0.98; behavioral intention to use was highest at 3.92 (SD = 1.07), t(63) = 6.90, p < .001, d = 1.07.
  • Concerns and critiques averaged 2.98 (SD = 1.00) and did not differ significantly from the neutral midpoint. The highest-rated concern was over-reliance on technology for programming assignments, M = 3.66, SD = 1.30.
  • Study 2 (n = 147, Germany): students scored significantly lower on LLM-generated debugging tasks than on instructor-designed tasks in Summer 2023 and Winter 2023/2024. For example, Operations Research Summer 2023: LLM M = 0.57 (SD = 0.70) vs instructor-designed M = 7.00 (SD = 6.48), t(23) = 2.61, p = .01; Cryptography Summer 2023: LLM M = 7.52 (SD = 3.15) vs instructor-designed M = 9.27 (SD = 11.22), t(16) = 2.83, p = .01.
  • In Winter 2022/2023, no significant differences appeared between task types; the paper notes overall performance was lower across all components, with passing rates of 20% in Operations Research (5/25) and 29.41% in Cryptography (5/17).

Methods

  • Two complementary quantitative studies. Study 1 used an anonymous online questionnaire distributed in March 2024 to 64 undergraduate students in Israel (17% male, 83% female). The questionnaire was informed by the Technology Acceptance Model and included 16 Likert-scale statements across four dimensions: usefulness in coursework, creativity enhancement, behavioral intention to use, and concerns/critiques. Reliability was high, Cronbach’s α = 0.83–0.91. Analysis used descriptive statistics and one-sample t-tests against the neutral midpoint of 3.
  • Study 2 analyzed exam performance from 147 undergraduate students in Germany (81% male, 19% female) enrolled in Operations Research and Cryptography. Six examinations across Winter 2022/2023, Summer 2023, and Winter 2023/2024 included LLM-generated Java debugging tasks with embedded semantic and syntactic errors, alongside instructor-designed programming and theoretical tasks. Scores were normalized to task weight and compared using paired-samples t-tests.

Limitations

  • The paper states that findings are based on two distinct undergraduate populations from different countries, limiting generalizability across cultural, institutional, and curricular contexts.
  • The paper states that Study 2 compared LLM-generated debugging tasks with instructor-designed programming tasks involving code construction, which represent distinct cognitive demands and may have influenced observed performance differences.
  • The paper states that the research was short-term, did not assess long-term skill development or professional readiness, and did not systematically investigate pedagogical strategies for integrating GenAI.
  • Evident from the text: Study 1 had a gender imbalance (83% female) and Study 2 had a different imbalance (81% male); some per-course/term subgroups were small, such as Summer 2023 Operations Research (n = 7) and Winter 2023/2024 Cryptography (n = 9). The discussion also notes that Study 2’s LLM debugging task was embedded in a high-stakes summative exam without prior instruction or training, which may have contributed to the performance gap.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s40593-025-00496-4

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

7 tests recomputed · 0 with the reported and recomputed p on opposite sides of .05

Claims the paper makes

19 of 52 cited sources checked · 2 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Explicit instruction on how to critically assess, refine, and debug AI-generated code fosters the development of essential competencies ranging from algorithmic reasoning to software testing.

    “Explicit instruction on how to critically assess, refine, and debug AI-generated code fosters the development of essential competencies -ranging from algorithmic reasoning to software testing (Denny et al., 2024;Kendon et al., 2023).”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Denny et al., 2024 source

      Section 5.2.2 reports students’ perceptions that writing prompts develops analytical and problem-solving skills, and Section 6 says Prompt Problems may encourage a critical eye toward generated code. Section 4.1 describes students refining prompts after automated tests fail. The paper does not establish that explicit instruction in debugging AI-generated code develops students’ software-testing competence; Section 4 says the generated code is not editable in the current exercise. (LLM, conf=0.85)

      Experimental — accuracy not yet measured
    • cites Kendon et al., 2023 source

      Section 3.2 proposes having students assess AI-generated code, while Sections 3.1.3, 3.1.7, and 3.1.8 describe finding errors, creating tests, and fixing bugs in code. These support the proposed activities, but the paper does not establish that explicit instruction in refining and debugging AI-generated code develops the claimed range of competencies, particularly algorithmic reasoning. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
  • Premisecitation

    Excessive dependence on GenAI tools may undermine the development of algorithmic thinking, error detection, and analytical reasoning.

    “While research has demonstrated that GenAI can support learning and foster creativity, excessive dependence on these tools may undermine the development of algorithmic thinking, error detection, and analytical reasoning (Kazemitabaar et al., 2023;Mogavi et al., 2024;Zviel-Girshin, 2024).”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Kazemitabaar et al., 2023 source

      Section 6.2.2 raises the possibility that over-utilization of AI code generators might impede learning, and the Introduction warns that learners may become dependent on them. But Section 7 calls their effect on algorithmic thinking an unanswered question; the paper does not establish that dependence undermines error detection or analytical reasoning. Its Discussion reports no reduction in the learning outcomes it measured. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites Mogavi et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Zviel-Girshin, 2024 source

      Section 4.3 (‘The Bad’; page not provided) says excessive reliance can weaken independent problem-solving and understanding of programming principles, leaving students more vulnerable to algorithmic errors. Section 3.5 reports unsuccessful bug identification and fixing, but the paper does not establish that dependence undermines the development of error-detection skills specifically. Section 6 notes that critical-thinking skills were not directly measured. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
  • Premisecitation

    As LLMs increasingly automate routine and syntactic coding tasks, educators face the challenge of preserving essential programming competencies such as algorithmic thinking, debugging, and independent problem-solving.

    “As LLMs increasingly automate routine and syntactic coding tasks, educators face the challenge of preserving essential programming competencies, such as algorithmic thinking, debugging, and independent problem-solving (Valový & Buchalcevova, 2023;Bull & Kharrufa, 2024;French et al., 2023).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Valový & Buchalcevova, 2023 source
    • cites Bull & Kharrufa, 2024 source
    • cites French et al., 2023 source

      The Introduction (p. 2) says the authors used OpenAI with games-programming students to develop technical and problem-solving skills, and Final Comments (p. 19) warns that reliance on AI might erode higher-order cognitive skills. The paper also describes AI-generated code (Example 1, p. 9), but it does not specifically characterize the automated work as routine or syntactic, or identify algorithmic thinking and independent debugging as competencies educators must preserve. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
  • Premisecitation

    When students lean too heavily on AI-generated code, their ability to develop core competencies such as logical reasoning, collaboration, and debugging may be compromised.

    “When students lean too heavily on AI-generated code, their ability to develop core competencies -such as logical reasoning, collaboration, and debugging -may be compromised (Groothuijsen et al., 2024;Manorat et al., 2025;Tedre et al., 2021).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Groothuijsen et al., 2024 source
    • cites Manorat et al., 2025 source
    • cites Tedre et al., 2021 source

      The paper’s “Pitfalls and Weaknesses” section warns that low-floor machine-learning tools can produce shallow learning because children learn workflows rather than underlying mechanisms. That supports a general concern about learning through opaque AI tools, but the paper does not address reliance on AI-generated code or say that it compromises logical reasoning, collaboration, or debugging. (LLM, conf=0.76)

      Experimental — accuracy not yet measured
  • Premisecitation

    By automating routine or repetitive coding tasks, AI allows students to allocate more cognitive effort toward higher-order activities such as design, optimization, and conceptual exploration.

    “For instance, by automating routine or repetitive coding tasks, AI allows students to allocate more cognitive effort toward higher-order activities such as design, optimization, and conceptual exploration (French et al., 2023;Marrone et al., 2022).”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites French et al., 2023 source

      The Discussion (p. 17) suggests that generative AI could speed game development and create more opportunities for programmers to be creative. It does not establish that automating routine coding tasks frees students’ cognitive effort for design, optimization, or conceptual exploration; the paper presents the benefit as a possibility, not an observed effect. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Marrone et al., 2022 source

      The Results and Discussion (§3.4, page not available) report that students felt AI could encourage independent thinking and create opportunities to be creative. The paper does not say that AI automates routine coding tasks or that such automation frees cognitive effort for design, optimization, or conceptual exploration. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
  • May combine two claims. 18
  • Classified as probably not a substantial claim. 17
  • The quote only partly states this claim. 1

  • May combine two claims.

    Premisecitation

    Tools such as GitHub Copilot can accelerate coding by generating syntactically correct suggestions but are prone to introducing vulnerabilities, inefficiencies, and logic errors, especially in complex or unfamiliar contexts.

    “In programming education, tools such as GitHub Copilot can accelerate coding by generating syntactically correct suggestions; however, they are also prone to introducing vulnerabilities, inefficiencies, and logic errors -especially in complex or unfamiliar contexts (Hilario et al., 2024;Zhang et al., 2023).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Hilario et al., 2024 source

      Section 2.3.1 (page unavailable) says generative AI can save time in penetration testing, and Section 2.4.1 (page unavailable) warns that AI results can contain false positives or negatives. Those points support the broad efficiency-and-error framing, but the paper does not establish the claim about GitHub Copilot’s coding suggestions, syntactic correctness, or their propensity to introduce vulnerabilities, inefficiencies, and logic errors in complex or unfamiliar programming contexts. (LLM, conf=0.78)

      Experimental — accuracy not yet measured
    • cites Zhang et al., 2023 source

  • May combine two claims.

    Premisecitation

    The inconsistency and occasional inaccuracy of AI outputs, particularly in conceptually complex or creative tasks, highlight the need for human oversight and strong foundational knowledge.

    “Moreover, the inconsistency and occasional inaccuracy of AI outputs, particularly in conceptually complex or creative tasks, highlight the need for human oversight and strong foundational knowledge (Holmes et al., 2022;Usher, 2025).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Holmes et al., 2022 source

      The Discussion (page not available) warns that incorrect judgments by AIED systems could harm learners, and Most Ignored Yet Critical Ethical Issues for AIED (page not available) cautions against replacing human teachers with autonomous systems. These passages support concern about errors and a continuing human role, but they do not address inconsistent outputs in conceptually complex or creative tasks or a need for strong foundational knowledge. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Usher, 2025 source

  • May combine two claims.

    Premisecitation

    Such tools may produce insecure or suboptimal code, requiring users to review and revise outputs carefully to avoid compounding mistakes.

    “Research has shown that such tools may produce insecure or suboptimal code, requiring users to review and revise outputs carefully to avoid compounding mistakes (Groothuijsen et al., 2024;Laux, 2024;Nguyen & Nadi, 2022).”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Groothuijsen et al., 2024 source
    • cites Laux, 2024 source

      The Introduction and “Two challenges for human oversight” (page unavailable) discuss the need to assess AI-supported decisions and the risk of over-relying on AI outputs. They do not address code-generation tools, insecure or suboptimal code, or revising code to prevent compounding mistakes. The paper supports only the general need for careful oversight. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Nguyen & Nadi, 2022 source

  • May combine two claims.

    Premisecitation

    LLMs can automate routine programming tasks, but their limitations become apparent when addressing nuanced or ill-structured problems.

    “Although LLMs can automate routine programming tasks, their limitations become apparent when addressing nuanced or ill-structured problems (Zhang et al., 2023;Denny et al. 2024).”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Zhang et al., 2023 source
    • cites Denny et al. 2024 source

      The Introduction says LLMs can generate code for typical introductory programming problems, and Related Work says they can struggle with longer, higher-level specifications. These passages support the contrast between routine tasks and more demanding ones, but the paper does not specifically establish a limitation for ill-structured problems. The supplied passages have section labels but no page numbers. (LLM, conf=0.82)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation

    The questionnaire design was informed by the Technology Acceptance Model (TAM) and recent studies on GenAI adoption in higher education.

    “The questionnaire design was informed by the Technology Acceptance Model (TAM; Venkatesh & Davis, 2000) and recent studies on GenAI adoption in higher education (e.g., Faraon et al., 2025).”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Venkatesh & Davis, 2000 source
    • cites Faraon et al., 2025 source

  • May combine two claims.

    Premisecitation

    GenAI tools hold substantial potential to enhance creativity and accelerate learning when used as a complement to traditional instruction.

    “At the same time, GenAI tools hold substantial potential to enhance creativity and accelerate learning when used as a complement to traditional instruction.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites French et al., 2023 source

      The Discussion (p. 16) says students used OpenAI to support their coursework and demonstrated creative and problem-solving skills; Limitations and Implications (p. 18) describes incorporating AI tools into teaching. These passages support the potential for AI to complement instruction and foster creativity, but they do not establish that it accelerates learning. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites Marrone et al., 2022 source

      The Results and Discussion (§3.4; page not provided) say students thought AI could assist creativity during learning, and the Conclusions (§4; page not provided) report that most thought AI could help develop their creativity. The paper does not establish that generative AI accelerates learning or that this benefit depends on its complementing traditional instruction. (LLM, conf=0.88)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Students' attitudes toward GenAI tools are shaped by their level of familiarity and the nature of their experiences: positive engagement tends to foster adoption, while limited exposure or difficulty using the tools may lead to caution or skepticism.

    “Previous research has shown that students' attitudes toward GenAI tools are shaped by their level of familiarity and the nature of their experiences: positive engagement tends to foster adoption, while limited exposure or difficulty using the tools may lead to caution or skepticism (Zhang et al., 2023;Ayanwale et al., 2022;Wang et al., 2021).”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Zhang et al., 2023 source
    • cites Ayanwale et al., 2022 source

      Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

      Experimental — accuracy not yet measured
    • cites Wang et al., 2021

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation

    Integrating AI tools alongside conventional instructional methods enables students to build a comprehensive skill set while developing the capacity to critique and refine automated outputs.

    “Research suggests that integrating AI tools alongside conventional instructional methods enables students to build a comprehensive skill set while developing the capacity to critique and refine automated outputs (Kazemitabaar et al., 2023).”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Kazemitabaar et al., 2023 source

      The AI-Assisted Learning Environment section describes AI code generation alongside programming tasks, documentation, and instructor feedback. The Discussion and Conclusion report better code-authoring performance without reduced manual code-modification performance. They do not establish that this combination builds a comprehensive skill set or develops students’ capacity to critique and refine AI-generated outputs. (LLM, conf=0.86)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    These mixed views reflect broader debates in the literature on the dual nature of AI in education.

    “These mixed views reflect broader debates in the literature on the dual nature of AI in education -simultaneously enhancing productivity and creativity while raising concerns about over-dependence and diminished cognitive engagement (Mogavi et al., 2024;Usher, 2025;Zviel-Girshin, 2024).”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Mogavi et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Usher, 2025 source
    • cites Zviel-Girshin, 2024 source

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    In the Winter Term 2022/2023 Operations Research course, students' scores on LLM tasks (M=6.96, SD=3.28) were comparable to their scores on instructor-designed tasks (M=6.98, SD=7.15), with no significant difference.

    “In the Operations Research course, students' scores on LLM tasks (M = 6.96,SD = 3.28) were comparable to their scores on instructor-designed tasks (M = 6.98,SD = 7.15).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Some participants acknowledged that frequent reliance on GenAI could impede the development of essential skills such as independent problem-solving and critical thinking.

    “Nonetheless, some participants acknowledged that frequent reliance on GenAI could impede the development of essential skills such as independent problem-solving and critical thinking.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Students reported moderate-to-high levels of perceived usefulness of GenAI, particularly valuing its ability to enhance learning efficiency and support creative thinking during programming tasks.

    “Students reported moderate-to-high levels of perceived usefulness, particularly valuing GenAI's ability to enhance learning efficiency and support creative thinking during programming tasks.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    All students in both studies provided informed consent prior to participation.

    “All students in both studies provided informed consent prior to participation.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Encouraging students to engage with errors, explore alternative solutions, and refine their code through hands-on troubleshooting strengthens their understanding of how and why code functions as intended.

    “Encouraging students to engage with errors, explore alternative solutions, and refine their code through hands-on troubleshooting strengthens their understanding of how and why code functions as intended (Zhang et al., 2023).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Zhang et al., 2023 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    The structured grading approach ensured that students' performance was evaluated comprehensively across diverse tasks.

    “This structured grading approach ensured that students' performance was evaluated comprehensively across diverse tasks, reflecting their competencies in both traditional programming and AI-augmented contexts.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    The same study reported concerns from instructors about decreased code quality and reduced student collaboration, particularly in pair programming.

    “While these tools promoted efficiency and understanding, the same study reported concerns from instructors about decreased code quality and reduced student collaboration, particularly in pair programming.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Groothuijsen et al. (2024) source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The research received approval from the respective faculties' ethics committees.

    “The research received approval from the respective faculties' ethics committees, ensuring compliance with institutional ethical standards.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Two distinct patterns emerged: students either asked ChatGPT to guide their own debugging process or used it to identify errors directly.

    “Two distinct patterns emerged: students either asked ChatGPT to guide their own debugging process or used it to identify errors directly.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Groothuijsen et al. (2024) source

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Programming scores were normalized to reflect each task's relative weight within the overall exam to ensure comparability.

    “To ensure comparability, programming scores were normalized to reflect each task's relative weight within the overall exam.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

    Interpretationreasoning

    The cultural, institutional, and curricular differences inherent in these contexts may not fully represent the diversity of educational environments globally.

    “The cultural, institutional, and curricular differences inherent in these contexts may not fully represent the diversity of educational environments globally.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationreasoning

    When thoughtfully implemented, GenAI tools have the potential to enhance programming education by complementing traditional instruction and preparing students for success in increasingly AI-augmented learning environments.

    “When thoughtfully implemented, GenAI tools have the potential to enhance programming education by complementing traditional instruction and preparing students for success in increasingly AI-augmented learning environments.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductreasoning

    In Study 2, student performance was evaluated on an LLM-generated debugging task and compared with instructor-designed programming tasks involving code construction; these represent distinct cognitive demands (debugging versus algorithmic development), which may have influenced the observed differences in performance.

    “Another methodological limitation lies in the task comparison in Study 2. Student performance was evaluated on an LLM-generated debugging task and compared with instructor-designed programming tasks involving code construction.These represent distinct cognitive demands (debugging versus algorithmic development), which may have influenced the observed differences in performance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The findings are based on two distinct undergraduate student populations from different countries, which may limit the generalizability of the results.

    “First, the findings are based on two distinct undergraduate student populations from different countries, which may limit the generalizability of the results.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation

    Prior studies on GenAI integration in higher education have underscored the importance of scaffolding, transparent assessment criteria, and guided practice, demonstrating that the way GenAI tools are introduced critically shapes students' abilities to use them effectively.

    “Prior studies on GenAI integration in higher education have underscored the importance of scaffolding, transparent assessment criteria, and guided practice, demonstrating that the way GenAI tools are introduced critically shapes students' abilities to use them effectively (Garg et al., 2025;Usher, 2025).”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites Garg et al., 2025 source
    • cites Usher, 2025 source

  • May combine two claims.

    Interpretationreasoning

    The pattern of students underperforming on LLM-generated debugging tasks was consistent across two academic terms, suggesting the task posed greater cognitive demands.

    “This pattern was consistent across two academic terms, suggesting that the LLM-generated debugging task posed greater cognitive demands.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    This study did not systematically investigate the pedagogical strategies used to integrate GenAI tools.

    “Additionally, this study did not systematically investigate the pedagogical strategies used to integrate GenAI tools.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Overall student performance during Winter 2022/2023 was lower across all exam components, which may have masked potential differences between task types.

    “A closer examination suggests that overall student performance during Winter 2022/2023 was lower across all exam components, which may have masked potential differences between task types.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The short-term scope of the research focused on students' immediate perceptions and performance; long-term effects such as sustained skill development, adaptability to AI-augmented learning environments, and professional readiness were not assessed.

    “A third limitation lies in the short-term scope of the research, which focused on students' immediate perceptions and performance.Long-term effects, such as sustained skill development, adaptability to AI-augmented learning environments, and professional readiness, were not assessed.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationcitation

    Gradual, scaffolded integration strategies are key to maximizing the benefits of GenAI while preserving and strengthening essential programming and problem-solving skills.

    “Applying these insights to programming education suggests that gradual, scaffolded integration strategies are key to maximizing the benefits of GenAI while preserving and strengthening essential programming and problem-solving skills.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    The benefits of GenAI are maximized when students are taught how to use these tools critically and strategically.

    “Their findings suggest that the benefits of GenAI are maximized when students are taught how to use these tools critically and strategically.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Garg et al., 2025 source

  • May combine two claims.

    Interpretationreasoning

    The setting of Study 2 (a high-stakes summative exam without prior instruction or training) placed greater cognitive demands on students and offered no opportunity for guided practice compared to exploratory or formative use cases.

    “In contrast, our second study examined students' performance on a debugging task involving LLM-generated code, embedded in a high-stakes sum-mative exam, without prior instruction or training.Unlike exploratory or formative use cases, this setting placed greater cognitive demands on students and offered no opportunity for guided practice.”
    Quote in the paperSubstantial claim
31 of 31

  • The paper relies on this without support.

    Debugging plays a central role in developing computational thinking and technical resilience.

    “Debugging, in particular, plays a central role in developing computational thinking and technical resilience.”

  • The paper relies on this without support.

    The educational effectiveness of GenAI tools depends significantly on how they are implemented and supported.

    “However, their educational effectiveness depends significantly on how they are implemented and supported.”

  • The paper relies on this without support.

    The LLM-generated debugging tasks and instructor-designed tasks are comparable in difficulty except for the AI-generated aspect, so performance differences reflect the challenge of correcting AI errors.

    “Students' performance on the LLM-based tasks was compared with their performance on traditional programming and theoretical components within the same exams.”

  • The paper relies on this without support.

    The semantic and syntactic errors intentionally embedded in the LLM-generated code are representative of typical LLM errors.

    “Several semantic and syntactic errors were intentionally embedded in the code.”

  • The paper relies on this without support.

    Students' performance on the LLM-generated debugging task reflects their ability to correct AI errors rather than other factors such as task format or time constraints.

    “Students were instructed to analyze the class, identify the errors (e.g., incorrect variable assignments, improper use of mathematical operations), and correct them to produce a functional RSA implementation (see Fig. 1).”
5 of 5

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

  • p-value fits an unstated one-tailed test: t(23) = 2.61, p = .01

    Minor

    The paper reports p = .01 here; recomputing from the test statistic and its degrees of freedom gives p = .016 (two-tailed). The reported value is what a one-tailed test gives, but the paper does not say near this result that a one-tailed test was used.

    Reported p = .01 · recomputed p = .016

1 of 1

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upt(23) = 2.61reported p = .01 · recomputed p = .016

    “t(23) = 2.61, p = .01”

  • Consistentt(63) = 5.76reported p < .001 · recomputed p < .001

    “t(63) = 5.76, p < .001”

  • Consistentt(63) = 5.38reported p < .001 · recomputed p < .001

    “t(63) = 5.38, p < .001”

  • Consistentt(63) = 6.9reported p < .001 · recomputed p < .001

    “t(63) = 6.90, p < .001”

  • Consistentt(16) = 2.83reported p = .01 · recomputed p = .012

    “t(16) = 2.83, p = .01”

  • Consistentt(42) = 5.71reported p < .001 · recomputed p < .001

    “t(42) = 5.71, p < .001”

  • Consistentt(31) = 3.54reported p = .01 · recomputed p = .001

    “t(31) = 3.54, p = .01”

    The paper prints p at a floor (such as p < .001)

7 of 7

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Winter 2023/2024 Cryptography t-test uses the wrong sample size

    Major

    The reported t(31) implies 32 paired observations, whereas Table 1 lists 9 Cryptography participants for that term. Nine paired observations would give 8 degrees of freedom.

  • Summer 2023 Cryptography t-test uses the wrong sample size

    Major

    The reported t(16) implies 17 paired observations, whereas Table 1 lists 56 Cryptography participants for Summer 2023. Fifty-six paired observations would give 55 degrees of freedom.

  • Summer 2023 Operations Research t-test uses the wrong sample size

    Major

    The reported t(23) implies 24 paired observations, but Table 1 lists only 7 Operations Research participants for that term. Seven paired observations would give 6 degrees of freedom.

  • Winter 2023/2024 Operations Research t-test uses the wrong sample size

    Major

    The reported t(42) implies 43 paired observations, but Table 1 lists 33 Operations Research participants for that term. Thirty-three paired observations would give 32 degrees of freedom.

  • Winter 2022/2023 performance was not lower across all components

    Minor

    The claim of lower performance across all components conflicts with Table 6: the Operations Research LLM-task mean was 6.96 in Winter 2022/2023, compared with 0.57 in Summer 2023.

5 of 5

Cites

Flagged first, then by citations

Research works printed but not linked: 1

  1. Wang, 2021, Factors affecting the adoption of AI-Based applications in higher education, Educational Technology & Society, 24, 116

Cited by

Sorted by citations

Related

Authors
Kohen-Vacs, D., Usher, M., & Jansen, M.
Affiliations
Holon Institute of Technology (IL)
Corresponding author
Usher, Maya
ORCID
Venue
International Journal of Artificial Intelligence in Education
Publisher
Elsevier BV
Published
2025-12
Volume
35
Issue
5
Pages
3166-3184
ISSN
1560-4292 (ISSN-L)
DOI
10.1007/s40593-025-00496-4
OpenAlex
W4412441827
Cited by
39
References
41
Citations by year
2026: 33 · 2025: 6
FWCI
36.57
Type
journal article
Language
en
Open access
Licence
Locations
International Journal of Artificial Intelligence in Education (publishedVersion, cc-by)
Funders
Technion - Israel Institute of Technology
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
Keywords
generative AI, programming education, student perceptions, large language models, code correction, AI-generated code, perceived usefulness, behavioral intention, overreliance on AI, critical evaluation of AI outputs, undergraduate students

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-23.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-23.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review