CoChat
Sign Up
Cited by 366Teaching and Learning ProgrammingSoftware Engineering ResearchArtificial Intelligence in Education

Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming

Open access
Kazemitabaar, Majeed, Chow, Justin, Ma, Carl Ka To, Ericson, Barbara J., Weintrop, David, Grossman, Tovi·2023·Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3544548.3580919
Open this paper in CoChat — free accountOpen PDF

Abstract

AI code generators like OpenAI Codex have the potential to assist novice programmers by generating code from natural language descriptions, however, over-reliance might negatively impact learning and retention. To explore the implications that AI code generators have on introductory programming, we conducted a controlled experiment with 69 novices (ages 10-17). Learners worked on 45 Python code-authoring tasks, for which half of the learners had access to Codex, each followed by a code-modification task. Our results show that using Codex significantly increased code-authoring performance (1.15x increased completion rate and 1.8x higher scores) while not decreasing performance on manual code-modification tasks. Additionally, learners with access to Codex during the training phase performed slightly better on the evaluation post-tests conducted one week later, although this difference did not reach statistical significance. Of interest, learners with higher Scratch pre-test scores performed significantly better on retention post-tests, if they had prior access to Codex.

Integrity signals

Registry

Crossref · 10.1145/3544548.3580919

Cites

102 of 102 references verified0 citation errors · 0 not checkable

Statistics

31 tests recomputed · 0 with the reported and recomputed p on opposite sides of .05

Claims the paper makes

2 of 10 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Prior work in introductory programming has shown that code-reading and tracing skills are essential to problem-solving activities including code-writing.

    “Prior work in introductory programming has shown that code-reading and tracing skills are essential to problem-solving activities including code-writing [49,55,57].”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites [49] 49. Amruth N Kumar. 2013. A study of the infuence of code-tracing problems on code-writing skills. In Proceedings of the 18th ACM conference on Innovation and technology in computer science education. 183-188. source
    • cites [55] 55. Raymond Lister, Colin Fidge, and Donna Teague. 2009. Further evidence of a relationship between explaining, tracing and writing skills in introductory programming. Acm sigcse bulletin 41, 3 (2009), 161-165. source
    • cites [57] 57. Mike Lopez, Jacqueline Whalley, Phil Robbins, and Raymond Lister. 2008. Rela- tionships between reading, tracing and writing skills in introductory program- ming. In Proceedings of the fourth international workshop on computing education research. 101-112. source

      The Discussion (§6) reports significant associations between code-reading, code-tracing, and code-writing performance. But the paper presents a hierarchy of skills as a possibility, not a demonstrated necessity; its findings do not establish that reading and tracing are essential to problem-solving activities generally. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Findingown data

    Learners in the Codex group were able to successfully generate code during the training phase.

    “learners who had access to the AI code generator (the Codex group) were able to successfully generate code”
    • Also stated in: § 1 INTRODUCTION · plain
      “Our results show that learners who had access to the AI code generator (the Codex group) were able to successfully generate code”
    Quote in the paperSubstantial claim
  • Findingown data

    The Codex group had 1.8x higher correctness on code-authoring tasks.

    “1.8x higher correctness”
    Quote in the paperSubstantial claim
  • Findingown data

    The Codex group performed significantly better on code-authoring tasks, with 1.15x increased progress.

    “They performed signifcantly better on code-authoring tasks (1.15x increased progress”
    Quote in the paperSubstantial claim
  • Findingown data

    The Codex group had 0.57x less time on code-authoring tasks.

    “0.57x less time”
    Quote in the paperSubstantial claim
  • May combine two claims. 21
  • Classified as probably not a substantial claim. 20

  • May combine two claims.

    Interpretationcitation

    Step-by-step usage of the AI code generator could potentially reduce cognitive load and allow novices to learn faster due to more availability of mental resources, specifically for germane cognitive load.

    “This could potentially explain how step-by-step usage of the AI code generator could potentially reduce cognitive load and allow novices to learn faster due to more availability of their mental resources, specifcally for germane cognitive load which is responsible for creating mental models and storing long-term memory [46].”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [46] 46. Paul Kirschner, John Sweller, and Richard E Clark. 2006. Why unguided learning does not work: An analysis of the failure of discovery learning, problem-based learning, experiential learning and inquiry-based learning. Educational Psycholo- gist 41, 2 (2006), 75-86. source

      In “Current Research Supporting Direct Guidance” (page unavailable), the paper says worked examples reduce working-memory load and direct resources toward learning problem-solving schemas. That supports the proposed mechanism for guided, step-by-step learning, but the paper does not address AI code generators or identify the freed resources specifically as germane cognitive load. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    The Codex group scored on average 12.7% higher on modifying tasks in the retention post-test, but this did not reach statistical significance.

    “and 12.7% higher on modifying tasks (Codex: M=47.5%,SD=31.6%;Baseline: M=34.8%,SD=29.0%;t(67)=1.70,p=.092, d=0.41), but neither of these reached statistical signifcance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Codex group scored on average 16% higher on loops in multiple-choice questions in the immediate post-test, but this did not reach statistical significance.

    “and on average 16% higher on loops, but this did not reach statistical signifcance (Codex: M=50.2%,SD=35.7%;Baseline: M=33.8%,SD=27.7%;t(58)=2.08,p=.041, d=0.51).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Codex group encountered slightly higher errors on modifying tasks in the retention post-test, but this did not reach statistical significance.

    “Learners in the Codex group also encountered slightly higher errors on modifying tasks, but this did not reach statistical signifcance (Codex: M=0.81, SD=0.85;Baseline: M=0.57, SD=0.70; t(67)=1.21,p=.229, d=0.30).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Codex group scored 10.5% higher on loops in correctness on modifying tasks during the training phase, but this did not reach statistical significance.

    “and 10.5% higher on loops (Codex: M=61%, SD=30.5%;Baseline: M=50.5%,SD=29.3%;t(58)=1.41,p=.161, d=0.34) however, neither of these results reached statistical signifcance.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    64 participants indicated using a block-based programming environment like Scratch or Code.org.

    “64 indicated using a block-based programming environment like Scratch or Code.org”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Some learners felt that the AI-generated code needed some explanations.

    “Some learners felt that the AI-generated code needed some explanations”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Codex group scored on average 9.0% higher on authoring tasks in the retention post-test, but this did not reach statistical significance.

    “learners from the Codex group on average scored 9.0% higher on authoring tasks (Codex: M=59.1%,SD=34.8%;Baseline: M=49.8%,SD=32.4%;t(67)=1.13,p=.262, d=0.28)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    27 participants indicated taking a programming-related class in the past.

    “27 indicated taking a programming-related class in the past”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Semantic errors occurred infrequently in both groups on authoring tasks during the training phase.

    “However, semantic errors occurred infrequently in both groups (Codex: M=0.01, SD=0.02;Baseline: M=0.03, SD=0.05; t(67)=1.84,p=.071, d=0.44).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Learners in the Codex group felt that generating the code they needed did not require a lot of practice.

    “They also felt that generating the code that they needed did not require a lot of practice.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the retention test one week later, learners from the Codex group performed slightly better on coding tasks and multiple-choice questions, but these results did not reach statistical significance.

    “In the retention test which was conducted one week later, learners from the Codex group performed slightly better on coding tasks and multiple-choice questions, although these results did not reach statistical signifcance”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Neither of the differences on loops and arrays in multiple-choice questions in the retention post-test reached statistical significance.

    “although neither of these diferences reached statistical signifcance.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Learners from both groups fully completed the first two topics (basics and data-types) and one learner from each group did not complete conditionals.

    “learners from both groups fully completed the frst two topics (basics and data-types) and one learner from each group did not complete conditionals.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Codex group had more learners that fully completed the 18 tasks on loops (Codex: 27/33, Baseline: 23/36) and the 7 tasks on arrays (Codex: 23/33, Baseline: 9/36).

    “the Codex group had more learners that fully completed the 18 tasks on loops (Codex: 27/33, Baseline: 23/36) and the 7 tasks on arrays (Codex: 23/33, Baseline: 9/36)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Several metrics had differences between the two groups that did not reach statistical significance but there did seem to be some difference in means and confidence intervals.

    “When discussing our results, there were several metrics in which the diferences between the two groups did not reach statistical signifcance but there did seem to be some diference in means and confdence intervals.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Learners employed various methods to generate code, such as breaking down the task into subgoals and using the code generator to generate code for each subgoal step-by-step.

    “Learners employed various methods to generate code such as breaking down the task into subgoals and using the code generator to generate code for each of the subgoals step-by-step.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    The study included many non-native English speakers, but they were mostly comfortable explaining things in English.

    “Furthermore, although our study included many non-native English speakers, but they were mostly comfortable explaining things in English.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The two graders fully agreed on 79% of the submissions.

    “The two graders fully agreed on 79% of the submissions”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    For 16% of disagreements with a difference of 25%, the two grades were averaged.

    “with 16% of disagreements having a diference of 25% in which we averaged the two grades”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    No common factors were identified among the 21 students who dropped out in terms of disability, native language, and computer or internet access.

    “No common factors were identifed among the 21 students who dropped out in terms of disability, native language (six non-English) and computer or internet access.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    One learner in each pair was randomly assigned to either the Codex or Baseline group.

    “One learner in each pair was randomly assigned to either the Codex or Baseline group.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    Although the Codex group used documentation less and relied heavily on AI-generated code for authoring tasks, they still did just as well, and in some cases better, in manual code modification tasks.

    “But overall, this is a promising result: although learners in the Codex group used the documentation less and relied heavily on AI-generated code for the authoring tasks, they still did just as well, and in some cases better, in manual code modifcation tasks.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Two independent researchers graded each submitted solution using a simple and consistent grading rubric.

    “two independent researchers graded each submitted solution using a simple and consistent grading rubric of deducting 25% for each major issue or missing part in their fnal submission”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    The AI code generator produces high quality results, but this depends on the quality of the prompt message.

    “This shows that the AI code generator produces high quality results, but this depends on the quality of the prompt message.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    Participants were divided into two groups using a matched-groups design to have similar means and variances based on their pre-test scores on Scratch.

    “participants were divided into two groups using a matched-groups design [13,60].The ninety participants who completed the frst session were divided into two groups to have similar means and variances based on their pre-test scores on Scratch.”
    Quote in the paperSubstantial claim0 of 2 sources checked
    • cites [13] 13. Miriam Bruhn and David McKenzie. 2009. In pursuit of balance: Randomization in practice in development feld experiments. American economic journal: applied economics 1, 4 (2009), 200-232. source
    • cites [60] 60. I Scott MacKenzie. 2012. Human-computer interaction: An empirical research perspective. (2012).

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    The grading dashboard did not display any information that would reveal the identity of learners or their group, to ensure consistent and unbiased feedback.

    “To ensure both groups received consistent and unbiased feedback on their submissions, the grading dashboard did not display any information that would reveal the identity of learners or their group”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    For 5% of disagreements more than 25%, the two graders met again to resolve disagreements.

    “For the 5% of the disagreements which were more than 25%, the two graders met again to resolve each of their disagreements.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    AI code generators may lower the barriers to entry into programming by generating code from simple sentences instead of formal specifications.

    “By generating code from simple sentences instead of formal and syntactically fxed specifcations, these AI Coding Assistants may lower the barriers to entry into programming.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    69 learners successfully completed all three phases of the study.

    “69 learners (21 female/48 male) ages 10-17 (M=12.5;SD=1.8) successfully completed all three phases of the study”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data

    AI code generators allowed novice programmers to perform better and faster with less frustration when writing code.

    “AI code generators allowed novice programmers to perform better and faster with less frustration when writing code”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationown data

    The lack of statistical findings for such metrics does not necessarily mean there is no effect present; it could potentially be due to the sample size of the study.

    “The lack of statistical fndings for such metrics does not necessarily mean there is no efect present, it could potentially be due to the sample size of our study.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    Having access to the AI code generator reduced stress and discouragement and improved eagerness to continue programming later on.

    “Furthermore, our qualitative feedback shows that having access to the AI code generator reduced stress and discouragement in addition to improving their eagerness to continue programming later on.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Alpha levels for per-topic comparisons were adjusted using Bonferroni correction ( = 0.05/5 = 0.01).

    “Alpha levels for per-topic comparisons are adjusted using Bonferroni correction ( = 0.05/5 = 0.01).”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Prior research has shown that subgoal-labeled instructional material reduces extraneous cognitive load imposed on novices learning programming by chunking problem-solving steps and promoting self-explanation.

    “Prior research has shown that subgoallabeled instructional material reduces extraneous cognitive load imposed on novices learning programming by chunking problemsolving steps and promoting self-explanation [61].”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [61] 61. Lauren E Margulieux, Mark Guzdial, and Richard Catrambone. 2012. Subgoal- labeled instructional material improves performance and transfer in learning to develop mobile applications. In Proceedings of the ninth annual international conference on International computing education research. 71-78. source

  • May combine two claims.

    Interpretationown data

    Using AI code generators can significantly increase task completion, improve correctness score, reduce encountered errors, and reduce task completion time on code-authoring tasks during the training phase.

    “Our results from code-authoring tasks during the training phase indicate that using AI code generators can significantly increase task completion, improve correctness score, reduce encountered errors, and reduce task completion time.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Although the Baseline group used the documentation more frequently, they had to start each task by creating a new program from scratch before getting to the modify portion of the activity, and thus encountered more errors.

    “Although learners in the Baseline group used the documentation more frequently, they had to start each task by creating a new program from scratch before getting to the modify portion of the activity, and thus encountered more errors.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Prior research has argued for switching away from statistical significance testing in favor of reporting informative charts with effect sizes and confidence intervals and nuanced interpretations.

    “prior research has even argued for switching away from statistical signifcance testing in favor of reporting informative charts with efect sizes and confdence intervals and nuanced interpretations [20]”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [20] 20. Pierre Dragicevic. 2015. HCI Statistics without p-values. Ph. D. Dissertation. Inria.

  • May combine two claims.

    Interpretationreasoning

    Higher prior programming knowledge meant learners probably had more code-reading and tracing skills, which means they were able to understand code better, and then perform better during the next modification or code-creation tasks.

    “Their higher prior programming knowledge meant that they probably had more code-reading and tracing skills which means that they were able to understand that code better, and then perform better during the next modifcation or code-creation tasks.”
    Quote in the paperSubstantial claim
39 of 39

  • The paper relies on this without support.

    The AI code generator (Codex) used in the study is representative of AI code generators in general.

    “Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming”
    “Similarly, AI code generators may reduce the cognitive load when learning to program, but its impact on the learning experience has not been studied.”

  • The paper relies on this without support.

    Performance on the Coding Steps tasks and multiple-choice questions reflects learning of introductory Python programming.

    “To answer these questions, we conducted a controlled study with 69 young learners, ages 10-17 (M=12.5,SD=1.8) with no prior textbased programming experience.Half of them used an AI Coding Assistant (Codex) to learn Python, and the other half did not.”
    “The tasks in this phase included fve code-authoring tasks, fve code modifcation tasks, and then 40 multiple-choice questions”

  • The paper relies on this without support.

    The AI code generator's quality measured on the 45 tasks before the study is representative of its quality during the actual study sessions.

    “To measure the quality of the AI code generator, we entered the task description of the 45 code-authoring tasks into the pre-conditioned model (described in Figure 3) and evaluated the accuracy and amount of required manual modifcations on the generated code.”

  • The paper relies on this without support.

    The learning effects and support provided by AI code generators have not been studied in the context of introductory programming.

    “To date, the learning efects and support provided by AI code generators have not been studied in the context of introductory programming.”
4 of 4

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The propensity of learners to use GenAI outputs varies significantly by exercise topic.

    “The propensity of learners to use GenAI outputs has been shown to vary significantly by exercise topic [Kazemitabaar et al. 2023; Mailach et al. 2025].”

    From How Do Programming Students Use Generative AI? (2025) · cites this paper as Kazemitabaar et al. 2023

    Section 5.1.3 (page unavailable) reports that learners used the AI code generator more often on later topics—48% of basics tasks versus 84% of loops and arrays tasks—and that unmodified AI-generated code was submitted more often for loops and arrays. It does not report a statistical test showing that use of GenAI outputs varied significantly by topic, so the significance claim is unsupported. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Generative AI can change how much students learn through two margins: time allocation and learning productivity.

    “Generative AI can change how much students learn through two margins.”

    From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Kazemitabaar et al., 2023

    The Discussion (section 6, page not provided) reports faster task completion with AI code generators and improved retention-test performance for learners with higher prior programming knowledge. The paper does not describe time allocation and learning productivity as two distinct ways AI changes how much students learn; faster task completion is not itself evidence of changed learning-time allocation. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

  • Excessive dependence on GenAI tools may undermine the development of algorithmic thinking, error detection, and analytical reasoning.

    “While research has demonstrated that GenAI can support learning and foster creativity, excessive dependence on these tools may undermine the development of algorithmic thinking, error detection, and analytical reasoning (Kazemitabaar et al., 2023;Mogavi et al., 2024;Zviel-Girshin, 2024).”

    From Integrating Generative AI into Programming Education: Student Perceptions and the Challenge of Correcting AI Errors (2025) · cites this paper as Kazemitabaar et al., 2023

    Section 6.2.2 raises the possibility that over-utilization of AI code generators might impede learning, and the Introduction warns that learners may become dependent on them. But Section 7 calls their effect on algorithmic thinking an unanswered question; the paper does not establish that dependence undermines error detection or analytical reasoning. Its Discussion reports no reduction in the learning outcomes it measured. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Integrating AI tools alongside conventional instructional methods enables students to build a comprehensive skill set while developing the capacity to critique and refine automated outputs.

    “Research suggests that integrating AI tools alongside conventional instructional methods enables students to build a comprehensive skill set while developing the capacity to critique and refine automated outputs (Kazemitabaar et al., 2023).”

    From Integrating Generative AI into Programming Education: Student Perceptions and the Challenge of Correcting AI Errors (2025) · cites this paper as Kazemitabaar et al., 2023

    The AI-Assisted Learning Environment section describes AI code generation alongside programming tasks, documentation, and instructor feedback. The Discussion and Conclusion report better code-authoring performance without reduced manual code-modification performance. They do not establish that this combination builds a comprehensive skill set or develops students’ capacity to critique and refine AI-generated outputs. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Reliance on code generation can negatively impact code authoring skills.

    “Furthermore, reliance on code generation can negatively impact code authoring skills [Jošt et al. 2024; Kazemitabaar et al. 2023].”

    From How Do Programming Students Use Generative AI? (2025) · cites this paper as Kazemitabaar et al. 2023

Statistics that don’t add up

Each reported p-value recomputed from its test statistic and degrees of freedom.

  • Inconsistent p-value: t(39)=2.78,p = .007

    Minor

    The paper reports p = .007 here; recomputing from the test statistic and its degrees of freedom gives p = .008 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p = .007 · recomputed p = .008

  • Inconsistent p-value: t(39)=2.34,p = .022

    Minor

    The paper reports p = .022 here; recomputing from the test statistic and its degrees of freedom gives p = .024 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p = .022 · recomputed p = .024

  • Inconsistent p-value: t(67)=1.70,p = .092

    Minor

    The paper reports p = .092 here; recomputing from the test statistic and its degrees of freedom gives p = .094 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p = .092 · recomputed p = .094

  • Inconsistent p-value: t(58)=1.41,p = .161

    Minor

    The paper reports p = .161 here; recomputing from the test statistic and its degrees of freedom gives p = .164 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p = .161 · recomputed p = .164

  • Inconsistent p-value: t(67)=0.54,p = .67

    Minor

    The paper reports p = .67 here; recomputing from the test statistic and its degrees of freedom gives p = .591 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p = .67 · recomputed p = .591

  • Inconsistent p-value: t(39)=2.02,p = .047

    Minor

    The paper reports p = .047 here; recomputing from the test statistic and its degrees of freedom gives p = .050 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p = .047 · recomputed p = .050

  • Inconsistent p-value: t(67)=1.28,p = .202

    Minor

    The paper reports p = .202 here; recomputing from the test statistic and its degrees of freedom gives p = .205 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.

    Reported p = .202 · recomputed p = .205

7 of 7

Statistics recomputed

Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.

  • Doesn’t add upt(67) = 0.54reported p = .67 · recomputed p = .591

    “t(67)=0.54,p = .67”

  • Doesn’t add upt(67) = 1.28reported p = .202 · recomputed p = .205

    “t(67)=1.28,p = .202”

  • Doesn’t add upt(39) = 2.34reported p = .022 · recomputed p = .024

    “t(39)=2.34,p = .022”

  • Doesn’t add upt(58) = 1.41reported p = .161 · recomputed p = .164

    “t(58)=1.41,p = .161”

  • Doesn’t add upt(39) = 2.78reported p = .007 · recomputed p = .008

    “t(39)=2.78,p = .007”

  • Doesn’t add upt(67) = 1.7reported p = .092 · recomputed p = .094

    “t(67)=1.70,p = .092”

  • Doesn’t add upt(39) = 2.02reported p = .047 · recomputed p = .050

    “t(39)=2.02,p = .047”

  • Not confirmedt(67) = 0.058reported p = .953 · recomputed p = .954

    “t(67)=0.058,p = .953”

  • Not confirmedt(67) = 1.21reported p = .228 · recomputed p = .231

    “t(67)=1.21,p = .228”

  • Consistentt(67) = 2.8reported p = .006 · recomputed p = .007

    “t(67)=2.8,p = .006”

  • Consistentt(67) = 0.25reported p = .805 · recomputed p = .803

    “t(67)=0.25,p = .805”

  • Consistentt(67) = 0.53reported p = .6 · recomputed p = .598

    “t(67)=0.53,p = .60”

  • Consistentt(67) = 1.15reported p = .25 · recomputed p = .254

    “t(67)=1.15,p = .25”

  • Consistentt(67) = 6.92reported p < .001 · recomputed p < .001

    “t(67)=6.92,p < .001”

  • Consistentt(67) = 6.4reported p < .001 · recomputed p < .001

    “t(67)=6.40,p < .001”

  • Consistentt(67) = 5.48reported p < .001 · recomputed p < .001

    “t(67)=5.48,p < .001”

  • Consistentt(67) = 3.03reported p = .004 · recomputed p = .003

    “t(67)=3.03,p = .004”

  • Consistentt(67) = 3.65reported p = .001 · recomputed p < .001

    “t(67)=3.65,p = .001”

  • Consistentt(67) = 0.98reported p = .331 · recomputed p = .331

    “t(67)=0.98,p = .331”

  • Consistentt(67) = 1.84reported p = .071 · recomputed p = .070

    “t(67)=1.84,p = .071”

  • Consistentt(67) = 1.53reported p = .129 · recomputed p = .131

    “t(67)=1.53,p = .129”

  • Consistentt(67) = 0.91reported p = .367 · recomputed p = .366

    “t(67)=0.91,p = .367”

  • Consistentt(39) = 3.53reported p = .001 · recomputed p = .001

    “t(39)=3.53,p = .001”

  • Consistentt(67) = 0.2reported p = .838 · recomputed p = .842

    “t(67)=0.20,p = .838”

  • Consistentt(58) = 2.08reported p = .041 · recomputed p = .042

    “t(58)=2.08,p = .041”

  • Consistentt(67) = 1.13reported p = .262 · recomputed p = .263

    “t(67)=1.13,p = .262”

  • Consistentt(67) = 2.08reported p = .042 · recomputed p = .041

    “t(67)=2.08,p = .042”

  • Consistentt(67) = 1.21reported p = .229 · recomputed p = .231

    “t(67)=1.21,p = .229”

  • Consistentt(67) = 4.1reported p < .001 · recomputed p < .001

    “t(67)=4.10,p < .001”

  • Consistentt(67) = 1.54reported p = .129 · recomputed p = .128

    “t(67)=1.54,p = .129”

  • Consistentt(58) = 2.09reported p = .04 · recomputed p = .041

    “t(58)=2.09,p = .04”

31 of 31

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • The first session has two stated durations

    Minor

    The procedure describes all ten sessions as 90 minutes long, but later specifies that the first session lasted two hours.

1 of 1

Cites

Flagged first, then by citations

Research works printed but not linked: 6

  1. Amazon Web Services. 2022. CodeWhisperer: ML-powered coding companion. https://aws.amazon.com/codewhisperer/. [Online; accessed 9-September-2022].
  2. 10.5555/364133.364161 · DOI
  3. Paul Jaccard. 1901. Étude comparative de la distribution florale dans une portion des Alpes et des Jura. Bull Soc Vaudoise Sci Nat 37 (1901), 547–579.
  4. Mitchel Resnick. 2014. Give P’sa chance: Projects, peers, passion, play. In Constructionism and creativity: Proceedings of the third international constructionism conference. Austrian computer society, Vienna. 13–20.
  5. Mitchel Resnick and David Siegel. 2015. A different approach to coding. International Journal of People-Oriented Programming 4, 1(2015), 1–4.
  6. Tabnine. 2022. Tabnine: AI assistant for software developers. https://www.tabnine.com/. [Online; accessed 9-September-2022].

Cited by

Sorted by citations

Related

Authors
Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T.
Affiliations
  • University of Toronto (CA)
  • Department of Computer Science, University of Toronto, Canada
  • University of Michigan (US)
  • College of Education, College of Information Studies, University of Maryland, United States
ORCID
Venue
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2023-04-19
Pages
1-23
DOI
10.1145/3544548.3580919
OpenAlex
W4321162272
Cited by
366
References
96
Citations by year
2026: 117 · 2025: 118 · 2024: 104 · 2023: 27
FWCI
110.39
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
Keywords
OpenAI Codex, novice programmers, introductory programming, code modification, knowledge retention, overreliance on AI, Scratch, Python, programming education

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-11.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-11.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review