The DOI matches its Crossref record; 102 of the 102 references it cites were found in a registry or an index; 2 of the 10 sources its claims cite were checked for support.
Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Open accessAbstract
AI code generators like OpenAI Codex have the potential to assist novice programmers by generating code from natural language descriptions, however, over-reliance might negatively impact learning and retention. To explore the implications that AI code generators have on introductory programming, we conducted a controlled experiment with 69 novices (ages 10-17). Learners worked on 45 Python code-authoring tasks, for which half of the learners had access to Codex, each followed by a code-modification task. Our results show that using Codex significantly increased code-authoring performance (1.15x increased completion rate and 1.8x higher scores) while not decreasing performance on manual code-modification tasks. Additionally, learners with access to Codex during the training phase performed slightly better on the evaluation post-tests conducted one week later, although this difference did not reach statistical significance. Of interest, learners with higher Scratch pre-test scores performed significantly better on retention post-tests, if they had prior access to Codex.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 of 10 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Prior work in introductory programming has shown that code-reading and tracing skills are essential to problem-solving activities including code-writing [49,55,57].”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6.1 How Novices Beneft from AI Coding Assistants
What the sources say
- Not checked — no full text of the sourcecites [49] 49. Amruth N Kumar. 2013. A study of the infuence of code-tracing problems on code-writing skills. In Proceedings of the 18th ACM conference on Innovation and technology in computer science education. 183-188. source
- Not checked — no full text of the sourcecites [55] 55. Raymond Lister, Colin Fidge, and Donna Teague. 2009. Further evidence of a relationship between explaining, tracing and writing skills in introductory programming. Acm sigcse bulletin 41, 3 (2009), 161-165. source
- Partly supportedcites [57] 57. Mike Lopez, Jacqueline Whalley, Phil Robbins, and Raymond Lister. 2008. Rela- tionships between reading, tracing and writing skills in introductory program- ming. In Proceedings of the fourth international workshop on computing education research. 101-112. sourceExperimental — accuracy not yet measured
The Discussion (§6) reports significant associations between code-reading, code-tracing, and code-writing performance. But the paper presents a hierarchy of skills as a possibility, not a demonstrated necessity; its findings do not establish that reading and tracing are essential to problem-solving activities generally. (LLM, conf=0.88)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“learners who had access to the AI code generator (the Codex group) were able to successfully generate code”
- Also stated in: § 1 INTRODUCTION · plain
“Our results show that learners who had access to the AI code generator (the Codex group) were able to successfully generate code”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“1.8x higher correctness”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They performed signifcantly better on code-authoring tasks (1.15x increased progress”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“0.57x less time”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“0.59x less errors”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The comparison is showing that most of the diferences between conditions (in terms of efect size) are appearing on the left side of this table, for the high performers.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.3 The Efect of Prior Programming Competency
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Learners submitted the AI-generated code without any modifcation for 49% of the tasks (503/1015).”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.3 AI Code Generator Usage.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, learners in the Codex group on average scored 20.8% higher on arrays in terms of correctness score (Codex: M=50%, SD=36.8%;Baseline: M=29.2%,SD=36.3%;t(39)=2.34,p=.022, d=0.57)”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.4 Modifying Tasks (Training Phase).
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, learners mostly felt that they learned about Python programming concepts after using the code generator.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.4.2 Perceptions on Using AI Code Generator.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- It may state two separate claims as one. 21
- Probably not a substantial claim. 20
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This could potentially explain how step-by-step usage of the AI code generator could potentially reduce cognitive load and allow novices to learn faster due to more availability of their mental resources, specifcally for germane cognitive load which is responsible for creating mental models and storing long-term memory [46].”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6.1 How Novices Beneft from AI Coding Assistants
What the sources say
- Partly supportedcites [46] 46. Paul Kirschner, John Sweller, and Richard E Clark. 2006. Why unguided learning does not work: An analysis of the failure of discovery learning, problem-based learning, experiential learning and inquiry-based learning. Educational Psycholo- gist 41, 2 (2006), 75-86. sourceExperimental — accuracy not yet measured
In “Current Research Supporting Direct Guidance” (page unavailable), the paper says worked examples reduce working-memory load and direct resources toward learning problem-solving schemas. That supports the proposed mechanism for guided, step-by-step learning, but the paper does not address AI code generators or identify the freed resources specifically as germane cognitive load. (LLM, conf=0.85)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and 12.7% higher on modifying tasks (Codex: M=47.5%,SD=31.6%;Baseline: M=34.8%,SD=29.0%;t(67)=1.70,p=.092, d=0.41), but neither of these reached statistical signifcance.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.2 Retention Post-Test.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and on average 16% higher on loops, but this did not reach statistical signifcance (Codex: M=50.2%,SD=35.7%;Baseline: M=33.8%,SD=27.7%;t(58)=2.08,p=.041, d=0.51).”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.1 Immediate Post-Test.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Learners in the Codex group also encountered slightly higher errors on modifying tasks, but this did not reach statistical signifcance (Codex: M=0.81, SD=0.85;Baseline: M=0.57, SD=0.70; t(67)=1.21,p=.229, d=0.30).”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.2 Retention Post-Test.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and 10.5% higher on loops (Codex: M=61%, SD=30.5%;Baseline: M=50.5%,SD=29.3%;t(58)=1.41,p=.161, d=0.34) however, neither of these results reached statistical signifcance.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.4 Modifying Tasks (Training Phase).
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“64 indicated using a block-based programming environment like Scratch or Code.org”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2 Participants
Strength:plain· not yet calibratedn = 64Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some learners felt that the AI-generated code needed some explanations”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.4.2 Perceptions on Using AI Code Generator.
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“learners from the Codex group on average scored 9.0% higher on authoring tasks (Codex: M=59.1%,SD=34.8%;Baseline: M=49.8%,SD=32.4%;t(67)=1.13,p=.262, d=0.28)”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.2 Retention Post-Test.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“27 indicated taking a programming-related class in the past”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2 Participants
Strength:plain· not yet calibratedn = 27Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, semantic errors occurred infrequently in both groups (Codex: M=0.01, SD=0.02;Baseline: M=0.03, SD=0.05; t(67)=1.84,p=.071, d=0.44).”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.2 Authoring Tasks (Training Phase).
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They also felt that generating the code that they needed did not require a lot of practice.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.4.2 Perceptions on Using AI Code Generator.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the retention test which was conducted one week later, learners from the Codex group performed slightly better on coding tasks and multiple-choice questions, although these results did not reach statistical signifcance”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“although neither of these diferences reached statistical signifcance.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.2 Retention Post-Test.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“learners from both groups fully completed the frst two topics (basics and data-types) and one learner from each group did not complete conditionals.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.1 Overall Completion Rates and Direct Hints.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the Codex group had more learners that fully completed the 18 tasks on loops (Codex: 27/33, Baseline: 23/36) and the 7 tasks on arrays (Codex: 23/33, Baseline: 9/36)”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.1 Overall Completion Rates and Direct Hints.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When discussing our results, there were several metrics in which the diferences between the two groups did not reach statistical signifcance but there did seem to be some diference in means and confdence intervals.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 7 LIMITATIONS AND FUTURE WORK
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Learners employed various methods to generate code such as breaking down the task into subgoals and using the code generator to generate code for each of the subgoals step-by-step.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6 DISCUSSION
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Furthermore, although our study included many non-native English speakers, but they were mostly comfortable explaining things in English.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 7 LIMITATIONS AND FUTURE WORK
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The two graders fully agreed on 79% of the submissions”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.4 Data Analysis
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“with 16% of disagreements having a diference of 25% in which we averaged the two grades”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.4 Data Analysis
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“No common factors were identifed among the 21 students who dropped out in terms of disability, native language (six non-English) and computer or internet access.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2 Participants
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“One learner in each pair was randomly assigned to either the Codex or Baseline group.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2.1 Participant Condition Assignment.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But overall, this is a promising result: although learners in the Codex group used the documentation less and relied heavily on AI-generated code for the authoring tasks, they still did just as well, and in some cases better, in manual code modifcation tasks.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.4 Modifying Tasks (Training Phase).
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“two independent researchers graded each submitted solution using a simple and consistent grading rubric of deducting 25% for each major issue or missing part in their fnal submission”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.4 Data Analysis
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This shows that the AI code generator produces high quality results, but this depends on the quality of the prompt message.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 3.4 Quality of AI Generated Code
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“participants were divided into two groups using a matched-groups design [13,60].The ninety participants who completed the frst session were divided into two groups to have similar means and variances based on their pre-test scores on Scratch.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2.1 Participant Condition Assignment.
What the sources say
- Not checked — no full text of the sourcecites [13] 13. Miriam Bruhn and David McKenzie. 2009. In pursuit of balance: Randomization in practice in development feld experiments. American economic journal: applied economics 1, 4 (2009), 200-232. source
- Not checked — no full text of the sourcecites [60] 60. I Scott MacKenzie. 2012. Human-computer interaction: An empirical research perspective. (2012).
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To ensure both groups received consistent and unbiased feedback on their submissions, the grading dashboard did not display any information that would reveal the identity of learners or their group”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.1.2 Training Phase.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the 5% of the disagreements which were more than 25%, the two graders met again to resolve each of their disagreements.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.4 Data Analysis
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By generating code from simple sentences instead of formal and syntactically fxed specifcations, these AI Coding Assistants may lower the barriers to entry into programming.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“69 learners (21 female/48 male) ages 10-17 (M=12.5;SD=1.8) successfully completed all three phases of the study”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2 Participants
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“AI code generators allowed novice programmers to perform better and faster with less frustration when writing code”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 8 CONCLUSION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The lack of statistical fndings for such metrics does not necessarily mean there is no efect present, it could potentially be due to the sample size of our study.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 7 LIMITATIONS AND FUTURE WORK
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, our qualitative feedback shows that having access to the AI code generator reduced stress and discouragement in addition to improving their eagerness to continue programming later on.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6 DISCUSSION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Alpha levels for per-topic comparisons are adjusted using Bonferroni correction ( = 0.05/5 = 0.01).”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 45B
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Prior research has shown that subgoallabeled instructional material reduces extraneous cognitive load imposed on novices learning programming by chunking problemsolving steps and promoting self-explanation [61].”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6.1 How Novices Beneft from AI Coding Assistants
What the sources say
- Not checked — no full text of the sourcecites [61] 61. Lauren E Margulieux, Mark Guzdial, and Richard Catrambone. 2012. Subgoal- labeled instructional material improves performance and transfer in learning to develop mobile applications. In Proceedings of the ninth annual international conference on International computing education research. 71-78. source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results from code-authoring tasks during the training phase indicate that using AI code generators can significantly increase task completion, improve correctness score, reduce encountered errors, and reduce task completion time.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6 DISCUSSION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although learners in the Baseline group used the documentation more frequently, they had to start each task by creating a new program from scratch before getting to the modify portion of the activity, and thus encountered more errors.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6.1 How Novices Beneft from AI Coding Assistants
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“prior research has even argued for switching away from statistical signifcance testing in favor of reporting informative charts with efect sizes and confdence intervals and nuanced interpretations [20]”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 7 LIMITATIONS AND FUTURE WORK
What the sources say
- Not checked — no full text of the sourcecites [20] 20. Pierre Dragicevic. 2015. HCI Statistics without p-values. Ph. D. Dissertation. Inria.
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Their higher prior programming knowledge meant that they probably had more code-reading and tracing skills which means that they were able to understand that code better, and then perform better during the next modifcation or code-creation tasks.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6.1 How Novices Beneft from AI Coding Assistants
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper relies on this without support.
The AI code generator (Codex) used in the study is representative of AI code generators in general.
“Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming”
“Similarly, AI code generators may reduce the cognitive load when learning to program, but its impact on the learning experience has not been studied.”
The paper relies on this without support.
Performance on the Coding Steps tasks and multiple-choice questions reflects learning of introductory Python programming.
“To answer these questions, we conducted a controlled study with 69 young learners, ages 10-17 (M=12.5,SD=1.8) with no prior textbased programming experience.Half of them used an AI Coding Assistant (Codex) to learn Python, and the other half did not.”
“The tasks in this phase included fve code-authoring tasks, fve code modifcation tasks, and then 40 multiple-choice questions”
The paper relies on this without support.
The AI code generator's quality measured on the 45 tasks before the study is representative of its quality during the actual study sessions.
“To measure the quality of the AI code generator, we entered the task description of the 45 code-authoring tasks into the pre-conditioned model (described in Figure 3) and evaluated the accuracy and amount of required manual modifcations on the generated code.”
The paper relies on this without support.
The learning effects and support provided by AI code generators have not been studied in the context of introductory programming.
“To date, the learning efects and support provided by AI code generators have not been studied in the context of introductory programming.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The propensity of learners to use GenAI outputs has been shown to vary significantly by exercise topic [Kazemitabaar et al. 2023; Mailach et al. 2025].”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Kazemitabaar et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Generative AI can change how much students learn through two margins.”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Kazemitabaar et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While research has demonstrated that GenAI can support learning and foster creativity, excessive dependence on these tools may undermine the development of algorithmic thinking, error detection, and analytical reasoning (Kazemitabaar et al., 2023;Mogavi et al., 2024;Zviel-Girshin, 2024).”
From Integrating Generative AI into Programming Education: Student Perceptions and the Challenge of Correcting AI Errors (2025) · cites this paper as Kazemitabaar et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research suggests that integrating AI tools alongside conventional instructional methods enables students to build a comprehensive skill set while developing the capacity to critique and refine automated outputs (Kazemitabaar et al., 2023).”
From Integrating Generative AI into Programming Education: Student Perceptions and the Challenge of Correcting AI Errors (2025) · cites this paper as Kazemitabaar et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Furthermore, reliance on code generation can negatively impact code authoring skills [Jošt et al. 2024; Kazemitabaar et al. 2023].”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Kazemitabaar et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“These numbers align with findings by Kazemitabaar et al. [2023], who observed an average similarity of 63% ( 𝜎 = 42%) between generated and submitted code, albeit using the Jaccard similarity measure.”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Kazemitabaar et al. [2023]
The cited source says what the claim attributes to it.
In the citing paper
“Third, in many designs students choose how much time to spend on the task (e.g., Kazemitabaar et al., 2023; Kestin et al., 2025; Kreijkes et al., 2026), so the estimated effects combine the two margins: time allocation and learning productivity.”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Kazemitabaar et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“no detectable difference on an immediate unassisted post-test ( -0.05 SD; the baseline group scored slightly higher, 62.9 versus 61.3 percent)”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Kazemitabaar et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“only a marginal difference on a one-week retention test of code modification (0.41 SD).”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Kazemitabaar et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“AI access produced large performance gains during training (1.67 SD)”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Kazemitabaar et al. (2023)
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, as the results are fairly consistent with similar surveys around the time [Kazemitabaar et al. 2023; Prather et al. 2023a; Zastudil et al. 2023] and we were commonly able to observe behavior that could be considered unethical in an academic context [Prather et al. 2023a], we do not believe this to be a major limitation.”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Kazemitabaar et al. 2023
Experimental — accuracy not yet measuredIn the citing paper
“Interacting with an LLM has been observed to lead to higher performance in code writing tasks but to little differences in learning gains [18].”
From Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study (2024) · cites this paper as [18]
In the citing paper
“They found that students who used AI code generators performed significantly better (1.15x progress, 0.59x errors, 1.8x higher correctness, 0.57x time spent) without negative effects on learning.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [108]
In the citing paper
“Recent research has shown promising possibilities for providing students with LLM partners in pair programming, given the right context and with the right scaffolding and support [44,108,159].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [108]
In the citing paper
“The models might generate code students do not understand [108] or may distract them with large blocks of text they did not write [159].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [108]
In the citing paper
“Also very concerning, Kazemitabaar et al. [108] found that students with more knowledge benefited more from code generation than students with less knowledge.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [108]
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom.
Inconsistent p-value: t(39)=2.78,p = .007
MinorThe paper reports p = .007 here; recomputing from the test statistic and its degrees of freedom gives p = .008 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.
Reported p = .007 · recomputed p = .008
“t(39)=2.78,p=.007”
Inconsistent p-value: t(39)=2.34,p = .022
MinorThe paper reports p = .022 here; recomputing from the test statistic and its degrees of freedom gives p = .024 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.
Reported p = .022 · recomputed p = .024
“t(39)=2.34,p=.022”
Inconsistent p-value: t(67)=1.70,p = .092
MinorThe paper reports p = .092 here; recomputing from the test statistic and its degrees of freedom gives p = .094 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.
Reported p = .092 · recomputed p = .094
“t(67)=1.70,p=.092”
Inconsistent p-value: t(58)=1.41,p = .161
MinorThe paper reports p = .161 here; recomputing from the test statistic and its degrees of freedom gives p = .164 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.
Reported p = .161 · recomputed p = .164
“t(58)=1.41,p=.161”
Inconsistent p-value: t(67)=0.54,p = .67
MinorThe paper reports p = .67 here; recomputing from the test statistic and its degrees of freedom gives p = .591 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.
Reported p = .67 · recomputed p = .591
“t(67)=0.54,p=.67”
Inconsistent p-value: t(39)=2.02,p = .047
MinorThe paper reports p = .047 here; recomputing from the test statistic and its degrees of freedom gives p = .050 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.
Reported p = .047 · recomputed p = .050
“t(39)=2.02,p=.047”
Inconsistent p-value: t(67)=1.28,p = .202
MinorThe paper reports p = .202 here; recomputing from the test statistic and its degrees of freedom gives p = .205 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.
Reported p = .202 · recomputed p = .205
“t(67)=1.28,p=.202”
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upt(67) = 0.54reported p = .67 · recomputed p = .591
“t(67)=0.54,p = .67”
- Doesn’t add upt(67) = 1.28reported p = .202 · recomputed p = .205
“t(67)=1.28,p = .202”
- Doesn’t add upt(39) = 2.34reported p = .022 · recomputed p = .024
“t(39)=2.34,p = .022”
- Doesn’t add upt(58) = 1.41reported p = .161 · recomputed p = .164
“t(58)=1.41,p = .161”
- Doesn’t add upt(39) = 2.78reported p = .007 · recomputed p = .008
“t(39)=2.78,p = .007”
- Doesn’t add upt(67) = 1.7reported p = .092 · recomputed p = .094
“t(67)=1.70,p = .092”
- Doesn’t add upt(39) = 2.02reported p = .047 · recomputed p = .050
“t(39)=2.02,p = .047”
- Not confirmedt(67) = 0.058reported p = .953 · recomputed p = .954
“t(67)=0.058,p = .953”
- Not confirmedt(67) = 1.21reported p = .228 · recomputed p = .231
“t(67)=1.21,p = .228”
- Consistentt(67) = 2.8reported p = .006 · recomputed p = .007
“t(67)=2.8,p = .006”
- Consistentt(67) = 0.25reported p = .805 · recomputed p = .803
“t(67)=0.25,p = .805”
- Consistentt(67) = 0.53reported p = .6 · recomputed p = .598
“t(67)=0.53,p = .60”
- Consistentt(67) = 1.15reported p = .25 · recomputed p = .254
“t(67)=1.15,p = .25”
- Consistentt(67) = 6.92reported p < .001 · recomputed p < .001
“t(67)=6.92,p < .001”
- Consistentt(67) = 6.4reported p < .001 · recomputed p < .001
“t(67)=6.40,p < .001”
- Consistentt(67) = 5.48reported p < .001 · recomputed p < .001
“t(67)=5.48,p < .001”
- Consistentt(67) = 3.03reported p = .004 · recomputed p = .003
“t(67)=3.03,p = .004”
- Consistentt(67) = 3.65reported p = .001 · recomputed p < .001
“t(67)=3.65,p = .001”
- Consistentt(67) = 0.98reported p = .331 · recomputed p = .331
“t(67)=0.98,p = .331”
- Consistentt(67) = 1.84reported p = .071 · recomputed p = .070
“t(67)=1.84,p = .071”
- Consistentt(67) = 1.53reported p = .129 · recomputed p = .131
“t(67)=1.53,p = .129”
- Consistentt(67) = 0.91reported p = .367 · recomputed p = .366
“t(67)=0.91,p = .367”
- Consistentt(39) = 3.53reported p = .001 · recomputed p = .001
“t(39)=3.53,p = .001”
- Consistentt(67) = 0.2reported p = .838 · recomputed p = .842
“t(67)=0.20,p = .838”
- Consistentt(58) = 2.08reported p = .041 · recomputed p = .042
“t(58)=2.08,p = .041”
- Consistentt(67) = 1.13reported p = .262 · recomputed p = .263
“t(67)=1.13,p = .262”
- Consistentt(67) = 2.08reported p = .042 · recomputed p = .041
“t(67)=2.08,p = .042”
- Consistentt(67) = 1.21reported p = .229 · recomputed p = .231
“t(67)=1.21,p = .229”
- Consistentt(67) = 4.1reported p < .001 · recomputed p < .001
“t(67)=4.10,p < .001”
- Consistentt(67) = 1.54reported p = .129 · recomputed p = .128
“t(67)=1.54,p = .129”
- Consistentt(58) = 2.09reported p = .04 · recomputed p = .041
“t(58)=2.09,p = .04”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
The first session has two stated durations
MinorThe procedure describes all ten sessions as 90 minutes long, but later specifies that the first session lasted two hours.
“The study lasted three weeks and included ten 90-minute sessions, with one session per day, spanning over three (consecutive) weeks.”
“The frst session in the training phase was 2-hours long and included a one-hour introduction to the basic concepts of programming and computational thinking using Scratch.”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
(1978) · Medical Entomology and Zoology · Cited by 53,488
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Wing, J. M. (2006) · Communications of the ACM · doi:10.1145/1118178.1118215 · Cited by 8,102
Kirschner, P. A., Sweller, J., & Clark, R. E. (2006) · Educational Psychologist · doi:10.1207/s15326985ep4102_1 · Cited by 7,776
Using Effect Size—or Why the P Value Is Not Enough
Sullivan, G. M., & Feinn, R. (2012) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-12-00156.1 · Cited by 5,553 · cited 1×
“Finally, Cohen’s d is reported for effect size [78].”
Resnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
Computational Thinking in K–12
Grover, S., & Pea, R. D. (2013) · Educational Researcher · doi:10.3102/0013189x12463051 · Cited by 3,034
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824 · cited 2×
“For example, models like OpenAI Codex [14], Microsoft CodeBERT [27], Google PaLM [16], DeepMind AlphaCode [53] have been trained on millions of lines of publicly available code (e.g., on Github).”
Flagged first, then by citations
Research works printed but not linked: 6
- Amazon Web Services. 2022. CodeWhisperer: ML-powered coding companion. https://aws.amazon.com/codewhisperer/. [Online; accessed 9-September-2022].
- 10.5555/364133.364161 · DOI
- Paul Jaccard. 1901. Étude comparative de la distribution florale dans une portion des Alpes et des Jura. Bull Soc Vaudoise Sci Nat 37 (1901), 547–579.
- Mitchel Resnick. 2014. Give P’sa chance: Projects, peers, passion, play. In Constructionism and creativity: Proceedings of the third international constructionism conference. Austrian computer society, Vienna. 13–20.
- Mitchel Resnick and David Siegel. 2015. A different approach to coding. International Journal of People-Oriented Programming 4, 1(2015), 1–4.
- Tabnine. 2022. Tabnine: AI assistant for software developers. https://www.tabnine.com/. [Online; accessed 9-September-2022].
Cited by
Sorted by citations
Items 101–110 of 110
Ten Essential Pillars in Artificial Intelligence for University Science Education: A Scoping Review
Deroncele-Acosta, Á., Valdiviezo, O. B., Trujillo, M. D. L. Á. S., Palacios-Núñez, M. L., Rueda-Garcés, H., & Brito-Garcías, J. G. (2024) · SAGE Open · doi:10.1177/21582440241272016 · Cited by 30
“…it was intended to include in the artificial intelligence block the descriptors: ‘‘ChatGPT,’’ ‘‘Generative AI,’’ and ‘‘Generative Artificial Intelligence,’’ but only one document was included in the search (Kazemitabaar et al., 2023) and although it addresses an interesting perspective, it…”
How Much Does AI Impact Development Speed? an Enterprise-Based Randomized Controlled Trial
Paradis, E., Grey, K., Madison, Q., Nam, D., Macvean, A., Meimand, V., Zhang, N., Ferrari-Church, B., & Chandra, S. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00060 · Cited by 20
“Recent work has explored the potential of these tools to assist students and novice programmers [8], [23], [24], as well as program-mers’ perceptions of [25], [26], [27] and trust in [28], [29], [30] AI tools.”
Lyu, W., Wang, Y., Sun, Y., & Zhang, Y. (2025) · ACM Conference on Learning @ Scale (L@S) · doi:10.1145/3698205.3729544 · Cited by 26
“…tools in computer science learning contexts, such as automated code explanations [34], enhanced debugging support [24, 27, 44], and generating exercises to help students practice [25, 40, 44, 46], as well as examining how students and instructors perceive GenAI in educational contexts [26, 58, 63].”
InfluentialAnalysis of Generative AI Policies in Computing Course Syllabi
Ali, A., Collier, A. H., Dewan, U., McDonald, N., & Johri, A. (2025) · Proceedings of the 56th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3641554.3701823 · Cited by 14
Mailach, A., Gorgosch, D., Siegmund, N., & Siegmund, J. (2025) · Empirical Software Engineering · doi:10.1007/s10664-024-10561-6 · Cited by 13
CreAItive Collaboration? Users' Misjudgment of AI-Creativity Affects Their Collaborative Performance
Bangerl, M. M., Disch, L., David, T., & Pammer-Schindler, V. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713886 · Cited by 13
Navigating the Ethical and Societal Impacts of Generative AI in Higher Computing Education
Mak, J., Nakatumba-Nabende, J., Clear, T., Clear, A., Abluwi, I., Andrei, O., Angeli, L., MacNeil, S., Oyelere, S. S., Hale Rattigan, M., et al. (2025) · Proceedings of the 2025 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3760545.3783975 · Cited by 12
Zhu, S., & Liu, Z. (2026) · Proceedings of the 2026 International Conference on Big Data and Informatization Education · doi:10.1145/3806980.3806985
Esnaola, L. M., Ramón, H. D., & Lanzarini, L. C. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10813-7 · Cited by 2
Experimental Evidence on the Learning Impact of Generative AI
Contractor, Z., & Reyes, G. (2026) · arXiv · doi:10.48550/arxiv.2607.08849
Metadata
- Authors
- Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T.
- Affiliations
- University of Toronto (CA)
- Department of Computer Science, University of Toronto, Canada
- University of Michigan (US)
- College of Education, College of Information Studies, University of Maryland, United States
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-23
- OpenAlex
- W4321162272
- Cited by
- 366
- References
- 96
- Citations by year
- 2026: 117 · 2025: 118 · 2024: 104 · 2023: 27
- FWCI
- 110.39
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- OpenAI Codex, novice programmers, introductory programming, code modification, knowledge retention, overreliance on AI, Scratch, Python, programming education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
