The DOI matches its Crossref record; 102 of the 102 references it cites were found in a registry or an index; 2 of the 10 sources its claims cite were checked for support.
Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Open accessAbstract
AI code generators like OpenAI Codex have the potential to assist novice programmers by generating code from natural language descriptions, however, over-reliance might negatively impact learning and retention. To explore the implications that AI code generators have on introductory programming, we conducted a controlled experiment with 69 novices (ages 10-17). Learners worked on 45 Python code-authoring tasks, for which half of the learners had access to Codex, each followed by a code-modification task. Our results show that using Codex significantly increased code-authoring performance (1.15x increased completion rate and 1.8x higher scores) while not decreasing performance on manual code-modification tasks. Additionally, learners with access to Codex during the training phase performed slightly better on the evaluation post-tests conducted one week later, although this difference did not reach statistical significance. Of interest, learners with higher Scratch pre-test scores performed significantly better on retention post-tests, if they had prior access to Codex.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
84 Own result · 31 Own interpretation · 11 Own method · 4 No full text of the sources · 3 No basis given · 2 Partly supported · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Prior work in introductory programming has shown that code-reading and tracing skills are essential to problem-solving activities including code-writing [49,55,57].”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6.1 How Novices Beneft from AI Coding Assistants
What the sources say
- Not checked — no full text of the sourcecites [49] 49. Amruth N Kumar. 2013. A study of the infuence of code-tracing problems on code-writing skills. In Proceedings of the 18th ACM conference on Innovation and technology in computer science education. 183-188. source
- Not checked — no full text of the sourcecites [55] 55. Raymond Lister, Colin Fidge, and Donna Teague. 2009. Further evidence of a relationship between explaining, tracing and writing skills in introductory programming. Acm sigcse bulletin 41, 3 (2009), 161-165. source
- Partly supportedcites [57] 57. Mike Lopez, Jacqueline Whalley, Phil Robbins, and Raymond Lister. 2008. Rela- tionships between reading, tracing and writing skills in introductory program- ming. In Proceedings of the fourth international workshop on computing education research. 101-112. sourceExperimental — accuracy not yet measured
The Discussion (§6) reports significant associations between code-reading, code-tracing, and code-writing performance. But the paper presents a hierarchy of skills as a possibility, not a demonstrated necessity; its findings do not establish that reading and tracing are essential to problem-solving activities generally. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“learners who had access to the AI code generator (the Codex group) were able to successfully generate code”
- Also stated in: § 1 INTRODUCTION · plain
“Our results show that learners who had access to the AI code generator (the Codex group) were able to successfully generate code”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“1.8x higher correctness”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They performed signifcantly better on code-authoring tasks (1.15x increased progress”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“0.57x less time”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“0.59x less errors”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The comparison is showing that most of the diferences between conditions (in terms of efect size) are appearing on the left side of this table, for the high performers.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.3 The Efect of Prior Programming Competency
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Learners submitted the AI-generated code without any modifcation for 49% of the tasks (503/1015).”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.3 AI Code Generator Usage.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, learners in the Codex group on average scored 20.8% higher on arrays in terms of correctness score (Codex: M=50%, SD=36.8%;Baseline: M=29.2%,SD=36.3%;t(39)=2.34,p=.022, d=0.57)”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.4 Modifying Tasks (Training Phase).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, learners mostly felt that they learned about Python programming concepts after using the code generator.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.4.2 Perceptions on Using AI Code Generator.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 21
- Probably not a substantial claim. 20
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This could potentially explain how step-by-step usage of the AI code generator could potentially reduce cognitive load and allow novices to learn faster due to more availability of their mental resources, specifcally for germane cognitive load which is responsible for creating mental models and storing long-term memory [46].”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6.1 How Novices Beneft from AI Coding Assistants
What the sources say
- Partly supportedcites [46] 46. Paul Kirschner, John Sweller, and Richard E Clark. 2006. Why unguided learning does not work: An analysis of the failure of discovery learning, problem-based learning, experiential learning and inquiry-based learning. Educational Psycholo- gist 41, 2 (2006), 75-86. sourceExperimental — accuracy not yet measured
In “Current Research Supporting Direct Guidance” (page unavailable), the paper says worked examples reduce working-memory load and direct resources toward learning problem-solving schemas. That supports the proposed mechanism for guided, step-by-step learning, but the paper does not address AI code generators or identify the freed resources specifically as germane cognitive load. (LLM, conf=0.85)
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and 12.7% higher on modifying tasks (Codex: M=47.5%,SD=31.6%;Baseline: M=34.8%,SD=29.0%;t(67)=1.70,p=.092, d=0.41), but neither of these reached statistical signifcance.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.2 Retention Post-Test.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and on average 16% higher on loops, but this did not reach statistical signifcance (Codex: M=50.2%,SD=35.7%;Baseline: M=33.8%,SD=27.7%;t(58)=2.08,p=.041, d=0.51).”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.1 Immediate Post-Test.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Learners in the Codex group also encountered slightly higher errors on modifying tasks, but this did not reach statistical signifcance (Codex: M=0.81, SD=0.85;Baseline: M=0.57, SD=0.70; t(67)=1.21,p=.229, d=0.30).”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.2 Retention Post-Test.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and 10.5% higher on loops (Codex: M=61%, SD=30.5%;Baseline: M=50.5%,SD=29.3%;t(58)=1.41,p=.161, d=0.34) however, neither of these results reached statistical signifcance.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.4 Modifying Tasks (Training Phase).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“64 indicated using a block-based programming environment like Scratch or Code.org”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2 Participants
plainn = 64- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some learners felt that the AI-generated code needed some explanations”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.4.2 Perceptions on Using AI Code Generator.
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“learners from the Codex group on average scored 9.0% higher on authoring tasks (Codex: M=59.1%,SD=34.8%;Baseline: M=49.8%,SD=32.4%;t(67)=1.13,p=.262, d=0.28)”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.2 Retention Post-Test.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“27 indicated taking a programming-related class in the past”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2 Participants
plainn = 27- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, semantic errors occurred infrequently in both groups (Codex: M=0.01, SD=0.02;Baseline: M=0.03, SD=0.05; t(67)=1.84,p=.071, d=0.44).”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.2 Authoring Tasks (Training Phase).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They also felt that generating the code that they needed did not require a lot of practice.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.4.2 Perceptions on Using AI Code Generator.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the retention test which was conducted one week later, learners from the Codex group performed slightly better on coding tasks and multiple-choice questions, although these results did not reach statistical signifcance”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“although neither of these diferences reached statistical signifcance.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.2 Retention Post-Test.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“learners from both groups fully completed the frst two topics (basics and data-types) and one learner from each group did not complete conditionals.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.1 Overall Completion Rates and Direct Hints.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the Codex group had more learners that fully completed the 18 tasks on loops (Codex: 27/33, Baseline: 23/36) and the 7 tasks on arrays (Codex: 23/33, Baseline: 9/36)”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.1 Overall Completion Rates and Direct Hints.
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When discussing our results, there were several metrics in which the diferences between the two groups did not reach statistical signifcance but there did seem to be some diference in means and confdence intervals.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 7 LIMITATIONS AND FUTURE WORK
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Learners employed various methods to generate code such as breaking down the task into subgoals and using the code generator to generate code for each of the subgoals step-by-step.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6 DISCUSSION
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Furthermore, although our study included many non-native English speakers, but they were mostly comfortable explaining things in English.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 7 LIMITATIONS AND FUTURE WORK
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The two graders fully agreed on 79% of the submissions”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.4 Data Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“with 16% of disagreements having a diference of 25% in which we averaged the two grades”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.4 Data Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“No common factors were identifed among the 21 students who dropped out in terms of disability, native language (six non-English) and computer or internet access.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2 Participants
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“One learner in each pair was randomly assigned to either the Codex or Baseline group.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2.1 Participant Condition Assignment.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But overall, this is a promising result: although learners in the Codex group used the documentation less and relied heavily on AI-generated code for the authoring tasks, they still did just as well, and in some cases better, in manual code modifcation tasks.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.4 Modifying Tasks (Training Phase).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“two independent researchers graded each submitted solution using a simple and consistent grading rubric of deducting 25% for each major issue or missing part in their fnal submission”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.4 Data Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This shows that the AI code generator produces high quality results, but this depends on the quality of the prompt message.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 3.4 Quality of AI Generated Code
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“participants were divided into two groups using a matched-groups design [13,60].The ninety participants who completed the frst session were divided into two groups to have similar means and variances based on their pre-test scores on Scratch.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2.1 Participant Condition Assignment.
What the sources say
- Not checked — no full text of the sourcecites [13] 13. Miriam Bruhn and David McKenzie. 2009. In pursuit of balance: Randomization in practice in development feld experiments. American economic journal: applied economics 1, 4 (2009), 200-232. source
- Not checked — no full text of the sourcecites [60] 60. I Scott MacKenzie. 2012. Human-computer interaction: An empirical research perspective. (2012).
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To ensure both groups received consistent and unbiased feedback on their submissions, the grading dashboard did not display any information that would reveal the identity of learners or their group”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.1.2 Training Phase.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the 5% of the disagreements which were more than 25%, the two graders met again to resolve each of their disagreements.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.4 Data Analysis
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By generating code from simple sentences instead of formal and syntactically fxed specifcations, these AI Coding Assistants may lower the barriers to entry into programming.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“69 learners (21 female/48 male) ages 10-17 (M=12.5;SD=1.8) successfully completed all three phases of the study”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2 Participants
plainn = 90- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“AI code generators allowed novice programmers to perform better and faster with less frustration when writing code”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 8 CONCLUSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The lack of statistical fndings for such metrics does not necessarily mean there is no efect present, it could potentially be due to the sample size of our study.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 7 LIMITATIONS AND FUTURE WORK
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, our qualitative feedback shows that having access to the AI code generator reduced stress and discouragement in addition to improving their eagerness to continue programming later on.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6 DISCUSSION
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Alpha levels for per-topic comparisons are adjusted using Bonferroni correction ( = 0.05/5 = 0.01).”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 45B
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Prior research has shown that subgoallabeled instructional material reduces extraneous cognitive load imposed on novices learning programming by chunking problemsolving steps and promoting self-explanation [61].”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6.1 How Novices Beneft from AI Coding Assistants
What the sources say
- Not checked — no full text of the sourcecites [61] 61. Lauren E Margulieux, Mark Guzdial, and Richard Catrambone. 2012. Subgoal- labeled instructional material improves performance and transfer in learning to develop mobile applications. In Proceedings of the ninth annual international conference on International computing education research. 71-78. source
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results from code-authoring tasks during the training phase indicate that using AI code generators can significantly increase task completion, improve correctness score, reduce encountered errors, and reduce task completion time.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6 DISCUSSION
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although learners in the Baseline group used the documentation more frequently, they had to start each task by creating a new program from scratch before getting to the modify portion of the activity, and thus encountered more errors.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6.1 How Novices Beneft from AI Coding Assistants
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“prior research has even argued for switching away from statistical signifcance testing in favor of reporting informative charts with efect sizes and confdence intervals and nuanced interpretations [20]”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 7 LIMITATIONS AND FUTURE WORK
What the sources say
- Not checked — no full text of the sourcecites [20] 20. Pierre Dragicevic. 2015. HCI Statistics without p-values. Ph. D. Dissertation. Inria.
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Their higher prior programming knowledge meant that they probably had more code-reading and tracing skills which means that they were able to understand that code better, and then perform better during the next modifcation or code-creation tasks.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6.1 How Novices Beneft from AI Coding Assistants
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper relies on this without support.
The AI code generator (Codex) used in the study is representative of AI code generators in general.
“Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming”
“Similarly, AI code generators may reduce the cognitive load when learning to program, but its impact on the learning experience has not been studied.”
The paper relies on this without support.
Performance on the Coding Steps tasks and multiple-choice questions reflects learning of introductory Python programming.
“To answer these questions, we conducted a controlled study with 69 young learners, ages 10-17 (M=12.5,SD=1.8) with no prior textbased programming experience.Half of them used an AI Coding Assistant (Codex) to learn Python, and the other half did not.”
“The tasks in this phase included fve code-authoring tasks, fve code modifcation tasks, and then 40 multiple-choice questions”
The paper relies on this without support.
The AI code generator's quality measured on the 45 tasks before the study is representative of its quality during the actual study sessions.
“To measure the quality of the AI code generator, we entered the task description of the 45 code-authoring tasks into the pre-conditioned model (described in Figure 3) and evaluated the accuracy and amount of required manual modifcations on the generated code.”
The paper relies on this without support.
The learning effects and support provided by AI code generators have not been studied in the context of introductory programming.
“To date, the learning efects and support provided by AI code generators have not been studied in the context of introductory programming.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The propensity of learners to use GenAI outputs has been shown to vary significantly by exercise topic [Kazemitabaar et al. 2023; Mailach et al. 2025].”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Kazemitabaar et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Generative AI can change how much students learn through two margins.”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Kazemitabaar et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While research has demonstrated that GenAI can support learning and foster creativity, excessive dependence on these tools may undermine the development of algorithmic thinking, error detection, and analytical reasoning (Kazemitabaar et al., 2023;Mogavi et al., 2024;Zviel-Girshin, 2024).”
From Integrating Generative AI into Programming Education: Student Perceptions and the Challenge of Correcting AI Errors (2025) · cites this paper as Kazemitabaar et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research suggests that integrating AI tools alongside conventional instructional methods enables students to build a comprehensive skill set while developing the capacity to critique and refine automated outputs (Kazemitabaar et al., 2023).”
From Integrating Generative AI into Programming Education: Student Perceptions and the Challenge of Correcting AI Errors (2025) · cites this paper as Kazemitabaar et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Furthermore, reliance on code generation can negatively impact code authoring skills [Jošt et al. 2024; Kazemitabaar et al. 2023].”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Kazemitabaar et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“These numbers align with findings by Kazemitabaar et al. [2023], who observed an average similarity of 63% ( 𝜎 = 42%) between generated and submitted code, albeit using the Jaccard similarity measure.”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Kazemitabaar et al. [2023]
The cited source says what the claim attributes to it.
In the citing paper
“Third, in many designs students choose how much time to spend on the task (e.g., Kazemitabaar et al., 2023; Kestin et al., 2025; Kreijkes et al., 2026), so the estimated effects combine the two margins: time allocation and learning productivity.”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Kazemitabaar et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“no detectable difference on an immediate unassisted post-test ( -0.05 SD; the baseline group scored slightly higher, 62.9 versus 61.3 percent)”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Kazemitabaar et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“only a marginal difference on a one-week retention test of code modification (0.41 SD).”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Kazemitabaar et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“AI access produced large performance gains during training (1.67 SD)”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Kazemitabaar et al. (2023)
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, as the results are fairly consistent with similar surveys around the time [Kazemitabaar et al. 2023; Prather et al. 2023a; Zastudil et al. 2023] and we were commonly able to observe behavior that could be considered unethical in an academic context [Prather et al. 2023a], we do not believe this to be a major limitation.”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Kazemitabaar et al. 2023
Experimental — accuracy not yet measuredIn the citing paper
“Interacting with an LLM has been observed to lead to higher performance in code writing tasks but to little differences in learning gains [18].”
From Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study (2024) · cites this paper as [18]
In the citing paper
“They found that students who used AI code generators performed significantly better (1.15x progress, 0.59x errors, 1.8x higher correctness, 0.57x time spent) without negative effects on learning.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [108]
In the citing paper
“Recent research has shown promising possibilities for providing students with LLM partners in pair programming, given the right context and with the right scaffolding and support [44,108,159].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [108]
In the citing paper
“The models might generate code students do not understand [108] or may distract them with large blocks of text they did not write [159].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [108]
In the citing paper
“Also very concerning, Kazemitabaar et al. [108] found that students with more knowledge benefited more from code generation than students with less knowledge.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [108]
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“t(39)=2.78,p=.007”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.1 Immediate Post-Test.
Reported p = .007 · recomputed p = .008
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“t(39)=2.34,p=.022”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.4 Modifying Tasks (Training Phase).
Reported p = .022 · recomputed p = .024
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“t(67)=1.70,p=.092”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.2 Retention Post-Test.
Reported p = .092 · recomputed p = .094
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“t(58)=1.41,p=.161”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.4 Modifying Tasks (Training Phase).
Reported p = .161 · recomputed p = .164
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“t(67)=0.54,p=.67”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2.1 Participant Condition Assignment.
Reported p = .67 · recomputed p = .591
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“t(39)=2.02,p=.047”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.2 Retention Post-Test.
Reported p = .047 · recomputed p = .050
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“t(67)=1.28,p=.202”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.4 Modifying Tasks (Training Phase).
Reported p = .202 · recomputed p = .205
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upt(67) = 0.54reported p = .67 · recomputed p = .591
“t(67)=0.54,p = .67”
- Doesn’t add upt(67) = 1.28reported p = .202 · recomputed p = .205
“t(67)=1.28,p = .202”
- Doesn’t add upt(39) = 2.34reported p = .022 · recomputed p = .024
“t(39)=2.34,p = .022”
- Doesn’t add upt(58) = 1.41reported p = .161 · recomputed p = .164
“t(58)=1.41,p = .161”
- Doesn’t add upt(39) = 2.78reported p = .007 · recomputed p = .008
“t(39)=2.78,p = .007”
- Doesn’t add upt(67) = 1.7reported p = .092 · recomputed p = .094
“t(67)=1.70,p = .092”
- Doesn’t add upt(39) = 2.02reported p = .047 · recomputed p = .050
“t(39)=2.02,p = .047”
- Not confirmedt(67) = 0.058reported p = .953 · recomputed p = .954
“t(67)=0.058,p = .953”
- Not confirmedt(67) = 1.21reported p = .228 · recomputed p = .231
“t(67)=1.21,p = .228”
- Consistentt(67) = 2.8reported p = .006 · recomputed p = .007
“t(67)=2.8,p = .006”
- Consistentt(67) = 0.25reported p = .805 · recomputed p = .803
“t(67)=0.25,p = .805”
- Consistentt(67) = 0.53reported p = .6 · recomputed p = .598
“t(67)=0.53,p = .60”
- Consistentt(67) = 1.15reported p = .25 · recomputed p = .254
“t(67)=1.15,p = .25”
- Consistentt(67) = 6.92reported p < .001 · recomputed p < .001
“t(67)=6.92,p < .001”
- Consistentt(67) = 6.4reported p < .001 · recomputed p < .001
“t(67)=6.40,p < .001”
- Consistentt(67) = 5.48reported p < .001 · recomputed p < .001
“t(67)=5.48,p < .001”
- Consistentt(67) = 3.03reported p = .004 · recomputed p = .003
“t(67)=3.03,p = .004”
- Consistentt(67) = 3.65reported p = .001 · recomputed p < .001
“t(67)=3.65,p = .001”
- Consistentt(67) = 0.98reported p = .331 · recomputed p = .331
“t(67)=0.98,p = .331”
- Consistentt(67) = 1.84reported p = .071 · recomputed p = .070
“t(67)=1.84,p = .071”
- Consistentt(67) = 1.53reported p = .129 · recomputed p = .131
“t(67)=1.53,p = .129”
- Consistentt(67) = 0.91reported p = .367 · recomputed p = .366
“t(67)=0.91,p = .367”
- Consistentt(39) = 3.53reported p = .001 · recomputed p = .001
“t(39)=3.53,p = .001”
- Consistentt(67) = 0.2reported p = .838 · recomputed p = .842
“t(67)=0.20,p = .838”
- Consistentt(58) = 2.08reported p = .041 · recomputed p = .042
“t(58)=2.08,p = .041”
- Consistentt(67) = 1.13reported p = .262 · recomputed p = .263
“t(67)=1.13,p = .262”
- Consistentt(67) = 2.08reported p = .042 · recomputed p = .041
“t(67)=2.08,p = .042”
- Consistentt(67) = 1.21reported p = .229 · recomputed p = .231
“t(67)=1.21,p = .229”
- Consistentt(67) = 4.1reported p < .001 · recomputed p < .001
“t(67)=4.10,p < .001”
- Consistentt(67) = 1.54reported p = .129 · recomputed p = .128
“t(67)=1.54,p = .129”
- Consistentt(58) = 2.09reported p = .04 · recomputed p = .041
“t(58)=2.09,p = .04”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.1 Study Procedure
“The study lasted three weeks and included ten 90-minute sessions, with one session per day, spanning over three (consecutive) weeks.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.1.1 Introduction Phase.
“The frst session in the training phase was 2-hours long and included a one-hour introduction to the basic concepts of programming and computational thinking using Scratch.”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
(1978) · Medical Entomology and Zoology · Cited by 53,488
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Wing, J. M. (2006) · Communications of the ACM · doi:10.1145/1118178.1118215 · Cited by 8,102
Kirschner, P. A., Sweller, J., & Clark, R. E. (2006) · Educational Psychologist · doi:10.1207/s15326985ep4102_1 · Cited by 7,776
Using Effect Size—or Why the P Value Is Not Enough
Sullivan, G. M., & Feinn, R. (2012) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-12-00156.1 · Cited by 5,553 · cited 1×
“Finally, Cohen’s d is reported for effect size [78].”
Resnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
Computational Thinking in K–12
Grover, S., & Pea, R. D. (2013) · Educational Researcher · doi:10.3102/0013189x12463051 · Cited by 3,034
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824 · cited 2×
“For example, models like OpenAI Codex [14], Microsoft CodeBERT [27], Google PaLM [16], DeepMind AlphaCode [53] have been trained on millions of lines of publicly available code (e.g., on Github).”
Research works printed but not linked: 6
- Amazon Web Services. 2022. CodeWhisperer: ML-powered coding companion. https://aws.amazon.com/codewhisperer/. [Online; accessed 9-September-2022].
- 10.5555/364133.364161 · DOI
- Paul Jaccard. 1901. Étude comparative de la distribution florale dans une portion des Alpes et des Jura. Bull Soc Vaudoise Sci Nat 37 (1901), 547–579.
- Mitchel Resnick. 2014. Give P’sa chance: Projects, peers, passion, play. In Constructionism and creativity: Proceedings of the third international constructionism conference. Austrian computer society, Vienna. 13–20.
- Mitchel Resnick and David Siegel. 2015. A different approach to coding. International Journal of People-Oriented Programming 4, 1(2015), 1–4.
- Tabnine. 2022. Tabnine: AI assistant for software developers. https://www.tabnine.com/. [Online; accessed 9-September-2022].
Cited by
Sorted by citations
Bahroun, Z., Anane, C., Ahmed, V., & Zacca, A. (2023) · Sustainability · doi:10.3390/su151712983 · Cited by 784
“Kazemitabaar et al. (2023) [19] Demonstrates the positive influence of AI code generators, like OpenAI Codex, on novice programmers, highlighting improvements in performance, computational thinking skills, and motivation.”
BackgroundThe Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
“studied how novice programmer productivity and learning is affected by the use of AI code generators [101].”
BackgroundInfluentialUsing an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
Computing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316
“Depending on the prompt, the resulting programs used varied programmatic structures, while ultimately favoring expected methods for each problem variation (i.e., for-loops for processing lists, and while-loops for processing standard input).”
BackgroundThe Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
Kazemitabaar, M., Ye, R., Wang, X., Henley, A. Z., Denny, P., Craig, M., & Grossman, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642773 · Cited by 290
“Kazemitabaar et al. developed Coding Steps to explore the use of LLM-based code generators for supporting learners in introductory programming [27].”
MethodSelf-citationLau, S., & Guo, P. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600138 · Cited by 241
“ran a controlled study with 69 pre-college students where half used Codex to learn Python and the other half did not [53].”
BackgroundCodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes
Liffiton, M., Sheese, B. E., Savelka, J., & Denny, P. (2023) · Proceedings of the 23rd Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631830 · Cited by 192
“[17], we deployed our tool in the classroom; however, our evaluation ran for 12 weeks, and we explore how students interact with it outside of scheduled class sessions.”
MethodPrompt Problems: A New Programming Exercise for the Generative AI Era
Denny, P., Leinonen, J., Prather, J., Luxton-Reilly, A., Amarouche, T., Becker, B. A., & Reeves, B. N. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630909 · Cited by 187
Samala, A. D., Rawas, S., Wang, T., Reed, J. M., Kim, J., Howard, N.-J., & Ertz, M. (2025) · Education and Information Technologies · doi:10.1007/s10639-024-12936-0 · Cited by 229
“Caution should be advised, however, since ChatGPT often “lacks the medical expertise and context needed to understand the complex relationships between different conditions and treatment fully” (p.33) (Kazemitabaar et al., 2023).”
Trust in Generative AI among students: An Exploratory Study
Amoozadeh, M., Daniels, D., Nam, D., Kumar, A., Chen, S., Hilton, M., Ragavan, S. S., & Alipour, M. A. (2023) · arXiv · doi:10.48550/arxiv.2310.04631 · Cited by 143
Savelka, J., Agarwal, A., An, M., Bogart, C., & Sakr, M. (2023) · Proceedings of the 2023 ACM Conference on International Computing Education Research V.1 · doi:10.1145/3568813.3600142 · Cited by 141
“studied learners using OpenAI’s Codex during traditional code creation tasks, and demonstrated the use of Codex did not harm their performance, with experienced students performing significantly better [19].”
MethodBridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs
Subramonyam, H., Pea, R. D., Pondoc, C., Agrawala, M., & Seifert, C. M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642754 · Cited by 247
“Without a sense of the outcome of its learning process, it is impossible for another mind to predict LLM output.”
BackgroundKazemitabaar, M., Hou, X., Henley, A. Z., Ericson, B. J., Weintrop, D., & Grossman, T. (2023) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631806 · Cited by 202
Self-citationA Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education
Doughty, J., Wan, Z., Bompelli, A., Qayum, J., Wang, T., Zhang, J., Zheng, Y., Doyle, A., Sridhar, P., Agarwal, A., et al. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636256 · Cited by 132
How Large Language Models Will Disrupt Data Management
Fernandez, R. C., Elmore, A. J., Franklin, M. J. M., Krishnan, S., & Tan, C. (2023) · Proceedings of the VLDB Endowment · doi:10.14778/3611479.3611527 · Cited by 134
“While the technology is still evolving, a few aspects of it are clear right now: (1) LLMs can provide a bridge between programming languages and natural language that have far more functionality than past efforts [40, 43, 67]; (2) LLMs consolidate intelligence – the same model/tooling is useful across the entire data…”
BackgroundHow Knowledge Workers Think Generative AI Will (Not) Transform Their Industries
Woodruff, A., Shelby, R. M., Kelley, P. G., Rousso-Schindler, S., Smith-Loud, J., & Wilcox, L. G. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642700 · Cited by 162
“A limited body of research has examined impacts on specific professional sectors, notably creative [44, 68, 77] and educational [5, 85, 100] domains.”
BackgroundDoes ChatGPT Help With Introductory Programming?An Experiment of Students Using ChatGPT in CS1
Xue, Y., Chen, H., Bai, G. R., Tairas, R., & Huang, Y. (2024) · Proceedings of the 46th International Conference on Software Engineering: Software Engineering Education and Training · doi:10.1145/3639474.3640076 · Cited by 96
Prather, J. E., Leinonen, J., Kiesler, N., Benario, J. G., Lau, S., MacNeil, S., Norouzi, N., Opel, S., Pettit, V., Porter, L., et al. (2025) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3689187.3709614 · Cited by 167
The Effects of Generative AI on Computing Students’ Help-Seeking Preferences
Hou, I., Mettille, S., Man, O., Li, Z., Zastudil, C., & MacNeil, S. (2024) · Proceedings of the 26th Australasian Computing Education Conference · doi:10.1145/3636243.3636248 · Cited by 97
“Especially as many other use cases are rapidly emerging in computing education contexts [3, 9, 14, 29, 35, 41, 52, 53, 58, 59, 61].”
BackgroundStudents’ Experiences of Using ChatGPT in an Undergraduate Programming Course
Haindl, P., & Weinberger, G. (2024) · IEEE Access · doi:10.1109/access.2024.3380909 · Cited by 108
“Itcanbe summarized that only two studies [20], [21] incorporated students’ experiences about using ChatGPT in programming education.”
BackgroundMethodDesirable Characteristics for AI Teaching Assistants in Programming Education
Denny, P. C., MacNeil, S., Šavelka, J., Porter, L., & Luxton-Reilly, A. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653574 · Cited by 134
Large Language Models in Computer Science Education: A Systematic Literature Review
Raihan, M. N., Siddiq, M. L., Santos, J. C. S., & Zampieri, M. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701863 · Cited by 162
“Regarding the successful completion of the tasks, we found mixed results regarding the use of LLMs. Studies have shown that LLMs can help students solve introductory programming problems, repair buggy code [68, 80, 117], and help write better code [62, 64, 93].”
Patterns of Student Help-Seeking When Using a Large Language Model-Powered Programming Assistant
Sheese, B. E., Liffiton, M., Šavelka, J., & Denny, P. C. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636249 · Cited by 103
“This behavior somewhat mirrors the low-effort submissions noted by Kazemitabaar et al. in their LLM-powered tool [15].”
MethodResultExploring the Potential of Large Language Models to Generate Formative Programming Feedback
Kiesler, N., Lohr, D., & Keuning, H. (2023) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie58773.2023.10343457 · Cited by 113
“Kazemitabaar et al. [17], for example, conducted experiments with novice programmers.”
BackgroundSaihi, A., Ben-Daya, M., Hariga, M., & As'ad, R. (2024) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2024.100274 · Cited by 75
ChatGPT in the Classroom. Exploring Its Potential and Limitations in a Functional Programming Course
Popovici, M.-D. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2023.2269006 · Cited by 70
“The paper of Kazemitabaar et al. (2023) performs a study aimed at assessing how students improve when using Copilot.”
Boguslawski, S., Deer, R., & Dawson, M. G. (2025) · Information and Learning Sciences · doi:10.1108/ils-10-2023-0163 · Cited by 66
“There are also concerns that overreliance on generative AI early in the learning process may hinder the development of the meta-cognitive skills essential to good programming (Denny et al. , 2023) and reduce satisfaction derived from solving a problem without support (Kazemitabaar et al. , 2023).”
BackgroundHow Beginning Programmers and Code LLMs (Mis)read Each Other
Nguyen, S., Babe, H. M., Zi, Y., Guha, A., Anderson, C. J., & Feldman, M. Q. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642706 · Cited by 120
Charting the Future of AI in Project-Based Learning: A Co-Design Exploration with Students
Zheng, C., Yuan, K., Guo, B., Hadi Mogavi, R., Peng, Z., Ma, S., & Ma, X. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642807 · Cited by 70
"I look at it as the king of knowledge": How Blind People Use and Understand Generative AI Tools
Adnin, R., & Das, M. (2024) · The 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675631 · Cited by 73
“With the proliferation of large language models (LLMs), GenAI tools like ChatGPT, Copilot, and Gemini have gained immense popularity across various domains, including education [36, 49, 83, 99], programming [16, 54, 92], communication [14], and creative work [45, 56, 84].”
MethodCambaz, D., & Zhang, X. (2024) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3626252.3630958 · Cited by 99
“Six studies [2, 5, 7, 11, 20, 25] used OpenAI”
BackgroundMethodInfluentialProgramming with ChatGPT: How far can we go?
Bucaioni, A., Ekedahl, H., Helander, V., & Nguyen, P. T. (2024) · Machine Learning with Applications · doi:10.1016/j.mlwa.2024.100526 · Cited by 64
How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering
Choudhuri, R., Liu, D., Steinmacher, I., Gerosa, M. A., & Sarma, A. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639201 · Cited by 99
Saihi, A., Ben‐Daya, M., & Hariga, M. A. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-13023-0 · Cited by 67
Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study
Sheard, J., Denny, P., Hellas, A., Leinonen, J., Malmi, L., & Simon (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630880 · Cited by 54
“Interacting with an LLM has been observed to lead to higher performance in code writing tasks but to little differences in learning gains [18].”
BackgroundPang, R. Y., Schroeder, H., Smith, K. S., Barocas, S., Xiao, Z., Tseng, E., & Bragg, D. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713726 · Cited by 112
Next-Step Hint Generation for Introductory Programming Using Large Language Models
Roest, L., Keuning, H., & Jeuring, J. T. (2024) · Australasian Computing Education Conference · doi:10.1145/3636243.3636259 · Cited by 95
“GPT-4 outperforms other existing LLMs on several academic benchmarks [30].”
BackgroundPan, Z., Xie, Z., Liu, T., & Xia, T. (2024) · Systems · doi:10.3390/systems12050176 · Cited by 63
Automatic Programming: Large Language Models and Beyond
Lyu, M. R.-T., Ray, B., Roychoudhury, A., Tan, S. H., & Thongtanunam, P. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3708519 · Cited by 83
“They may also execute the code or run a static analyzer to help them validate the suggestions.”
Koutcheme, C., Dainese, N., Sarsa, S., Hellas, A., Leinonen, J., & Denny, P. C. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653612 · Cited by 82
Denecke, K., Glauser, R., & Reichenpfader, D. (2023) · Trends in Higher Education · doi:10.3390/higheredu2040039 · Cited by 75
Cheng, R., Wang, R., Zimmermann, T., & Ford, D. (2024) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3651990 · Cited by 47
“In the context of novice learning programming, a recent study found that learners with access to OpenAI Codex were able to produce code of higher accuracy and greater efficiency [ 38 ].”
BackgroundDunder, N., Lundborg, S., Wong, J., & Viberg, O. (2024) · International Conference on Learning Analytics and Knowledge (LAK) · doi:10.1145/3636555.3636882 · Cited by 42
“On the one hand, early research exploring teachers’ attitudes toward using ChatGPT has shown that educators are generally cautious in their approach to using ChatGPT [15].”
BackgroundStudent-AI Interaction: A Case Study of CS1 students
Amoozadeh, M., Nam, D., Prol, D., Alfageeh, A., Prather, J. E., Hilton, M. L., Ragavan, S. S., & Alipour, M. A. (2024) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699567 · Cited by 68
“Liu et al. [25] investigated the integration of Generative AI tools in the CS50 course at Harvard University, demonstrating how they can significantly enhance the learning experience for students in introductory programming classes.”
Bhattacharjee, A., Zeng, Y., Xu, S. Y., Kulzhabayeva, D., Ma, M., Kornfield, R., Ahmed, S. I., Mariakakis, A. T., Czerwinski, M. P., Kuzminykh, A., et al. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642081 · Cited by 58
“In programming courses, LLMs might act as adaptable tutors that provide hints or suggest debugging strategies when students encounter errors in their code [30, 55, 68, 70].”
BackgroundJordan, M., Ly, K., & Soosai Raj, A. G. (2024) · Proceedings of the 55th ACM Technical Symposium on Computer Science Education V. 1 · doi:10.1145/3626252.3630897 · Cited by 39
Kohen-Vacs, D., Usher, M., & Jansen, M. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00496-4 · Cited by 39
“While research has demonstrated that GenAI can support learning and foster creativity, excessive dependence on these tools may undermine the development of algorithmic thinking, error detection, and analytical reasoning (Kazemitabaar et al., 2023; Mogavi et al., 2024; Zviel-Girshin, 2024).”
Lepp, M., & Kaimre, J. (2025) · Computers in Human Behavior Reports · doi:10.1016/j.chbr.2025.100642 · Cited by 73
Ouyang, F., Guo, M., Zhang, N., Bai, X., & Jiao, P. (2024) · IEEE Transactions on Learning Technologies · doi:10.1109/tlt.2024.3486749 · Cited by 39
Investigates the Impact of AI-generated Code Tools on Software Readability Code Quality Factor
Fawareh, H. J., Al-Shdaifat, H. M., Al-Refai, M., et al. (2024) · International Arab Conference on Information Technology (ACIT) · doi:10.1109/acit62805.2024.10876897 · Cited by 39
“The research highlighted the potential of AI tools to enhance learning and code quality for novice programmers [9].”
Kim, Y. W., Cha, M. C., Yoon, S. H., & Lee, S. C. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2375701 · Cited by 43
“In a study conducted by Kazemitabaar et al. (2023), beginner programmers utilizing Codex experienced a 1.15 times increase in the rate of code completion and attained scores 1.8 times higher than those in the baseline condition.”
InfluentialBjelobaba, S., Waddington, L., Perkins, M., Foltýnek, T., Bhattacharyya, S., & Weber-Wulff, D. (2025) · International Journal for Educational Integrity · doi:10.1007/s40979-025-00191-w · Cited by 96
“One recent study showed that these tools significantly improved novice learners’ code-authoring performance without negatively affecting their manual coding skills or retention (Kazemitabaar et al. 2023).”
Garcia, M. B. (2025) · Education and Information Technologies · doi:10.1007/s10639-025-13452-5 · Cited by 54
“Kazemitabaar et al. (2023) observed that excessive reliance on such tools could hinder the learning process.”
Software engineering education in the era of conversational AI: current trends and future directions
Sengul, C., Neykova, R., & Destefanis, G. (2024) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2024.1436350 · Cited by 63
To Rely or Not to Rely? Evaluating Interventions for Appropriate Reliance on Large Language Models
Bo, J. Y., Wan, S., & Anderson, A. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714097 · Cited by 36
Padiyath, A., Hou, X., Pang, A., Viramontes Vargas, D., Gu, X., Nelson-Fromm, T., Wu, Z., Guzdial, M., & Ericson, B. (2024) · Proceedings of the 2024 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3632620.3671098 · Cited by 35
“Although previous studies have investigated novice use of modified LLM assistants [33, 69], relatively few have explored the effects of using the LLMs in novice students’ programming education [43, 44].”
Self-citationDebugging with an AI Tutor: Investigating Novice Help-seeking Behaviors and Perceived Learning
Yang, S., Zhao, H., Xu, Y., Brennan, K. A., & Schneider, B. (2024) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3632620.3671092 · Cited by 37
How Do Programming Students Use Generative AI?
Rahe, C., & Maalej, W. (2025) · Proceedings of the ACM on Software Engineering · doi:10.1145/3715762 · Cited by 45
“Furthermore, reliance on code generation can negatively impact code authoring skills [Jošt et al. 2024; Kazemitabaar et al. 2023].”
InfluentialComputing education using generative artificial intelligence tools: A systematic literature review
Agbo, F. J., Olivia, C., Oguibe, G. F., Sanusi, I. T., & Sani, G. (2025) · Computers and Education Open · doi:10.1016/j.caeo.2025.100266 · Cited by 59
Kazemitabaar, M., Huang, O., Suh, S., Henley, A. Z., & Grossman, T. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712104 · Cited by 70
“They have been found particularly effective in reducing the time needed to reach proficiency [81], leading to the development of interactive versions [32].”
Self-citationStudent Perspectives on Using a Large Language Model (LLM) for an Assignment on Professional Ethics
Grande, V., Kiesler, N., & Rodríguez, M. A. F. (2024) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3649217.3653624 · Cited by 35
“(5) Studying the interactions between programmers (including students) and LLMs w.r.t. interaction patterns [4, 33, 44], productivity [19, 28], code security [37], and students’ use of code explanations [27, 29].”
BackgroundPirzado, F. A., Ahmed, A., Mendoza-Urdiales, R. A., & Terashima‐Marín, H. (2024) · IEEE Access · doi:10.1109/access.2024.3443621 · Cited by 41
“[1], [3], [14], [22], [37], [40], [43], [44], [48], [50], [51], [52], [53], [66], [67], [72], [78], [90].”
ResultVirvou, M. K., & Tsihrintzis, G. A. (2023) · International Conference on Information, Intelligence, Systems & Applications (IISA) · doi:10.1109/iisa59645.2023.10345949 · Cited by 34
Hou, I., Man, O., Hamilton, K., Muthusekaran, S., Johnykutty, J., Zadeh, L., & MacNeil, S. (2025) · Proceedings of the 30th ACM Conference on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3724363.3729024 · Cited by 37
“This growth may beexplained bythe many associatedbenefits, suchas personalizedexplanations [5, 30, 34], intelligent teaching assistants [12, 24, 25, 31], and support for identifying bugs and debugging code [33, 57].”
Pu, K., Lazaro, D., Arawjo, I., Xia, H., Xiao, Z., Grossman, T., & Chen, Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713357 · Cited by 67
“Large language models (LLMs) have enabled generative programming assistance tools to provide powerful in-situ developer support for novices and experts alike [54–56, 70, 86].”
Self-citationExploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices
Xiao, R., Hou, X., & Stamper, J. C. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650937 · Cited by 46
“Meanwhile, Kazemitabaar et al. [9] pointed out that having access to the AI code generator and generated code snippets did not impede learning gains.”
KOGI: A Seamless Integration of ChatGPT into Jupyter Environments for Programming Education
Kuramitsu, K., Obara, Y., Sato, M., & Obara, M. (2023) · ACM SIGPLAN International Symposium on SPLASH-E (SPLASH-E) · doi:10.1145/3622780.3623648 · Cited by 30
Keuning, H., Alpizar-Chacon, I., Lykourentzou, I., Beehler, L., Köppe, C., de Jong, I., & Sosnovsky, S. (2024) · Proceedings of the 24th Koli Calling International Conference on Computing Education Research · doi:10.1145/3699538.3699546 · Cited by 32
“While the first papers mainly focused on GPT’s and Codex’s abilities to generate solutions to standard programming problems, the topics expanded to educational content generation [22], student’s use of GenAI in the classroom [13, 15– 17], and its implications for computing education [2, 3, 18].”
Zamfirescu-Pereira, J. D., Qi, L., Hartmann, B., DeNero, J., & Norouzi, N. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701864 · Cited by 28
AI chatbots in programming education: guiding success or encouraging plagiarism
Akçapınar, G., & Sidan, E. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00203-7 · Cited by 52
“For example, in an experimental study conducted by Kazemitabaar et al. [13], it was aimed to measure the coding performance of using an AI programming assistant on students who were new to programming and had no previous programming knowledge.”
Choi, S., & Kim, H. (2024) · Education and Information Technologies · doi:10.1007/s10639-024-13107-x · Cited by 44
“For example, Ma et al. (2023) examined the relative advantages of interacting with an LLM versus a human counterpart, emphasizing the benefits of collaborative learning with LLMs. Kazemitabaar et al. (2023) showed the positive impact of using these tools in beginner programming classes in higher education.”
InfluentialEnhancing Programming Error Messages in Real Time with Generative AI
Kimmel, B., Geisert, A. L., Yaro, L., Gipson, B., Hotchkiss, R., Osae-Asante, S. K., Vaught, H., Wininger, G., & Yamaguchi, C. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3647967 · Cited by 38
“Kazemitabaar et al. found that students will utilize it to either explore new ways of doing a task or to attempt to accelerate their task completion time [10].”
Prather, J. E., Denny, P. C., Leinonen, J., Becker, B. A., Al-Bluwi, I., Caspersen, M. E., Craig, M., Keuning, H., Kiesler, N., Kohn, T., et al. (2023) · ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE) · doi:10.1145/3587103.3594206 · Cited by 26
“Students using these tools can also quickly become lost while reading code they didn’t write [5] or lazily drift from code suggestion to code suggestion without understanding what they are doing [9].”
BackgroundDoes ChatGPT Help Novice Programmers Write Better Code? Results From Static Code Analysis
Haindl, P., & Weinberger, G. (2024) · IEEE Access · doi:10.1109/access.2024.3445432 · Cited by 45
“In a recent study, Kazemitabaar et al. [24] examined the suitability of OpenAI Codex on supporting introductory programming for novice programmers.”
BackgroundGadde, A. (2025) · Journal of Computer Science and Technology Studies · doi:10.32996/jcsts.2025.7.4.66 · Cited by 31
Need Help? Designing Proactive AI Assistants for Programming
Chen, V., Zhu, A., Zhao, S., Mozannar, H., Sontag, D. A., & Talwalkar, A. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714002 · Cited by 72
“Prior studies investigating chat dialogue for programming assistance have always looked at the setting where people initiate questions to the assistant [10, 23, 24, 38, 44, 52].”
CodeA11y: Making AI Coding Assistants Useful for Accessible Web Development
Mowar, P., Peng, Y.-H., Wu, J., Steinfeld, A., & Bigham, J. P. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713335 · Cited by 31
“The percentages in the distribution column indicate the proportion of participants who either disagree (including ‘strongly disagree’, ‘disagree’ and ‘slightly disagree’) or agree (including ‘strongly agree’, ‘agree’ and ‘slightly agree’) with the provided statements.”
van den Broek, S., Sankaran, S., de Wit, J., & de Rooij, A. (2024) · Participatory Design Conference · doi:10.1145/3661455.3669868 · Cited by 27
“While this has been explored in several ways such as using AI as a copilot programmer [33, 54] collaboratively writing or designing with AI [47, 51], its potential has not been explored in multi-stakeholder co-design contexts.”
Rocks Coding, Not Development: A Human-Centric, Experimental Evaluation of LLM-Supported SE Tasks
Wang, W., Ning, H., Zhang, G., Liu, L., & Wang, Y. (2024) · Proceedings of the ACM on software engineering. · doi:10.1145/3643758 · Cited by 40
A Map of Exploring Human Interaction Patterns with LLM: Insights into Collaboration and Creativity
Li, J., Li, J., & Su, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-60615-1_5 · Cited by 29
Non-Expert Programmers in the Generative AI Future
Feldman, M. Q., & Anderson, C. J. (2024) · Annual Symposium on Human-Computer Interaction for Work (CHIWORK) · doi:10.1145/3663384.3663393 · Cited by 34
Literature Review on the Integration of Generative AI in Programming Education
Nathaniel, J., Oyelere, S. S., Suhonen, J. T., & Tedre, M. (2025) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-025-00524-3 · Cited by 45
“The development of higher-order thinking skills is important in programming edu - cation, as it advances critical thinking (Smolansky et al., 2023), problem-solving (Kazemitabaar et al., 2023a, 2023b), and creativity (Kothari, 2023).”
Tankelevitch, L., Glassman, E. L., He, J., Kazemitabaar, M., Kittur, A., Lee, M., Palani, S., Sarkar, A., Ramos, G., Rogers, Y., et al. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3706745 · Cited by 32
Self-citationThe Role of AI in Transforming Assessment Practices in Education
Vetrivel, S. C., Vidhyapriya, P., & Arun, V. P. (2024) · Advances in educational marketing, administration, and leadership book series · doi:10.4018/979-8-3693-5443-8.ch003 · Cited by 24
AI-Assisted Programming Tasks Using Code Embeddings and Transformers
Kotsiantis, S. B., Verykios, V. S., & Tzagarakis, M. M. (2024) · Electronics · doi:10.3390/electronics13040767 · Cited by 29
The Impact of Generative AI Coding Assistants on Developers Who Are Visually Impaired
Flores-Saviaga, C., Hanrahan, B. V., Imteyaz, K., Clarke, S., & Savage, S. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3714008 · Cited by 32
“• Programming Task: Following prior work [46], we asked participants to write a program that calculates the number of days until the user’s next birthday.”
Evaluating Language Models for Generating and Judging Programming Feedback
Koutcheme, C., Dainese, N., Sarsa, S., Hellas, A., Leinonen, J., Ashraf, S., & Denny, P. C. (2025) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3641554.3701791 · Cited by 43
“These include tools to generate improved error messages [25], aid with real-time debugging [13], and to help explain code [24, 26] and tailor next-step hints [37].”
Generative AI in Computer Science Education: Challenges and Opportunities
Franklin, D., Denny, P., Gonzalez-Maldonado, D. A., & Tran, M. (2025) · doi:10.1017/9781009581738 · Cited by 20
Nathaniel, J., Oyelere, S. S., Suhonen, J. T., & Tedre, M. (2025) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2025.100460 · Cited by 42
“…GenAI tools, uncritical use risks fostering overreliance, potentially weakening understanding of programming logic and higher-order thinking skills (HOTS) such as problem-solving, critical thinking, and creativity (Dickey, Bejarano, & Chirayu, 2024;Kazemitabaar et al., 2023; Zastudil et al., 2023).”
DBox: Scaffolding Algorithmic Programming Learning through Learner-LLM Co-Decomposition
Ma, S., Wang, J., Zhang, Y., Ma, X., & Wang, A. Y. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713748 · Cited by 39
“Participants experienced two conditions in a randomly assigned order: • DBox : Participants solved problems using the proposed DBox.”
InfluentialExperiences from Integrating Large Language Model Chatbots into the Classroom
Hellas, A., Leinonen, J., & Leppänen, L. (2024) · ACM Virtual Global Computing Education Conference (SIGCSE Virtual) · doi:10.1145/3649165.3690101 · Cited by 32
“Even so, students may quickly accept incorrect code suggestions and tinker with the code before discovering they do not need it, only to start over again [13, 32, 37].”
Students' Perspectives on AI Code Completion: Benefits and Challenges
Takerngsaksiri, W., Warusavitarne, C., Yaacoub, C., Keng Hou, M. H., & Tantithamthavorn, C. (2024) · 2024 IEEE 48th Annual Computers, Software, and Applications Conference (COMPSAC) · doi:10.1109/compsac61105.2024.00252 · Cited by 18
“Align with the previous work, Kazemitabaar et al. [9] investigated the performance of novice programmers using OpenAI Codex [17].”
BackgroundMethodWhodunit: Classifying Code as Human Authored or GPT-4 Generated - A case study on CodeChef problems
Idialu, O. J., Mathews, N. S., Maipradit, R., Atlee, J. M., & Nagappan, M. (2024) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3643991.3644926 · Cited by 33
“The use of AI assistants in introductory programming courses has been shown to increase the productivity of novice programmers in solving introductory programming problems, both in terms of improving the quality of their programs and easing the cognitive load and development effort required [32].”
BackgroundGenerating Programs Trivially: Student Use of Large Language Models
Prasad, S., Greenman, B., Nelson, T., & Krishnamurthi, S. (2023) · ACM Global Computing Education Conference (CompEd) · doi:10.1145/3576882.3617921 · Cited by 26
Generative AI Uses and Risks for Knowledge Workers in a Science Organization
Wagman, K. B., Dearing, M. T., & Chetty, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713827 · Cited by 23
“Multiple studies have found that generative AI tools increase productivity more for less skilled workers [2, 32], including for workers in consulting [9] and programming [6, 22, 36].”
Providing tailored reflection instructions in collaborative learning using large language models
Naik, A., Yin, J. R., Kamath, A., Ma, Q., Wu, T., Murray, R. C., Bogart, C., Sakr, M. F., & Rosé, C. P. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13548 · Cited by 16
Enhancing Computer Programming Education using ChatGPT- A Mini Review
Deriba, F. G., Sanusi, I. T., & Sunday, A. O. (2023) · Koli Calling International Conference on Computing Education Research · doi:10.1145/3631802.3631848 · Cited by 15
Performance, Workload, Emotion, and Self-Efficacy of Novice Programmers Using AI Code Generation
Gardella, N., Pettit, R., & Riggs, S. L. (2024) · Proceedings of the 2024 on Innovation and Technology in Computer Science Education V. 1 · doi:10.1145/3649217.3653615 · Cited by 15
“The AI-assisted students who performed best on their entrance exam also showed significant improvements in learning based on the summative exams, indicating that AIDEs can potentially provide learning benefits to more advanced learners [21] .”
InfluentialExamining Student and Teacher Perspectives on Undisclosed Use of Generative AI in Academic Work
Adnin, R., Pandkar, A., Yao, B., Wang, D., & Das, M. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713393 · Cited by 16
Metadata
- Authors
- Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T.
- Affiliations
- University of Toronto (CA)
- Department of Computer Science, University of Toronto, Canada
- University of Michigan (US)
- College of Education, College of Information Studies, University of Maryland, United States
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-23
- OpenAlex
- W4321162272
- Cited by
- 366
- References
- 96
- Citations by year
- 2026: 117 · 2025: 118 · 2024: 104 · 2023: 27
- FWCI
- 110.39
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- OpenAI Codex, novice programmers, introductory programming, code modification, knowledge retention, overreliance on AI, Scratch, Python, programming education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
