The DOI matches its Crossref record; 102 of the 102 references it cites were found in a registry or an index; 2 of the 10 sources its claims cite were checked for support.
Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Open accessAbstract
AI code generators like OpenAI Codex have the potential to assist novice programmers by generating code from natural language descriptions, however, over-reliance might negatively impact learning and retention. To explore the implications that AI code generators have on introductory programming, we conducted a controlled experiment with 69 novices (ages 10-17). Learners worked on 45 Python code-authoring tasks, for which half of the learners had access to Codex, each followed by a code-modification task. Our results show that using Codex significantly increased code-authoring performance (1.15x increased completion rate and 1.8x higher scores) while not decreasing performance on manual code-modification tasks. Additionally, learners with access to Codex during the training phase performed slightly better on the evaluation post-tests conducted one week later, although this difference did not reach statistical significance. Of interest, learners with higher Scratch pre-test scores performed significantly better on retention post-tests, if they had prior access to Codex.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
2 of 10 cited sources checked · 0 supported · 0 flagged · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Prior work in introductory programming has shown that code-reading and tracing skills are essential to problem-solving activities including code-writing [49,55,57].”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6.1 How Novices Beneft from AI Coding Assistants
What the sources say
- Not checked — no full text of the sourcecites [49] 49. Amruth N Kumar. 2013. A study of the infuence of code-tracing problems on code-writing skills. In Proceedings of the 18th ACM conference on Innovation and technology in computer science education. 183-188. source
- Not checked — no full text of the sourcecites [55] 55. Raymond Lister, Colin Fidge, and Donna Teague. 2009. Further evidence of a relationship between explaining, tracing and writing skills in introductory programming. Acm sigcse bulletin 41, 3 (2009), 161-165. source
- Partly supportedcites [57] 57. Mike Lopez, Jacqueline Whalley, Phil Robbins, and Raymond Lister. 2008. Rela- tionships between reading, tracing and writing skills in introductory program- ming. In Proceedings of the fourth international workshop on computing education research. 101-112. sourceExperimental — accuracy not yet measured
The Discussion (§6) reports significant associations between code-reading, code-tracing, and code-writing performance. But the paper presents a hierarchy of skills as a possibility, not a demonstrated necessity; its findings do not establish that reading and tracing are essential to problem-solving activities generally. (LLM, conf=0.88)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“learners who had access to the AI code generator (the Codex group) were able to successfully generate code”
- Also stated in: § 1 INTRODUCTION · plain
“Our results show that learners who had access to the AI code generator (the Codex group) were able to successfully generate code”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Also stated in: § 1 INTRODUCTION · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“1.8x higher correctness”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They performed signifcantly better on code-authoring tasks (1.15x increased progress”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“0.57x less time”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“0.59x less errors”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The comparison is showing that most of the diferences between conditions (in terms of efect size) are appearing on the left side of this table, for the high performers.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.3 The Efect of Prior Programming Competency
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Learners submitted the AI-generated code without any modifcation for 49% of the tasks (503/1015).”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.3 AI Code Generator Usage.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In fact, learners in the Codex group on average scored 20.8% higher on arrays in terms of correctness score (Codex: M=50%, SD=36.8%;Baseline: M=29.2%,SD=36.3%;t(39)=2.34,p=.022, d=0.57)”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.4 Modifying Tasks (Training Phase).
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, learners mostly felt that they learned about Python programming concepts after using the code generator.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.4.2 Perceptions on Using AI Code Generator.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passes
- It may state two separate claims as one. 21
- Probably not a substantial claim. 20
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This could potentially explain how step-by-step usage of the AI code generator could potentially reduce cognitive load and allow novices to learn faster due to more availability of their mental resources, specifcally for germane cognitive load which is responsible for creating mental models and storing long-term memory [46].”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6.1 How Novices Beneft from AI Coding Assistants
What the sources say
- Partly supportedcites [46] 46. Paul Kirschner, John Sweller, and Richard E Clark. 2006. Why unguided learning does not work: An analysis of the failure of discovery learning, problem-based learning, experiential learning and inquiry-based learning. Educational Psycholo- gist 41, 2 (2006), 75-86. sourceExperimental — accuracy not yet measured
In “Current Research Supporting Direct Guidance” (page unavailable), the paper says worked examples reduce working-memory load and direct resources toward learning problem-solving schemas. That supports the proposed mechanism for guided, step-by-step learning, but the paper does not address AI code generators or identify the freed resources specifically as germane cognitive load. (LLM, conf=0.85)
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and 12.7% higher on modifying tasks (Codex: M=47.5%,SD=31.6%;Baseline: M=34.8%,SD=29.0%;t(67)=1.70,p=.092, d=0.41), but neither of these reached statistical signifcance.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.2 Retention Post-Test.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and on average 16% higher on loops, but this did not reach statistical signifcance (Codex: M=50.2%,SD=35.7%;Baseline: M=33.8%,SD=27.7%;t(58)=2.08,p=.041, d=0.51).”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.1 Immediate Post-Test.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Learners in the Codex group also encountered slightly higher errors on modifying tasks, but this did not reach statistical signifcance (Codex: M=0.81, SD=0.85;Baseline: M=0.57, SD=0.70; t(67)=1.21,p=.229, d=0.30).”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.2 Retention Post-Test.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and 10.5% higher on loops (Codex: M=61%, SD=30.5%;Baseline: M=50.5%,SD=29.3%;t(58)=1.41,p=.161, d=0.34) however, neither of these results reached statistical signifcance.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.4 Modifying Tasks (Training Phase).
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“64 indicated using a block-based programming environment like Scratch or Code.org”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2 Participants
Strength:plain· not yet calibratedn = 64Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Some learners felt that the AI-generated code needed some explanations”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.4.2 Perceptions on Using AI Code Generator.
Strength:plain· not yet calibratedSingle caseTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“learners from the Codex group on average scored 9.0% higher on authoring tasks (Codex: M=59.1%,SD=34.8%;Baseline: M=49.8%,SD=32.4%;t(67)=1.13,p=.262, d=0.28)”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.2 Retention Post-Test.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“27 indicated taking a programming-related class in the past”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2 Participants
Strength:plain· not yet calibratedn = 27Tests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, semantic errors occurred infrequently in both groups (Codex: M=0.01, SD=0.02;Baseline: M=0.03, SD=0.05; t(67)=1.84,p=.071, d=0.44).”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.2 Authoring Tasks (Training Phase).
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“They also felt that generating the code that they needed did not require a lot of practice.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.4.2 Perceptions on Using AI Code Generator.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In the retention test which was conducted one week later, learners from the Codex group performed slightly better on coding tasks and multiple-choice questions, although these results did not reach statistical signifcance”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“although neither of these diferences reached statistical signifcance.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.2.2 Retention Post-Test.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“learners from both groups fully completed the frst two topics (basics and data-types) and one learner from each group did not complete conditionals.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.1 Overall Completion Rates and Direct Hints.
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the Codex group had more learners that fully completed the 18 tasks on loops (Codex: 27/33, Baseline: 23/36) and the 7 tasks on arrays (Codex: 23/33, Baseline: 9/36)”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.1 Overall Completion Rates and Direct Hints.
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When discussing our results, there were several metrics in which the diferences between the two groups did not reach statistical signifcance but there did seem to be some diference in means and confdence intervals.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 7 LIMITATIONS AND FUTURE WORK
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Learners employed various methods to generate code such as breaking down the task into subgoals and using the code generator to generate code for each of the subgoals step-by-step.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6 DISCUSSION
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“Furthermore, although our study included many non-native English speakers, but they were mostly comfortable explaining things in English.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 7 LIMITATIONS AND FUTURE WORK
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The two graders fully agreed on 79% of the submissions”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.4 Data Analysis
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“with 16% of disagreements having a diference of 25% in which we averaged the two grades”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.4 Data Analysis
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“No common factors were identifed among the 21 students who dropped out in terms of disability, native language (six non-English) and computer or internet access.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2 Participants
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“One learner in each pair was randomly assigned to either the Codex or Baseline group.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2.1 Participant Condition Assignment.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“But overall, this is a promising result: although learners in the Codex group used the documentation less and relied heavily on AI-generated code for the authoring tasks, they still did just as well, and in some cases better, in manual code modifcation tasks.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 5.1.4 Modifying Tasks (Training Phase).
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“two independent researchers graded each submitted solution using a simple and consistent grading rubric of deducting 25% for each major issue or missing part in their fnal submission”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.4 Data Analysis
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This shows that the AI code generator produces high quality results, but this depends on the quality of the prompt message.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 3.4 Quality of AI Generated Code
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A claim is checked only against the cited paper’s full text, never its abstract, and none is available for its sources.
In the paper
“participants were divided into two groups using a matched-groups design [13,60].The ninety participants who completed the frst session were divided into two groups to have similar means and variances based on their pre-test scores on Scratch.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2.1 Participant Condition Assignment.
What the sources say
- Not checked — no full text of the sourcecites [13] 13. Miriam Bruhn and David McKenzie. 2009. In pursuit of balance: Randomization in practice in development feld experiments. American economic journal: applied economics 1, 4 (2009), 200-232. source
- Not checked — no full text of the sourcecites [60] 60. I Scott MacKenzie. 2012. Human-computer interaction: An empirical research perspective. (2012).
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“To ensure both groups received consistent and unbiased feedback on their submissions, the grading dashboard did not display any information that would reveal the identity of learners or their group”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.1.2 Training Phase.
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the 5% of the disagreements which were more than 25%, the two graders met again to resolve each of their disagreements.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.4 Data Analysis
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“By generating code from simple sentences instead of formal and syntactically fxed specifcations, these AI Coding Assistants may lower the barriers to entry into programming.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 1 INTRODUCTION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“69 learners (21 female/48 male) ages 10-17 (M=12.5;SD=1.8) successfully completed all three phases of the study”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 4.2 Participants
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“AI code generators allowed novice programmers to perform better and faster with less frustration when writing code”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 8 CONCLUSION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The lack of statistical fndings for such metrics does not necessarily mean there is no efect present, it could potentially be due to the sample size of our study.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 7 LIMITATIONS AND FUTURE WORK
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Furthermore, our qualitative feedback shows that having access to the AI code generator reduced stress and discouragement in addition to improving their eagerness to continue programming later on.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6 DISCUSSION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Alpha levels for per-topic comparisons are adjusted using Bonferroni correction ( = 0.05/5 = 0.01).”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 45B
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Prior research has shown that subgoallabeled instructional material reduces extraneous cognitive load imposed on novices learning programming by chunking problemsolving steps and promoting self-explanation [61].”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6.1 How Novices Beneft from AI Coding Assistants
What the sources say
- Not checked — no full text of the sourcecites [61] 61. Lauren E Margulieux, Mark Guzdial, and Richard Catrambone. 2012. Subgoal- labeled instructional material improves performance and transfer in learning to develop mobile applications. In Proceedings of the ninth annual international conference on International computing education research. 71-78. source
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our results from code-authoring tasks during the training phase indicate that using AI code generators can significantly increase task completion, improve correctness score, reduce encountered errors, and reduce task completion time.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6 DISCUSSION
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although learners in the Baseline group used the documentation more frequently, they had to start each task by creating a new program from scratch before getting to the modify portion of the activity, and thus encountered more errors.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6.1 How Novices Beneft from AI Coding Assistants
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“prior research has even argued for switching away from statistical signifcance testing in favor of reporting informative charts with efect sizes and confdence intervals and nuanced interpretations [20]”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 7 LIMITATIONS AND FUTURE WORK
What the sources say
- Not checked — no full text of the sourcecites [20] 20. Pierre Dragicevic. 2015. HCI Statistics without p-values. Ph. D. Dissertation. Inria.
Strength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Their higher prior programming knowledge meant that they probably had more code-reading and tracing skills which means that they were able to understand that code better, and then perform better during the next modifcation or code-creation tasks.”
§ Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming > 6.1 How Novices Beneft from AI Coding Assistants
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
The paper relies on this without support.
The AI code generator (Codex) used in the study is representative of AI code generators in general.
“Studying the efect of AI Code Generators on Supporting Novice Learners in Introductory Programming”
“Similarly, AI code generators may reduce the cognitive load when learning to program, but its impact on the learning experience has not been studied.”
The paper relies on this without support.
Performance on the Coding Steps tasks and multiple-choice questions reflects learning of introductory Python programming.
“To answer these questions, we conducted a controlled study with 69 young learners, ages 10-17 (M=12.5,SD=1.8) with no prior textbased programming experience.Half of them used an AI Coding Assistant (Codex) to learn Python, and the other half did not.”
“The tasks in this phase included fve code-authoring tasks, fve code modifcation tasks, and then 40 multiple-choice questions”
The paper relies on this without support.
The AI code generator's quality measured on the 45 tasks before the study is representative of its quality during the actual study sessions.
“To measure the quality of the AI code generator, we entered the task description of the 45 code-authoring tasks into the pre-conditioned model (described in Figure 3) and evaluated the accuracy and amount of required manual modifcations on the generated code.”
The paper relies on this without support.
The learning effects and support provided by AI code generators have not been studied in the context of introductory programming.
“To date, the learning efects and support provided by AI code generators have not been studied in the context of introductory programming.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The propensity of learners to use GenAI outputs has been shown to vary significantly by exercise topic [Kazemitabaar et al. 2023; Mailach et al. 2025].”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Kazemitabaar et al. 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Generative AI can change how much students learn through two margins.”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Kazemitabaar et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While research has demonstrated that GenAI can support learning and foster creativity, excessive dependence on these tools may undermine the development of algorithmic thinking, error detection, and analytical reasoning (Kazemitabaar et al., 2023;Mogavi et al., 2024;Zviel-Girshin, 2024).”
From Integrating Generative AI into Programming Education: Student Perceptions and the Challenge of Correcting AI Errors (2025) · cites this paper as Kazemitabaar et al., 2023
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Research suggests that integrating AI tools alongside conventional instructional methods enables students to build a comprehensive skill set while developing the capacity to critique and refine automated outputs (Kazemitabaar et al., 2023).”
From Integrating Generative AI into Programming Education: Student Perceptions and the Challenge of Correcting AI Errors (2025) · cites this paper as Kazemitabaar et al., 2023
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Furthermore, reliance on code generation can negatively impact code authoring skills [Jošt et al. 2024; Kazemitabaar et al. 2023].”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Kazemitabaar et al. 2023
The cited source says what the claim attributes to it.
In the citing paper
“These numbers align with findings by Kazemitabaar et al. [2023], who observed an average similarity of 63% ( 𝜎 = 42%) between generated and submitted code, albeit using the Jaccard similarity measure.”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Kazemitabaar et al. [2023]
The cited source says what the claim attributes to it.
In the citing paper
“Third, in many designs students choose how much time to spend on the task (e.g., Kazemitabaar et al., 2023; Kestin et al., 2025; Kreijkes et al., 2026), so the estimated effects combine the two margins: time allocation and learning productivity.”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Kazemitabaar et al., 2023
The cited source says what the claim attributes to it.
In the citing paper
“no detectable difference on an immediate unassisted post-test ( -0.05 SD; the baseline group scored slightly higher, 62.9 versus 61.3 percent)”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Kazemitabaar et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“only a marginal difference on a one-week retention test of code modification (0.41 SD).”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Kazemitabaar et al. (2023)
The cited source says what the claim attributes to it.
In the citing paper
“AI access produced large performance gains during training (1.67 SD)”
From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Kazemitabaar et al. (2023)
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“However, as the results are fairly consistent with similar surveys around the time [Kazemitabaar et al. 2023; Prather et al. 2023a; Zastudil et al. 2023] and we were commonly able to observe behavior that could be considered unethical in an academic context [Prather et al. 2023a], we do not believe this to be a major limitation.”
From How Do Programming Students Use Generative AI? (2025) · cites this paper as Kazemitabaar et al. 2023
Experimental — accuracy not yet measuredIn the citing paper
“Interacting with an LLM has been observed to lead to higher performance in code writing tasks but to little differences in learning gains [18].”
From Instructor Perceptions of AI Code Generation Tools - A Multi-Institutional Interview Study (2024) · cites this paper as [18]
In the citing paper
“They found that students who used AI code generators performed significantly better (1.15x progress, 0.59x errors, 1.8x higher correctness, 0.57x time spent) without negative effects on learning.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [108]
In the citing paper
“Recent research has shown promising possibilities for providing students with LLM partners in pair programming, given the right context and with the right scaffolding and support [44,108,159].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [108]
In the citing paper
“The models might generate code students do not understand [108] or may distract them with large blocks of text they did not write [159].”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [108]
In the citing paper
“Also very concerning, Kazemitabaar et al. [108] found that students with more knowledge benefited more from code generation than students with less knowledge.”
From The Robots Are Here: Navigating the Generative AI Revolution in Computing Education (2023) · cites this paper as [108]
Statistics that don’t add up
Each reported p-value recomputed from its test statistic and degrees of freedom.
Inconsistent p-value: t(39)=2.78,p = .007
MinorThe paper reports p = .007 here; recomputing from the test statistic and its degrees of freedom gives p = .008 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.
Reported p = .007 · recomputed p = .008
“t(39)=2.78,p=.007”
Inconsistent p-value: t(39)=2.34,p = .022
MinorThe paper reports p = .022 here; recomputing from the test statistic and its degrees of freedom gives p = .024 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.
Reported p = .022 · recomputed p = .024
“t(39)=2.34,p=.022”
Inconsistent p-value: t(67)=1.70,p = .092
MinorThe paper reports p = .092 here; recomputing from the test statistic and its degrees of freedom gives p = .094 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.
Reported p = .092 · recomputed p = .094
“t(67)=1.70,p=.092”
Inconsistent p-value: t(58)=1.41,p = .161
MinorThe paper reports p = .161 here; recomputing from the test statistic and its degrees of freedom gives p = .164 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.
Reported p = .161 · recomputed p = .164
“t(58)=1.41,p=.161”
Inconsistent p-value: t(67)=0.54,p = .67
MinorThe paper reports p = .67 here; recomputing from the test statistic and its degrees of freedom gives p = .591 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.
Reported p = .67 · recomputed p = .591
“t(67)=0.54,p=.67”
Inconsistent p-value: t(39)=2.02,p = .047
MinorThe paper reports p = .047 here; recomputing from the test statistic and its degrees of freedom gives p = .050 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.
Reported p = .047 · recomputed p = .050
“t(39)=2.02,p=.047”
Inconsistent p-value: t(67)=1.28,p = .202
MinorThe paper reports p = .202 here; recomputing from the test statistic and its degrees of freedom gives p = .205 (two-tailed). The check cannot tell whether the statistic or the p-value is the misprint.
Reported p = .202 · recomputed p = .205
“t(67)=1.28,p=.202”
Statistics recomputed
Every reported test result, recomputed from its test statistic and degrees of freedom. A result shows as not adding up only once an independent check has confirmed it.
- Doesn’t add upt(67) = 0.54reported p = .67 · recomputed p = .591
“t(67)=0.54,p = .67”
- Doesn’t add upt(67) = 1.28reported p = .202 · recomputed p = .205
“t(67)=1.28,p = .202”
- Doesn’t add upt(39) = 2.34reported p = .022 · recomputed p = .024
“t(39)=2.34,p = .022”
- Doesn’t add upt(58) = 1.41reported p = .161 · recomputed p = .164
“t(58)=1.41,p = .161”
- Doesn’t add upt(39) = 2.78reported p = .007 · recomputed p = .008
“t(39)=2.78,p = .007”
- Doesn’t add upt(67) = 1.7reported p = .092 · recomputed p = .094
“t(67)=1.70,p = .092”
- Doesn’t add upt(39) = 2.02reported p = .047 · recomputed p = .050
“t(39)=2.02,p = .047”
- Not confirmedt(67) = 0.058reported p = .953 · recomputed p = .954
“t(67)=0.058,p = .953”
- Not confirmedt(67) = 1.21reported p = .228 · recomputed p = .231
“t(67)=1.21,p = .228”
- Consistentt(67) = 2.8reported p = .006 · recomputed p = .007
“t(67)=2.8,p = .006”
- Consistentt(67) = 0.25reported p = .805 · recomputed p = .803
“t(67)=0.25,p = .805”
- Consistentt(67) = 0.53reported p = .6 · recomputed p = .598
“t(67)=0.53,p = .60”
- Consistentt(67) = 1.15reported p = .25 · recomputed p = .254
“t(67)=1.15,p = .25”
- Consistentt(67) = 6.92reported p < .001 · recomputed p < .001
“t(67)=6.92,p < .001”
- Consistentt(67) = 6.4reported p < .001 · recomputed p < .001
“t(67)=6.40,p < .001”
- Consistentt(67) = 5.48reported p < .001 · recomputed p < .001
“t(67)=5.48,p < .001”
- Consistentt(67) = 3.03reported p = .004 · recomputed p = .003
“t(67)=3.03,p = .004”
- Consistentt(67) = 3.65reported p = .001 · recomputed p < .001
“t(67)=3.65,p = .001”
- Consistentt(67) = 0.98reported p = .331 · recomputed p = .331
“t(67)=0.98,p = .331”
- Consistentt(67) = 1.84reported p = .071 · recomputed p = .070
“t(67)=1.84,p = .071”
- Consistentt(67) = 1.53reported p = .129 · recomputed p = .131
“t(67)=1.53,p = .129”
- Consistentt(67) = 0.91reported p = .367 · recomputed p = .366
“t(67)=0.91,p = .367”
- Consistentt(39) = 3.53reported p = .001 · recomputed p = .001
“t(39)=3.53,p = .001”
- Consistentt(67) = 0.2reported p = .838 · recomputed p = .842
“t(67)=0.20,p = .838”
- Consistentt(58) = 2.08reported p = .041 · recomputed p = .042
“t(58)=2.08,p = .041”
- Consistentt(67) = 1.13reported p = .262 · recomputed p = .263
“t(67)=1.13,p = .262”
- Consistentt(67) = 2.08reported p = .042 · recomputed p = .041
“t(67)=2.08,p = .042”
- Consistentt(67) = 1.21reported p = .229 · recomputed p = .231
“t(67)=1.21,p = .229”
- Consistentt(67) = 4.1reported p < .001 · recomputed p < .001
“t(67)=4.10,p < .001”
- Consistentt(67) = 1.54reported p = .129 · recomputed p = .128
“t(67)=1.54,p = .129”
- Consistentt(58) = 2.09reported p = .04 · recomputed p = .041
“t(58)=2.09,p = .04”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
The first session has two stated durations
MinorThe procedure describes all ten sessions as 90 minutes long, but later specifies that the first session lasted two hours.
“The study lasted three weeks and included ten 90-minute sessions, with one session per day, spanning over three (consecutive) weeks.”
“The frst session in the training phase was 2-hours long and included a one-hour introduction to the basic concepts of programming and computational thinking using Scratch.”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Jakubův, J., Chvalovský, K., Goertzel, Z., et al. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,648
(1978) · Medical Entomology and Zoology · Cited by 53,488
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Wing, J. M. (2006) · Communications of the ACM · doi:10.1145/1118178.1118215 · Cited by 8,102
Kirschner, P. A., Sweller, J., & Clark, R. E. (2006) · Educational Psychologist · doi:10.1207/s15326985ep4102_1 · Cited by 7,776
Using Effect Size—or Why the P Value Is Not Enough
Sullivan, G. M., & Feinn, R. (2012) · Journal of Graduate Medical Education · doi:10.4300/jgme-d-12-00156.1 · Cited by 5,553 · cited 1×
“Finally, Cohen’s d is reported for effect size [78].”
Resnick, M., Maloney, J., Monroy-Hernández, A., Rusk, N., Eastmond, E., Brennan, K., Millner, A., Rosenbaum, E., Silver, J., Silverman, B., et al. (2009) · Communications of the ACM · doi:10.1145/1592761.1592779 · Cited by 3,333
Computational Thinking in K–12
Grover, S., & Pea, R. D. (2013) · Educational Researcher · doi:10.3102/0013189x12463051 · Cited by 3,034
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824 · cited 2×
“For example, models like OpenAI Codex [14], Microsoft CodeBERT [27], Google PaLM [16], DeepMind AlphaCode [53] have been trained on millions of lines of publicly available code (e.g., on Github).”
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590 · cited 2×
“For example, models like OpenAI Codex [14], Microsoft CodeBERT [27], Google PaLM [16], DeepMind AlphaCode [53] have been trained on millions of lines of publicly available code (e.g., on Github).”
MethodCognitive Architecture and Instructional Design: 20 Years Later
Sweller, J., van Merriënboer, J. J. G., & Paas, F. (2019) · Educational Psychology Review · doi:10.1007/s10648-019-09465-5 · Cited by 2,337 · cited 1×
“Cognitive load theory defines the demand that a situation or task places on a learner’s working memory [80] and is based in part on the learner’s prior knowledge and the complexity of the task or material [22].”
Henderson, P. B., Cortina, T. J., & Wing, J. M. (2007) · ACM SIGCSE Bulletin · doi:10.1145/1227504.1227378 · Cited by 1,580
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 10×
“After clicking on the generate button, the code generated from OpenAI Codex would be inserted at the beginning of the line in which the cursor was located (and shifting any existing code below).”
BackgroundMethodInfluentialSeq2SQL: Generating Structured Queries from Natural Language using Reinforcement LearningPreprint
Zhong, V., Xiong, C., & Socher, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.00103 · Cited by 1,384 · cited 1×
“Various methods have been employed, such as using semantic parsers [4, 9, 11, 18, 33, 34, 47, 50, 51, 56, 68, 80] and machine learning [5, 54, 71–73, 93, 95].”
Direct Manipulation Interfaces
Hutchins, E. L., Hollan, J. D., & Norman, D. A. (1985) · Human-Computer Interaction · doi:10.1207/s15327051hci0104_2 · Cited by 1,374
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043
Automating string processing in spreadsheets using input-output examples
Gulwani, S. (2011) · ACM-SIGACT Symposium on Principles of Programming Languages · doi:10.1145/1926385.1926423 · Cited by 1,039 · cited 1×
“Various methods have been employed, such as using semantic parsers [4, 9, 11, 18, 33, 34, 47, 50, 51, 56, 68, 80] and machine learning [5, 54, 71–73, 93, 95].”
Taking the Load Off a Learner's Mind: Instructional Design for Complex Learning
van Merrienboer, J. J. G., Kirschner, P. A., & Kester, L. (2003) · Educational Psychologist · doi:10.1207/s15326985ep3801_2 · Cited by 857 · cited 2×
“Several approaches have been developed to reduce this cognitive load in introductory programming, such as the use of worked examples and similar practice problems [72, 84], or mixed-up code (Parsons) problems [64], where learners are given mixed up fragments that must be placed in the correct order to solve a problem.”
Computational thinking for youth in practice
Lee, I., Martin, F., Denner, J., Coulter, B., Allan, W., Erickson, J., Malyn-Smith, J., & Werner, L. (2011) · ACM Inroads · doi:10.1145/1929887.1929902 · Cited by 799 · cited 1×
“The benefit of AI code generators for novice learners could be explained by the effective employment of the use-modify-create pedagogical strategy often used in introductory programming [30, 52].”
In Pursuit of Balance: Randomization in Practice in Development Field Experiments
Bruhn, M., & McKenzie, D. (2009) · American Economic Journal: Applied Economics · doi:10.1257/app.1.4.200 · Cited by 788 · cited 1×
“Following the Introduc-tion Phase using Scratch, participants were divided into two groups using a matched-groups design [13, 60].”
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Some Difficulties of Learning to Program
Du Boulay, B. (1986) · Journal of Educational Computing Research · doi:10.2190/3lfx-9rrf-67t8-uvk9 · Cited by 648
DeepCoder: Learning to Write ProgramsPreprint
Balog, M., Gaunt, A. L., Brockschmidt, M., Nowozin, S., & Tarlow, D. (2016) · International Conference on Learning Representations · doi:10.48550/arxiv.1611.01989 · Cited by 641
A Syntactic Neural Model for General-Purpose Code Generation
Yin, P., & Neubig, G. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1041 · Cited by 623
A Syntactic Neural Model for General-Purpose Code Generation
Yin, P., & Neubig, G. (2017) · arXiv · doi:10.48550/arxiv.1704.01696 · Cited by 585
Automating string processing in spreadsheets using input-output examples
Gulwani, S. (2011) · ACM SIGPLAN Notices · doi:10.1145/1925844.1926423 · Cited by 581
CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation
Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., et al. (2021) · arXiv · doi:10.48550/arxiv.2102.04664 · Cited by 536 · cited 2×
“code operations such as code explanation and search [56, 77].”
Students’ Misconceptions and Other Difficulties in Introductory Programming: A Literature Review
Qian, Y., & Lehman, J. (2018) · ACM Transactions on Computing Education · doi:10.1145/3077618 · Cited by 529 · cited 1×
“Programming is not easy to learn [21, 70].”
Introductory programming: a systematic literature review
Luxton-Reilly, A., Simon, Al-Bluwi, I., Becker, B. A., Giannakos, M. N., Kumar, A. N., Ott, L. M., Paterson, J. H., Scott, M., Sheard, J., et al. (2018) · doi:10.1145/3293881.3295779 · Cited by 511 · cited 1×
“A full review of introductory programming research is beyond the scope of this paper, and instead we refer the reader to recent surveys on the topic [7, 59].”
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models
Sarsa, S., Denny, P., Hellas, A., & Leinonen, J. (2022) · Proceedings of the 2022 ACM Conference on International Computing Education Research - Volume 1 · doi:10.1145/3501385.3543957 · Cited by 497 · cited 4×
“Additionally, these tools could support educators in curriculum development by generating programming exercises and explanations to solutions [79].”
InfluentialAlice: a 3-D tool for introductory programming concepts
Cooper, S., Dann, W. P., & Pausch, R. F. (2000) · Journal of computing sciences in colleges · Cited by 489
Fesakis, G., Gouli, E., & Mavroudi, E. (2012) · Computers & Education · doi:10.1016/j.compedu.2012.11.016 · Cited by 480 · cited 1×
“Research has provided evidence that computing education offers a platform to practice and improve problemsolving [28, 41, 62, 69], critical thinking [26], collaboration [77], and active learning [40] skills.”
The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming
Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A., & Prather, J. (2022) · Proceedings of the 24th Australasian Computing Education Conference · doi:10.1145/3511861.3511863 · Cited by 463 · cited 2×
“[30] showed that OpenAI Codex outperforms most students on code writing questions that are used in typical first-year programming exams.”
A new way of teaching programming skills to K-12 students: Code.org
Kalelıoğlu, F. (2015) · Computers in Human Behavior · doi:10.1016/j.chb.2015.05.047 · Cited by 437 · cited 4×
“Research has provided evidence that computing education offers a platform to practice and improve problemsolving [29, 41, 60, 67], critical thinking [27], collaboration [75], and active learning [40] skills.”
InfluentialIn Pursuit Of Balance: Randomization In Practice In Development Field Experiments
McKenzie, D. J., & Bruhn, M. (2008) · World Bank, Washington, DC eBooks · doi:10.1596/1813-9450-4752 · Cited by 380
(2012) · Cited by 370
Relationships between reading, tracing and writing skills in introductory programming
Lopez, M., Whalley, J., Robbins, P., & Lister, R. (2008) · Proceedings of the Fourth international Workshop on Computing Education Research · doi:10.1145/1404520.1404531 · Cited by 363 · cited 1×
“Prior work in introductory programming has shown that code-reading and tracing skills are essential to problem-solving activities including code-writing [49, 55, 57].”
Computer science in K-12 school curricula of the 2lst century: Why, what and when?
Webb, M., Davis, N. E., Bell, T., Katz, Y. J., Reynolds, N., Chambers, D. P., & Sysło, M. M. (2016) · Education and Information Technologies · doi:10.1007/s10639-016-9493-x · Cited by 355
Kalelıoğlu, F., & Gülbahar, Y. (2014) · Informatics in Education · doi:10.15388/infedu.2014.03 · Cited by 352
Latent Predictor Networks for Code Generation
Ling, W., Blunsom, P., Grefenstette, E., Hermann, K. M., Kočiský, T., Wang, F., & Senior, A. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1057 · Cited by 307 · cited 1×
“Various methods have been employed, such as using semantic parsers [4, 9, 11, 18, 33, 34, 47, 50, 51, 56, 68, 80] and machine learning [5, 54, 71–73, 93, 95].”
Direct Manipulation Interfaces
(2011) · Springer eBooks · doi:10.1007/springerreference_64794 · Cited by 298
Altadmri, A., & Brown, N. C. C. (2015) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/2676723.2677258 · Cited by 261 · cited 1×
“One main approach to improving the learning experience is the use of assistive programming environments, which can help alleviate misconceptions in syntax [1, 35], and conceptual knowledge [81].”
Learning to code or coding to learn? A systematic review
Popat, S., & Starkey, L. (2019) · Computers & Education · doi:10.1016/j.compedu.2018.10.005 · Cited by 260 · cited 1×
“Computing is now being integrated into K-12 education of many countries [67, 90].”
Natural programming languages and environments
Myers, B. A., Pane, J. F., & Ko, A. J. (2004) · Communications of the ACM · doi:10.1145/1015864.1015888 · Cited by 247 · cited 1×
“In contrast, there are also arguments supporting the idea of bringing programming closer to how people think [63, 65] through natural language programming.”
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243
The Worked-Out Examples Principle in Multimedia Learning
Renkl, A. (2005) · Cambridge University Press eBooks · doi:10.1017/cbo9780511816819.016 · Cited by 231
Investigating Explainability of Generative AI for Code through Scenario-based Design
Sun, J., Liao, Q. V., Muller, M., Agarwal, M., Houde, S., Talamadupula, K., & Weisz, J. D. (2022) · 27th International Conference on Intelligent User Interfaces · doi:10.1145/3490099.3511119 · Cited by 219 · cited 2×
“To explore the explainability needs of AI code generators, Jiao et al. [83] conducted 9 semi-structured workshops with 43 software engineers using AI Coding Assistants and identified 11 categories of explainability needs such as types of input that the model can take, and the data that these models…”
Psycharis, S., & Kallia, M. (2017) · Instructional Science · doi:10.1007/s11251-017-9421-5 · Cited by 216
Parson's programming puzzles: a fun and effective learning tool for first programming courses
Parsons, D., & Haden, P. (2006) · Australasian Computing Education Conference · Cited by 195
App inventor and real-world motivation
Wolber, D. (2011) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/1953163.1953329 · Cited by 182 · cited 1×
“For example, many introductory programming courses for K-12 learners start with block-based programming environments (BBPEs) like LogoBlocks [8], Alice [17], Scratch [76], and AppInven-tor [92].”
Falloon, G. (2016) · Journal of Computer Assisted Learning · doi:10.1111/jcal.12155 · Cited by 180 · cited 1×
“Research has provided evidence that computing education offers a platform to practice and improve problemsolving [29, 41, 60, 67], critical thinking [27], collaboration [75], and active learning [40] skills.”
Language to Code: Learning Semantic Parsers for If-This-Then-That Recipes
Quirk, C., Mooney, R., & Galley, M. (2015) · Annual Meeting of the Association for Computational Linguistics · doi:10.3115/v1/p15-1085 · Cited by 169 · cited 1×
“Various methods have been employed, such as using semantic parsers [4, 9, 11, 18, 33, 34, 47, 50, 51, 56, 68, 80] and machine learning [5, 54, 71–73, 93, 95].”
Raghothaman, M., Wei, Y., & Hamadi, Y. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884808 · Cited by 165
Experiencing programming assignments in CS1: the emotional toll
Kinnunen, P., & Simon, B. (2010) · Proceedings of the Sixth international workshop on Computing education research · doi:10.1145/1839594.1839609 · Cited by 156 · cited 1×
“Novices can be overwhelmed by the complexity of coding tasks [44, 84] and spend an unexpectedly large amount of time on them [10].”
Kölling, M., Brown, N. C. C., & Altadmri, A. (2015) · Workshop in Primary and Secondary Computing Education · doi:10.1145/2818314.2818331 · Cited by 153 · cited 1×
“To support this transition, various tools have been developed, including: dual-modality programming environments like PencilCode [6] and MakeCode [3], and hybrid environments such as Frame-based editing [47] and CodeStruct [42].”
Margulieux, L. E., Guzdial, M., & Catrambone, R. (2012) · Proceedings of the ninth annual international conference on International computing education research · doi:10.1145/2361276.2361291 · Cited by 149 · cited 1×
“Prior research has shown that subgoal-labeled instructional material reduces extraneous cognitive load imposed on novices learning programming by chunking problem-solving steps and promoting self-explanation [61].”
Bau, D., Bau, D. A., Dawson, M., & Pickens, C. S. (2015) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/2771839.2771875 · Cited by 145 · cited 2×
“To support this transition, various tools have been developed, including: dual-modality programming environments like PencilCode [6] and MakeCode [3], and hybrid environments such as Frame-based editing [48] and CodeStruct [42, 43].”
Coarse-grained detection of student frustration in an introductory programming course
Rodrigo, M. M. T., & Baker, R. S. (2009) · doi:10.1145/1584322.1584332 · Cited by 129 · cited 1×
“This can be a frus-trating experience [78] for students and repeated failures, especially early on, can lower their self-efficacy with respect to programming [45].”
Solving parsons problems versus fixing and writing code
Ericson, B. J., Margulieux, L. E., & Rick, J. (2017) · Proceedings of the 17th Koli Calling International Conference on Computing Education Research · doi:10.1145/3141880.3141895 · Cited by 121 · cited 1×
“Learner’s typically complete Parsons problems significantly more quickly than fixing or writing code and have similar learning gains [24, 25, 94].”
Self-citationIdentifying and correcting Java programming errors for introductory computer science students
Hristova, M., Misra, A., Rutter, M., & Mercuri, R. T. (2003) · ACM SIGCSE Bulletin · doi:10.1145/792548.611956 · Cited by 120
Lister, R., Fidge, C., & Teague, D. (2009) · Proceedings of the 14th annual ACM SIGCSE conference on Innovation and technology in computer science education · doi:10.1145/1562877.1562930 · Cited by 118 · cited 1×
“Prior work in introductory programming has shown that code-reading and tracing skills are essential to problem-solving activities including code-writing [48, 54, 55].”
BackgroundProgram synthesis using natural language
Desai, A., Gulwani, S., Hingorani, V., Jain, N., Karkare, A., Marron, M., Sailesh, R., & Roy, S. (2016) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2884781.2884786 · Cited by 116 · cited 1×
“Various methods have been employed, such as using semantic parsers [4, 9, 11, 18, 33, 34, 47, 50, 51, 56, 68, 80] and machine learning [5, 54, 71–73, 93, 95].”
CS majors' self-efficacy perceptions in CS1: results in light of social cognitive theory
Kinnunen, P., & Simon, B. (2011) · Proceedings of the seventh international workshop on Computing education research · doi:10.1145/2016911.2016917 · Cited by 116 · cited 1×
“This can be a frustrating experience [76] for students and repeated failures, especially early on, can lower their self-efficacy with respect to programming [43].”
50 Years of CS1 at SIGCSE: A Review of the Evolution of Introductory Programming Education Research
Becker, B. A., & Quille, K. (2019) · Proceedings of the 50th ACM Technical Symposium on Computer Science Education · doi:10.1145/3287324.3287432 · Cited by 107
Exploring programming misconceptions
Sirkiä, T., & Sorva, J. (2012) · Koli Calling International Conference on Computing Education Research · doi:10.1145/2401796.2401799 · Cited by 99 · cited 1×
“One main approach to improving the learning experience is the use of assistive programming environments, which can help alleviate misconceptions in syntax [1, 35], and conceptual knowledge [81].”
Gulwani, S., & Marron, M. (2014) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/2588555.2612177 · Cited by 97
Cognitive Load Theory in Computing Education Research: A Review
Duran, R., Zavgorodniaia, A., & Sorva, J. (2022) · ACM Transactions on Computing Education · doi:10.1145/3483843 · Cited by 92 · cited 1×
“The learning environment included 45 two-part programming tasks that were designed to avoid overwhelming learners through too much cognitive load [23] by gradually increasing complexity and introducing new concepts as they made progress.”
LogoBlocks: A Graphical Programming Language for Interacting with the World
Begel, A., & Resnick, M. (2000) · Cited by 89
Price, D., Rilofff, E., Zachary, J. L., et al. (2000) · doi:10.1145/325737.325845 · Cited by 87 · cited 1×
“Various methods have been employed, such as using semantic parsers [4, 9, 11, 18, 33, 34, 47, 50, 51, 56, 68, 80] and machine learning [5, 54, 71–73, 93, 95].”
A programming system for children that is designed for usability
Pane, J. F., Myers, B. A., & Garlan, D. (2002) · Cited by 84
Le, V., Gulwani, S., & Su, Z. (2013) · International Conference on Mobile Systems, Applications, and Services · doi:10.1145/2462456.2464443 · Cited by 83 · cited 1×
“Various methods have been employed, such as using semantic parsers [4, 9, 11, 18, 33, 34, 47, 50, 51, 56, 68, 80] and machine learning [5, 54, 71–73, 93, 95].”
Latent Predictor Networks for Code GenerationPreprint
Ling, W., Grefenstette, E., Hermann, K. M., Kočiský, T., Senior, A., Wang, F., & Blunsom, P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06744 · Cited by 82
Jiang, E., Toh, E., Molina, A., Olson, K., Kayacik, C., Donsbach, A., Cai, C. J., & Terry, M. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501870 · Cited by 79 · cited 1×
“Jiang et al. created GenLine [39] and conducted a study with 12 participants that worked on two web-programming tasks to explore the user experience of natural language programming with AI code generators.”
Translating keyword commands into executable code
Little, G., & Miller, R. C. (2006) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1166253.1166275 · Cited by 77 · cited 1×
“Various methods have been employed, such as using semantic parsers [4, 9, 11, 18, 33, 34, 47, 50, 51, 56, 68, 80] and machine learning [5, 54, 71–73, 93, 95].”
Tarr, P., & Cook, W. R. (2006) · doi:10.1145/1176617 · Cited by 76
An experimental study of natural language programming
Biermann, A. W., Ballard, B. W., & Sigmon, A. H. (1983) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(83)80005-4 · Cited by 73 · cited 1×
“Various methods have been employed, such as using semantic parsers [4, 9, 11, 18, 33, 34, 47, 50, 51, 56, 68, 80] and machine learning [5, 54, 71–73, 93, 95].”
Evaluating the Efficiency and Effectiveness of Adaptive Parsons Problems
Ericson, B. J., Foley, J. D., & Rick, J. (2018) · Proceedings of the 2018 ACM Conference on International Computing Education Research · doi:10.1145/3230977.3231000 · Cited by 70 · cited 1×
“Learner’s typically complete Parsons problems significantly more quickly than fixing or writing code and have similar learning gains [24, 25, 94].”
Self-citationProgramming in natural language
Ballard, B. W., & Biermann, A. W. (1979) · doi:10.1145/800177.810072 · Cited by 66
Lister, R., Fidge, C., & Teague, D. (2009) · ACM SIGCSE Bulletin · doi:10.1145/1595496.1562930 · Cited by 62
Benda, K., Bruckman, A., & Guzdial, M. (2012) · ACM Transactions on Computing Education · doi:10.1145/2382564.2382567 · Cited by 53 · cited 1×
“Novices can be overwhelmed by the complexity of coding tasks [44, 84] and spend an unexpectedly large amount of time on them [10].”
On the foolishness of “natural language programming”
Dijkstra, E. W. (1979) · Lecture notes in computer science · doi:10.1007/bfb0014656 · Cited by 51 · cited 1×
“Historically, many computer scientists have been arguing that programming a computer should remain a completely formal process with syntactically fxed specifcations, similar to math, as it would encourage people to think more like computers and therefore, perform better [19].”
Understanding Conversational Programmers
Chilana, P. K., Singh, R., & Guo, P. J. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858323 · Cited by 51 · cited 1×
“But there still exist many questions about the implications of AI Coding Assistants with other populations such as conversational programmers [15] who are motivated to improve their efficacy in technical conversations, or experienced programmers when they want to learn a new programming language or programming library.”
Identifying and correcting Java programming errors for introductory computer science students
Hristova, M., Misra, A., Rutter, M., et al. (2003) · doi:10.1145/611953.611956 · Cited by 49
Microsoft MakeCode: embedded programming for education, in blocks and TypeScript
Ball, T. J., Chatra, A., de Halleux, P., et al. (2019) · doi:10.1145/3358711.3361630 · Cited by 48 · cited 2×
“To support this transition, various tools have been developed, including: dual-modality programming environments like PencilCode [6] and MakeCode [3], and hybrid environments such as Frame-based editing [48] and CodeStruct [42, 43].”
Compositional program synthesis from natural language and examples
Raza, M., Gulwani, S., & Milić-Frayling, N. (2015) · International Conference on Artificial Intelligence · Cited by 47
Towards an Analysis of Program Complexity From a Cognitive Perspective
Duran, R., Sorva, J., & Leite, S. (2018) · Proceedings of the 2018 ACM Conference on International Computing Education Research · doi:10.1145/3230977.3230986 · Cited by 44 · cited 1×
“Cognitive load theory defines the demand that a situation or task places on a learner’s working memory [84] and is based in part on the learner’s prior knowledge and the complexity of the task or material [22].”
A study of the influence of code-tracing problems on code-writing skills
Kumar, A. N. (2013) · Proceedings of the 18th ACM conference on Innovation and technology in computer science education · doi:10.1145/2462476.2462507 · Cited by 43 · cited 1×
“Prior work in introductory programming has shown that code-reading and tracing skills are essential to problem-solving activities including code-writing [48, 54, 55].”
Franklin, D., Coenraad, M., Palmer, J., Eatinger, D., Zipp, A., Anaya, M., White, M., Pham, H., Gökdemir, O., & Weintrop, D. (2020) · Proceedings of the 2020 ACM Conference on International Computing Education Research · doi:10.1145/3372782.3406256 · Cited by 43 · cited 1×
“The benefit of AI code generators for novice learners could be explained by the effective employment of the use-modify-create pedagogical strategy often used in introductory programming [30, 52].”
Self-citationEvaluating the Effectiveness of Parsons Problems for Block-based Programming
Zhi, R., Chi, M., Barnes, T., & Price, T. W. (2019) · Proceedings of the 2019 ACM Conference on International Computing Education Research · doi:10.1145/3291279.3339419 · Cited by 42 · cited 1×
“Learner’s typically complete Parsons problems significantly more quickly than fixing or writing code and have similar learning gains [24, 25, 94].”
Effects of Logo computer programming experience on problem solving and spatial relations ability
Miller, R. B., Kelly, G. N., & Kelly, J. T. (1988) · Contemporary Educational Psychology · doi:10.1016/0361-476x(88)90034-3 · Cited by 41 · cited 1×
“Research has provided evidence that computing education offers a platform to practice and improve problemsolving [29, 41, 60, 67], critical thinking [27], collaboration [75], and active learning [40] skills.”
Exploring the Learnability of Program Synthesizers by Novice Programmers
Jayagopal, D., Lubin, J., & Chasins, S. E. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545659 · Cited by 40 · cited 1×
“Finally, in a recent study to explore the learnability of program synthesizers, Jayagopal et al. [38] found that participants preferred program synthesis tools where users can also write code manually and liked the triggerless initiation and triggerless result communication mechanisms of Copilot.”
More Natural Programming Languages and Environments
Pane, J. F., & Myers, B. A. (2006) · Human-computer interaction series · doi:10.1007/1-4020-5386-x_3 · Cited by 38
Begel, A., & Graham, S. L. (2005) · doi:10.1109/vlhcc.2005.58 · Cited by 36
NLCI: a natural language command interpreter
Landhäußer, M., Weigelt, S., & Tichy, W. F. (2016) · Automated Software Engineering · doi:10.1007/s10515-016-0202-1 · Cited by 35 · cited 1×
“Various methods have been employed, such as using semantic parsers [4, 9, 11, 18, 33, 34, 47, 50, 51, 56, 68, 80] and machine learning [5, 54, 71–73, 93, 95].”
(2015) · HAL · Cited by 20
Kazemitabaar, M., Chyhir, V., Weintrop, D., & Grossman, T. (2022) · ACM International Conference on Interaction Design and Children (IDC) · doi:10.1145/3501712.3529733 · Cited by 17 · cited 2×
“To support this transition, various tools have been developed, including: dual-modality programming environments like PencilCode [6] and MakeCode [3], and hybrid environments such as Frame-based editing [48] and CodeStruct [42, 43].”
Self-citationSchlegel, V., Lang, B., Handschuh, S., & Freitas, A. (2019) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3301275.3302267 · Cited by 13
Ji, Y., & Gao, S. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 10
Kazemitabaar, M., Chyhir, V., Weintrop, D., & Grossman, T. (2023) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3545945.3569723 · Cited by 9 · cited 1×
“To support this transition, various tools have been developed, including: dual-modality programming environments like PencilCode [6] and MakeCode [3], and hybrid environments such as Frame-based editing [48] and CodeStruct [42, 43].”
Self-citationProgram Synthesis using Natural LanguagePreprint
Desai, A., Gulwani, S., Hingorani, V., Jain, N., Karkare, A., Marron, M., Sailesh, R., & Roy, S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.00413 · Cited by 2
Knöll, R., & Mezini, M. (2006) · Companion to the 21st ACM SIGPLAN symposium on Object-oriented programming systems, languages, and applications · doi:10.1145/1176617.1176628
Flagged first, then by citations
Research works printed but not linked: 6
- Amazon Web Services. 2022. CodeWhisperer: ML-powered coding companion. https://aws.amazon.com/codewhisperer/. [Online; accessed 9-September-2022].
- 10.5555/364133.364161 · DOI
- Paul Jaccard. 1901. Étude comparative de la distribution florale dans une portion des Alpes et des Jura. Bull Soc Vaudoise Sci Nat 37 (1901), 547–579.
- Mitchel Resnick. 2014. Give P’sa chance: Projects, peers, passion, play. In Constructionism and creativity: Proceedings of the third international constructionism conference. Austrian computer society, Vienna. 13–20.
- Mitchel Resnick and David Siegel. 2015. A different approach to coding. International Journal of People-Oriented Programming 4, 1(2015), 1–4.
- Tabnine. 2022. Tabnine: AI assistant for software developers. https://www.tabnine.com/. [Online; accessed 9-September-2022].
Cited by
Sorted by citations
Bahroun, Z., Anane, C., Ahmed, V., & Zacca, A. (2023) · Sustainability · doi:10.3390/su151712983 · Cited by 784
“Kazemitabaar et al. (2023) [19] Demonstrates the positive influence of AI code generators, like OpenAI Codex, on novice programmers, highlighting improvements in performance, computational thinking skills, and motivation.”
BackgroundThe Robots Are Here: Navigating the Generative AI Revolution in Computing Education
Prather, J., Denny, P., Leinonen, J., Becker, B. A., Albluwi, I., Craig, M., Keuning, H., Kiesler, N., Kohn, T., Luxton-Reilly, A., et al. (2023) · Proceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education · doi:10.1145/3623762.3633499 · Cited by 396
“studied how novice programmer productivity and learning is affected by the use of AI code generators [101].”
BackgroundInfluentialUsing an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
Computing Education in the Era of Generative AI
Denny, P., Prather, J., Becker, B. A., Finnie-Ansley, J., Hellas, A., Leinonen, J., Luxton-Reilly, A., Reeves, B. N., Santos, E. A., & Sarsa, S. (2024) · Communications of the ACM · doi:10.1145/3624720 · Cited by 316
“Depending on the prompt, the resulting programs used varied programmatic structures, while ultimately favoring expected methods for each problem variation (i.e., for-loops for processing lists, and while-loops for processing standard input).”
BackgroundThe Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
Metadata
- Authors
- Kazemitabaar, M., Chow, J., Ma, C. K. T., Ericson, B. J., Weintrop, D., & Grossman, T.
- Affiliations
- University of Toronto (CA)
- Department of Computer Science, University of Toronto, Canada
- University of Michigan (US)
- College of Education, College of Information Studies, University of Maryland, United States
- Venue
- Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2023-04-19
- Pages
- 1-23
- OpenAlex
- W4321162272
- Cited by
- 366
- References
- 96
- Citations by year
- 2026: 117 · 2025: 118 · 2024: 104 · 2023: 27
- FWCI
- 110.39
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- OpenAI Codex, novice programmers, introductory programming, code modification, knowledge retention, overreliance on AI, Scratch, Python, programming education
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
