The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index; 15 of the 46 sources its claims cite were checked for support.
How Do Programming Students Use Generative AI?
Open accessAbstract
Exploratory experiment with 37 programming students. Majority prompted ChatGPT to generate full solutions. Students got trapped in a vicious cycle of submitting wrong generated code and asking the bot for a fix. Regular AI users more likely to prompt for full solutions.
Overview
CoChat Summary
TL;DR
This paper combines a GPT task-solving evaluation of an introductory programming course with a monitored experiment in which 37 students could use ChatGPT while solving a code-comprehension exercise. It finds that GPT can solve most course exercises, but students often used the chatbot to request full code solutions, fell into ineffective error-fix cycles, and showed over-reliance rather than asking for explanations.
Key findings
- In the task-solving evaluation, both GPT models produced syntactically correct code in all responses, but GPT-3.5 had more logical errors and failed exercise requirements more than twice as often as GPT-4; the authors conclude a student could not pass the course by relying solely on GenAI because of in-person reviews and secondary assessments.
- In the student experiment, 42 participated and 37 were analyzed; 62.2% completed the entire exercise correctly and 78.4% solved at least one problem. 37.8% did not use the chatbot at all, while among chatbot users the prompt ratio was 62% Codegen to 38% Support, and 73.9% submitted a Solve prompt at some point.
- Manual analysis of code changes with chatbot interactions found 54.3% of submissions semantically identical to generated code (Copy All), 13.6% reused part of the generated solution (Idea), 11.1% copied a language construct (Syntax), 2.5% were based on generated explanation, and 18.5% were unrelated to chatbot interactions.
- Consecutive submissions were significantly less similar when chatbot interactions occurred between them: 64.1% vs 82.2% without chatbot interactions (t = 5.76, p < 0.0001); in 62% of cases with generations, the modified submission was more similar to generated code than to the previous submission. GPT-3.5 generated correct solutions for only 8.8% of 80 P1 solutions, but 76.9% of 13 P2 solutions.
Methods
- Task-solving evaluation: all course assignments were converted into prompts for GPT-3.5 (`gpt-3.5-turbo-1106`) and GPT-4 (`gpt-4-1106-preview`) with temperature 0.0; generated solutions were inserted into project files and evaluated against tutor criteria including syntax, logic, requirements, style, learnability, and intent.
- Student experiment: an in-person controlled study in the SE1 course at University of Hamburg, November–December 2023; participants solved a two-problem Moodle/CodeRunner exercise with optional access to a ChatGPT-like interface using `gpt-3.5-turbo-1106`, while code changes, chatbot conversations, copy/paste events, and submissions were recorded.
- Analysis: quantitative metrics and manual labeling of prompts and responses; code similarity used Ratcliff/Obershelp with a >90% threshold for close match; sequence pattern mining used NOSEP and NETLAP to identify interaction patterns.
Limitations
- Stated: the referenced LLMs may be updated and yield different outputs; prompt phrasing and the entirely German material may affect model performance; participant self-selection may bias the pool toward better-performing students; the sample came from one university course in one year; the experiment setting and recording may have influenced chatbot use and self-reported GenAI use; only one fairly complex exercise was tested.
- Stated: the authors recommend future studies test students’ understanding after completion and use exit surveys or think-aloud observations, because the study did not capture much information on students’ thoughts and impressions during the experiment.
- Evident from the text: the analyzed student sample is small (37 participants), and the findings are tied to a single course, task, and chatbot configuration, limiting generalizability beyond that setting.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
10 Valid · 6 Not valid · 20 Not checked · 171 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“This supports prior findings by Mailach et al. [2025] and Kruse et al. [2024] that some students do not understand what information precisely needs to be shared with the chatbot.”
What the sources say
- Not checked — no full text of the sourcecites Mailach et al. [2025] source
- Supportedcites Kruse et al. [2024] source
In C. Code Comprehension (page unavailable), the paper reports that less experienced students initially omitted relevant documentation keywords from their prompts and received less useful explanations. The Abstract (page unavailable) also says students wanted more support in formulating prompts. Together, these support the claim in the context of prompting a chatbot-like tool for code documentation. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Using LLMs for code generation shifts the developer's focus from ideation [Wei et al. 2024] to parsing and debugging unfamiliar code.”
What the sources say
- Partly supportedcites Wei et al. 2024 sourceExperimental — accuracy not yet measured
The UI Code Generation section describes LLM-generated HTML, and the Conclusion identifies UI design as a process involving ideation. But neither passage says that code generation shifts developers’ focus from ideation to parsing and debugging unfamiliar code; the UI Code Adjustment section discusses fixing generated HTML through additional prompts instead. The cited paper supports the code-generation and ideation context, not the claimed shift. (LLM, conf=0.82)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“However, as the results are fairly consistent with similar surveys around the time [Kazemitabaar et al. 2023; Prather et al. 2023a; Zastudil et al. 2023] and we were commonly able to observe behavior that could be considered unethical in an academic context [Prather et al. 2023a], we do not believe this to be a major limitation.”
What the sources say
- Could not verifycites Kazemitabaar et al. 2023 sourceExperimental — accuracy not yet measured
The citation is attached to the citing authors’ assertion that their results are consistent with similar surveys, not directly to their conclusion about social desirability bias. Sections 4.3 and 5.4 describe a post-study survey of learners’ perceptions of a Codex code generator, while Section 5.1.3 reports measured generator use. I found no passage addressing self-reported ChatGPT usage, its consistency with the citing survey, or whether social desirability bias is a major limitation. (LLM, conf=0.75)
- Not matchedcites Prather et al. 2023a
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Not checked — no full text of the sourcecites Zastudil et al. 2023 source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Could not verify
The cited sources that could be checked couldn’t confirm or dispute this claim.
In the paper
“Our findings suggest that once learners decide turn to a ChatGPT-like assistant for help, especially if they already have prior experience with generative AI, they predominantly seek code solutions rather than support in creatively working through the problem [Wei et al. 2024].”
What the sources say
- Could not verifycites Wei et al. 2024 sourceExperimental — accuracy not yet measured
The UI Code Generation section describes generating HTML, while Boosting Creativity with AI and the design-process discussion concern designers and practitioner teams. Our copy does not address learners’ help-seeking choices, whether prior generative-AI experience affects them, or whether learners predominantly seek code solutions rather than creative support. Searches for learners, prior experience, and code-solution preferences found no passage on those points; the cited passages have no page numbers. (LLM, conf=0.75)
hedgedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
- Could not verify
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All participants had access to the chatbot interface. Still, 37.8% did not use it at all during the experiment.”
- Also stated in: § 3.2 Chatbot Use (RQ2) · plain
“Still, 37.8% did not use it at all during the experiment.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 3.2 Chatbot Use (RQ2) · plain
109 more claims
Create a free account to see all 207 claims and every source verdict.
- Borderline: one of the tests for a substantial claim is unclear. 1
- Probably not a substantial claim. 49
- It may state two separate claims as one. 63
- The quote only partly states this claim. 1
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“In total, 177 prompts were sent to the chatbot and 172 responses were received. The remaining 5 prompts went unanswered due to network interruptions. On average, each participant submitted 4.57 prompts (M = 2, 𝜎 = 5.96).”
Reported 4.57 · recomputed 4.783784
Citations
Cites
The Content Analysis Guidebook
Neuendorf, K. A. (2017) · doi:10.4135/9781071802878 · Cited by 9,286 · cited 1×
“The labels were initially created through a deductive approach [Maalej and Robillard 2013; Neuendorf 2017], and then consolidated after reviewing the actual chat logs.”
ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education
Kasneci, E., Seßler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · doi:10.35542/osf.io/5er8f · Cited by 7,552
ChatGPT for good? On opportunities and challenges of large language models for education
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., et al. (2023) · Learning and Individual Differences · doi:10.1016/j.lindif.2023.102274 · Cited by 7,326 · cited 2×
“…highlight the risk of learners relying too heavily on the model, noting that “the model simplifies the acquisition of answers or information, which can amplify laziness and counteract the learners’ interest to [...] come to their own conclusions and solutions” [Kasneci et al. 2023, p. 5].”
InfluentialChatting and cheating: Ensuring academic integrity in the era of ChatGPT
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2024) · Innovations in Education and Teaching International · doi:10.1080/14703297.2023.2190148 · Cited by 3,087
Chatting and Cheating. Ensuring academic integrity in the era of ChatGPT
Cotton, D., Cotton, P., & Shipway, J. R. (2023) · doi:10.35542/osf.io/mrz8h · Cited by 3,079
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · cited 1×
“…as ChatGPT [OpenAI 2022] and GitHub Copilot [GitHub 2022], students in programming courses now have access to code authoring tools capable of solving coding exercises [Berrezueta-Guzman and Krusche 2023; Chen et al. 2021; Savelka et al. 2023] and exam questions entirely [Finnie-Ansley et al. 2023].”
The role of trust in automation reliance
Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., & Beck, H. P. (2003) · International Journal of Human-Computer Studies · doi:10.1016/s1071-5819(03)00038-7 · Cited by 1,276
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660 · cited 1×
“Prior work has shown that this is a challenge for professional developers [Sarkar et al. 2022; Vaithilingam et al. 2022].”
How is ChatGPT's behavior changing over time?
Chen, L., Zaharia, M., & Zou, J. (2023) · arXiv · doi:10.48550/arxiv.2307.09009 · Cited by 507 · cited 1×
“5 and GPT-4 over time [Chen et al. 2023], though it is not clear whether these results are based on architectural changes, continuous training, or simply the non-deterministic nature of the LLMs.”
How Is ChatGPT’s Behavior Changing Over Time?
Chen, L., Zaharia, M., & Zou, J. (2024) · Harvard Data Science Review · doi:10.1162/99608f92.5317da47 · Cited by 507
Research works printed but not linked: 3
- David J Barnes and Michael Kölling. 2009. Java lernen mit BlueJ: Eine Einführung in die objektorientierte Programmierung. Pearson Deutschland GmbH.
- Marlo Häring and Walid Maalej. 2019. A Socio-Technical Framework for Face-to-Face Teaching in Large Software Development Courses.. In Software Engineering (Workshops). 3–6.
- John W. Ratcliff and DM Metzener. 1988. Gestalt: an introduction to the Ratcliff/Obershelp pattern matching algorithm. Dr. Dobbs Journal 7 (1988), 46.
Cited by
Reihanian, I., Hou, Y., & Sun, Q. (2025) · AI · doi:10.3390/ai7010006 · Cited by 15
Stoyanova, D., Stoyanova-Petrova, S. V., Shotarova, S., Lyubomirov, S. Y., & Mileva, N. S. (2025) · Education Sciences · doi:10.3390/educsci16010019 · Cited by 13
``I Would Have Written My Code Differently': Beginners Struggle to Understand LLM-Generated Code
Zi, Y., Li, L., Guha, A., Anderson, C. J., & Feldman, M. Q. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3731663 · Cited by 33
Examining the Usage of Generative AI Models in Student Learning Activities for Software Programming
Chen, R., Jiang, S., Shen, J., Moon, A., & Wei, L. (2025) · ACM International Conference on AI-Powered Software (AIware) · doi:10.1109/aiware69974.2025.00016 · Cited by 3
Rizvi, S. W. F., Yankouskaya, A., Alshakhsi, S. A. A., Al‐Thani, D., & Ali, R. (2025) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm67465.2025.11391082 · Cited by 2
“In programming context, an experimental study showed concerns of students who regularly use AI chatbot, highlighting potential overreliance and reduced engagement in problem-solving tasks [18].”
Roy, N., Horielko, O., & Omojokun, O. (2025) · ACM Conference on International Computing Education Research (ICER) · doi:10.1145/3702653.3744328 · Cited by 2
Aimicheva, G., Bekmanova, G., Omarbekova, A., Nazyrova, A., Khamzina, A., & Kadyr, Y. (2025) · Technologies · doi:10.3390/technologies14010005 · Cited by 3
Recalibrating Academic Expertise in the Age of Generative AIPreprint
Sohail, A., & Lin, Z. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5224452 · Cited by 1
Preliminary Study on a Multi-Source RAG Chatbot Using Lecture Materials and QA Logs
Ochi, Y., & Hamada, Y. (2025) · IEEE Global Conference on Consumer Electronics (GCCE) · doi:10.1109/gcce65946.2025.11275388
“Rahe and Maalej [3] observed that many students primarily used AI tools to generate code rather than as aids for debugging or conceptual reinforcement, raising concerns about diminished critical thinking.”
Generative AI and IT Professionals: Work Tasks and Ethical Reflections
Spens, A.-S. F., Mikkelsen, K. G., Lykke, M., & Svarre, T. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-26717-7_18
WIP: Context-Aware AI in Learning Management Systems for Computer Science Courses
Simmons, A., de Holanda, M. T., & Silva, D. D. (2025) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie63693.2025.11328241
Bamfo, K. A., Hall-Holt, O. A., Ola, O., Owusu, D. A., & Yeluripati, G. R. (2026) · ACM Technical Symposium on Computer Science Education (SIGCSE) · doi:10.1145/3770762.3772659 · Cited by 3
PockeTA: Virtual Teaching Assistant Platform with Course Context
Sin, Z. P. T., Cheung, M. H. T., Jia, Y., Wang, X. E., Lai, M. W. H., Li, C., Huang, X., Ng, P. H. F., Baciu, G., & Li, Q. (2026) · Lecture notes in computer science · doi:10.1007/978-981-92-0042-9_4
Inoue, S., Morisaki, S., Wada, T., & Hattori, Y. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3702933
“Practices or approaches for using generative AI without offloading are necessary to prevent developers from incorrectly comprehending source code owing to comprehension outsourcing [59], [60].”
How students use generative AI for software testing: An observational study
Ardıç, B., Dilavrec, Q. L., & Zaidman, A. (2026) · Empirical Software Engineering · doi:10.1007/s10664-026-10898-0 · Cited by 8
“[32] conducted an in-depth study on how undergraduate programming students use ChatGPT during a programming exercise.”
Oyelere, S. S. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100633
AI Is Available, Partnership Is Optional: Student Agency in Human–AI Interaction
Retes, R. M., Escalona, S. G., Webster, M., Giotto, F. M., & Wayllace, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-29770-9_7 · Cited by 1
How Large Language Models Shape Programming Skill Development Beyond Task Completion
Orehovački, T. (2026) · AI · doi:10.3390/ai7070252 · Cited by 1
“Related uses are visible beyond formal instructional interventions, including advanced web development and routine programming workflows, where generative AI is used during debugging, implementation refinement, code explanation, and problem solving [1,30].”
Fast and Fleeting: Evaluating ChatGPT’s Impact on Students’ Computational Thinking Skills
Mahmoud, M., Asare, E., Shahid, N., Sakr, N., & Nadi, S. (2026) · International Workshop on Cooperative and Human Aspects of Software Engineering (CHASE) · doi:10.1145/3794860.3794894 · Cited by 1
Prompting Without Principles: Are Students Transferring Software Engineering Knowledge to LLM Use?
Sousa, L., Ozkaya, I., Ivers, J., Cywinska, C., Xie, B., Kostial, M., Dey, T., & Edman, R. (2026) · SEET@ICSE · doi:10.1145/3786580.3786990
Pombo, N. (2026) · IEEE International Conference on Software Testing, Verification and Validation (ICST) · doi:10.1109/icst69053.2026.00024
“They may also find it difficult to identify implicit assumptions or hidden defects in generated code, leading to over-reliance on automated outputs without sufficient verification [14].”
Keeping Humans in the Driver’s Seat: A Conceptual Framework for GenAI and Competence Debt
Lüders, C. M., Krużycki, O., & Brandt, C. (2026) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3803437.3806707
How Well Can AI Generate Backlogs from App Mockups?
Airaldi, A. L., Romera, L., & Maalej, W. (2026) · IEEE International Requirements Engineering Conference Workshops (REW) · doi:10.1109/rew72749.2026.00017
“The team in Project 6 worried about over-reliance on AI, warning that it might discourage critical thinking [22].”
Self-citationApplications, challenges and opportunities of AI integration in STEM education
Atchia, S. M. C., & Chuen, J. A. (2026) · Elsevier eBooks · doi:10.1016/b978-0-443-36775-5.00001-1
Mellado, R., Cubillos, C., & Roncagliolo, S. (2026) · Behavioral Sciences · doi:10.3390/bs16081405
“…have been reported by students who value feedback they describe as specific and corrective (Hernandez et al., 2025; H.-J. Li et al., 2025), whereas Rahe and Maalej (2025) identified learners who proceed by trial and error and request complete solutions without examining the code they receive, a…”
How can large language models help novice programmers? An exploratory study in educational setting
Zhou, Y., Peng, J., Zhang, M., Tao, Y., & Liu, Y. (2026) · Journal of Systems and Software · doi:10.1016/j.jss.2026.113073
Understanding Generative AI-Supported Problem Solving and Learning in Programming Education
Tian, Y., & Jiang, P. (2026) · International Conference on Distance Education and Learning (ICDEL) · doi:10.1109/icdel70250.2026.11663855
“For instance, generative AI can be used to explain programming concepts, interpret code snippets, provide debugging suggestions, and generate alternative solutions [6].”
Lai, C.-H., Lin, Y.-H., Chien, W., & Yang, Y.-S. (2026) · Electronics · doi:10.3390/electronics15173967 · Cited by 1
Keskinkılıç, F., & Kalelıoğlu, F. (2026) · Journal of Educational Technology Systems · doi:10.1177/00472395261484623
Lee, E. (2026) · The Journal of Korean Association of Computer Education · doi:10.32431/kace.2026.29.8.001
CodePalz: GenAI Powered Adaptive AI Learning
Sable, N. P., Darda, A., Anthony, A., Akulwar, S. P., & Doshi, A. (2026) · International Conference on Trends in Quantum Computing and Emerging Business Technologies (TQCEBT) · doi:10.1109/tqcebt67648.2026.11681022
“At the same times, researchers mention the importance of safety when integrating these models into real-world [19,22].”
Analisis Persepsi Siswa terhadap Pemanfaatan Sublime Text dalam Pembelajaran Pemrograman Website SMK
Senia, S., Saputra, H. N., & Razilu, Z. (2026) · Journal of Instructional and Development Researches · doi:10.53621/jider.v6i3.800
“Kajian tentang code completion dan alat pendukung pemrograman juga memperlihatkan bahwa fitur editor dapat membantu produktivitas, tetapi dapat pula menimbulkan ketergantungan apabila tidak dibarengi pemahaman konseptual (Chen et al., 2023; Rahe & Maalej, 2025; Takerngsaksiri et al., 2023).”
Hadjichristofi, C., Gerokonstantis, D., Papadeas, P. V., & Vescoukis, V. (2026) · Annals of Computer Science and Information Systems · doi:10.15439/2026f9512
“Related work examines how students incorporate generative AI into programming activities, identifying recurring usage patterns and interaction strategies [13], [14].”
Teaching Scalability Reasoning through a Complexity-First Approach to Algorithms and Data Structures
Ungureanu, V. (2026) · ACM Transactions on Computing Education · doi:10.1145/3849390
“Empirical studies document the use of generative AI systems by programming students as part of their problem-solving and coding workflows [Rahe and Maalej 2025].”
Generative AI and the balance of effort in group tasks: Appropriate, over-, and under-scaffolding
Korchak, A., Hughes, C., & Costley, J. (2026) · Journal of Research on Technology in Education · doi:10.1080/15391523.2026.2724351
Zhu, L., Qian, Y., & Wang, M. (2026) · ACM Turing Celebration Conference - China (ACM TURC) · doi:10.1145/3833142.3833154
“It may fail to fully identify issues in students’ code, generate misleading suggestions, or directly provide complete answers despite prompt constraints, which may increase students’ dependence and weaken their autonomous problem-solving processes [1, 6, 17].”
Metadata
- Authors
- Rahe, C., & Maalej, W.
- Affiliations
- Universität Hamburg (DE)
- Venue
- Proceedings of the ACM on Software Engineering
- Published
- 2025-06-19
- Volume
- 2
- Issue
- FSE
- Pages
- 978-1000
- ISSN
- 2994-970X (ISSN-L)
- OpenAlex
- W4406604115
- Cited by
- 45
- References
- 43
- Citations by year
- 2026: 33 · 2025: 12
- FWCI
- 39.98
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Proceedings of the ACM on software engineering. (publishedVersion, cc-by-nc)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- generative AI, ChatGPT, programming education, student use of AI, overreliance on AI, critical thinking, program comprehension, code reasoning, student learning behavior
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-01, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-01 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.