CoChat
Sign Up
PreprintCited by 10Teaching and Learning ProgrammingSoftware Engineering ResearchHuman-Automation Interaction and Safety

How AI Impacts Skill Formation

Open access
Shen, Judy Hanwen, Tamkin, Alex·2026·arXiv
DOI 10.48550/arxiv.2601.20245arXiv 2601.20245
Open this paper in CoChat — free accountOpen PDF

Abstract

AI assistance produces significant productivity gains across professional domains, particularly for novice workers. Yet how this assistance affects the development of skills required to effectively supervise AI remains unclear. Novice workers who rely heavily on AI to complete unfamiliar tasks may compromise their own skill acquisition in the process. We conduct randomized experiments to study how developers gained mastery of a new asynchronous programming library with and without the assistance of AI. We find that AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average. Participants who fully delegated coding tasks showed some productivity improvements, but at the cost of learning the library. We identify six distinct AI interaction patterns, three of which involve cognitive engagement and preserve learning outcomes even when participants receive AI assistance. Our findings suggest that AI-enhanced productivity is not a shortcut to competence and AI assistance should be carefully adopted into workflows to preserve skill formation -- particularly in safety-critical domains.

AI snapshot

Written from the paper’s full text.

TL;DR

AI assistance reduced skill formation for developers learning a new asynchronous Python library, with lower quiz scores and no statistically significant average speedup. The paper identifies six AI interaction patterns, three of which preserve learning by keeping users cognitively engaged.

Key findings

  • AI-assisted participants scored 17% lower on the 27-point quiz, a 4.15-point difference or two grade points (Cohen’s d = 0.738, p = 0.010); controlling for warm-up task time, the effect remained significant (d = 0.725, p = 0.016).
  • The main study found no statistically significant acceleration in task completion time with AI assistance, despite the AI assistant being able to generate full correct code.
  • Six AI interaction patterns emerged: low-scoring patterns (24%–39% quiz scores) involved heavy reliance on AI, while high-scoring patterns (65%–86%) involved more cognitive effort, such as asking conceptual questions or requesting explanations alongside generated code.
  • The AI group encountered fewer errors (median 1 vs. 3 in control), and the largest treatment–control quiz gap was on debugging questions, with the smallest gap on code reading.

Methods

  • Between-subjects randomized experiment with 52 participants (26 AI treatment, 26 no-AI control), recruited with >1 year Python experience, weekly Python use, prior AI coding assistance use, and no prior Trio library use.
  • Participants completed a warm-up coding task, then two Trio asynchronous programming tasks within a 35-minute limit; the AI condition had access to a GPT-4o chat assistant with access to their code.
  • Skill formation was measured by a 14-question, 27-point quiz covering conceptual, code reading, and debugging questions; qualitative analysis annotated screen recordings for 51/52 participants.

Limitations

  • Stated: the study uses a single task and chat-based interface, which the authors call a lower bound for cognitive offloading; skill formation was measured over about one hour rather than months or years; participants lacked real-job incentives to learn the library; prompting skills were not measured beyond self-report; evaluation relied on a quiz; and no human-assistance counterfactual was included.
  • Evident from the text: the main sample is small (n=52), interaction-pattern clusters are very small (n=2–7), pilot studies had non-compliance (35% and 25% using AI in no-AI conditions), one participant was disqualified, and screen recording was unavailable for one participant.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2601.20245

Cites

43 of 43 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

12 of 26 cited sources checked · 6 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Knowledge workers reported lower cognitive effort and confidence when using generative AI tools.

    “Furthermore, knowledge workers reported a lower cognitive effort and confidence when using generative AI tools [Lee et al., 2025].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Lee et al., 2025 source

      Section 5.1 reports that knowledge workers generally perceived less effort for cognitive activities associated with critical thinking when using GenAI. The paper does not report that using GenAI lowered their confidence; its Discussion (Section 6.1) instead describes confidence in AI and self-confidence as factors associated with critical-thinking effort. Page numbers were not available in our copy. (LLM, conf=0.89)

      Experimental — accuracy not yet measured
  • Premisecitation

    For software engineering, AI assistance significantly improves productivity in daily tasks.

    “Software engineering, in particular, has been identified as a profession in which AI tools can be readily applied and AI assistance significantly improves productivity in daily tasks [Peng et al., 2023, Cui et al., 2024].”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Peng et al., 2023 source

      The Introduction reports that developers using GitHub Copilot completed a programming task significantly faster than controls. The Discussion specifies that this was one standardized task and cautions that benefits may vary across tasks, so the paper does not establish significant productivity gains in daily software-engineering work generally. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
    • cites Cui et al., 2024 source
  • Premisecitation

    Generative AI can produce incorrect or hallucinated content.

    “the reality is that generative AI can produce incorrect [Longwell et al., 2024] or hallucinated content [Maleki et al., 2024].”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites Longwell et al., 2024 source
    • cites Maleki et al., 2024 source
  • Findingown data

    Contrary to previous work, our results do not show a significant improvement in productivity if we only look at the total completion time across the treatment and control groups.

    “Contrary to previous work finding significant uplift or speedup of AI assistance for coding [Peng et al., 2023, Cui et al., 2024], our results do not show a significant improvement in productivity if we only look at the total completion time across the treatment and control groups.”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites Peng et al., 2023 source
    • cites Cui et al., 2024 source
  • Findingown data

    There are six distinct AI interaction patterns.

    “We identify six distinct AI interaction patterns, three of which involve cognitive engagement and preserve learning outcomes even when participants receive AI assistance.”
    • Also stated in: § 1.1 Our Results · plain
      “We categorize AI interaction behavior into six common patterns”
    Quote in the paperSubstantial claim
  • Classified as probably not a substantial claim. 60
  • May combine two claims. 50
  • The quote only partly states this claim. 2

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Premisecitation

    AI assistance produces significant productivity gains across professional domains, particularly for novice workers.

    “AI assistance produces significant productivity gains across professional domains, particularly for novice workers.”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Dell'Acqua et al., 2023 source
    • cites Peng et al., 2023 source

      The Introduction reports a statistically significant productivity gain for developers using GitHub Copilot on a JavaScript programming task, and the Results say less experienced developers benefited more. The paper does not establish gains across professional domains or for novice workers generally; the Discussion cautions that benefits may vary across tasks and programming languages. (LLM, conf=0.90)

      Experimental — accuracy not yet measured
    • cites Cui et al., 2024 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Prior works have presented mixed results on whether AI helps or hinders coding productivity.

    “Prior works have presented mixed results on whether AI helps or hinders coding productivity [Peng et al., 2023, Becker et al., 2025]; our study differs from prior results in that it is designed to study how AI affects skill formation while performing a task requiring new knowledge.”
    Quote in the paperNot a substantial claim2 of 2 sources checked
    • cites Peng et al., 2023 source

      The Introduction reports that developers using GitHub Copilot completed a programming task 55.8% faster than controls, supporting the claim’s ‘helps’ side. The Discussion cautions that benefits may vary by task and language, but it does not report a finding that AI hinders coding productivity or establish that prior results were mixed. This verdict assesses Peng et al. only, not the other citation in the sentence. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites Becker et al., 2025 source

  • May combine two claims.

    Premisecitation

    Frequent use of AI has been associated with worse critical thinking abilities and increased cognitive offloading in surveys of knowledge workers.

    “In surveys given to knowledge workers, frequent use of AI has been associated with worse critical thinking abilities and increased cognitive offloading [Gerlich, 2025].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Gerlich, 2025 source

      The correlation analysis (section 4.3) reports that greater AI use was associated with more cognitive offloading and lower critical-thinking scores. But the participants section (3.2) describes a diverse UK sample, and the occupation frequencies include 185 students; the paper does not establish these associations specifically among knowledge workers. (LLM, conf=0.89)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Prior works have found that humans use AI in many different ways for coding: from question answering to writing code, to debugging.

    “Prior works have found that humans use AI in many different ways for coding: from question answering to writing code, to debugging [Poitras et al., 2024, Wang et al., 2020, Pinto et al., 2024].”
    Quote in the paperNot a substantial claim1 of 3 sources checked
    • cites Poitras et al., 2024 source
    • cites Wang et al., 2020 source
    • cites Pinto et al., 2024 source

      The Introduction and Conclusion describe developers using AI assistants to answer contextual questions and generate code. The Discussion also describes refining code suggestions through chat. The passages from our copy do not establish debugging as a use, so the full list of uses is only partially supported (page numbers not supplied). (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The shift from coding to reading and understanding has also been found in previous work.

    “This shift from coding to reading and understanding has also been found in previous work [Becker et al., 2025].”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Becker et al., 2025 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationcitation

    Expressions of appreciation reflect that politeness in human-AI interaction also appears in the context of AI for coding assistance.

    “These expressions of appreciation, even at the cost of additional task time, reflect that politeness in the human-AI interaction [Druga et al., 2017, Ribino, 2023] also appears in the context of AI for coding assistance.”
    Quote in the paperNot a substantial claim1 of 2 sources checked
    • cites Druga et al., 2017 source
    • cites Ribino, 2023 source

  • May combine two claims.

    Findingown data

    Some participants asked up to 15 questions or spent more than 30% of the total available task time on composing queries.

    “Some participants asked up to 15 questions or spent more than 30% of the total available task time on composing queries (Figure 12).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    For a 27-point quiz, the 4.15 point difference translates into a 17% score difference or 2 grade points.

    “For a 27-point quiz, this translates into a 17% score difference or 2 grade points.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    AI use impairs conceptual understanding, code reading, and debugging abilities.

    “We find that AI use impairs conceptual understanding, code reading, and debugging abilities, without delivering significant efficiency gains on average.”
    • Also stated in: § 1 Introduction · plain
      “We find a significant decrease in library-specific skills (conceptual understanding, code reading, and debugging) among workers using AI assistance for completing tasks with a new python library.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    In the main study, 4 of 26 control participants did not complete the second task within the 35-minute limit, while every participant in the AI condition completed the second task.

    “4 of the 26 participants in the control (No AI) group did not complete the second task within the 35-minute limit, while every participant in the AI condition completed the second task.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data

    There is a 4.15 point difference between the means of the treatment and control groups on the quiz.

    “There is a 4.15 point difference between the means of the treatment and control groups.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    Among the different types of questions, the largest score gap occurs in the debugging questions and the smallest score gap in the code reading questions.

    “Among the different types of questions, the largest score gap occurs in the debugging questions and the smallest score gap in the code reading questions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Among novice programmers (1-3 years experience), the AI group had a slightly lower average completion time, but the difference in task time was not significant due to the small group size (n=4).

    “While we do observe a slightly lower average completion time in the AI group among novice programmers, due to the small group size of the 1-3 year participant group (n=4), the difference in task time was not significant.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Three of the six AI interaction patterns involve cognitive engagement and preserve learning outcomes even when participants receive AI assistance.

    “We identify six distinct AI interaction patterns, three of which involve cognitive engagement and preserve learning outcomes even when participants receive AI assistance.”
    • Also stated in: § 1.1 Our Results · plain
      “find three AI interaction patterns that best preserve skill development”
    • Also stated in: § 1.1 Our Results · plain
      “These three patterns of interaction with AI, which resulted in higher scores in our skill evaluation, involve more cognitive effort and independent thinking”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A higher fraction of debugging queries correlates with slower completion times and lower quiz scores.

    “A higher fraction of debugging queries correlates with slower completion times (Figure 18) and lower quiz scores (Figure 19).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    A smaller group of participants in the AI condition (n=4) mostly wrote their own code without copying or pasting the generated code; these participants were relatively fast and demonstrated high proficiency by only asking AI assistant clarification questions.

    “There was a smaller group of participants in the AI condition who mostly wrote their own code without copying or pasting the generated code (n = 4); these participants were relatively fast and demonstrated high proficiency by only asking AI assistant clarification questions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    In response to these questions, the AI assistant clarified that they could see the code and instructions.

    “In response to these questions, the AI assistance clarified that they could see the code and instructions.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    A quarter of the participants left feedback after the task and quiz were completed.

    “A quarter of the participants left feedback after the task and quiz were completed.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The AI group encountered fewer errors than the control group: the median participant in the treatment group encountered only one error in the entire task, while the median for the control group was three errors.

    “The AI group encountered fewer errors than the control group: the median participant in the treatment group encountered only one error in the entire task, while the median for the control group was three errors.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    High-scoring interaction patterns were clusters of behaviors where the average quiz score is 65% or higher.

    “High-scoring interaction patterns were clusters of behaviors where the average quiz score is 65% or higher.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Conceptual Inquiry group (n=7) only asked conceptual questions and relied on their improved understanding to complete the task.

    “Conceptual Inquiry (n=7): Participants in this group only asked conceptual questions and relied on their improved understanding to complete the task.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    A third category of common queries was debugging (q=9).

    “A third category of common queries was debugging (q=9).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Although participants in the AI condition also encounter errors, there are much fewer Trio-related errors encountered.

    “Although participants in the AI condition also encounter errors (Figure 15), there are much fewer Trio-related errors encountered.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The Hybrid Code-Explanation group (n=3) composed hybrid queries in which they asked for code generation along with explanations of the generated code.

    “Hybrid Code-Explanation (n=3): Participants in this group composed hybrid queries in which they asked for code generation along with explanations of the generated code.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    There were questions (q=4) about whether the assistant could see the existing code and whether the assistant had knowledge of the specific library.

    “Although we only recruited participants who have used AI assistants before, there were still questions (q=4) about whether the assistant could see the existing code and whether the assistant had knowledge of the specific library.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Conceptual Inquiry group encountered many errors but independently resolved them.

    “Although this group encountered many errors, they also independently resolved these errors.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The most common errors are not directly related to the Trio library: NameError and AttributeError are typically typos made on variable names or function names that are quickly corrected.

    “Figure 14 shows that the most common errors are not directly related to the Trio library: NameError and AttributeError are typically typos made on variable names or function names that are quickly corrected.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingnone given

    In general, most participants ran the code for the first time after trying to complete most of the question and ran the code again only after the changes were made.

    “In general, most of the participants ran the code for the first time after trying to complete most of the question and ran the code again only after the changes were made.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Generation-Then-Comprehension participants were not particularly fast when using AI, but demonstrated a high level of understanding on the quiz.

    “These participants were not particularly fast when using AI, but demonstrated a high level of understanding on the quiz.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The second most common queries were asking for code to be generated (q=51).

    “The second most common were queries asking for code to be generated (q=51); some participants asked for an entire task to be completed, while other participants asked for specific functions to be implemented.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The tasks were designed to be straightforward, but participants still encountered various errors.

    “Our tasks were designed to be straightforward, but the participants still encountered various errors (Section 6.2).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    On average, the Conceptual Inquiry mode was the fastest among high-scoring patterns and second fastest overall after the AI Delegation mode.

    “On average, this mode was the fastest among high-scoring patterns and second fastest overall after the AI Delegation mode.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The Iterative AI Debugging group made a higher number of queries to the AI assistant, but relied on the assistant to solve problems rather than clarifying their own understanding.

    “This group made a higher number of queries to the AI assistant, but relied on the assistant to solve problems, rather than clarifying their own understanding.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The six AI interaction patterns yield different outcomes for both completion time and skill formation (quiz score).

    “As a result of this categorization, these six patterns yield different outcomes for both completion time and skill formation (i.e., quiz score).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The six AI interaction patterns based on query types, number of queries, queries per task, and active time adequately capture meaningful differences in AI use.

  • May combine two claims.

    Findingown data

    A handful of participants asked more than five questions and spent up to six minutes composing a single query during the 35-minute assignment.

    “Since participants could ask the AI assistant as many questions as time allowed, a handful of participants asked more than five questions and spent up to six minutes composing a single query during this 35-minute assignment (Figure 12).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Several participants expressed their appreciation for the assistant after the task was completed correctly.

    “Several participants also expressed their appreciation for the assistant after the task was completed correctly.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    The most common type of query was explanations (q=79).

    “The most common type of query was explanations (q=79); users requested more information about the trio library, details about asynchronous operations, and high-level conceptual introductions.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    Some participants directly paste AI-written code, while other participants manually typed in (i.e., copied) the AI generated code into their own file.

    “Copying Another pattern that differs between participants is that some participants directly paste AI-written code, while other participants manually typed in (i.e., copied) the the AI generated code into their own file.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning

    As a result, the Iterative AI Debugging group scored poorly on the quiz and were relatively slower at completing the two tasks.

    “As a result, they scored poorly on the quiz and were relatively slower at completing the two tasks.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    The Generation-Then-Comprehension group (n=2) first generated code, then manually copied or pasted the code, and then asked follow-up questions to improve understanding.

    “Generation-Then-Comprehension (n=2): Participants in this group first generated code and then manually copied or pasted the code into their work.After their code was generated, they then asked the AI assistant follow-up questions to improve understanding.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The AI Delegation group completed the task the fastest and encountered few or no errors.

    “This group completed the task the fastest and encountered few or no errors in the process.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The control group achieves high quiz scores with a higher total active time without the use of AI.

    “When we look at quiz score, the control group achieves high quiz scores with a higher total active time without the use of AI.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data

    Participants reported feeling 'lazy' and that there are still a lot of gaps in their understanding.

    “Specifically, participants reported feeling 'lazy' and that 'there are still a lot of gaps in (their) understanding'.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingnone given

    As the number of queries the total completion time increases, users who ask a high fraction of debugging queries also tend to use more time to complete the task.

    “As the number of queries the total completion time increases, users who ask a high fraction of debugging queries also tend to use more time to complete the task.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    While using AI improved the average completion time of the task, the improvement in efficiency was not significant in our study.

    “While using AI improved the average completion time of the task, the improvement in efficiency was not significant in our study, despite the AI Assistant being able to generate the complete code solution when prompted.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    Participants using AI by directly pasting outputs experience the most significant speed ups while participants who manually copied the AI-generated output were similar in pace to the control (No AI) group.

    “Participants using AI by directly pasting outputs experience the most significant speed ups while participants who manually copied the AI-generated output were similar in pace to the control (No AI) group.”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data

    In the main study (n=52), there was a difference in means of overall task time and quiz score between the control (No AI) and treatment (AI Assistant) groups.

    “Difference in means of overall task time and quiz score between the control (No AI) and treatment (AI Assistant) groups in main study (n=52).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Findingnone given

    The average completion time was faster for the AI group.

    “Since the average completion time was faster for the AI group, the AI group left more comments since they felt like they had more time at the end of the task.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingnone given

    In the study, users who relied on AI without thinking performed the worst on the evaluation.

    “In our work, users who relied on AI without thinking performed the worst on the evaluation; a completely agentic tool would create a similar effect.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    Most participants are professional programmers.

    “Most participants are professional programmers.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    In the control group (No AI), participants remarked that they found the task fun and that the task instructions were good at helping develop an understanding of Trio.

    “In the control group (No AI), participants remarked that they found the task fun and that the tasks instructions were good at helping develop an understanding of Trio.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data

    In the treatment group (AI Assistance), participants remarked that they wished they had paid more attention to the details of the Trio library during the task.

    “In the treatment group (AI Assistance), participants remarked that they wished they had paid more attention to the details of the Trio library during the task, either by reading the generated code or by generating explanations in more depth.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data

    The No AI condition participants spent more active time coding and achieved higher quiz scores.

    “The No AI condition participants spent more active time coding and achieved higher quiz scores.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Following preregistered disqualification criteria, 1 participant was disqualified after leaving four blank questions on the quiz due to not realizing there were multiple parts and running out of time.

    “Following our preregistered disqualification criteria, 1 participant was disqualified after leaving four blank questions on the quiz due to not realizing that there were multiple parts of the quiz and subsequently running out of time.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data

    Participants were recruited only if they self-reported having more than one year of Python experience, code in Python at least once a week, have tried AI coding assistance at least a few times, and have never used the Trio library before.

    “For all our pilot studies and the main study, we only recruited participants who self-reported having more than one year of Python experience, code in Python at least once a week, have tried AI coding assistance at least a few times, and have never used the Trio library before (Table 1).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    Participants were balanced across years of coding experience, years of Python experience, prior usage of the Python Asyncio library, frequency of Python use in the past year, and an asynchronous programming familiarity score.

    “Participants were balanced across the following attributes (recorded through a separate recruitment survey): years of coding experience, years of Python experience, prior usage of the Python Asyncio library, frequency of Python use in the past year, and an asynchronous programming familiarity score (a 5-question, multiple-choice concept check).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premiseown data

    The Trio library is less well known than asyncio and involves new concepts beyond Python fluency.

    “This library is less well known than asyncio (according to the number of StackOverflow questions) and involves new concepts (e.g., structured concurrency) beyond just Python fluency.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductown data

    Using screen recordings, it was verified that participants did not use AI in the control group nor for the quiz on platform P2.

    “Using screen recordings of participant progress, we verified that participants did not use AI in the control group nor for the quiz.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Premisecitation

    Novice programmers benefited more from AI coding assistance.

    “For software engineering in particular, Peng et al. found that crowd-sourced software developers using copilot completed a task 55.5% faster than the control group and novice programmers benefited more from AI coding assistance.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Peng et al.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    53 participants completed all three parts of the study.

    “53 participants completed all three parts of the study.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    The largest gap in debugging and smallest in code reading is expected because treatment and control groups may have similar exposure to reading code, but the control group encountered more errors and became more capable at debugging.

    “This outcome is expected since treatment and control groups may have similar exposure to reading code through the task, but the control group with no access to AI assistance encountered more errors during the task and became more capable at debugging.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The grading rubric for the quiz was submitted in the study pre-registration before running the experiment.

    “We submitted the grading rubric for the quiz in our study pre-registration before running the experiment.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given

    How AI assistance affects the development of skills required to effectively supervise AI remains unclear.

    “Yet how this assistance affects the development of skills required to effectively supervise AI remains unclear.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The two tasks in the study are standalone; sufficient instructions and usage examples are provided so that participants can complete one task without the other.

    “These two tasks are standalone; we provide sufficient instructions and usage examples so that participants can complete one task without the other.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The choice of tasks and task design did not critically hinge on participants' experience level but presented new skills to be acquired for every experience group.

    “This shows that our choice of tasks and task design did not critically hinge on the participants' experience level of the but presented new skills to be acquired for every experience group.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The coding task requires new knowledge or skill for all participants.

  • Probably not a substantial claim: may not pass S2 (checkable), S4 (based).

    Interpretationnone given

    Our original motivation for the importance of preserving debugging skills may hinge on the acquisition of these skills without relying on AI.

    “Moreover, our original motivation for the importance of preserving debugging skills may hinge on the acquisition of these skills without relying on AI.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationnone given

    Certain errors require a deeper understanding of the Trio library, which may account for differences in learning outcomes.

    “Certain errors require a deeper understanding of the Trio library, which may account for differences in learning outcomes.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    High-scoring patterns generally involve more cognitive effort and less AI reliance.

    “We can divide the interaction pattern into two categories: low-and high-scoring interaction patterns; the high-scoring patterns generally involve more cognitive effort and less AI reliance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    The Generation-Then-Comprehension approach looks nearly the same as the AI delegation group, but additionally uses AI to check their own understanding.

    “Importantly, this approach looks nearly the same as the AI delegation group, but additionally uses AI to check their own understanding.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    If all participants in the AI group were to use AI solely for generating code, the skill-formation differences compared to the control group would be even greater.

    “In fact, 3 of the 8 lowest-scoring participants asked AI to generate code without asking for explanations, suggesting that if all participants in the AI group were to use AI for solely generating code, the skill-formation differences compared to the control group would be even greater.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    Higher active time correlates with lower quiz score because more experienced programmers spend less time actively coding while having better base knowledge compared to novice programmers.

    “Within each condition, higher active time correlates with lower quiz score, this is because the more experienced programmers spend less time actively coding while having better base knowledge compared to novice programmers.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    These errors force an understanding of key concepts on how the Trio library handles coroutines and the usage of await keywords that are tested in the evaluation.

    “These errors force an understanding of key concepts on how the trio library handles corountines and the usage of await keywords that are tested in the evaluation.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    A more significant difference in completion time due to shorter interactions with AI assistance would likely translate to an even larger negative impact on skill formation.

    “A more significant difference in completion time due to shorter interactions with AI assistance would likely translate to an even larger negative impact on skill formation.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Not all errors carry the same weight in skill development in our study.

    “Not all errors carry the same weight in skill development in our study.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationown data

    The Progressive AI Reliance group scored poorly on the quiz largely due to not mastering any of the concepts in the second task.

    “This group scored poorly on the quiz largely due to not mastering any of the concepts in the second task.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Becker et al. found a slowdown effect for expert coders on longer tasks when participants waiting for AI-written code may become distracted.

    “Although this effect might be due to the short duration of our task, Becker et al. also found a slowdown effect for expert coders on longer tasks when participants waiting for AI-written code may become distracted.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Becker et al.

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning

    The AI group left more comments because they felt like they had more time at the end of the task.

    “Since the average completion time was faster for the AI group, the AI group left more comments since they felt like they had more time at the end of the task.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The study measured skill formation for a specific Python library over a one-hour period.

    “We measured skill formation for a specific Python library over a one-hour period.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The study collected self-reported familiarity with AI coding tools but did not measure differences in prompting techniques.

    “We collect self-reported familiarity with AI coding tools, but we do not actually measure differences in prompting techniques.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationnone given

    While participants in the study were professional or freelance programmers, there was not the same incentive to learn the library as if it were required for their actual job.

    “While participants in our study were professional or freelance programmers, there was not the same incentive to learn the library as if it were required for their actual job.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given

    Participants were not exposed to any risks during this study.

    “Participants were not exposed to any risks during this study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The study focuses on a single task using a chat-based interface.

    “This study focuses on a single task using a chat-based interface.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

    Interpretationreasoning

    The single chat-based task should be a lower bound for cognitive offloading because agentic AI coding tools would require even less human participation.

    “This should be a lower bound for cognitive offloading since agentic AI coding tools would require even less human participation.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Agentic AI coding tools require less human participation than chat-based interfaces.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    51 out of 52 participants uploaded screen recordings of their work in the warm-up task, main coding task, and quiz.

    “51/52 participants uploaded screen recordings of their work in the warm-up task, main coding task, and quiz.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    In a randomized controlled trial, participants were assigned to the treatment condition (using an AI assistant, web search, and instructions) or the control condition (completing tasks with web search and instructions alone).

    “In a randomized controlled trial, participants were assigned to the treatment condition (using an AI assistant, web search, and instructions) or the control condition (completing tasks with web search and instructions alone).”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    Participants were given the right to withdraw consent at any time without penalty.

    “We gave participants the right to withdraw consent at any time without penalty.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    All stored information is completely anonymized; only the data platform can use the IDs to identify the participants for payment.

    “All stored information is completely anonymized; only the data platform can use the IDs to identify the participants for payment.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The screen recording for one participant in the AI condition was not available.

    “The screen recording for one participant in the AI condition was not available.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The protocol was reviewed and approved by internal reviewers at Anthropic.

    “The protocol was reviewed and approved by internal reviewers at Anthropic.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    Informed consent for participation in the study was collected during the prescreening stage.

    “We collected informed consent for participation in the study during the prescreening stage.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The study does not include the counterfactual of how skill formation would be impacted by receiving assistance from humans.

    “We do not include the counterfactual of how skill formation would be impacted by receiving assistance from humans.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S2 (checkable).

    Interpretationown data

    For novice workers in software engineering or any other industry, the study can be viewed as a small piece of evidence toward the value of intentional skill development despite the ubiquity of AI tools.

    “For novice workers in software engineering or any other industry, our study can be viewed as a small piece of evidence toward the value of intentional skill development despite the ubiquity of AI tools.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning

    Exerting cognitive effort can be assisted by AI.

    “Exerting cognitive effort can be assisted by AI; beyond the patterns we describe, major LLM services also provide learning modes (e.g., ChatGPT Study Mode, Claude Code Learning / Explanatory mode).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    The study demonstrates the benefits of deploying cognitive effort when encountering a learning opportunity to master a new tool even if barriers such as errors may be encountered in the process of mastery.

    “Our study demonstrates the benefits of deploying cognitive effort when encountering a learning opportunity to master a new tool even if barriers (e.g., errors) may be encountered in the process of mastery.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning

    Given time constraints and organizational pressures, junior developers or other professionals may rely on AI to complete tasks as fast as possible at the cost of real skill development.

    “Given time constraints and organizational pressures, junior developers or other professionals may rely on AI to complete tasks as fast as possible at the cost of real skill development.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The study was pre-registered at https://osf.io/pk6a5.

    “Pre-registration: https://osf.io/pk6a5”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationown data

    A completely agentic tool would create a similar effect to users relying on AI without thinking.

    “In our work, users who relied on AI without thinking performed the worst on the evaluation; a completely agentic tool would create a similar effect.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The evaluation was designed to test three different software skills: conceptual understanding, code reading, and code writing.

    “We designed the evaluation to test three different software skills: conceputal understanding, code reading, and code writing.”
    Quote in the paperSubstantial claim
98 of 98

  • The paper relies on this without support.

    The recruited participants are representative of novice software engineers in on-the-job learning.

    “For all our pilot studies and the main study, we only recruited participants who self-reported having more than one year of Python experience, code in Python at least once a week, have tried AI coding assistance at least a few times, and have never used the Trio library before (Table 1).”

  • The paper relies on this without support.

    The AI assistant (GPT-4o) is representative of AI coding assistants used in professional settings.

    “The base model used for this assistant is GPT-4o, and the model is prompted to be an intelligent coding assistant.”

  • The paper relies on this without support.

    The quiz measures the skills required to supervise AI.

    “Returning to our initial motivation of developing and retaining the skills required for supervising automation, proficiency in some of these areas may be more important than others for the oversight of AI-generated code.”

  • The paper relies on this without support.

    The quiz score is a valid measure of skill formation or learning.

    “the level of skill formation gained by completing the task, measured by our quiz, is significantly reduced (Cohen d=0.738, p=0.01).”

  • The paper relies on this without support.

    The coding task requires new knowledge or skill for all participants.

    “our study differs from prior results in that it is designed to study how AI affects skill formation while performing a task requiring new knowledge.”

  • The paper relies on this without support.

    Manual annotation of screen recordings is reliable and accurate.

    “In the initial coding phase of our qualitative analysis, we manually annotated screen recordings of the 51 participants in the main study.”

  • The paper relies on this without support.

    The six AI interaction patterns based on query types, number of queries, queries per task, and active time adequately capture meaningful differences in AI use.

    “Using these axes, we develop a typology of six AI interaction patterns based on query types, number of queries, queries per task, and active time.”

  • The paper relies on this without support.

    Screen recordings accurately capture every error encountered by participants.

    “We recorded every error encountered by each participant as we watched the screen recordings of the task progress.”

  • The paper relies on this without support.

    Skill formation takes place over months to years.

    “Ideally, skill formation takes place over months to years.”

  • The paper relies on this without support.

    Agentic AI coding tools require less human participation than chat-based interfaces.

    “This should be a lower bound for cognitive offloading since agentic AI coding tools would require even less human participation.”

  • The paper relies on this without support.

    With AI assistance becoming more ubiquitous in the workplace, novice workers may complete tasks without the same learning outcomes.

    “With AI assistance becoming more ubiquitous in the workplace, novice workers may complete tasks without the same learning outcomes.”
11 of 11

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Programmers who ask the assistant to explain generated code perform better than those who delegate or debug by repeated queries (Shen and Tamkin, 2026).

    “programmers who ask the assistant to explain generated code perform better than those who delegate or debug by repeated queries (Shen and Tamkin, 2026)”

    From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Shen and Tamkin, 2026

    The Conceptual Inquiry section reports high quiz scores for participants who sought explanations of generated code and poor quiz scores for those who delegated or repeatedly used AI to debug. But “perform better” is unqualified: the AI Delegation group completed the tasks fastest, so the comparison holds for quiz performance, not completion speed. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

  • Shen and Tamkin (2026) find that developers who used AI to learn an unfamiliar programming library scored substantially worse on a subsequent unassisted test.

    “Shen and Tamkin (2026) find that developers who used AI to learn an unfamiliar programming library scored substantially worse on a subsequent unassisted test.”

    From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Shen and Tamkin (2026)

  • Four studies compare learners by their observed use of the same tool.

    “We also identified four studies that compare learners by their observed use of the same tool, as we do (Liu et al., 2026; Shen and Tamkin, 2026; Franco et al., 2026; Strömberg et al., 2026).”

    From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Shen and Tamkin, 2026

  • Comparisons by observed use of the same tool point the same way as the paper's comparisons.

    “Comparisons by observed use of the same tool point the same way as ours.”

    From Experimental Evidence on the Learning Impact of Generative AI (2026) · cites this paper as Shen and Tamkin, 2026

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Control-group async-quiz count conflicts with its proportion

    Minor

    The table lists 18 control participants in the 3-4-score category but gives their proportion as 0.577. With 26 control participants, 18 corresponds to 0.692.

    Reported 0.577 · recomputed 0.692308

  • Number of screen recordings analyzed differs

    Minor

    The main text says recordings of 51 participants were annotated, but the appendix says recordings were watched for 25 participants in each condition, or 50 total.

  • Coding-experience counts do not match group sizes

    Minor

    The treatment counts across the three experience categories sum to 27, although the paper states that each group has 26 participants. The corresponding control counts sum to 25.

    Reported 26 · recomputed 27

  • Async-quiz category shows a difference of three for equal counts

    Minor

    The 3-4-score category lists 18 participants in each group, so its count difference is zero, not the reported three.

    Reported 3 · recomputed 0

  • Figure caption says the evaluation tested code writing

    Minor

    The methods explicitly say code-writing questions were excluded, while the Figure 20 caption names code writing as one of the three skills tested.

5 of 5

Cites

Flagged first, then by citations

1 of the 44 references couldn’t be matched to a known paper.

Research works printed but not linked: 2

  1. David A Kolb. Experiential learning: Experience as the source of learning and development. FT press, 2014.
  2. Emma Wiles, Lisa Krayer, Mohamed Abbadi, Urvi Awasthi, Ryan Kennedy, Pamela Mishkin, Daniel Sack, and François Candelon. Genai as an exoskeleton: Experimental evidence on knowledge workers using genai on new skills. Available at SSRN 4944588, 2024.

Cited by

Sorted by citations

Authors
Shen, J. H., & Tamkin, A.
ORCID
Tamkin, Alex
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2601.20245
arXiv
2601.20245
OpenAlex
W7126029612
Cited by
4
References
44
Citations by year
2026: 4
Type
posted content
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-nd)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Social Sciences › Psychology › Social Psychology › Human-Automation Interaction and Safety
Keywords
AI-assisted programming, skill development, novice programmers, developer productivity, conceptual understanding, code reading, debugging, asynchronous programming, cognitive engagement, safety-critical domains

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-09.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-09.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review