CoChat
Sign Up
PreprintCited by 2Artificial Intelligence in EducationIntelligent Tutoring Systems and Adaptive LearningArtificial Intelligence in Healthcare and Education

Experimental Evidence on the Learning Impact of Generative AI

Open access
Contractor, Zara, Reyes, Germán·2026·arXiv
DOI 10.48550/arxiv.2607.08849arXiv 2607.08849
Open this paper in CoChat — free accountOpen PDF

Abstract

We study whether access to off-the-shelf generative AI changes how much students learn in a timed assignment. In a proctored, in-person randomized experiment, undergraduates study an unfamiliar topic and write an essay with or without AI, then complete unaided assessments immediately and one week later. AI access raises test scores, and a smaller gain persists a week later. Gains remain for students who use AI as a tutor ("augmentation") and fade for students who have AI write for them ("automation"). We document two mechanisms: students shift time from writing toward reading and searching for information, and report greater learning enjoyment.

Verified 2026-10-18 · 2 references not checked yet

AI snapshot

Written from the paper’s full text.

TL;DR

In a proctored randomized experiment, undergraduates given access to off-the-shelf generative AI during a timed learning task scored higher on unaided tests immediately and one week later. Gains were concentrated among students who used AI as a tutor rather than to automate writing.

Key findings

  • AI access raised immediate test scores by 7.2 pp (0.28 SD; control mean 56.3%) and Session Two scores by 3.9 pp (0.21 SD; control mean 50.0%); TOT estimates were 10.4 pp (0.41 SD) and 5.5 pp (0.30 SD).
  • Essay quality was 0.34 SD higher in Session One and 0.24 SD higher in Session Two; the share of text flagged as AI-generated rose 22.2 pp in Session One but the effect disappeared in Session Two (0.7 pp).
  • Treated students shifted 5.3 pp of learning time away from writing and 4.4 pp toward reading/searching, and reported 11% higher enjoyment; total learning time did not change.
  • Among AI users, 61% were augmentation users, 9% automation, and 27% mixed; automation users’ Session One essay advantage (1.50 SD) disappeared in Session Two (−0.14 SD), while augmentation users’ Session Two test difference remained positive (0.24 SD).

Methods

  • In-person RCT with 211 Middlebury undergraduates in Spring 2025; two sessions about one week apart (mean 6.98 days), with random assignment to AI-allowed or AI-forbidden during a 35-minute learning phase.
  • Students learned one of three technical topics and wrote an essay; all tests and the Session Two essay were completed without AI or other resources.
  • ITT and 2SLS/TOT estimates used double-lasso controls and strata fixed effects; outcomes included tests, human- and AI-graded essays, ChatGPT logs, AI-detection, time use, and beliefs.

Limitations

  • The paper states the lab setting offered few competing uses of time, so it estimates learning gains holding time on task fixed and cannot say whether AI raises or lowers total learning when students reallocate saved time.
  • The paper describes the automation-versus-augmentation results as descriptive, not randomized by use type.
  • Evident from the text: the sample is one selective college with high AI familiarity, follow-up is only one week, and treated students had more Session One integrity violations, though the paper’s robustness checks reduce but do not eliminate the test-score effect.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

DataCite · 10.48550/arxiv.2607.08849

Cites

0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

55 of 108 cited sources checked · 44 supported · 0 flagged · accuracy not yet measured

  • Premisecitation
    § 1 Introduction

    In many designs students choose how much time to spend on the task, so estimated effects combine time allocation and learning productivity.

    “Third, in many designs students choose how much time to spend on the task (e.g., Kazemitabaar et al., 2023; Kestin et al., 2025; Kreijkes et al., 2026), so the estimated effects combine the two margins: time allocation and learning productivity.”
    Quote in the paperSubstantial claim2 of 3 sources checked
    • cites Kazemitabaar et al., 2023 source
    • cites Kestin et al., 2025 source
    • cites Kreijkes et al., 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Premisecitation
    § 3.4 Main Outcomes

    Pangram achieves near-zero false positive and false negative rates among four AI text detectors evaluated, even when AI-generated text is modified using humanizer tools.

    “Jabarian and Imas (2025) evaluate four AI text detectors and find that Pangram achieves near-zero false positive and false negative rates, even when AI-generated text is modified using 'humanizer' tools.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Jabarian and Imas (2025) source
  • Premisecitation
    § 3.1 Sample and Recruitment

    At the time of the experiment, AI adoption at Middlebury exceeded 80 percent.

    “At the time of the experiment, AI adoption at Middlebury exceeded 80 percent (Contractor and Reyes, 2025), so our treatment varies AI access among students already familiar with the tool.”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Contractor and Reyes, 2025 source
  • Premisecitation
    § 1 Introduction

    Shen and Tamkin (2026) find that developers who used AI to learn an unfamiliar programming library scored substantially worse on a subsequent unassisted test.

    “Shen and Tamkin (2026) find that developers who used AI to learn an unfamiliar programming library scored substantially worse on a subsequent unassisted test.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Shen and Tamkin (2026) source
  • Premisecitation
    § 3.4 Main Outcomes

    LLM ratings of text quality track human ratings.

    “a growing literature finds that LLM ratings of text quality track human ratings (Chiang and Lee, 2023; Liu et al., 2023; Zheng et al., 2023)”
    Quote in the paperSubstantial claim3 of 3 sources checked
    • cites Chiang and Lee, 2023 source
    • cites Liu et al., 2023 source
    • cites Zheng et al., 2023 source
  • Classified as probably not a substantial claim. 93
  • May combine two claims. 113
  • The quote only partly states this claim. 18

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § 1 Introduction

    A few studies examine whether AI-assisted work improves subsequent unassisted performance, with mixed results.

    “A few studies examine whether AI-assisted work improves subsequent unassisted performance, with mixed results.”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Brynjolfsson et al. (2025) source
    • cites Cruces et al. (2026) source
    • cites Autor et al. (2026) source

  • May combine two claims.

    Premisecitation
    § 1 Introduction

    Generative AI can change how much students learn through two margins: time allocation and learning productivity.

    “Generative AI can change how much students learn through two margins.”
    • Also stated in: § 1 Introduction · plain
      “The second margin is how much students learn from a fixed amount of study time (henceforth, 'learning productivity').”
    Quote in the paperSubstantial claim1 of 3 sources checked
    • cites Kazemitabaar et al., 2023 source

      The Discussion (section 6, page not provided) reports faster task completion with AI code generators and improved retention-test performance for learners with higher prior programming knowledge. The paper does not describe time allocation and learning productivity as two distinct ways AI changes how much students learn; faster task completion is not itself evidence of changed learning-time allocation. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites Nickow et al., 2024 source
    • cites Chi and Wylie, 2014 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 1 Introduction

    Autor et al. (2026) find that an AI drafting assistant improves subsequent unassisted performance among senior patent lawyers, though not among junior patent lawyers.

    “Autor et al. (2026) also find that an AI drafting assistant improves subsequent unassisted performance among senior patent lawyers, though not among junior patent lawyers.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Autor et al. (2026) source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation
    § 3.2 Summary Statistics and Balance

    The study uses the double-lasso procedure to address residual imbalance.

    “To address any residual imbalance, we use the double-lasso procedure (Belloni et al., 2014), as described below.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Belloni et al., 2014 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 4.1 First Stage: AI Adoption and Usage Patterns

    The near-exclusive use of ChatGPT is consistent with surveys of adoption among college students.

    “The near-exclusive use of ChatGPT is consistent with surveys of adoption among college students (Hirabayashi et al., 2024; Stöhr et al., 2024; Contractor and Reyes, 2025).”
    Quote in the paperNot a substantial claim2 of 3 sources checked
    • cites Hirabayashi et al., 2024 source
    • cites Stöhr et al., 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Contractor and Reyes, 2025 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 5.1 Effects on How Students Learn

    Prior work in professional settings found that AI shifts effort from task execution to task stewardship.

    “This pattern is consistent with a broader finding in professional settings: AI shifts effort from task execution to task stewardship (Lee et al., 2025).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Lee et al., 2025 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 6 Beliefs About AI's Impact on Learning

    Even experienced professionals misjudge AI's productivity effects.

    “even experienced professionals misjudge AI's productivity effects.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Becker et al. (2025) source

  • May combine two claims.

    Premisecitation
    § 6 Beliefs About AI's Impact on Learning

    Experienced open-source developers predicted AI tools would speed them up by 24 percent, after the study still believed AI had accelerated their work by roughly 20 percent, while the actual measured effect was a 19 percent slowdown.

    “Becker et al. (2025) find that experienced open-source developers predicted AI tools would speed them up by 24 percent and, after the study, still believed AI had accelerated their work by roughly 20 percent. The actual measured effect was a 19 percent slowdown .”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites Becker et al. (2025) source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductcitation
    § Figures and Tables

    The Table 7 specifications used controls selected by double-lasso on the full sample and included strata fixed effects.

    “All specifications use controls selected by double-lasso on the full sample (Belloni et al., 2014), with strata fixed effects.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Belloni et al., 2014 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation
    § Figures and Tables

    All specifications included controls selected through a double-lasso procedure.

    “All specifications include controls selected through a double-lasso procedure (Belloni et al., 2014).”
    • Also stated in: § Figures and Tables · plain
      “All specifications include controls selected through a double-lasso procedure (Belloni et al., 2014).”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Belloni et al., 2014 source

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § C.1 Validating the Open-Ended Response Measure

    Responses to the open-ended question skew positive, with most students expressing net-positive sentiment about AI's effect on learning and a minority expressing negative sentiment.

    “Responses skew positive, with most students expressing net-positive sentiment about AI's effect on learning and a minority expressing negative sentiment.”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites Hutto and Gilbert, 2014 source

      Section 3.2 analyzes positive and negative tweets, and Section 4 evaluates VADER’s sentiment classification. Neither reports open-ended responses from students or their views on AI’s effect on learning. Searches for student responses and AI in learning found no relevant passage. (LLM, conf=0.78)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data
    § 1 Introduction

    Most students name both a channel through which AI helps learning and one through which it harms it, say the effect depends on how AI is used, and organize the channels around the augmentation-versus-automation distinction along which the use-type results differ.

    “Most students name both a channel through which AI helps learning and one through which it harms it, say the effect depends on how AI is used, and organize the channels around the augmentation-versus-automation distinction along which our use-type results differ.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 1 Introduction

    Both treated and control students expect AI access to have positive effects on test scores, but only the treated group's prediction comes close to the paper's estimate; the control group overestimates it.

    “Both treated and control students expect positive effects, but only the treated group's prediction comes close to our estimate; the control group overestimates it.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 1 Introduction

    AI access shifts the composition of learning time away from writing: treated students spend 5.3 percentage points less of their learning time on writing and 4.4 percentage points more on reading and searching for information, relative to control shares of 53.4 and 44.5 percent.

    “treated students report spending 5.3 pp less of their learning time on writing activities and 4.4 pp more on reading and searching for information”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Students' self-reported learning-time allocation and enjoyment ratings accurately measure their actual time use and enjoyment.

  • May combine two claims.

    Findingown data
    § 1 Introduction

    Among treated students, the most common substantive AI use was explaining concepts (56 percent), followed by drafting responses (29 percent), editing responses (23 percent), and summarizing readings (16 percent).

    “ChatGPT conversation logs reveal that students use AI most often to explain concepts (56 percent of treated students). Drafting responses (29 percent), editing responses (23 percent), and summarizing readings (16 percent) are the next most common substantive uses.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: ChatGPT conversation logs accurately reflect students' AI use and permit valid classification into augmentation, automation, and mixed use types.

  • May combine two claims.

    Findingown data
    § 1 Introduction

    Automation and mixed users submit higher-rated Session One essays, but automation users' essay-quality advantage is absent from their unaided Session Two essays one week later.

    “Automation and mixed users submit higher-rated essays in Session One, but automation users' advantage is absent from their unaided essays one week later.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: ChatGPT conversation logs accurately reflect students' AI use and permit valid classification into augmentation, automation, and mixed use types.
    • Rests on an assumption: Graders' overall quality ratings and five dimension scores validly measure essay quality and understanding.

  • May combine two claims.

    Findingown data
    § 1 Introduction

    In Session Two essays, point estimates for all five quality dimensions are positive, with the largest effects on use of evidence and examples and on writing style and clarity.

    “the point estimates for all five dimensions are positive, with the largest effects on use of evidence and examples and on writing style and clarity”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Graders' overall quality ratings and five dimension scores validly measure essay quality and understanding.

  • May combine two claims.

    Findingown data
    § 1 Introduction

    Augmentation and mixed users show positive test-score differences relative to the control group in both sessions, and both groups' point estimates exceed automation users' on every Session Two outcome.

    “Augmentation and mixed users show positive test-score differences relative to the control group in both sessions, and both groups' point estimates exceed automation users' on every Session Two outcome.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: ChatGPT conversation logs accurately reflect students' AI use and permit valid classification into augmentation, automation, and mixed use types.

  • May combine two claims.

    Findingown data
    § 1 Introduction

    Among AI users, 61 percent are classified as augmentation users, 9 percent as automation users, and 27 percent as mixed users.

    “we use treated students' ChatGPT conversation logs and classify 61 percent of AI users as 'augmentation' users (who work with AI), 9 percent as 'automation' users (who use AI to do the work for them), and 27 percent as 'mixed' users (who do both).”
    Quote in the paperSubstantial claim
    • Rests on an assumption: ChatGPT conversation logs accurately reflect students' AI use and permit valid classification into augmentation, automation, and mixed use types.

  • May combine two claims.

    Findingown data
    § 4.1 First Stage: AI Adoption and Usage Patterns

    ChatGPT use increased by 60.2 pp, versus just a 2.9 pp increase in the use of other AI models.

    “AI adoption was almost entirely ChatGPT: a 60 . 2 pp increase in ChatGPT use ( p < 0 . 001), versus just a 2 . 9 pp increase in the use of other AI models ( p = 0 . 182).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 3.2 Summary Statistics and Balance

    Mean GPA (3.69 versus 3.67) and SAT scores (1392 versus 1383) are nearly identical across the AI-forbidden and AI-allowed groups.

    “Measures of academic ability, strong predictors of learning outcomes, are well balanced: mean GPA (3.69 versus 3.67) and SAT scores (1392 versus 1383) are nearly identical across groups.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.1 First Stage: AI Adoption and Usage Patterns

    Self-assessed proficiency predicts AI take-up in bivariate specifications, while early adoption does not.

    “Self-assessed proficiency also predicts takeup in bivariate specifications, while early adoption does not.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.1 First Stage: AI Adoption and Usage Patterns

    In ChatGPT logs, explaining concepts is the most common use (56 percent of treated students), followed by drafting responses (29 percent), editing responses (23 percent), and summarizing readings (16 percent).

    “In the logs, explaining concepts is the most common use (56 percent of treated students), followed among the substantive categories by drafting responses (29 percent), editing responses (23 percent), and summarizing readings (16 percent).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Effects on Test Scores

    The ITT of AI access on self-assessed knowledge is -0.10 points, statistically indistinguishable from zero.

    “The rise is nearly identical among treated students, leaving an ITT of ˆ β = -0 . 10 points, which is statistically indistinguishable from zero.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Effects on Test Scores

    AI access adds 7.2 pp to test performance, a 13 percent improvement over the control mean.

    “AI access adds a further ˆ β = 7 . 2 pp, a 13 percent improvement over the control mean.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 1 Introduction

    The gains in knowledge from AI access are evident on unassisted assessments immediately and one week later.

    “We provide evidence that AI access can raise learning productivity in an academic context: given the same amount of time, students acquire more knowledge, with gains evident on unassisted assessments immediately and one week later.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.1 First Stage: AI Adoption and Usage Patterns

    There is no significant association between AI take-up and either AI's perceived benefits or GPA.

    “We find no significant association between take-up and either AI's perceived benefits or GPA.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 2.2 Session Two

    Only 5 percent of students reported looking up extra information about the topic between sessions, with no significant difference between treatment and control groups.

    “only 5 percent of students reported looking up extra information about the topic, with no significant difference between treatment and control groups”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Effects on Test Scores

    The Session Two ITT of AI access on test scores is 3.9 pp (p = 0.068), 54 percent of the 7.2 pp Session One effect.

    “The resulting ITT of ˆ β = 3 . 9 pp ( p = 0 . 068) is 54 percent of the 7.2 pp Session One effect”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.1 First Stage: AI Adoption and Usage Patterns

    Treated students use AI most often to explain concepts, but also to draft and edit text.

    “Treated students use AI for a mix of purposes: most often to explain concepts, but also to draft and edit text.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.1 First Stage: AI Adoption and Usage Patterns

    Self-reports show explaining concepts as the most common use (46 percent), with drafting (17 percent) and editing (14 percent) reported less often than the logs show.

    “The self-reports give a similar picture except for writing: explaining concepts is again the most common use (46 percent), but students report drafting (17 percent) and editing (14 percent) less often than the logs show.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.1 Effects on How Students Learn

    AI access did not change how skilled or effective students felt; the effect was 0.06 points (p=0.834).

    “The effect on how skilled or effective students feel is also positive, but small and statistically insignificant ( ˆ β = 0 . 06 points, p = 0 . 834).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.3 Effects on Essay-Based Outcomes

    In Session Two, relevance to the prompt improved by 0.29 points (0.20 SD, p=0.133), a smaller and imprecise estimate.

    “The estimates for relevance to the prompt ( ˆ β = 0 . 29 points, or 0 . 20 SD, p = 0 . 133) and accuracy of content ( ˆ β = 0 . 26 points, or 0 . 22 SD, p = 0 . 124) are smaller and imprecise”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.3 Effects on Essay-Based Outcomes

    In Session Two, accuracy of content improved by 0.26 points (0.22 SD, p=0.124), a smaller and imprecise estimate.

    “The estimates for relevance to the prompt ( ˆ β = 0 . 29 points, or 0 . 20 SD, p = 0 . 133) and accuracy of content ( ˆ β = 0 . 26 points, or 0 . 22 SD, p = 0 . 124) are smaller and imprecise”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Effects on Test Scores

    In Session One, treated students are 8.3 pp more likely to score at least 40 percent correct and 10.6 pp more likely to score at least 60 percent correct.

    “In Session One, treated students are 8.3 pp more likely to score at least 40 percent correct ( p = 0 . 071) and 10.6 pp more likely to score at least 60 percent correct ( p = 0 . 094).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingreasoning
    § 4.2 Effects on Test Scores

    The implied contribution of violations to the ITT is 1.2 pp under the proctor measure and 1.6 pp under the self-report measure, or 17 and 22 percent of the estimate.

    “The implied contribution of violations to the ITT is 1.2 pp under the proctor measure and 1.6 pp under the self-report measure, or 17 and 22 percent of the estimate.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The test-score gap between rule violators and non-violators reflects the causal effect of violations, so that multiplying it by the treatment effect on violations bounds the contribution of violations to the ITT.

  • May combine two claims.

    Findingown data
    § 4.2 Effects on Test Scores

    Adding a violation indicator to the regression, the Session One ITT falls to 0.25 SD under the proctor measure and 0.21 SD under self-reports.

    “Adding a violation indicator to the regression gives similar results: the Session One ITT falls to 0.25 SD under the proctor measure and 0.21 SD under self-reports.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3 Effects on Essay-Based Outcomes

    The effects of AI access on the length, readability, and lexical diversity indices of student essays are small and statistically insignificant.

    “we find little evidence of stylistic changes in student essays: the effects on the length, readability, and lexical diversity indices are small and statistically insignificant”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3 Effects on Essay-Based Outcomes

    The near-zero effects on both within-group similarity and similarity to reading material persisted in Session Two.

    “Both similarity effects remain near zero in Session Two (columns 4-6).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Effects on Test Scores

    Excluding students who violated the rules as captured by proctors, by self-reports, or by both reduces the learning effect to 0.24 SD, 0.15 SD, and 0.27 SD, respectively.

    “Excluding students who violated the rules as captured by proctors, by self-reports, or by both a proctor and a self-report reduces the learning effect to 0.24 SD, 0.15 SD, and 0.27 SD, respectively.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3 Effects on Essay-Based Outcomes

    In this experiment, AI access did not increase within-group textual similarity (homogeneity); the estimated treatment effect on cosine similarity was 0.003 (0.4% of control mean 0.788) and statistically insignificant.

    “Despite concerns that AI homogenizes writing (Doshi and Hauser, 2024; Meincke et al., 2025; Moon et al., 2025, 2026), we see no such effect in our data (Table 5, Panel B).”
    • Also stated in: § 4.3 Effects on Essay-Based Outcomes · plain
      “Treatment effects on cosine similarity, our measure of homogeneity, are near zero and statistically insignificant: within-group textual similarity changes by ˆ β = 0 . 003, a 0.4 percent change relative to the control mean of 0.788.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3 Effects on Essay-Based Outcomes

    In Session One, relevance to the prompt improved by 0.17 points (0.12 SD, p=0.448), a smaller and imprecise estimate.

    “The estimates for use of evidence and examples ( ˆ β = 0 . 23 points, or 0 . 16 SD, p = 0 . 271) and relevance to the prompt ( ˆ β = 0 . 17 points, or 0 . 12 SD, p = 0 . 448) are smaller and imprecise.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Effects on Test Scores

    AI access increases the likelihood of rule violations on the Session One test: proctor-observed violations rise by 5.6 pp and self-reported violations rise by 9.6 pp.

    “Proctor-observed violations rise by ˆ β = 5 . 6 pp ( p = 0 . 076), while self-reported violations rise by ˆ β = 9 . 6 pp ( p = 0 . 003).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3 Effects on Essay-Based Outcomes

    The fraction of text flagged by Pangram's plagiarism checker is near zero in both groups, with no significant treatment effect in either session.

    “We find no evidence of copying from online sources: the fraction of text flagged by Pangram's plagiarism checker is near zero in both groups, with no significant treatment effect in either session.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 4.2 Effects on Test Scores

    AI access shifts the middle of the performance distribution rightward while leaving the tails largely unchanged.

    “AI access shifts the middle of the performance distribution rightward while leaving the tails largely unchanged.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 4.3 Effects on Essay-Based Outcomes

    The fraction of text classified as AI-generated rises by 22.2 pp in the AI-allowed group, a 337 percent increase over the control mean of 6.6 percent.

    “The fraction of text classified as AI-generated rises by ˆ β = 22 . 2 pp in the AIallowed group ( p < 0 . 001), a 337 percent increase over the control mean of 6.6 percent.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.1 Effects on How Students Learn

    Enjoyment rose by 0.58 points on the 0-10 scale (p=0.063), an 11% improvement over the control mean of 5.22.

    “Enjoyment of the learning task rises by ˆ β = 0 . 58 points on the 0-10 scale ( p = 0 . 063), an 11 percent improvement over the control mean of 5.22.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.3 Outcomes by Type of AI Use

    Mixed users' Session Two essay-quality difference is positive but imprecise at 0.25 SD (p = 0.320).

    “Mixed users' Session Two differences are positive but imprecise ( ˆ β = 0 . 28 SD, p = 0 . 220, on the test and ˆ β = 0 . 25 SD, p = 0 . 320, on essay quality).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.3 Outcomes by Type of AI Use

    Of treated AI users, 61% were pure augmentation users, 9% pure automation users, 27% mixed, and 3% Other.

    “Of treated AI users, 61 percent are pure augmentation users, 9 percent are pure automation users, 27 percent are mixed, and the remaining 3 percent are classified as Other .”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.3 Outcomes by Type of AI Use

    Augmentation users' essay-quality difference is positive but imprecise in both sessions, with estimates of 0.09 SD and 0.23 SD.

    “A week later, when students work without AI, augmentation users' test-score difference remains positive ( ˆ β = 0 . 24 SD, p = 0 . 095) and does not change significantly between sessions, and their essay-quality difference is positive but imprecise in both sessions ( ˆ β = 0 . 09 SD and 0 . 23 SD).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.3 Outcomes by Type of AI Use

    Automation users' test-score difference is negative but imprecise at -0.24 SD (p = 0.538).

    “In contrast, automation users' advantages disappear: their essay-quality difference falls by 1.64 SD ( p = 0 . 023) to ˆ β = -0 . 14 SD, and their test-score difference is negative but imprecise ( ˆ β = -0 . 24 SD, p = 0 . 538).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § 5.3 Outcomes by Type of AI Use

    Mixed users' Session Two test-score difference is positive but imprecise at 0.28 SD (p = 0.220).

    “Mixed users' Session Two differences are positive but imprecise ( ˆ β = 0 . 28 SD, p = 0 . 220, on the test and ˆ β = 0 . 25 SD, p = 0 . 320, on essay quality).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 5.3 Outcomes by Type of AI Use

    The three AI-use groups prompted AI differently: automation users mainly for drafting and editing, augmentation users mainly for explaining concepts, and mixed users for both.

    “First, the three groups prompt AI differently: automation users use AI mainly for drafting and editing, augmentation users mainly for explaining concepts, and mixed users for both (Appendix Figure A10).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.1 Beliefs About AI's Effect on Test Scores

    Both groups give similar predictions of their own raw score: control 37.8 percent correct, treated 37.2 percent, both below the actual mean of 51.2 percent.

    “Both groups give similar predictions of their own raw score (control 37.8 percent correct, treated 37.2 percent, both below the actual mean of 51.2 percent)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.1 Beliefs About AI's Effect on Test Scores

    The lower average perceived effect among treated students reflects two shifts in the distribution of beliefs relative to control students: more mass at zero (no perceived effect) and less mass at large positive values (five or more questions).

    “The lower average perceived effect among treated students reflects two shifts in the distribution of beliefs relative to control students (Appendix Figure A11): more mass at zero (no perceived effect) and less mass at large positive values (five or more questions).”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § Figures and Tables

    Assignment to AI access had no statistically significant effect on self-assessed knowledge in Session One (ITT -0.098, SE 0.203).

    “Self-assessed | 4.788 | - 0.098 (0.203) | - 0.142 (0.296)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.1 Beliefs About AI's Effect on Test Scores

    Predictions about others' performance show the same pattern in muted form: control students predict 14.2 pp and treated 11.2 pp, both still inflated relative to the actual effect.

    “Predictions about others' performance show the same pattern in muted form: control students predict 14.2 pp and treated 11.2 pp, both still inflated relative to the actual effect.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.1 Beliefs About AI's Effect on Test Scores

    Control students predict an AI gain for themselves of 25.5 pp, which is 6.5 times the actual effect of 3.9 pp.

    “Control students predict an AI gain for themselves of 25.5 pp, 6.5 times the actual effect of 3.9 pp.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § 7 Conclusion

    AI access increases the time students spend reading and searching for information while learning a new topic.

    “more time reading and searching for information”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.1 Beliefs About AI's Effect on Test Scores

    Treated students, most of whom used AI in Session One, predict an AI gain of only 3.0 pp, close to the effect we estimate.

    “Treated students, most of whom used AI in Session One, predict only 3.0 pp, close to the effect we estimate.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § 7 Conclusion

    AI access increases students' reported enjoyment.

    “report higher enjoyment”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § 6.1 Beliefs About AI's Effect on Test Scores

    Control students overestimate the effect of AI on their test performance, whereas treated students predict an effect close to the one we estimate.

    “Control students overestimate the effect of AI on their test performance, whereas treated students predict an effect close to the one we estimate (Figure 7).”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § 7 Conclusion

    AI access has no effect on total learning time.

    “We find no effect on total learning time”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data
    § Figures and Tables

    Assignment to the AI-allowed group increased enjoyment of learning (ITT 0.579, SE 0.310, p<0.10; TOT 0.847, SE 0.461, p<0.10).

    “Enjoyed learning (0-10) | 5.221 | 0.579 ∗ (0.310) | 0.847 ∗ (0.461)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Augmentation users

    Among AI-allowed students, high self-assessed AI proficiency is associated with higher AI use in bivariate regressions but not after adjusting for other characteristics.

    “| High AI proficiency (at or above median) | 0.208 ∗∗ (0.096) | 0.264 ∗∗ (0.103) | 0.096 (0.119) | 0.165 (0.148) |”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Figures and Tables

    Assignment to the AI-allowed group increased the quality index in both sets of estimates (ITT 0.344, SE 0.180, p<0.10 and ITT 0.301, SE 0.173, p<0.10; TOT 0.498, SE 0.262, p<0.10 and TOT 0.431, SE 0.253, p<0.10).

    “Quality index (average) | 5.758 | 0.344 ∗ (0.180) | 0.498 ∗ (0.262) | 4.645 | 0.301 ∗ (0.173) | 0.431 ∗ (0.253)”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § Figures and Tables

    Assignment to the AI-allowed group increased overall essay quality in both sets of estimates (ITT 0.453, SE 0.200, p<0.05 and ITT 0.327, SE 0.188, p<0.10; TOT 0.657, SE 0.290, p<0.05 and TOT 0.470, SE 0.274, p<0.10).

    “Overall quality (average) | 5.503 | 0.453 ∗∗ (0.200) | 0.657 ∗∗ (0.290) | 4.466 | 0.327 ∗ (0.188) | 0.470 ∗ (0.274)”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Figures and Tables

    No plagiarism was detected in either session or treatment group.

    “Plagiarism (fraction) | 0.000 | 0.000 (0.000) | 0.000 (0.000) | 0.000 | 0.000 (0.000) | 0.000 (0.001)”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § Figures and Tables

    Assignment to AI access had no statistically significant effect on essay similarity to the reading material in either session.

    “Reading material sim. | 0.814 | 0.000 (0.009) | 0.000 (0.013) | 0.761 | 0.010 (0.014) | 0.014 (0.020)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § Figures and Tables

    Assignment to the AI-allowed group decreased the share of learning time on writing activities by 5.347 percentage points (ITT, SE 2.407, p<0.05) and 7.719 percentage points (TOT, SE 3.450, p<0.05).

    “Share on writing activities (percent) | 53.381 | - 5.347 ∗∗ (2.407) | - 7.719 ∗∗ (3.450)”
    Quote in the paperSubstantial claim

  • The quote only partly states this claim.

    Findingown data
    § Figures and Tables

    Assignment to the AI-allowed group did not significantly change total time spent learning measured by Qualtrics (ITT -0.205, SE 0.678; TOT -0.292, SE 0.968).

    “Time spent learning (Qualtrics, minutes) | 32.611 | - 0.205 (0.678) | - 0.292 (0.968)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § Augmentation users

    Among AI-allowed students, being a frequent AI user is associated with higher ChatGPT use.

    “| Frequent AI user (at or above median) | 0.357 ∗∗∗ (0.113) | 0.359 ∗∗∗ (0.116) | 0.283 ∗ (0.148) | 0.325 ∗∗ (0.162) |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § B.6 Classification of ChatGPT Conversations

    Across all 435 student prompts, the distribution across categories was: Explaining concepts 49.0%, Other 21.6%, Editing responses 12.6%, Drafting responses 10.3%, Summarizing readings 5.5%, and Proofreading 0.5%.

    “Across all 435 student prompts, the distribution is: Explaining concepts (49.0 percent), Other (21.6 percent), Editing responses (12.6 percent), Drafting responses (10.3 percent), Summarizing readings (5.5 percent), and Proofreading (0.5 percent).”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The LLM-based classification of student prompts and full conversations into the defined categories is accurate.

  • May combine two claims.

    Findingown data
    § Figures and Tables

    Assignment to the AI-allowed group increased the share of learning time on research activities by 4.435 percentage points (ITT, SE 2.245, p<0.05) and 6.403 percentage points (TOT, SE 3.244, p<0.05).

    “Share on research activities (percent) | 44.462 | 4.435 ∗∗ (2.245) | 6.403 ∗∗ (3.244)”
    Quote in the paperSubstantial claim

  • May combine two claims. The quote only partly states this claim.

    Findingown data
    § Augmentation users

    AI access had no statistically significant effect on the lexical diversity index in either session.

    “| Lexical diversity index | 0.000 | 0.106 (0.115) | 0.154 (0.165) | 0.000 | - 0.115 (0.123) | - 0.162 (0.175) |”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § Augmentation users

    AI access had no statistically significant effect on the readability index in either session.

    “| Readability index | 0.000 | 0.061 (0.088) | 0.087 (0.127) | 0.000 | 0.004 (0.082) | 0.006 (0.116) |”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingnone given
    § Augmentation users

    AI access had no statistically significant effect on the length index in either session.

    “| Length index | 0.000 | 0.132 (0.120) | 0.190 (0.173) | 0.000 | 0.042 (0.115) | 0.059 (0.163) |”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Findingown data
    § Augmentation users

    Among AI-allowed students, social-science majors were more likely to self-report AI use.

    “| Social-science major | 0.118 (0.096) | 0.258 ∗∗ (0.101) | 0.125 (0.124) | 0.281 ∗∗ (0.130) |”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § Figures and Tables

    Assignment to the AI-allowed group did not significantly change students' rating that learning was effective (ITT 0.059, SE 0.283; TOT 0.084, SE 0.400).

    “Found effective (0-10) | 5.000 | 0.059 (0.283) | 0.084 (0.400)”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Findingown data
    § Figures and Tables

    Assignment to the AI-allowed group did not significantly change self-reported total time spent learning (ITT -0.532, SE 0.669; TOT -0.769, SE 0.966).

    “Time spent learning (self-reported, minutes) | 34.312 | - 0.532 (0.669) | - 0.769 (0.966)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § C.3 The Narratives of College Students

    The most common promoting and harming mechanisms are explanation and tutoring (40 percent) and shortcutting and cheating (43 percent).

    “the most common promoting and harming mechanisms are explanation and tutoring (40 percent) and shortcutting and cheating (43 percent).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § C.3 The Narratives of College Students

    The remaining type is a smaller, one-sided deskilling narrative (8 percent), in which skill atrophy and offloading erode critical thinking.

    “The remaining type is smaller and one-sided: a deskilling narrative (8 percent), in which skill atrophy and offloading erode critical thinking.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § C.3 The Narratives of College Students

    Many augmentation narratives also flag accuracy concerns.

    “though many of these narratives also flag accuracy concerns”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § C.3 The Narratives of College Students

    Framing varies with field of study and beliefs about AI.

    “Framing varies instead with field of study and beliefs about AI.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Findingown data
    § C.3 The Narratives of College Students

    The average narrative contains 3.1 mechanisms and 6.5 links.

    “The average narrative contains 3.1 mechanisms and 6.5 links”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § C.3 The Narratives of College Students

    Agreement rises with model capability: within the OpenAI line, GPT-5.5 reaches mechanism and link agreement of 0.88 and 0.73, whereas two smaller models reach 0.77 and 0.53, and 0.70 and 0.47.

    “Agreement rises with model capability: within the OpenAI line, GPT-5.5 reaches mechanism and link agreement of 0.88 and 0.73, whereas two smaller models reach 0.77 and 0.53, and 0.70 and 0.47.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Findingown data
    § C.3 The Narratives of College Students

    The most common single node is mode of use, at 55 percent.

    “The most common single node is mode of use (55 percent)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Findingown data
    § C.2 Narratives as Causal Graphs

    98 percent of open-ended responses can be represented as a causal graph, and the remaining responses are non-substantive.

    “The majority of responses (98 percent) can be represented as a causal graph; the rest are non-substantive (e.g., 'I don't know').”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data
    § D.5 Relationship Between the Model and the Experiment

    AI access raised test scores in both sessions.

    “AI access raised test scores in both sessions (Section 4.2).”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Section 4.2

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction

    Computer-assisted learning programs that adapt instruction to students' levels or provide targeted practice and immediate feedback have more encouraging evidence of improving academic achievement, particularly in mathematics.

    “More encouraging evidence comes from computer-assisted learning programs that adapt instruction to students' levels or provide targeted practice and immediate feedback, particularly in mathematics (Escueta et al., 2020).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Escueta et al., 2020 source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction

    Simply expanding access to computers and internet connectivity has not consistently improved academic achievement.

    “Simply expanding access to computers and internet connectivity has not consistently improved academic achievement (Bulman and Fairlie, 2016).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Bulman and Fairlie, 2016 source

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning
    § 1 Introduction

    The unaided tests and Session Two essay measure what students learned because students completed them without AI or other resources.

    “Students complete all tests and the Session Two essay without AI or other resources, so these assessments measure what they learned.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 1 Introduction

    Generative AI can do the work for students.

    “Generative AI combines features of both kinds of technology: it is a generalpurpose tool that can give personalized explanations and respond to students' questions in real time, but it can also do the work for students.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Premisenone given
    § 1 Introduction

    Much of the research on AI productivity measures task performance while AI is available.

    “Much of this research measures task performance while AI is available.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction

    Prior studies show that AI assistance can generate substantial productivity gains in professional writing, software development, consulting, customer support, legal work, sales, team-based problem solving, and taxi driving.

    “Prior studies show that AI assistance can generate substantial productivity gains in professional writing, software development, consulting, customer support, legal work, sales, team-based problem solving, and taxi driving.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [3]

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Premisenone given
    § 1 Introduction

    Generative AI is a general-purpose tool that can give personalized explanations and respond to students' questions in real time.

    “Generative AI combines features of both kinds of technology: it is a generalpurpose tool that can give personalized explanations and respond to students' questions in real time, but it can also do the work for students.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction

    Brynjolfsson et al. (2025) provide evidence consistent with learning among customer-support agents.

    “Brynjolfsson et al. (2025) provide evidence consistent with learning among customer-support agents”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Brynjolfsson et al. (2025) source

  • Probably not a substantial claim: may not pass S4 (based).

    Premisenone given
    § 1 Introduction

    Many studies evaluate AI systems that are scaffolded, restricted, or otherwise customized beyond a standard off-the-shelf chatbot.

    “First, many studies evaluate AI systems that are scaffolded, restricted, or otherwise customized beyond a standard off-the-shelf chatbot.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation
    § Abstract

    The randomized controlled trial was preregistered as AEARCTR-0015806.

    “This RCT was registered as AEARCTR-0015806 (Contractor and Reyes, 2026).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Contractor and Reyes, 2026

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 1 Introduction

    AI usage and compliance were monitored through direct proctor observation, ChatGPT conversation logs, and self-reports.

    “We monitor AI usage and compliance through direct proctor observation, ChatGPT conversation logs, and self-reports.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 3.3 Regression Model

    The coefficient of interest, beta, measures the causal effect of AI access on a given outcome.

    “The coefficient of interest, β , measures the causal effect of AI access on a given outcome.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § 3.1 Sample and Recruitment

    Recruitment materials framed the study as 'a study on learning' without mentioning AI, to minimize selection concerns.

    “To minimize selection concerns, we framed the study as 'a study on learning' without mentioning AI in any recruitment materials.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductown data
    § 3.4 Main Outcomes

    For main estimates, the paper uses the average of human and AI grader scores for each dimension and shows robustness to using only human or only AI ratings.

    “For our main estimates, we use the average of human and AI grader scores for each dimension and show robustness to using only the human or only the AI ratings.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § 3.2 Summary Statistics and Balance

    Most students (97.2 percent) returned for Session Two.

    “Most students (97.2 percent) returned for Session Two (Panel C).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § 2.2 Session Two

    The two Session Two assessments were presented in randomized order.

    “The session includes two assessments presented in randomized order (Figure 1, Panel B): a ten-item multiple-choice knowledge test and a 20-minute analytical essay that uses the complementary prompt developed for each topic.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductnone given
    § 3.4 Main Outcomes

    As specified in the pre-analysis plan, the overall quality rating and the quality index are the primary essay-based outcomes, and the individual dimensions are secondary outcomes.

    “As specified in our pre-analysis plan, the overall quality rating and the quality index are our primary essay-based outcomes, and the individual dimensions are secondary outcomes.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § 1 Introduction

    Cruces et al. (2026) find that adults with lower levels of education retain some performance gains on an unassisted follow-up task.

    “Cruces et al. (2026) find that adults with lower levels of education retain some performance gains on an unassisted follow-up task.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Cruces et al. (2026) source

  • May combine two claims.

    Study design and conductown data
    § 3.2 Summary Statistics and Balance

    There is no differential attendance in Session One (82.5 versus 82.3 percent) and no differential attrition between sessions.

    “We find no differential attendance in Session One (82.5 versus 82.3 percent) and no differential attrition between sessions.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § 2.2 Session Two

    Session Two measures knowledge retention when all students, regardless of Session One treatment assignment, work without access to AI or external resources.

    “This session measures knowledge retention when all students, regardless of Session One treatment assignment, work without access to AI or external resources.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § 3.4 Main Outcomes

    Each essay was independently evaluated on a 0-10 scale by three or four graders blind to treatment condition and student identity.

    “Each essay was independently evaluated on a 0-10 scale by three or four graders blind to treatment condition and student identity; each grader scored five essays.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data
    § 4.2 Effects on Test Scores

    The test-score estimates are robust to excluding any one time slot or any one test item.

    “The test-score estimates are robust to excluding any one time slot (Appendix Figure A4) or any one test item (Appendix Figure A5).”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § 3.4 Main Outcomes

    Twenty-three graders who failed a comprehension check were excluded from essay evaluation.

    “We exclude 23 graders who failed a comprehension check.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § 2.2 Session Two

    All analyses include session fixed effects to account for protocol variations.

    “All analyses include session fixed effects to account for these protocol variations.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: Session fixed effects fully account for the protocol variations in ways that do not bias the treatment-effect estimates.

  • May combine two claims.

    Study design and conductown data
    § 3.4 Main Outcomes

    The sentence embeddings capture content variation: average within-topic cosine similarity is 0.76 versus 0.16 across topics, and essays responding to the same prompt are more similar (0.79) than essays on the same topic but a different prompt (0.72).

    “As a validation exercise, we confirm that the embeddings capture content variation: average withintopic cosine similarity is 0.76, compared with 0.16 across topics, and essays responding to the same prompt are more similar to each other (0.79) than essays on the same topic but a different prompt (0.72).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 4.3 Effects on Essay-Based Outcomes

    Session One essay estimates reflect both AI-generated text and any learning that occurred during the session, while Session Two essays, written one week later without AI, isolate the learning channel.

    “Because Session One essays are written during the learning phase, when treated students have AI access, the Session One estimates reflect both AI-generated text and any learning that occurred during the session. With the Session Two essays, which students write one week later without AI, we isolate the learning channel.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 5.1 Effects on How Students Learn

    Noy and Zhang (2023) found that ChatGPT access increased enjoyment by 0.47 SD and self-efficacy by 0.19 SD among knowledge workers, though the self-efficacy effect was not statistically significant.

    “These findings align with Noy and Zhang (2023), who find that ChatGPT access increases enjoyment of the task by 0 . 47 SD and self-efficacy by 0 . 19 SD among knowledge workers (though the latter is not statistically significant).”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Noy and Zhang (2023) source

  • The quote only partly states this claim.

    Interpretationcitation
    § 4.2 Effects on Test Scores

    Our ITT of 0.28 SD is substantially larger than the median effect in the educational-intervention literature.

    “Our ITT of ˆ β = 0 . 28 SD is substantially larger.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationown data
    § 4.3 Effects on Essay-Based Outcomes

    Together with the test-score gains, the essay-quality estimates indicate that treated students learned more about the topic and expressed that learning both in facts they recall and in how clearly they explain and argue in writing.

    “Together with the test-score gains of Section 4.2, these estimates indicate that treated students learn more about the topic and express that learning in two ways: in the facts they recall and in how clearly they explain the topic and argue about it in their own writing.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation
    § 4.3 Effects on Essay-Based Outcomes

    Prior studies found convergence among customer-support agents and reductions in collective creative diversity in AI-assisted writing and brainstorming.

    “These null results contrast with the convergence patterns documented by Brynjolfsson et al. (2025) among customer-support agents and with the reductions in collective creative diversity observed in experiments on AI-assisted writing and brainstorming (Doshi and Hauser, 2024; Meincke et al., 2025).”
    Quote in the paperSubstantial claim0 of 3 sources checked
    • cites Brynjolfsson et al. (2025) source
    • cites Doshi and Hauser, 2024

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites Meincke et al., 2025 source

      Checked before — its earlier verdict is held back until a new check runs.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation
    § 5.1 Effects on How Students Learn

    GitHub Copilot users spend about a third of their coding time double-checking and editing AI-generated suggestions and more than half on Copilot-related activities.

    “For example, Mozannar et al. (2024) find that GitHub Copilot users spend about a third of their coding time double-checking and editing AI-generated suggestions, and more than half on Copilotrelated activities.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Mozannar et al. (2024) source

      Not checked in the free run.

  • May combine two claims.

    Study design and conductown data
    § 5.3 Outcomes by Type of AI Use

    LLM coding of conversations agreed with hand coding 87% on the automation indicator, 82% on the augmentation indicator, and 100% on the off-topic indicator.

    “Agreement is 87 percent on the automation indicator, 82 percent on the augmentation indicator, and 100 percent on the off-topic indicator.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 5.2 A Model of Learning with AI

    The shift of learning time from writing toward research follows from both AI capabilities: tutoring raises research effort, and delegation lowers writing effort by a larger proportion than research effort.

    “The shift of learning time from writing toward research (Table 6, Panel B) follows from both capabilities: tutoring raises research effort, and delegation lowers writing effort by a larger proportion than research effort, so each lowers the share of effort spent on writing.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: AI used as a personalized tutor raises the return to research effort: each unit of reading and searching produces more knowledge because the student can ask for explanations and follow-up questions.
    • The authors state this as an assumption: The student can delegate the essay to AI, replacing the student's own writing with an AI-written essay of a given quality.

  • The quote only partly states this claim.

    Interpretationreasoning
    § 5.2 A Model of Learning with AI

    The model can rationalize the three main facts documented so far: the test-score gain, the shift from writing to research, and the enjoyment gain.

    “The model can rationalize the three main facts documented so far.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: AI used as a personalized tutor raises the return to research effort: each unit of reading and searching produces more knowledge because the student can ask for explanations and follow-up questions.
    • The authors state this as an assumption: The student can delegate the essay to AI, replacing the student's own writing with an AI-written essay of a given quality.

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 5.2 A Model of Learning with AI

    The model predicts that outcomes should differ with how students use AI.

    “The model also predicts that outcomes should differ with how students use AI.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: AI used as a personalized tutor raises the return to research effort: each unit of reading and searching produces more knowledge because the student can ask for explanations and follow-up questions.
    • The authors state this as an assumption: The student can delegate the essay to AI, replacing the student's own writing with an AI-written essay of a given quality.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationreasoning
    § 5.2 A Model of Learning with AI

    The enjoyment gain is consistent with each AI capability raising the utility the student obtains from the task and with a lower cost of effort.

    “The enjoyment gain (Table 6, Panel C) is consistent with each capability raising the utility the student obtains from the task and with a lower cost of effort.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: AI used as a personalized tutor raises the return to research effort: each unit of reading and searching produces more knowledge because the student can ask for explanations and follow-up questions.
    • The authors state this as an assumption: The student can delegate the essay to AI, replacing the student's own writing with an AI-written essay of a given quality.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § 5.2 A Model of Learning with AI

    Delegation removes the essay-quality reward from the student's own work, so the student supplies less effort in both activities and forgoes the knowledge that effort would have produced.

    “Delegation removes the essay-quality reward from the student's own work, so the student supplies less effort in both activities and forgoes the knowledge that this effort would have produced.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The student can delegate the essay to AI, replacing the student's own writing with an AI-written essay of a given quality.
    • The authors state this as an assumption: A student chooses effort on research and writing to maximize a weighted sum of knowledge and essay quality net of the cost of effort.
    • The authors state this as an assumption: Both research and writing produce knowledge, and writing also produces the submitted essay, whose quality reflects both the student's writing and the knowledge the student has acquired.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § 5.2 A Model of Learning with AI

    Because students wrote Session Two essays without AI, the essay advantage of students who delegated in Session One should be absent from their unaided work one week later, whereas knowledge from tutoring should persist and show up on tests and in the unaided essay.

    “Because students write the Session Two essays without AI, the essay advantage of students who delegated in Session One should be absent from their unaided work one week later, whereas the knowledge that tutoring produces should persist and show up on the tests and in the unaided essay.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The student can delegate the essay to AI, replacing the student's own writing with an AI-written essay of a given quality.
    • The authors state this as an assumption: An AI-written essay raises the quality of the submitted essay without adding knowledge.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § 5.2 A Model of Learning with AI

    Tutoring raises knowledge because each unit of research effort produces more knowledge and because the student supplies more research effort in response.

    “Tutoring raises knowledge because each unit of research effort produces more of it and because the student endogenously supplies more research effort in response.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: AI used as a personalized tutor raises the return to research effort: each unit of reading and searching produces more knowledge because the student can ask for explanations and follow-up questions.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § Figures and Tables

    The order of the Endline Test and the Essay was randomized across students.

    “the order of the Endline Test and the Essay was randomized across students”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § Figures and Tables

    Students were randomly assigned to AI-allowed or AI-forbidden conditions.

    “students were randomly assigned to AI-allowed or AI-forbidden conditions”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § 6 Beliefs About AI's Impact on Learning

    The open-ended responses were validated against students' AI usage patterns.

    “We validate students' open-ended responses against their AI usage patterns (Appendix C.1)”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § 7 Conclusion

    The absence of an effect on total learning time is likely because the lab setting offered few competing uses of time.

    “likely because the lab setting offered few competing uses of time”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § 7 Conclusion

    In the model, using AI as a tutor raises knowledge.

    “In our model, using AI as a tutor raises knowledge”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § 7 Conclusion

    Whether AI raises or lowers total learning depends on how students reallocate the time they save and on whether learning-productivity gains more than compensate for any reduction in time spent learning.

    “Whether AI raises or lowers total learning therefore depends on how students reallocate the time they save and on whether learning-productivity gains more than compensate for any reduction in time spent learning.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: In ordinary academic work, students choose how long to spend on each task.
    • Rests on an assumption: Many students likely use AI to save time.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § Figures and Tables

    At baseline, the AI-allowed group had a 0.092 lower proportion of students who had already declared a major than the AI-forbidden group (SE 0.052, p<0.10).

    “Already declared major | 0.825 | 0.734 | - 0.092 ∗ | (0.052)”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data
    § B.3 Human Essay Grading

    Each essay was evaluated by three to four independent graders.

    “Each essay is evaluated by three to four independent graders.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Study design and conductnone given
    § A Appendix Figures and Tables

    The physical separation of treatment conditions across different buildings helped prevent cross-contamination between experimental groups.

    “The physical separation of treatment conditions across different buildings helped prevent cross-contamination between experimental groups.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning
    § B.4 AI Essay Grading

    AI grading used six separate API calls per essay, one per dimension, to avoid cross-contamination between scores.

    “For each essay, we make six separate API calls-one per grading dimension-to avoid cross-contamination between scores.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Study design and conductown data
    § Figures and Tables

    At baseline, the AI-allowed and AI-forbidden groups did not differ significantly in the fraction of correct answers (difference -0.007, SE 0.038).

    “Fraction correct (baseline) | 0.306 | 0.301 | - 0.007 | (0.038)”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § A Appendix Figures and Tables

    Each workstation was equipped with privacy dividers to minimize distractions and prevent participants from viewing other screens.

    “Each workstation was equipped with privacy dividers to minimize distractions and prevent participants from viewing other screens.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning
    § B.4 AI Essay Grading

    Human and AI graders shared a common scoring anchor through the same calibration essays, scores, and rationales.

    “The calibration essays, scores, and rationales are the ones shown to human graders during training, ensuring that human and AI graders share a common scoring anchor.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § B.6 Classification of ChatGPT Conversations

    LLM labels of ChatGPT conversations were used as a check on the research assistant's hand coding.

    “We use the LLM labels described here as a check on that coding.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given
    § A Appendix Figures and Tables

    For each session time slot, one laboratory was randomly assigned to the AI-allowed condition and the other to the AI-forbidden condition.

    “For each session time slot, one lab was randomly assigned to the AI-allowed condition and the other to the AI-forbidden condition.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given
    § A Appendix Figures and Tables

    Proctors recorded unauthorized resource use using seating charts, with two proctors assigned to each laboratory.

    “Proctors used these charts to record any instances of unauthorized resource use, with two proctors assigned to each laboratory.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data
    § B.5 SAT Score Imputation

    Imputed SAT scores were used only for bin assignment and did not enter the regression directly as a continuous covariate.

    “The imputed SAT scores are used only for bin assignment; they do not enter the regression directly as a continuous covariate.”
    Quote in the paperNot a substantial claim

  • The quote only partly states this claim.

    Study design and conductown data
    § Figures and Tables

    Attendance at both sessions did not differ significantly between the AI-allowed and AI-forbidden groups (difference -0.020, SE 0.050).

    “Attended both sessions | 0.810 | 0.792 | - 0.020 | (0.050)”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Study design and conductnone given
    § B.3 Human Essay Grading

    Human graders were blind to treatment condition, participant identity, and all other experimental variables.

    “Graders are blind to treatment condition, participant identity, and all other experimental variables.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Premisecitation
    § B.7 Literature Comparison

    In Ba et al. (2024), scores on a closed-book knowledge exam were statistically indistinguishable between ChatGPT-assisted instruction and standard bedside teaching (-0.07 SD), with both groups near the test ceiling.

    “Scores on a closed-book knowledge exam were statistically indistinguishable ( -0.07 SD), with both groups near the test ceiling.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Ba et al. (2024)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

    Study design and conductown data
    § Figures and Tables

    At baseline, the AI-allowed group had a 0.382-year higher mean age than the AI-forbidden group (SE 0.227, p<0.10).

    “Age | 20.016 | 20.400 | 0.382 ∗ | (0.227)”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § B.7 Literature Comparison

    In Fischer et al. (2025), restricted GPT-4 access (only after ten minutes of independent reading) produced a statistically insignificant 0.13 SD effect.

    “restricted access produced a statistically insignificant 0.13 SD.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Fischer et al. (2025)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation
    § B.7 Literature Comparison

    In Bassner et al. (2026), both AI arms scored higher on the exercise itself, but on the supervised, unassisted knowledge test neither arm learned more than the control (baseline-adjusted gains -0.07 SD for Iris and -0.01 SD for ChatGPT).

    “Both AI arms scored higher on the exercise itself, but on the supervised, unassisted knowledge test neither arm learned more than the control: baseline-adjusted gains are -0.07 SD (Iris) and -0.01 SD (ChatGPT)-the paper's titular dissociation between performance and learning.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites Bassner et al. (2026)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data
    § C Student Beliefs About AI and Learning

    All 205 students who attended both sessions answered the open-ended question about AI and learning.

    “All 205 students who attended both sessions answered the question.”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Premisecitation
    § B.7 Literature Comparison

    In Lehmann et al. (2025) Study 3, ChatGPT use during practice produced a 0.42 SD effect on an unassisted 20-question coding post-test, not individually significant.

    “Effects are 0.25 SD (Study 2) and 0.42 SD (Study 3), neither individually significant at conventional levels.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lehmann et al. (2025)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Premisecitation
    § B.7 Literature Comparison

    In Lehmann et al. (2025) Study 2, ChatGPT use during practice produced a 0.25 SD effect on an unassisted 20-question coding post-test, not individually significant.

    “Effects are 0.25 SD (Study 2) and 0.42 SD (Study 3), neither individually significant at conventional levels.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lehmann et al. (2025)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationcitation
    § C.2 Narratives as Causal Graphs

    The taxonomy of mechanisms maps onto the use classification in the main text, with explanation, feedback, and brainstorming as channels where AI works with the student and shortcutting and offloading as channels where AI works for the student.

    “The taxonomy also maps onto the use classification in the main text.”
    • Also stated in: § C.2 Narratives as Causal Graphs · plain
      “Explanation, feedback, and brainstorming are channels in which AI works with the student; shortcutting and offloading are channels in which AI works for the student.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Andre et al. (2026) source

  • May combine two claims.

    Interpretationreasoning
    § D.3 Introducing AI

    In the model, the effect of AI access on knowledge depends on whether the student delegates and, if so, on whether the tutoring gain exceeds the two losses from delegation.

    “The effect of AI access on knowledge therefore depends on whether the student delegates and, if so, on whether the gain from tutoring exceeds the two losses from delegation.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § D.3 Introducing AI

    In the model, delegation decreases both research effort and writing effort.

    “Thus, both types of effort decrease.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: Delegation is treated as the extreme case in which the submitted essay is AI-written with quality Qbar.

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data
    § C.2 Narratives as Causal Graphs

    Every response was coded a second time with a model from a different developer (OpenAI's GPT-5.5), and inter-rater agreement was computed.

    “As a validation, we code every response a second time with a model from a different developer (OpenAI's GPT-5.5) and compute their inter-rater agreement statistic, the pooled probability that a code assigned by one rater is also assigned by the other.”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § D.4 The Decision to Delegate Work to the AI

    In the model, the cost of delegation is the value of the knowledge the student gives up by delegating plus the reward the student's own essay would have earned net of the extra effort it would have taken.

    “The cost of delegation is the value of the knowledge that a student with AI gives up by delegating, ω K [ K ∗ ( τ, 0) -K ∗ ( τ, 1)], plus the reward the student's own essay would have earned, net of the extra effort it would have taken.”
    Quote in the paperSubstantial claim
    • The authors state this as an assumption: The student values knowledge at ω_K per unit and essay quality at ω_Q per unit.
    • The authors state this as an assumption: Delegation saves the student time that writing would have taken, valued at v ≥ 0.

  • May combine two claims.

    Study design and conductown data
    § C.2 Narratives as Causal Graphs

    Mechanism and link agreement statistics are far above their random-coding benchmarks of 0.33 and 0.22, respectively.

    “with the mechanism and link statistics far above their random-coding benchmarks of 0.33 and 0.22, respectively.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § D.4 The Decision to Delegate Work to the AI

    In the model, for a student who writes the essay, tutoring raises essay quality through the knowledge it produces, by φ times the gain from tutoring in equation (D8).

    “For a student who writes the essay, tutoring raises its quality through the knowledge it produces, by ϕ times the gain from tutoring in equation (D8), so a gain in essay quality can also come with a gain in knowledge.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Both research and writing effort produce knowledge (learning).
    • The authors state this as an assumption: Writing produces the submitted essay, and essay quality reflects the student's acquired knowledge.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § D.4 The Decision to Delegate Work to the AI

    In the model, a gain in essay quality can also come with a gain in knowledge.

    “For a student who writes the essay, tutoring raises its quality through the knowledge it produces, by ϕ times the gain from tutoring in equation (D8), so a gain in essay quality can also come with a gain in knowledge.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductcitation
    § B.7 Literature Comparison

    In Lehmann et al. (2025) Study 2, an unintended copy-paste restriction may have affected the treatment.

    “Study 2 included an unintended copy-paste restriction that may have affected the treatment.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Lehmann et al. (2025)

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • May combine two claims.

    Interpretationown data
    § C.3 The Narratives of College Students

    Framing tracks self-assessed proficiency rather than frequency of use or early adoption.

    “Framing thus tracks self-assessed proficiency rather than frequency of use or early adoption.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning
    § D.3 Introducing AI

    In the model, the effect of AI access on knowledge is ex ante ambiguous and depends on the relative size of three forces.

    “Equation (D8) shows that the impact of AI on learning is ex ante ambiguous and it depends on the relative size of three forces.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: AI used for augmentation raises the return to research effort by τ ≥ 0.
    • The authors state this as an assumption: Delegation is treated as the extreme case in which the submitted essay is AI-written with quality Qbar.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation
    § C.3 The Narratives of College Students

    Andre et al. (2026) document the same gender difference in narratives about inflation.

    “the same gender difference Andre et al. (2026) document in narratives about inflation.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites Andre et al. (2026) source

  • May combine two claims.

    Study design and conductown data
    § C.2 Narratives as Causal Graphs

    The two models agree on a response's overall valence 94 percent of the time and on its augmentation-versus-automation framing 85 percent of the time.

    “The two models agree on a response's overall valence 94 percent of the time and on its augmentation-versus-automation framing 85 percent of the time.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Interpretationnone given
    § D.2 The Student's Problem Without AI

    In the model without AI, the student's objective is strictly concave in research and writing effort, so the first-order conditions give a unique solution.

    “The objective is strictly concave in ( r, w ), so the first-order conditions give the unique solution”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Effort in each activity has a convex cost x^2/(2µ).

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § D.3 Introducing AI

    In the model, writing effort remains positive even with delegation because students still get the learning benefit of writing.

    “Notably, writing effort remains positive even with delegation ( m = 1) because students still get the learning benefit of writing, for example when they take notes or organize their ideas.”
    Quote in the paperNot a substantial claim
    • The authors state this as an assumption: Both research and writing effort produce knowledge (learning).

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

    Interpretationreasoning
    § D.5 Relationship Between the Model and the Experiment

    Automation users should stop earlier than control students.

    “automation users should write less than control students and stop earlier.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning
    § D.5 Relationship Between the Model and the Experiment

    Relative to a student without AI, delegation lowers writing effort and frees time.

    “whereas delegation lowers writing effort and frees time”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning
    § D.5 Relationship Between the Model and the Experiment

    Holding the tutoring gain fixed, automation users differ from augmentation users only by the delegation effect, which lowers effort in both activities.

    “Holding the tutoring gain fixed, however, automation users differ from augmentation users only by the delegation effect in equation (D6), which lowers effort in both activities.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Study design and conductnone given
    § D.5 Relationship Between the Model and the Experiment

    Random assignment equalizes the two groups' preferences and returns to effort in expectation.

    “Random assignment equalizes the two groups' preferences and returns to effort in expectation”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning
    § D.5 Relationship Between the Model and the Experiment

    Delegation lowers effort in both activities, with writing falling proportionally more.

    “delegation lowers effort in both activities, writing by proportionally more.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning
    § D.5 Relationship Between the Model and the Experiment

    Holding the delegation choice constant, less costly effort induces more research and writing effort and raises knowledge.

    “Less costly effort induces more research and writing effort and raises knowledge.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The learning-experience measures relate to the cost of effort.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning
    § D.5 Relationship Between the Model and the Experiment

    Students who delegate place a lower value on knowledge, or a higher value on recovered time, expect a better AI-written essay, or gain less from tutoring.

    “By the delegation condition, students who delegate place a lower value on knowledge or a higher value on recovered time, expect a better AI-written essay, or gain less from tutoring”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationnone given
    § D.5 Relationship Between the Model and the Experiment

    Relative to a student without AI, tutoring raises research effort and leaves writing effort unchanged.

    “Relative to a student without AI, tutoring raises research effort and leaves writing effort unchanged”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § D.5 Relationship Between the Model and the Experiment

    Knowledge gains can appear in both the follow-up test and the unaided essay, whereas features of AI-written text cannot.

    “Knowledge gains can appear in both the follow-up test and the unaided essay, whereas the features of AI-written text cannot.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Both tests measure knowledge K.
    • Rests on an assumption: The unaided Session Two essay's quality reflects the student's writing effort during the assessment and, through phi, the knowledge the student retains.

  • Classified as probably not a substantial claim. May combine two claims.

    Interpretationreasoning
    § D.5 Relationship Between the Model and the Experiment

    For a student who delegates, the submitted essay can be better or worse than the student's own.

    “For a student who delegates (and the rise in the share of text flagged as AI-generated shows that many treated students did), the submitted essay can be better or worse than the student's own.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: The Session One essay measures Q1.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationnone given
    § D.5 Relationship Between the Model and the Experiment

    For a student who writes the essay, tutoring raises essay quality through the knowledge it produces.

    “For a student who writes the essay, tutoring raises its quality through the knowledge it produces.”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The Session One essay measures Q1.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § D.5 Relationship Between the Model and the Experiment

    Augmentation users should do more research than control students.

    “Augmentation users should therefore do more research than control students”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § D.5 Relationship Between the Model and the Experiment

    The model can rationalize automation users' higher baseline test scores through a higher Qbar for better-prepared students.

    “which the model can rationalize through a higher ¯ Q for better-prepared students.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

    Interpretationreasoning
    § D.5 Relationship Between the Model and the Experiment

    A higher tutoring capability raises research effort without changing writing effort.

    “a higher τ raises research effort without changing writing effort”
    Quote in the paperNot a substantial claim

  • Classified as probably not a substantial claim.

    Interpretationreasoning
    § D.5 Relationship Between the Model and the Experiment

    Automation users should write less than control students.

    “and automation users should write less than control students”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning
    § D.5 Relationship Between the Model and the Experiment

    A delegating student's Session One essay advantage comes from Qbar and is absent from unaided work.

    “First, a delegating student's Session One essay advantage comes from ¯ Q and is absent from unaided work”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The Session One essay measures Q1.
    • Rests on an assumption: The unaided Session Two essay's quality reflects the student's writing effort during the assessment and, through phi, the knowledge the student retains.

  • Probably not a substantial claim: may not pass S4 (based). May combine two claims.

    Interpretationreasoning
    § D.5 Relationship Between the Model and the Experiment

    Knowledge from tutoring or from the student's own research and writing persists into unaided assessments, including the unaided essay.

    “whereas knowledge, whether from tutoring or from the student's own research and writing, persists into the unaided assessments, including the unaided essay, which expresses knowledge through ϕ .”
    Quote in the paperNot a substantial claim
    • Rests on an assumption: The unaided Session Two essay's quality reflects the student's writing effort during the assessment and, through phi, the knowledge the student retains.

  • Probably not a substantial claim: may not pass S4 (based).

    Interpretationreasoning
    § D.5 Relationship Between the Model and the Experiment

    In the laboratory, freed time could be used only by stopping early and browsing.

    “which in the laboratory students could use only by stopping early and browsing.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationreasoning
    § D.5 Relationship Between the Model and the Experiment

    In the model, both AI capabilities lower the share of effort devoted to writing.

    “In the model, both AI capabilities lower the share of effort devoted to writing: a higher τ raises research effort without changing writing effort, and delegation lowers effort in both activities, writing by proportionally more.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given
    § D.5 Relationship Between the Model and the Experiment

    The time-use survey reports minutes, which relate to effort only loosely.

    “The time-use survey reports minutes, which relate to effort loosely, as discussed below”
    Quote in the paperNot a substantial claim
189 of 189

  • The paper relies on this without support.

    Evidence on whether the off-the-shelf chatbots students use help them learn is scarce.

    “Despite this widespread use, evidence on whether the off-the-shelf chatbots students use help them learn is scarce.”

  • The paper relies on this without support.

    The state-of-the-art AI detector accurately flags AI-generated text.

    “the fraction flagged as AIgenerated by a state-of-the-art detector”
    “the share of treated students' text flagged as AI-generated rises by 22.2 pp”

  • The paper relies on this without support.

    Students' self-reported learning-time allocation and enjoyment ratings accurately measure their actual time use and enjoyment.

    “treated students report spending 5.3 pp less of their learning time on writing activities and 4.4 pp more on reading and searching for information”
    “treated students report 11 percent higher enjoyment ratings”

  • The paper relies on this without support.

    ChatGPT conversation logs accurately reflect students' AI use and permit valid classification into augmentation, automation, and mixed use types.

    “we use treated students' ChatGPT conversation logs and classify 61 percent of AI users as 'augmentation' users (who work with AI), 9 percent as 'automation' users (who use AI to do the work for them), and 27 percent as 'mixed' users (who do both).”

  • The paper relies on this without support.

    Graders' overall quality ratings and five dimension scores validly measure essay quality and understanding.

    “Graders give each essay an overall quality rating and scores on five dimensions: accuracy of content, use of evidence and examples, relevance to the prompt, organization and structure, and writing style and clarity.”

  • The authors state this as an assumption.

    A student divides effort between research (reading and searching) and writing.

    “we write down a model in which a student divides effort between research (reading and searching) and writing”

  • The authors state this as an assumption.

    Both research and writing activities produce knowledge, and writing also produces the essay, whose quality depends on the student's writing and on what the student knows.

    “Both activities produce knowledge, and writing also produces the essay, whose quality depends on the student's writing and on what the student knows.”

  • The authors state this as an assumption.

    AI used as a tutor raises the return to research effort, while AI used to write the essay lets the student delegate so that the student's own work no longer raises the essay's quality.

    “Used as a tutor, it raises the return to research effort; used to write the essay, it lets the student delegate, so the student's own work no longer raises the essay's quality.”

  • The paper relies on this without support.

    The randomly assigned treatment status satisfies the exclusion restriction and monotonicity required for 2SLS to identify the LATE for compliers.

    “These estimates identify the local average treatment effect (LATE) for compliers, students whose treatment assignment influenced their AI use.”

  • The paper relies on this without support.

    Students' self-reports about looking up information or studying between sessions are accurate.

    “only 5 percent of students reported looking up extra information about the topic, with no significant difference between treatment and control groups, and no student reported studying it.”

  • The paper relies on this without support.

    Session fixed effects fully account for the protocol variations in ways that do not bias the treatment-effect estimates.

    “All analyses include session fixed effects to account for these protocol variations.”

  • The paper relies on this without support.

    The test-score gap between rule violators and non-violators reflects the causal effect of violations, so that multiplying it by the treatment effect on violations bounds the contribution of violations to the ITT.

    “As a back-of-the-envelope calculation, we multiply each measure's treatment effect by the test-score gap between violators and non-violators (22.1 pp for proctor-observed violations and 16.4 pp for self-reported ones).”

  • The authors state this as an assumption.

    AI used as a personalized tutor raises the return to research effort: each unit of reading and searching produces more knowledge because the student can ask for explanations and follow-up questions.

    “First, we assume that AI used as a personalized tutor raises the return to research effort: with AI, each unit of reading and searching produces more knowledge, because the student can ask for explanations and pose follow-up questions.”

  • The authors state this as an assumption.

    The student can delegate the essay to AI, replacing the student's own writing with an AI-written essay of a given quality.

    “Second, we assume that the student can delegate the essay to AI, which replaces the student's own writing with an AI-written essay of a given quality.”

  • The authors state this as an assumption.

    A student chooses effort on research and writing to maximize a weighted sum of knowledge and essay quality net of the cost of effort.

    “In the model, a student chooses how much cognitive effort to exert on two activities: 'research' (which includes reading and searching) and 'writing' (which includes drafting, editing, and note-taking).”
    “The student chooses the two effort levels to maximize a weighted sum of knowledge and essay quality, net of the cost of effort.”

  • The authors state this as an assumption.

    Both research and writing produce knowledge, and writing also produces the submitted essay, whose quality reflects both the student's writing and the knowledge the student has acquired.

    “Both activities produce knowledge, and writing also produces the essay the student submits, whose quality reflects both the student's writing and the knowledge the student has acquired.”

  • The authors state this as an assumption.

    An AI-written essay raises the quality of the submitted essay without adding knowledge.

    “In the model, an AI-written essay raises the quality of the submitted essay without adding knowledge.”

  • The paper relies on this without support.

    A one-week unaided Session Two performance is an adequate measure of the durability of learning gains.

    “A week later, when students work without AI, augmentation users' test-score difference remains positive ( ˆ β = 0 . 24 SD, p = 0 . 095) and does not change significantly between sessions, and their essay-quality difference is positive but imprecise in both sessions ( ˆ β = 0 . 09 SD and 0 . 23 SD).”

  • The paper relies on this without support.

    Students' beliefs about AI's effect on learning shape their adoption decisions.

    “Students' beliefs about AI's effect on learning may shape their adoption decisions.”

  • The paper relies on this without support.

    In ordinary academic work, students choose how long to spend on each task.

    “In ordinary academic work, students choose how long to spend on each task”

  • The paper relies on this without support.

    Many students likely use AI to save time.

    “many likely use AI to save time”

  • The paper relies on this without support.

    Random assignment to AI access affects test scores and essay outcomes only through actual ChatGPT use (the exclusion restriction for the TOT estimates).

    “treatment-on-the-treated (TOT) estimates from two-stage least squares, instrumenting actual ChatGPT use with random treatment assignment”

  • The paper relies on this without support.

    The two prompts designed for each topic are of comparable difficulty.

    “For each topic, we designed two complementary prompts of comparable difficulty.”

  • The paper relies on this without support.

    The reading materials are self-contained and require no prior knowledge of the topic.

    “The texts are information-rich but selfcontained, requiring no prior knowledge of the topic.”

  • The paper relies on this without support.

    The LLM-based classification of student prompts and full conversations into the defined categories is accurate.

    “For each prompt, we provide the LLM with the full system prompt establishing the experimental context-that students were given a reading passage on a science topic and asked to write an essay-along with descriptions of each category. The classifier returns a single category label.”
    “For each conversation, we provide the LLM with the full exchange between the student and ChatGPT, along with descriptions of each category.”

  • The authors state this as an assumption.

    Both research and writing effort produce knowledge (learning).

    “Both activities produce knowledge (denoted by K ), that is, lead to learning.”

  • The authors state this as an assumption.

    Writing produces the submitted essay, and essay quality reflects the student's acquired knowledge.

    “Writing also produces the essay the student submits, and the quality of that essay reflects the knowledge the student has acquired.”

  • The authors state this as an assumption.

    The student values knowledge at ω_K per unit and essay quality at ω_Q per unit.

    “The student values knowledge at ω K &gt; 0 per unit and essay quality at ω Q &gt; 0 per unit.”

  • The authors state this as an assumption.

    Effort in each activity has a convex cost x^2/(2µ).

    “Effort x in each activity costs x 2 / (2 µ ), with µ &gt; 0. The cost is convex, so each additional unit of effort costs more than the last, and a higher µ lowers the marginal cost of effort.”

  • The authors state this as an assumption.

    AI used for augmentation raises the return to research effort by τ ≥ 0.

    “First, used for augmentation, AI raises the return to research effort by τ ≥ 0.”

  • The authors state this as an assumption.

    Delegation is modeled as a binary choice.

    “We model delegation as a binary decision for simplicity.”

  • The authors state this as an assumption.

    Delegation is treated as the extreme case in which the submitted essay is AI-written with quality Qbar.

    “It treats delegation as the extreme case in which the submitted essay is AI-written”

  • The authors state this as an assumption.

    Delegation saves the student time that writing would have taken, valued at v ≥ 0.

    “Delegation saves the student the time that writing the essay would have taken. Let v ≥ 0 denote the student's valuation of this time.”

  • The authors state this as an assumption.

    τ and Qbar are taken as given parameters when solving for effort and delegation.

    “For simplicity, we take them as given parameters when solving for effort and delegation.”

  • The paper relies on this without support.

    Both tests measure knowledge K.

    “Both tests measure K .”

  • The paper relies on this without support.

    The Session One essay measures Q1.

    “The Session One essay measures Q 1 .”

  • The paper relies on this without support.

    The unaided Session Two essay's quality reflects the student's writing effort during the assessment and, through phi, the knowledge the student retains.

    “The Session Two essay is written without AI, so its quality reflects the student's writing effort during the assessment and, through ϕ , the knowledge the student retains.”

  • The paper relies on this without support.

    The learning-experience measures relate to the cost of effort.

    “the learning-experience measures relate to the cost of effort.”
38 of 38

Cites

The publisher hasn’t deposited this paper’s reference list.

Research works printed but not linked: 4

  1. Bowden, A. B., Rodriguez, V., and Weingarten, Z. (2026). The unintended consequences of academic leniency. American Economic Journal: Economic Policy . Forthcoming.
  2. Carver, R. P. (1992). Reading rate: Theory, research, and practical implications. Journal of Reading , 36(2):84-95.
  3. Jensen, R. (2010). The (perceived) returns to education and the demand for schooling. The Quarterly Journal of Economics , 125(2):515-548.
  4. Lehmann, M., Cornelius, P. B., and Sting, F. J. (2025). AI meets the classroom: When do large language models harm learning? SSRN Working Paper 4941259.

Cited by

Sorted by citations

Authors
Contractor, Z., & Reyes, G.
Venue
arXiv
Publisher
arXiv
Published
2026
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2607.08849
arXiv
2607.08849
OpenAlex
W7168156755
Semantic Scholar
c561714dc6b97ff5306ff4e4db89fbcd54848a77
Cited by
0
References
0
Type
posted content
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Intelligent Tutoring Systems and Adaptive Learning
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
generative AI, student learning, AI-assisted learning, essay writing, knowledge retention, higher-order thinking skills, AI augmentation, AI automation, joyful learning, undergraduate education, information seeking

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-18.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-18.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review