Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 62 of the 75 references it cites were found in a registry or an index, and 6 are not checked yet; 25 of the 63 sources its claims cite were checked for support.

Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability

Open access
Borg, Markus, Hewett, Dave, Hagatulah, Nadim, Couderc, Noric, Söderberg, Emma, Graham, Donald, Kini, Uttam, Farley, Dave · 2026 · Empirical Software Engineering

Abstract

Abstract Context AI assistants, like GitHub Copilot and Cursor, are transforming software engineering. While several studies highlight productivity improvements, their impact on maintainability requires further investigation. Objective This study investigates whether co-development with AI assistants affects software maintainability, specifically how easily other developers can evolve the resulting source code. Method We conducted a two-phase, preregistered controlled experiment involving 151 participants, 95% of whom were professional developers. In Phase 1, participants added a new feature to a Java web application, with or without AI assistance. In Phase 2, a randomized controlled trial, new participants evolved these solutions without AI assistance. Results Phase 2 revealed no significant differences in subsequent evolution with respect to completion time or code quality. Bayesian analysis suggests that any speed or quality improvements from AI use were at most small and highly uncertain. Observational results from Phase 1 corroborate prior research: using an AI assistant yielded a 30.7% median reduction in completion time, and habitual AI users showed an estimated 55.9% speedup. Conclusions Overall, we did not detect systematic maintainability advantages or disadvantages when other developers evolved code co-developed with AI assistants. Within the scope of our tasks and measures, we observed no consistent warning signs of degraded code-level maintainability. Future work should examine risks such as code bloat from excessive code generation and cognitive debt as developers offload more mental effort to assistants.

Verified 2026-10-10 · 6 references not checked yet

Overview

CoChat Summary

TL;DR

A preregistered two-phase controlled experiment with 151 participants (95.4% professional developers) found no significant differences in Phase 2 when developers manually evolved code co-developed with AI assistants versus code developed without AI. Phase 1 observational results corroborated prior AI speedups: a 30.7% median reduction in completion time overall and an estimated 55.9% speedup for habitual AI users.

Key findings

  • Phase 2 RCT (N=75) showed no significant difference in manual evolution completion time: treatment median 136 minutes vs control 173 minutes, Wilcoxon p=0.56, Cliff’s δ=−0.079; Bayesian posterior mean −12.6% (95% CrI [−45%, +33%], P(∆<0)=76%).
  • CodeHealth showed no significant frequentist difference (treatment mean 8.49 vs control 8.44, p=0.24, Cohen’s d=0.28), but Bayesian analysis suggested a small improvement when the original solution was co-developed by a habitual AI user: +0.10 (95% CrI [+0.02, +0.19], P(∆>0)=98%). Test coverage was essentially identical (median 0.70 in both groups, p=0.49, Cliff’s δ=0.09).
  • Perceived productivity showed no significant frequentist difference (treatment mean 3.95 vs control 4.06, p=0.37, Cohen’s d=−0.21); Bayesian analysis for habitual AI users suggested a small uncertain disadvantage (−1.26 SD, 95% CrI [−2.46, +2.31], P(∆<0)=84%).
  • Phase 1 observational results: AI-devs had a 30.7% shorter median completion time (p=0.0029, Cliff’s δ=−0.42); Bayesian estimate for habitual AI users was about 59.6% faster (CrI [−77.11%, −30.64%]). AI-devs added comments more often (79% vs 54%) and resolved the Bumpy Road smell more often (72% vs 46%), but introduced more Complex Conditionals (13% vs 5%).

Methods

  • Preregistered two-phase controlled experiment. Phase 1: participants added a feature to a Java Spring Boot RecipeFinder application with or without AI assistance. Phase 2: a randomized controlled trial in which new participants manually evolved Phase 1 solutions without AI assistance.
  • 449 participants completed prescreening; 151 completed tasks (33.6%); Phase 2 RCT had 75 participants. Measures were completion time, CodeHealth (custom CodeScene configuration), test coverage (JaCoCo), and perceived productivity (SPACE-based Likert scale).
  • Analysis combined frequentist tests (Wilcoxon rank-sum, Welch’s t-test) with Bayesian causal models using priors, plus qualitative coding of free-text responses and source code.

Limitations

  • Stated by the paper: the study observes only short-term maintainability effects, focuses on individual developers rather than teams, uses a small RecipeFinder system, and does not investigate security. The Phase 2 RCT was underpowered for small-to-moderate effects (75 participants; 80% power only for d≈0.66; power ≈0.57 for d≈0.5 and ≈0.17 for d≈0.2). Remote participation limits protocol adherence, and many completion times relied on self-reported estimates. Sampling bias is acknowledged (95% male; missing China, Japan, South Korea, Iran).
  • Stated construct threats include possible differing interpretations of “AI-assisted development” and potential confirmation bias in perceived productivity among AI-devs.
  • Evident from the text: deliberately vague quality expectations led to varied participant effort and definitions of “good enough”; some participants found Task 2 simplistic; setup and learning-time variation was large; one AI-dev submitted two Task 1 solutions; and the study did not examine Phase 2 developers using AI assistants themselves.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10664-026-10889-1

Cites

62 of 75 references verified0 citation errors · 7 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

24 Valid · 8 Not valid · 36 Not checked · 378 Own results and interpretations

155 more claims

Create a free account to see all 446 claims and to verify the 8 references still open.

See all claims — free account
  • Probably not a substantial claim. 216
  • It may state two separate claims as one. 123
  • The quote only partly states this claim. 4

Create a free account to see these claims.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

5 of 7 · the rest aren’t loaded

2 more findings

Create a free account to see every finding of this check.

See all findings — free account

Citations

Cites

Show all 63 references

Cited by

See all 13

Metadata

Authors
Borg, M., Hewett, D., Hagatulah, N., Couderc, N., Söderberg, E., Graham, D., Kini, U., & Farley, D.
Affiliations
Lund University (SE)
Corresponding author
Borg, Markus
Venue
Empirical Software Engineering
Published
2026-06-09
Volume
31
Issue
6
ISSN
  • 1382-3256 (ISSN-L)
  • 1573-7616
OpenAlex
W4416888427
Cited by
4
References
63
Citations by year
2026: 4
FWCI
34.03
Type
journal article
Language
en
Open access
Licence
Locations
  • Empirical Software Engineering (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • Lund University Publications (Lund University) (submittedVersion)
  • arXiv (Cornell University)
Funders
Lund University
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
Keywords
AI assistants, software maintainability, code evolution, GitHub Copilot, Cursor, developer productivity, code quality, code bloat, cognitive debt, Java web application

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability. https://app.cochat.ai/papers/10.1007/s10664-026-10889-1
The paper
Borg, M., Hewett, D., Hagatulah, N., Couderc, N., Söderberg, E., Graham, D., Kini, U., & Farley, D. (2026). Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability. Empirical Software Engineering. https://doi.org/10.1007/s10664-026-10889-1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.