The DOI matches its Crossref record; 62 of the 75 references it cites were found in a registry or an index, and 6 are not checked yet; 25 of the 63 sources its claims cite were checked for support.
Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability
Open accessAbstract
Abstract Context AI assistants, like GitHub Copilot and Cursor, are transforming software engineering. While several studies highlight productivity improvements, their impact on maintainability requires further investigation. Objective This study investigates whether co-development with AI assistants affects software maintainability, specifically how easily other developers can evolve the resulting source code. Method We conducted a two-phase, preregistered controlled experiment involving 151 participants, 95% of whom were professional developers. In Phase 1, participants added a new feature to a Java web application, with or without AI assistance. In Phase 2, a randomized controlled trial, new participants evolved these solutions without AI assistance. Results Phase 2 revealed no significant differences in subsequent evolution with respect to completion time or code quality. Bayesian analysis suggests that any speed or quality improvements from AI use were at most small and highly uncertain. Observational results from Phase 1 corroborate prior research: using an AI assistant yielded a 30.7% median reduction in completion time, and habitual AI users showed an estimated 55.9% speedup. Conclusions Overall, we did not detect systematic maintainability advantages or disadvantages when other developers evolved code co-developed with AI assistants. Within the scope of our tasks and measures, we observed no consistent warning signs of degraded code-level maintainability. Future work should examine risks such as code bloat from excessive code generation and cognitive debt as developers offload more mental effort to assistants.
Verified 2026-10-10 · 6 references not checked yet
Overview
CoChat Summary
TL;DR
A preregistered two-phase controlled experiment with 151 participants (95.4% professional developers) found no significant differences in Phase 2 when developers manually evolved code co-developed with AI assistants versus code developed without AI. Phase 1 observational results corroborated prior AI speedups: a 30.7% median reduction in completion time overall and an estimated 55.9% speedup for habitual AI users.
Key findings
- Phase 2 RCT (N=75) showed no significant difference in manual evolution completion time: treatment median 136 minutes vs control 173 minutes, Wilcoxon p=0.56, Cliff’s δ=−0.079; Bayesian posterior mean −12.6% (95% CrI [−45%, +33%], P(∆<0)=76%).
- CodeHealth showed no significant frequentist difference (treatment mean 8.49 vs control 8.44, p=0.24, Cohen’s d=0.28), but Bayesian analysis suggested a small improvement when the original solution was co-developed by a habitual AI user: +0.10 (95% CrI [+0.02, +0.19], P(∆>0)=98%). Test coverage was essentially identical (median 0.70 in both groups, p=0.49, Cliff’s δ=0.09).
- Perceived productivity showed no significant frequentist difference (treatment mean 3.95 vs control 4.06, p=0.37, Cohen’s d=−0.21); Bayesian analysis for habitual AI users suggested a small uncertain disadvantage (−1.26 SD, 95% CrI [−2.46, +2.31], P(∆<0)=84%).
- Phase 1 observational results: AI-devs had a 30.7% shorter median completion time (p=0.0029, Cliff’s δ=−0.42); Bayesian estimate for habitual AI users was about 59.6% faster (CrI [−77.11%, −30.64%]). AI-devs added comments more often (79% vs 54%) and resolved the Bumpy Road smell more often (72% vs 46%), but introduced more Complex Conditionals (13% vs 5%).
Methods
- Preregistered two-phase controlled experiment. Phase 1: participants added a feature to a Java Spring Boot RecipeFinder application with or without AI assistance. Phase 2: a randomized controlled trial in which new participants manually evolved Phase 1 solutions without AI assistance.
- 449 participants completed prescreening; 151 completed tasks (33.6%); Phase 2 RCT had 75 participants. Measures were completion time, CodeHealth (custom CodeScene configuration), test coverage (JaCoCo), and perceived productivity (SPACE-based Likert scale).
- Analysis combined frequentist tests (Wilcoxon rank-sum, Welch’s t-test) with Bayesian causal models using priors, plus qualitative coding of free-text responses and source code.
Limitations
- Stated by the paper: the study observes only short-term maintainability effects, focuses on individual developers rather than teams, uses a small RecipeFinder system, and does not investigate security. The Phase 2 RCT was underpowered for small-to-moderate effects (75 participants; 80% power only for d≈0.66; power ≈0.57 for d≈0.5 and ≈0.17 for d≈0.2). Remote participation limits protocol adherence, and many completion times relied on self-reported estimates. Sampling bias is acknowledged (95% male; missing China, Japan, South Korea, Iran).
- Stated construct threats include possible differing interpretations of “AI-assisted development” and potential confirmation bias in perceived productivity among AI-devs.
- Evident from the text: deliberately vague quality expectations led to varied participant effort and definitions of “good enough”; some participants found Task 2 simplistic; setup and learning-time variation was large; one AI-dev submitted two Task 1 solutions; and the study did not examine Phase 2 developers using AI assistants themselves.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
24 Valid · 8 Not valid · 36 Not checked · 378 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“This is supported by a recent study on LLM creativity by Haase et al. (2025), which shows homogenization effects across LLMs, i.e., output from models lacks originality and clusters around similar solutions.”
What the sources say
- Partly supportedcites Haase et al. (2025) sourceExperimental — accuracy not yet measured
The Discussion (p. 10) finds that highly creative responses were rare and says LLM outputs often lack the originality needed to reach the top decile of human creativity. It describes homogenization in LLM-assisted output as an already documented trend, but does not show that outputs from different models cluster around similar solutions. The Discussion (p. 11) instead emphasizes substantial variability across and within models. (LLM, conf=0.86)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“In the context of essay writing, they found that participants assisted by ChatGPT showed significantly lower cognitive activity than those using web search or no tools at all.”
What the sources say
- Supportedcites Kosmyna et al. 2025 source
The EEG analysis reports significantly different neural connectivity patterns across the three essay-writing groups, with the ChatGPT-assisted group showing the weakest overall coupling, the web-search group intermediate engagement, and the no-tools group the strongest. The paper measures this difference as EEG connectivity, which supports the citing sentence’s paraphrase of lower cognitive activity. (LLM, conf=0.88)
plainstates a causen = 54- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Moreover, the LLM-assisted participants underperformed in remembering details of their work.”
What the sources say
- Supportedcites Kosmyna et al. 2025 source
The Discussion (p. 149) reports that participants originally assigned to the LLM group had more difficulty quoting their essays after AI support was removed. The quoting results (pp. 46–47) show that they reproduced accurate quotes less often than participants originally assigned to the Brain-only group, supporting the claim about remembering details of their work. (LLM, conf=0.93)
plainstates a causen = 54- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Bayesian analysis suggests that any speed or quality improvements from AI use were at most small and highly uncertain.”
hedgedn = 151- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“although Bayesian analysis suggests a small CodeHealth improvement when the original solution was co-developed with AI by a habitual AI user.”
hedgedstates a possible cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
155 more claims
Create a free account to see all 446 claims and to verify the 8 references still open.
- Probably not a substantial claim. 216
- It may state two separate claims as one. 123
- The quote only partly states this claim. 4
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ 5.1 RQ1: More Efficient Manual Evolution? · p. 54
“our Bayesian analysis of Task 1 supports this trend, showing that habitual AI users completed the task about 59.6% faster than the control group”
§ 7 Conclusion and Future Work · p. 62
“in Phase 1, the posterior mean effect on completion time for habitual AI users was a 55.9% speedup”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.2.2 Perceived Productivity · p. 35
“anon144 Evolved !AI-dev and finished the task in 3 hours. Changed 12 files in 21 commits.”
§ 4.2.2 Perceived Productivity · p. 35
“-anon144 , Task 2, Treatment, Pro+Advanced, R5, 290 LoC, (4.5h, CH=8.61, TC=69%, PP=5.0)”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.4.2 Quantitative Bayesian Analysis · p. 20
“the range of test coverages in our dataset is narrow (65% to 75%).”
§ 4.3.2 Test Coverage · p. 41
“The highest observed TC was 0.90, submitted by Participant anon050 , who evolved a Task 1 solution with TC=0.91 - the second highest score in Task 1. Note that we received no Task 2 solution building on the Task 1 solution with the highest TC (0.98). At the lower end, three participants submitted solutions with TC=0.59:”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 3.3.2 Data Validation and Correction · p. 18
“After inverting the two negatively worded items, the resulting alpha was 0.82, which shows a good internal consistency.”
§ 6.4 Conclusion Validity · p. 61
“We verified high internal consistency (Cronbach's α > 0 . 85 )”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 4.1.1 Task 1 Demographics · p. 24
“The task concluded with 39 valid AI-dev solutions and 37 !AI-dev solutions.”
§ 4.1.2 Task 2 Demographics · p. 28
“Of the 75 Task 1 solutions, 24 were not evolved further (see the black box in Fig. 9).”
2 more findings
Create a free account to see every finding of this check.
Citations
Cites
Statistical Rethinking: A Bayesian Course with Examples in R and Stan
McElreath, R. (2020) · doi:10.1201/9780429029608 · Cited by 1,959
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043
Competition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Gelman, A., Hill, J., & Vehtari, A. (2020) · doi:10.1017/9781139161879 · Cited by 613
Software measurement: a necessary scientific basis
Fenton, N. (1994) · IEEE Transactions on Software Engineering · doi:10.1109/32.268921 · Cited by 501
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436
On the naturalness of software
Hindle, A., Barr, E. T., Su, Z., Gabel, M., & Devanbu, P. (2012) · 2012 34th International Conference on Software Engineering (ICSE) · doi:10.1109/icse.2012.6227135 · Cited by 425
Quantifying the Effect of Code Smells on Maintenance Effort
Sjoberg, D. I. K., Yamashita, A., Anda, B. C. D., Mockus, A., & Dyba, T. (2013) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2012.89 · Cited by 316
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023) · arXiv · doi:10.48550/arxiv.2302.06590 · Cited by 287
Do Users Write More Insecure Code with AI Assistants?
Perry, N., Srivastava, M., Kumar, D., & Boneh, D. (2023) · Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security · doi:10.1145/3576915.3623157 · Cited by 259
Cited by
Rethinking Software Engineering for Agentic AI Systems
Alenezi, M. (2026) · arXiv.org · doi:10.48550/arxiv.2604.10599 · Cited by 8
“The empirical evidence that downstream maintainability depends on process quality rather than model capability [14] provides strong justification for investing in process infrastructure over model sophistication.”
InfluentialCode for Machines, Not Just Humans: Quantifying AI-Friendliness with Code Health Metrics
Borg, M., Hagatulah, N., Tornhill, A., & Söderberg, E. (2026) · IEEE/ACM International Conference on AI Foundation Models and Software Engineering (FORGE) · doi:10.1145/3793655.3793722 · Cited by 10
“The metric has also been used in recent industry-facing AI studies [8, 29].”
Self-citationPopescu, R. M., Gros, D., Botocan, A., Pandita, R., Dévanbu, P., & Izadi, M. (2026) · International Conference on Mining Software Repositories (MSR) · doi:10.1145/3793302.3793354 · Cited by 6
Martin-Lopez, A., Tufano, R., Guglielmi, E., Sánchez, A. B., Sanz, A. L., Scalabrino, S., Oliveto, R., Segura, S., & Bavota, G. (2026) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2026.3679627 · Cited by 7
“On average, WAI increased completeness by 42.99 points relative to NAI (95% CI [36.26, 49.72]), holding all covariates constant and accounting for clustering by participant and experiment.”
InfluentialMenezes, R. R. (2026) · Nexus Science Review · doi:10.66104/z67gjz21 · Cited by 1
“Later maintenance work must also be considered because a change can be easy to produce but costly for another developer to understand or extend (Borg et al., 2026; GitHub, 2024; Perry et al., 2023; Sandoval et al., 2023).”
Influential
Metadata
- Authors
- Borg, M., Hewett, D., Hagatulah, N., Couderc, N., Söderberg, E., Graham, D., Kini, U., & Farley, D.
- Affiliations
- Lund University (SE)
- Corresponding author
- Borg, Markus
- Venue
- Empirical Software Engineering
- Published
- 2026-06-09
- Volume
- 31
- Issue
- 6
- ISSN
- 1382-3256 (ISSN-L)
- 1573-7616
- OpenAlex
- W4416888427
- Cited by
- 4
- References
- 63
- Citations by year
- 2026: 4
- FWCI
- 34.03
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Empirical Software Engineering (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Lund University Publications (Lund University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Lund University
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Software › Advanced Software Engineering Methodologies
- Keywords
- AI assistants, software maintainability, code evolution, GitHub Copilot, Cursor, developer productivity, code quality, code bloat, cognitive debt, Java web application
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability. https://app.cochat.ai/papers/10.1007/s10664-026-10889-1
- The paper
- Borg, M., Hewett, D., Hagatulah, N., Couderc, N., Söderberg, E., Graham, D., Kini, U., & Farley, D. (2026). Echoes of AI: Investigating the downstream effects of AI assistants on software maintainability. Empirical Software Engineering. https://doi.org/10.1007/s10664-026-10889-1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.