CoChat

AI-assisted scoping review of code sharing in clinical prediction model research

Open access

The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.

Abstract

Clinical prediction models are increasingly deployed to support diagnostic and prognostic decisions, making reproducibility essential for assessing their reliability and generalizability. Analytical code supports independent assessment of these processes, yet its availability in the literature remains limited. This scoping review quantifies current practices in sharing analytical code to inform the development of TRIPOD-Code, a reporting guideline for code availability and reproducibility. Here a large-language-model-assisted pipeline was developed to screen articles citing TRIPOD or TRIPOD+AI, extract repository links and assess retrieved repositories against 14 predefined reproducibility-related features. Among 3,967 articles, 482 (12.2%) included code-sharing statements. Sharing prevalence varied widely by journal and country. Repository assessment showed substantial heterogeneity in reproducibility features. These findings underscore the need for clearer expectations beyond code availability, including documentation, dependency specification and executable structure. Strengthening these practices may improve the usability of clinical prediction model studies and support their deployment in real-world clinical settings.

Verification finished with gaps

Some checks didn’t finish. The results below cover the steps that did.

  1. Registry recordVerified
  2. Full textRead the full text
  3. Reference list40 references found
  4. Reference check40 of 40 references checked
  5. StatisticsRecomputed
  6. ClaimsDidn’t finish — the rest of the report still applies.
  7. Cited sourcesDidn’t finish — the rest of the report still applies.
  8. Self-checkChecked
  9. VerificationFindings verified

Overview

CoChat Summary

TL;DR

This AI-assisted scoping review found that among 3,967 clinical prediction model articles citing TRIPOD or TRIPOD+AI, only 12.2% shared code, and shared repositories often lacked documentation, dependency specification, licensing, and other reproducibility features. The authors conclude that code availability alone is insufficient for computational reproducibility and use these findings to inform the TRIPOD-Code reporting guideline.

Key findings

  • Among 3,967 included articles, 482 (12.2%) included code-sharing statements; code sharing increased over time (OR 1.12 per publication year, 95% CI 1.07–1.18, P < 0.001), rising from 6.3% in 2015 to 14.5% in 2024 and 15.8% in 2025.
  • Of 447 repository links, 27 pointed to unsupported providers, 38 could not be resolved, and 2 accessible repositories contained no source code, leaving 380 repositories characterized; 83.4% were hosted on GitHub, 3.6% on OSF, 3.4% on Zenodo, and 1.6% on GitLab.
  • Repository reproducibility features were heterogeneous: 80.5% had a README, but only 52.1% had a README describing purpose and expected outputs; 42.4% were modular/structured; 37.6% specified dependencies, 21.6% had versioned dependency constraints; 35.3% had licensing information; 3.9% had tests; 8.4% documented hardware requirements; 37.1% included data or sample datasets; among 286 repositories with stochastic components, 64.3% fixed random seeds.
  • Python and R were the most common languages (190 and 189 repositories, respectively); Python use increased over time (OR 1.31 per publication year, 95% CI 1.16–1.49, FDR-adjusted P < 0.001), while R use decreased (OR 0.85, 95% CI 0.75–0.96, FDR-adjusted P = 0.007), with Python overtaking R in 2023.

Methods

  • Scoping review of PubMed-indexed articles citing TRIPOD or TRIPOD+AI as of 11 August 2025: 6,762 TRIPOD-citing and 411 TRIPOD+AI-citing articles were identified; after removing 515 duplicates, 6,658 unique articles remained, of which 1,407 were not retrievable through the PMC Open Access API and 1,284 were excluded by LLM screening as not developing, updating, or validating a multivariable prediction model, leaving 3,967 articles.
  • An LLM-assisted pipeline using GPT-5.2 (2025-12-11) with a structured output schema performed eligibility screening, code-availability statement identification, repository link extraction, and first-author country extraction; validation against 500 randomly selected human-annotated articles yielded a weighted F1 score of 0.97 for selection criteria and 92.3% accuracy for repository link retrieval.
  • Retrieved repositories were compiled into structured text by a custom utility and characterized by an LLM against 14 predefined reproducibility-related features; validation against human labels for all retrieved repositories (n = 35) yielded a weighted F1 score of 0.83. Analyses included descriptive summaries, binary logistic regression for temporal trends, Pearson χ² tests for country and TRIPOD vs TRIPOD+AI comparisons, and Monte Carlo permutations for journal-level comparisons.

Limitations

  • The paper states that the LLM-assisted approach is imperfect despite good validation performance; misclassification could modestly affect point estimates, so estimates should be interpreted as indicators of overall trends rather than exact counts.
  • The paper states that the cohort was restricted to studies citing TRIPOD or TRIPOD+AI and retrievable through PubMed Central Open Access, likely over-representing studies attentive to reporting guidance and possibly overestimating code-sharing prevalence relative to broader prediction model literature; findings may not generalize to work citing other checklists such as STARD or CLAIM.
  • Evident from the text: repository characterization validation was limited to 35 repositories, with lower performance for subjective criteria such as adequacy of documentation (F1 = 0.71); additionally, articles reporting code in supplementary materials or linking to unsupported providers were counted as sharing code without independent verification of accessibility or content.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41591-026-04691-1

Cites

40 of 40 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Modified PRISMA flow diagram. We determined that an article was sharing code ( n = 482) if: it shared code in the appendix ( n = 75), its repository provider was not supported ( n = 27) or its repository was included in the review ( n = 380).
Fig. 1 Modified PRISMA flow diagram. We determined that an article was sharing code ( n = 482) if: it shared code in the appendix ( n = 75), its repository provider was not supported ( n = 27) or its repository was included in the review ( n = 380).CC-BY · From the PDF
Fig. 2 Code sharing over time. Proportion of articles sharing their code by year, for the article cohort ( n = 3,967). Articles that cited only the TRIPOD Statement are shown in blue, and articles that cited only the TRIPOD+AI Statement are shown in orange. Seventeen articles cited both statements, of which three reported their code, and are excluded from this figure.
Fig. 2 Code sharing over time. Proportion of articles sharing their code by year, for the article cohort ( n = 3,967). Articles that cited only the TRIPOD Statement are shown in blue, and articles that cited only the TRIPOD+AI Statement are shown in orange. Seventeen articles cited both statements, of which three reported their code, and are excluded from this figure.CC-BY · From the PDF
Fig. 3 Code sharing by country. Top: proportion of articles sharing their code by country. Bottom: number of articles by country, for the article cohort ( n = 3,967). Countries with more than ten articles in the article cohort are included.
Fig. 3 Code sharing by country. Top: proportion of articles sharing their code by country. Bottom: number of articles by country, for the article cohort ( n = 3,967). Countries with more than ten articles in the article cohort are included.CC-BY · From the PDF
Fig. 4 Code sharing by journal. Proportion of articles reporting their code by journal, for the article cohort ( n = 3,967). Journals with more than ten articles in the cohort are included.
Fig. 4 Code sharing by journal. Proportion of articles reporting their code by journal, for the article cohort ( n = 3,967). Journals with more than ten articles in the cohort are included.CC-BY · From the PDF
Image 5
CC-BY · From the PDF
Image 6
CC-BY · From the PDF
Extended Data Fig. 1 | Repository characteristics. Repository characteristics - quality criteria against percentage of repositories, for the repository cohort (n = 380). README files that provide an overview of the purpose and expected outputs of their code repository are shown as Complete, while those that do not are shown as Incomplete.
Extended Data Fig. 1 | Repository characteristics. Repository characteristics - quality criteria against percentage of repositories, for the repository cohort (n = 380). README files that provide an overview of the purpose and expected outputs of their code repository are shown as Complete, while those that do not are shown as Incomplete.CC-BY · From the PDF
Extended Data Fig. 2 | Distribution of languages used in the repositories. Distribution of the languages identified by the LLM in the set of repositories, for languages with more than one occurrence.
Extended Data Fig. 2 | Distribution of languages used in the repositories. Distribution of the languages identified by the LLM in the set of repositories, for languages with more than one occurrence.CC-BY · From the PDF

Verification

Claims the paper makes

This check didn’t finish.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 40 references

Research works printed but not linked: 2

  1. Bosma JS, et al. Reproducibility of training deep learning models for medical image analysis. In Proc. Machine Learning Research Vol. 227, 1269–1287 (PMLR, 2024).
  2. Sounack, T. et al. Code sharing in prediction model research: a protocol for a scoping review. INPLASY https://inplasy.com/wp-content/uploads/2026/02/INPLASY-Protocol-8832.pdf (2026).

Cited by

See all 2

Metadata

Authors
Sounack, T., Giancotti, R., Gao, C. A., Barreñada, L., Lee, H., Lee, H.-C., Celi, L. A., Moons, K. G. M., Collins, G. S., Lindvall, C., & Pollard, T.
Affiliations
  • Dana-Farber Cancer Institute (US)
  • University of Calabria (IT)
  • Northwestern University (US)
  • KU Leuven (BE)
  • Seoul National University (KR)
  • Harvard University (US)
  • University Medical Center Utrecht (NL)
  • University Hospitals Birmingham NHS Foundation Trust (GB)
Corresponding author
  • Sounack, Thomas
  • Pollard, Tom
Venue
Nature Medicine
Published
2026-10-09
ISSN
  • 1078-8956 (ISSN-L)
  • 1546-170X
OpenAlex
W7221195564
Cited by
2
References
38
Citations by year
2026: 2
FWCI
3.06
Type
journal article
Language
en
Open access
Licence
Locations
  • Nature Medicine (publishedVersion, cc-by)
  • Lirias (KU Leuven) (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • University of Birmingham Research Portal (University of Birmingham) (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Funders
  • American Thoracic Society
  • National Institute for Health and Care Research (MR/Z503873/1)
  • Department of Health and Social Care
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek
  • Korea Health Industry Development Institute (RS-2024-00403047)
  • National Institutes of Health (OT2OD032701, 1u24eb037545-01)
  • Medical Research Council (MR/Z503873/1)
  • Engineering and Physical Sciences Research Council (EP/Y018516/1)
  • Birmingham Biomedical Research Centre (MR/Z503873/1)
  • National Heart, Lung, and Blood Institute (K23HL169815)
  • National Institute of Biomedical Imaging and Bioengineering (R01EB030362)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
  • Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
Keywords
code sharing, clinical prediction model, reproducibility, tripod, reporting guidelines, large language models, software repositories, repository assessment, software documentation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: AI-assisted scoping review of code sharing in clinical prediction model research. https://app.cochat.ai/papers/10.1038/s41591-026-04691-1
The paper
Sounack, T., Giancotti, R., Gao, C. A., Barreñada, L., Lee, H., Lee, H.-C., Celi, L. A., Moons, K. G. M., Collins, G. S., Lindvall, C., & Pollard, T. (2026). AI-assisted scoping review of code sharing in clinical prediction model research. Nature Medicine. https://doi.org/10.1038/s41591-026-04691-1

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.