AI-assisted scoping review of code sharing in clinical prediction model research
Open accessThe DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Abstract
Clinical prediction models are increasingly deployed to support diagnostic and prognostic decisions, making reproducibility essential for assessing their reliability and generalizability. Analytical code supports independent assessment of these processes, yet its availability in the literature remains limited. This scoping review quantifies current practices in sharing analytical code to inform the development of TRIPOD-Code, a reporting guideline for code availability and reproducibility. Here a large-language-model-assisted pipeline was developed to screen articles citing TRIPOD or TRIPOD+AI, extract repository links and assess retrieved repositories against 14 predefined reproducibility-related features. Among 3,967 articles, 482 (12.2%) included code-sharing statements. Sharing prevalence varied widely by journal and country. Repository assessment showed substantial heterogeneity in reproducibility features. These findings underscore the need for clearer expectations beyond code availability, including documentation, dependency specification and executable structure. Strengthening these practices may improve the usability of clinical prediction model studies and support their deployment in real-world clinical settings.
Verification finished with gaps
Some checks didn’t finish. The results below cover the steps that did.
- Registry recordVerified
- Full textRead the full text
- Reference list40 references found
- Reference check40 of 40 references checked
- StatisticsRecomputed
- ClaimsDidn’t finish — the rest of the report still applies.
- Cited sourcesDidn’t finish — the rest of the report still applies.
- Self-checkChecked
- VerificationFindings verified
Overview
CoChat Summary
TL;DR
This AI-assisted scoping review found that among 3,967 clinical prediction model articles citing TRIPOD or TRIPOD+AI, only 12.2% shared code, and shared repositories often lacked documentation, dependency specification, licensing, and other reproducibility features. The authors conclude that code availability alone is insufficient for computational reproducibility and use these findings to inform the TRIPOD-Code reporting guideline.
Key findings
- Among 3,967 included articles, 482 (12.2%) included code-sharing statements; code sharing increased over time (OR 1.12 per publication year, 95% CI 1.07–1.18, P < 0.001), rising from 6.3% in 2015 to 14.5% in 2024 and 15.8% in 2025.
- Of 447 repository links, 27 pointed to unsupported providers, 38 could not be resolved, and 2 accessible repositories contained no source code, leaving 380 repositories characterized; 83.4% were hosted on GitHub, 3.6% on OSF, 3.4% on Zenodo, and 1.6% on GitLab.
- Repository reproducibility features were heterogeneous: 80.5% had a README, but only 52.1% had a README describing purpose and expected outputs; 42.4% were modular/structured; 37.6% specified dependencies, 21.6% had versioned dependency constraints; 35.3% had licensing information; 3.9% had tests; 8.4% documented hardware requirements; 37.1% included data or sample datasets; among 286 repositories with stochastic components, 64.3% fixed random seeds.
- Python and R were the most common languages (190 and 189 repositories, respectively); Python use increased over time (OR 1.31 per publication year, 95% CI 1.16–1.49, FDR-adjusted P < 0.001), while R use decreased (OR 0.85, 95% CI 0.75–0.96, FDR-adjusted P = 0.007), with Python overtaking R in 2023.
Methods
- Scoping review of PubMed-indexed articles citing TRIPOD or TRIPOD+AI as of 11 August 2025: 6,762 TRIPOD-citing and 411 TRIPOD+AI-citing articles were identified; after removing 515 duplicates, 6,658 unique articles remained, of which 1,407 were not retrievable through the PMC Open Access API and 1,284 were excluded by LLM screening as not developing, updating, or validating a multivariable prediction model, leaving 3,967 articles.
- An LLM-assisted pipeline using GPT-5.2 (2025-12-11) with a structured output schema performed eligibility screening, code-availability statement identification, repository link extraction, and first-author country extraction; validation against 500 randomly selected human-annotated articles yielded a weighted F1 score of 0.97 for selection criteria and 92.3% accuracy for repository link retrieval.
- Retrieved repositories were compiled into structured text by a custom utility and characterized by an LLM against 14 predefined reproducibility-related features; validation against human labels for all retrieved repositories (n = 35) yielded a weighted F1 score of 0.83. Analyses included descriptive summaries, binary logistic regression for temporal trends, Pearson χ² tests for country and TRIPOD vs TRIPOD+AI comparisons, and Monte Carlo permutations for journal-level comparisons.
Limitations
- The paper states that the LLM-assisted approach is imperfect despite good validation performance; misclassification could modestly affect point estimates, so estimates should be interpreted as indicators of overall trends rather than exact counts.
- The paper states that the cohort was restricted to studies citing TRIPOD or TRIPOD+AI and retrievable through PubMed Central Open Access, likely over-representing studies attentive to reporting guidance and possibly overestimating code-sharing prevalence relative to broader prediction model literature; findings may not generalize to work citing other checklists such as STARD or CLAIM.
- Evident from the text: repository characterization validation was limited to 35 repositories, with lower performance for subjective criteria such as adequacy of documentation (F1 = 0.71); additionally, articles reporting code in supplementary materials or linking to unsupported providers were counted as sharing code without independent verification of accessibility or content.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
This check didn’t finish.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ AI-assisted scoping review of code sharing in clinical prediction model research > Nature Medicine
“two independent reviewers with relevant expertise in prediction model research, clinical AI, and computational methods, including the development and evaluation of analytical code (T.S., R. G.) annotated all retrieved repositories (n = 35), according to the same predefined schema.”
§ AI-assisted scoping review of code sharing in clinical prediction model research > Repository characteristics
“Two accessible repositories contained no source code, leaving a total of 380 repositories to be characterized.”
Citations
Cites
PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation
Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., et al. (2018) · Annals of Internal Medicine · doi:10.7326/m18-0850 · Cited by 43,565
The FAIR Guiding Principles for scientific data management and stewardship
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., da Silva Santos, L. B., Bourne, P. E., et al. (2016) · Scientific Data · doi:10.1038/sdata.2016.18 · Cited by 18,423
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationCollins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718
Self-citationSTARD 2015: an updated list of essential items for reporting diagnostic accuracy studies
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2015) · BMJ · doi:10.1136/bmj.h5527 · Cited by 3,660
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Self-citationWong, A., Otles, E., Donnelly, J. P., Krumm, A., McCullough, J., DeTroyer-Cooley, O., Pestrue, J., Phillips, M., Konye, J., Penoza, C., et al. (2021) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2021.2626 · Cited by 995
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): 2024 Update
Tejani, A. S., Klontzas, M. E., Gatti, A. A., Mongan, J. T., Moy, L., Park, S. H., Kahn, C. E., Panel, F. T. C. 2. U., Abbara, S., Afat, S., et al. (2024) · Radiology: Artificial Intelligence · doi:10.1148/ryai.240300 · Cited by 986
Seyyed-Kalantari, L., Zhang, H., McDermott, M. B. A., Chen, I. Y., & Ghassemi, M. (2021) · Nature Medicine · doi:10.1038/s41591-021-01595-0 · Cited by 928
Panic, N., Leoncini, E., de Belvis, G., Ricciardi, W., & Boccia, S. (2013) · PLoS ONE · doi:10.1371/journal.pone.0083138 · Cited by 883
Research works printed but not linked: 2
- Bosma JS, et al. Reproducibility of training deep learning models for medical image analysis. In Proc. Machine Learning Research Vol. 227, 1269–1287 (PMLR, 2024).
- Sounack, T. et al. Code sharing in prediction model research: a protocol for a scoping review. INPLASY https://inplasy.com/wp-content/uploads/2026/02/INPLASY-Protocol-8832.pdf (2026).
Cited by
The complexities of patient-centred conversational artificial intelligencePreprint
de Matos, J. C. R. G., Buege, O., Cheung, D., Collins, G. S., Dhiman, P., Li, N., Mao, B., Nelson, B. W., Ouroutzoglou, M., Varghese, P., et al. (2026) · arXiv (Cornell University) · doi:10.48550/arxiv.2607.08625
Self-citationA systematic review of clinical translation of large language models using electronic health recordsPreprint
Zhu, T., Miresmaeili, M., Tomlinson, C., Qian, L., Teo, J., Ibrahim, Z., & Dobson, R. (2026) · Research Square · doi:10.21203/rs.3.rs-11651581/v1
Metadata
- Authors
- Sounack, T., Giancotti, R., Gao, C. A., Barreñada, L., Lee, H., Lee, H.-C., Celi, L. A., Moons, K. G. M., Collins, G. S., Lindvall, C., & Pollard, T.
- Affiliations
- Dana-Farber Cancer Institute (US)
- University of Calabria (IT)
- Northwestern University (US)
- KU Leuven (BE)
- Seoul National University (KR)
- Harvard University (US)
- University Medical Center Utrecht (NL)
- University Hospitals Birmingham NHS Foundation Trust (GB)
- Corresponding author
- Sounack, Thomas
- Pollard, Tom
- ORCID
- Venue
- Nature Medicine
- Published
- 2026-10-09
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- OpenAlex
- W7221195564
- Cited by
- 2
- References
- 38
- Citations by year
- 2026: 2
- FWCI
- 3.06
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Nature Medicine (publishedVersion, cc-by)
- Lirias (KU Leuven) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- University of Birmingham Research Portal (University of Birmingham) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- American Thoracic Society
- National Institute for Health and Care Research (MR/Z503873/1)
- Department of Health and Social Care
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek
- Korea Health Industry Development Institute (RS-2024-00403047)
- National Institutes of Health (OT2OD032701, 1u24eb037545-01)
- Medical Research Council (MR/Z503873/1)
- Engineering and Physical Sciences Research Council (EP/Y018516/1)
- Birmingham Biomedical Research Centre (MR/Z503873/1)
- National Heart, Lung, and Blood Institute (K23HL169815)
- National Institute of Biomedical Imaging and Bioengineering (R01EB030362)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- code sharing, clinical prediction model, reproducibility, tripod, reporting guidelines, large language models, software repositories, repository assessment, software documentation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: AI-assisted scoping review of code sharing in clinical prediction model research. https://app.cochat.ai/papers/10.1038/s41591-026-04691-1
- The paper
- Sounack, T., Giancotti, R., Gao, C. A., Barreñada, L., Lee, H., Lee, H.-C., Celi, L. A., Moons, K. G. M., Collins, G. S., Lindvall, C., & Pollard, T. (2026). AI-assisted scoping review of code sharing in clinical prediction model research. Nature Medicine. https://doi.org/10.1038/s41591-026-04691-1
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.