Code and Data Sharing Practices in the Radiology Artificial Intelligence Literature: A Meta-Research Study
Open accessAbstract
Purpose To evaluate code and data sharing practices in original artificial intelligence (AI) scientific manuscripts published in the Radiological Society of North America (RSNA) journals suite from 2017 through 2021. Materials and Methods A retrospective meta-research study was conducted of articles published in the RSNA journals suite from January 1, 2017, through December 31, 2021. A total of 218 articles were included and evaluated for code sharing practices, reproducibility of shared code, and data sharing practices. Categorical comparisons were conducted using Fisher exact tests with respect to year and journal of publication, author affiliation(s), and type of algorithm used. Results Of the 218 included articles, 73 (34%) shared code, with 24 (33% of code sharing articles and 11% of all articles) sharing reproducible code. Radiology and Radiology: Artificial Intelligence published the most code sharing articles (48 [66%] and 21 [29%], respectively). Twenty-nine articles (13%) shared data, and 12 of these articles (41% of data sharing articles) shared complete experimental data by using only public domain datasets. Four of the 218 articles (2%) shared both code and complete experimental data. Code sharing rates were statistically higher in 2020 and 2021 compared with earlier years (P < .01) and were higher in Radiology and Radiology: Artificial Intelligence compared with other journals (P < .01). Conclusion Original AI scientific articles in the RSNA journals suite had low rates of code and data sharing, emphasizing the need for open-source code and data to achieve transparent and reproducible science. Keywords: Meta-Analysis, AI in Education, Machine Learning Supplemental material is available for this article. © RSNA, 2022
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the current radiological literature, fewer than approximately 6% of all published articles in radiology journals partially or completely share the experimental data used to build AI models.
“In the current radiological literature, there are ongoing difficulties in sharing benchmark datasets, with fewer than approximately 6% of all published articles in radiology journals partially or completely sharing the experimental data used to build the AI models.125”
From Bias in artificial intelligence for medical imaging: fundamentals, detection, avoidance, mitigation, challenges, ethics, and prospects (2024) · cites this paper as [125]
Cites
Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., & Summers, R. M. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.369 · Cited by 4,616 · see citing works
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports
Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-Y., Mark, R. G., & Horng, S. (2019) · Scientific Data · doi:10.1038/s41597-019-0322-0 · Cited by 1,816 · see citing works
Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., & Oermann, E. K. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002683 · Cited by 1,719 · see citing works
Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443 · see citing works
PadChest: A large chest x-ray image dataset with multi-label annotated reports
Bustos, A., Pertusa, A., Salinas, J.-M., & de la Iglesia-Vayá, M. (2020) · Medical Image Analysis · doi:10.1016/j.media.2020.101797 · Cited by 691 · see citing works
Bluemke, D. A., Moy, L., Bredella, M. A., Ertl-Wagner, B. B., Fowler, K. J., Goh, V. J., Halpern, E. F., Hess, C. P., Schiebler, M. L., & Weiss, C. R. (2020) · Radiology · doi:10.1148/radiol.2019192515 · Cited by 343 · see citing works
Reproducibility in machine learning for health research: Still a ways to go
McDermott, M. B. A., Wang, S., Marinsek, N., Ranganath, R., Foschini, L., & Ghassemi, M. (2021) · Science Translational Medicine · doi:10.1126/scitranslmed.abb1655 · Cited by 273 · see citing works
AI-RADS: an artificial intelligence curriculum for residents
Lindqwister, A. L., Hassanpour, S., Lewis, P. J., & Sin, J. M. (2021) · Academic Radiology · doi:10.1016/j.acra.2020.09.017 · Cited by 103 · see citing works
Global Trend in Artificial Intelligence–Based Publications in Radiology From 2000 to 2018
West, E., Mutasa, S., Zhu, Z., & Ha, R. S. (2019) · American Journal of Roentgenology · doi:10.2214/ajr.19.21346 · Cited by 77 · see citing works
Challenge to scientists: does your ten-year-old code still run?
Perkel, J. M. (2020) · Nature · doi:10.1038/d41586-020-02462-7 · Cited by 60 · see citing works
Wiggins, W. F., Caton, M. T., Magudia, K. A., Glomski, S. A., George, E., Rosenthal, M. H., Gaviola, G. C., & Andriole, K. P. (2020) · Radiology Artificial Intelligence · doi:10.1148/ryai.2020200057 · Cited by 55 · see citing works
Magician’s Corner: How to Start Learning about Deep Learning
Erickson, B. J. (2019) · Radiology Artificial Intelligence · doi:10.1148/ryai.2019190072 · Cited by 22 · see citing works
Reproducible Artificial Intelligence Research Requires Open Communication of Complete Source Code
Kitamura, F., Pan, I., & Kline, T. L. (2020) · Radiology Artificial Intelligence · doi:10.1148/ryai.2020200060 · Cited by 16 · see citing works
AMERICAN ASSOCIATION OF PHYSICISTS IN MEDICINE
(chairman, G. S. L., Brezovich, I., Fessenden, P., et al. (2012) · Medical Physics · doi:10.1002/j.2473-4209.2012.tb00183.x · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Koçak, B., Ponsiglione, A., Stanzione, A., Bluethgen, C., Santinha, J., Ugga, L., Huisman, M., Klontzas, M. E., Cannella, R., & Cuocolo, R. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.242854 · Cited by 168 · see citing works
“125 Finally, data labeling for model training can be affected by the human image interpretation and diagnostic performance of the selected reference standard for the investigated condition.”
Towards reproducible radiomics research: introduction of a database for radiomics studies
Akinci D’Antonoli, T., Cuocolo, R., Baessler, B., & Pinto dos Santos, D. (2023) · European Radiology · doi:10.1007/s00330-023-10095-3 · Cited by 49 · see citing works
- Authors
- Venkatesh, K., Santomartino, S. M., Sulam, J., & Yi, P. H.
- Affiliations
- University of Maryland, Baltimore (US)
- Venue
- Radiology: Artificial Intelligence
- Published
- 2022-09-01
- Volume
- 4
- Issue
- 5
- Pages
- e220081-e220081
- ISSN
- 2638-6100 (ISSN-L)
- PMC
- PMC9530751
- OpenAlex
- W4292088819
- Cited by
- 31
- References
- 14
- Citations by year
- 2026: 4 · 2025: 10 · 2024: 11 · 2023: 6
- FWCI
- 6.68
- Type
- journal article
- Language
- en
- Open access
- green
- www.ncbi.nlm.nih.gov
- Locations
- Radiology Artificial Intelligence (publishedVersion, public-domain)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- code sharing, data sharing, reproducibility, artificial intelligence in radiology, open science, research transparency, open-source code, machine learning
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
