Towards reproducible radiomics research: introduction of a database for radiomics studies
Open accessAbstract
Objectives: To investigate the model-, code-, and data-sharing practices in the current radiomics research landscape and to introduce a radiomics research database. Methods: A total of 1254 articles published between January 1, 2021, and December 31, 2022, in leading radiology journals (European Radiology, European Journal of Radiology, Radiology, Radiology: Artificial Intelligence, Radiology: Cardiothoracic Imaging, Radiology: Imaging Cancer) were retrospectively screened, and 257 original research articles were included in this study. The categorical variables were compared using Fisher’s exact tests or chi-square test and numerical variables using Student’s t test with relation to the year of publication. Results: Half of the articles (128 of 257) shared the model by either including the final model formula or reporting the coefficients of selected radiomics features. A total of 73 (28%) models were validated on an external independent dataset. Only 16 (6%) articles shared the data or used publicly available open datasets. Similarly, only 20 (7%) of the articles shared the code. A total of 7 (3%) articles both shared code and data. All collected data in this study is presented in a radiomics research database (RadBase) and could be accessed at https://github.com/EuSoMII/RadBase. Conclusion: According to the results of this study, the majority of published radiomics models were not technically reproducible since they shared neither model nor code and data. There is still room for improvement in carrying out reproducible and open research in the field of radiomics. Clinical relevance statement: To date, the reproducibility of radiomics research and open science practices within the radiomics research community are still very low. Ensuring reproducible radiomics research with model-, code-, and data-sharing practices will facilitate faster clinical translation. Key Points: • There is a discrepancy between the number of published radiomics papers and the clinical implementation of these published radiomics models. • The main obstacle to clinical implementation is the lack of model-, code-, and data-sharing practices. • In order to translate radiomics research into clinical practice, the radiomics research community should adopt open science practices.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 1× · see citing works
“This trend resembles the reproducibility crisis in the field of psychology, where independent researchers attempted to replicate results of previously published prominent articles, but succeeded only 30% of the time [5].”
ResultRadiomics: Images Are More than Pictures, They Are Data
Gillies, R. J., Kinahan, P. E., & Hricak, H. (2016) · Radiology · doi:10.1148/radiol.2015151169 · Cited by 8,421 · see citing works
Radiomics: extracting more information from medical images using advanced feature analysis
Lambin, P., Rios-Velazquez, E., Leijenaar, R., Carvalho, S., van Stiphout, R. G. P. M., Granton, P., Zegers, C. M. L., Gillies, R., Boellard, R., Dekker, A., et al. (2012) · European Journal of Cancer · doi:10.1016/j.ejca.2011.11.036 · Cited by 6,084 · see citing works
Promoting an open research culture
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., et al. (2015) · Science · doi:10.1126/science.aab2374 · Cited by 2,708 · see citing works
Bias in error estimation when using cross-validation for model selection
Varma, S., & Simon, R. (2006) · BMC Bioinformatics · doi:10.1186/1471-2105-7-91 · Cited by 2,485 · cited 1× · see citing works
“If cross-validation is to be used alone, the entire pipeline must be nested, or a test set should be left out of the pipeline to obtain unbiased results; otherwise, the model will be overfitted and cannot be generalized [15, 16].”
MethodChecklist for Artificial Intelligence in Medical Imaging (CLAIM): a guide for authors and reviewers
Mongan, J., Moy, L., & Kahn, C. E. (2020) · Radiology: Artificial Intelligence · doi:10.1148/ryai.2020200029 · Cited by 1,443 · cited 1× · see citing works
“…checklists such as CheckList for EvaluAtion of Radiomics research (CLEAR) [8], the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) [9], or Must AI Criteria 10 Checklist (MAIC-10) [10], code and data sharing in radiomics and artificial intelligence (AI) research is increasingly…”
MethodWhat does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · cited 1× · see citing works
“In the absence of a universally accepted definition of reproducibility, in this paper, we accept the earlier description of “research reproducibility” [13, 14].”
MethodKocak, B., Baessler, B., Bakas, S., Cuocolo, R., Fedorov, A., Maier-Hein, L., Mercaldo, N., Müller, H., Orlhac, F., Pinto dos Santos, D., et al. (2023) · Insights into Imaging · doi:10.1186/s13244-023-01415-8 · Cited by 471 · see citing works
Self-citationChallenges to the Reproducibility of Machine Learning Models in Health Care
Beam, A. L., Manrai, A. K., & Ghassemi, M. (2020) · JAMA · doi:10.1001/jama.2019.20866 · Cited by 366 · see citing works
Reproducibility standards for machine learning in the life sciences
Heil, B. J., Hoffman, M. M., Markowetz, F. M., Lee, S., Greene, C. S., & Hicks, S. C. (2021) · Nature Methods · doi:10.1038/s41592-021-01256-7 · Cited by 261 · see citing works
Liu, Y., Kim, J., Qu, F., Liu, S., Wang, H. X., Balagurunathan, Y., Ye, Z., & Gillies, R. J. (2016) · Radiology · doi:10.1148/radiol.2016151455 · Cited by 232 · see citing works
Pineau, J., Vincent-Lamarre, P., Sinha, K., Larivière, V., Beygelzimer, A., d'Alché-Buc, F., Fox, E., & Larochelle, H. (2020) · arXiv · doi:10.48550/arxiv.2003.12206 · Cited by 232 · see citing works
Flanders, A. E., Prevedello, L. M. S., Shih, G., Halabi, S. S., Kalpathy–Cramer, J., Ball, R. L., Mongan, J., Stein, A., Kitamura, F., Lungren, M. P., et al. (2020) · Radiology Artificial Intelligence · doi:10.1148/ryai.2020190211 · Cited by 217 · see citing works
On the Dangers of Cross-Validation. An Experimental Evaluation
Rao, R. B., Fung, G., & Rosales, R. (2008) · SIAM International Conference on Data Mining · doi:10.1137/1.9781611972788.54 · Cited by 207 · see citing works
A decade of radiomics research: are images really data or just patterns in the noise?
dos Santos, D. P., Dietzel, M., & Baeßler, B. (2020) · European Radiology · doi:10.1007/s00330-020-07108-w · Cited by 129 · see citing works
Sustainable computational science: the ReScience initiative
Rougier, N. P., Hinsen, K., Alexandre, F., Arildsen, T., Barba, L. A., Benureau, F., Brown, C. T., de Buyl, P., Çağlayan, O., Davison, A. P., et al. (2017) · PeerJ Computer Science · doi:10.7717/peerj-cs.142 · Cited by 123 · see citing works
van Baardwijk, A. A. W., Reymen, B. J. T., Wanders, S. L., Borger, J. H., Öllers, M., Dingemans, A. C., Bootsma, G. P., Geraedts, W. H., Pitz, C. C. M., Lunde, R., et al. (2012) · European Journal of Cancer · doi:10.1016/j.ejca.2012.04.014 · Cited by 74 · see citing works
Challenge to scientists: does your ten-year-old code still run?
Perkel, J. M. (2020) · Nature · doi:10.1038/d41586-020-02462-7 · Cited by 60 · see citing works
Open science, the replication crisis, and environmental public health
Hicks, D. (2021) · Accountability in Research · doi:10.1080/08989621.2021.1962713 · Cited by 56 · see citing works
MAIC–10 brief quality checklist for publications using artificial intelligence and medical images
Cerdá-Alberich, L., Solana, J., Mallol, P., Ribas, G., García-Junco, M., Alberich‐Bayarri, Á., & Martí‐Bonmatí, L. (2023) · Insights into Imaging · doi:10.1186/s13244-022-01355-9 · Cited by 55 · see citing works
Flanders, A. E., Prevedello, L. M. S., Shih, G., et al. (2020) · Radiology Artificial Intelligence · doi:10.1148/ryai.2020209002 · Cited by 54 · see citing works
Murphy, Z. R., Venkatesh, K., Sulam, J., & Yi, P. H. (2022) · Radiology Artificial Intelligence · doi:10.1148/ryai.220012 · Cited by 46 · see citing works
To share or not to share? Expected pros and cons of data sharing in radiological research
Sardanelli, F., Alì, M., Hunink, M. G. M., Houssami, N., Sconfienza, L. M., & Leo, G. D. (2018) · European Radiology · doi:10.1007/s00330-017-5165-5 · Cited by 45 · see citing works
Why scientists are turning to Rust
Perkel, J. M. (2020) · Nature · doi:10.1038/d41586-020-03382-2 · Cited by 41 · see citing works
Reproducibility in Machine Learning for HealthPreprint
McDermott, M. B. A., Wang, S., Marinsek, N., Ranganath, R., Ghassemi, M., & Foschini, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1907.01463 · Cited by 34 · see citing works
Venkatesh, K., Santomartino, S. M., Sulam, J., & Yi, P. H. (2022) · Radiology: Artificial Intelligence · doi:10.1148/ryai.220081 · Cited by 31 · see citing works
Lin, Z., Tang, B., Cai, J., et al. (2021) · European Journal of Radiology · doi:10.1016/j.ejrad.2021.109693 · Cited by 29 · see citing works
Tsai, P., Lin, D.-C., Jan, Y., Liu, Y., Wu, T.-H., & Huang, S.-C. (2022) · European Radiology · doi:10.1007/s00330-022-09356-4 · Cited by 25 · see citing works
Lv, X., Chen, M., Kong, C., et al. (2021) · European Journal of Radiology · doi:10.1016/j.ejrad.2021.109955 · Cited by 21 · see citing works
Reproducible Artificial Intelligence Research Requires Open Communication of Complete Source Code
Kitamura, F., Pan, I., & Kline, T. L. (2020) · Radiology Artificial Intelligence · doi:10.1148/ryai.2020200060 · Cited by 16 · see citing works
Yu, J., Lin, S., Lü, H., et al. (2021) · European Journal of Radiology · doi:10.1016/j.ejrad.2021.110139 · Cited by 13 · see citing works
Li, X., Xu, N., Meng, X., et al. (2021) · European Journal of Radiology · doi:10.1016/j.ejrad.2021.110137 · Cited by 5 · see citing works
Lin, F.-Y., Chang, Y., Huang, H.-Y., et al. (2022) · European Radiology · doi:10.1007/s00330-021-08456-x · Cited by 5 · see citing works
dos Santos, D. P., & Santos, J. M. G. (2020) · European Radiology · doi:10.1007/s00330-020-07603-0 · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Koçak, B., Ponsiglione, A., Stanzione, A., Bluethgen, C., Santinha, J., Ugga, L., Huisman, M., Klontzas, M. E., Cannella, R., & Cuocolo, R. (2024) · Diagnostic and Interventional Radiology · doi:10.4274/dir.2024.242854 · Cited by 168 · see citing works
“50 Once the code has been scrutinized for potential bias, comprehensive testing should be initiated.”
Self-citation
- Authors
- Akinci D’Antonoli, T., Cuocolo, R., Baessler, B., & Pinto dos Santos, D.
- Affiliations
- University of Salerno (IT)
- Universitätsklinikum Würzburg (DE)
- Venue
- European Radiology
- Published
- 2023-08-12
- Volume
- 34
- Issue
- 1
- Pages
- 436-443
- ISSN
- 0938-7994 (ISSN-L)
- 1432-1084
- PMC
- PMC10791815
- OpenAlex
- W4385782881
- Cited by
- 49
- References
- 34
- Citations by year
- 2026: 8 · 2025: 16 · 2024: 21 · 2023: 4
- FWCI
- 8.08
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- European Radiology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Online Publication Service of Würzburg University (Würzburg University) (submittedVersion, cc-by)
- PUBLISSO (German National Library of Medicine) (submittedVersion, cc-by)
- Funders
- University of Basel
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- radiomics, reproducibility, open science, model sharing, code sharing, data sharing, research databases, external validation, clinical translation
- MeSH terms
- Radiomics, Artificial Intelligence, Humans, Radiography, Retrospective Studies, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
