The DOI matches its Crossref record; 78 of the 78 references it cites were found in a registry or an index.
Repeatability and Reproducibility of Radiomic Features: A Systematic Review
Open accessAbstract
PURPOSE: An ever-growing number of predictive models used to inform clinical decision making have included quantitative, computer-extracted imaging biomarkers, or "radiomic features." Broadly generalizable validity of radiomics-assisted models may be impeded by concerns about reproducibility. We offer a qualitative synthesis of 41 studies that specifically investigated the repeatability and reproducibility of radiomic features, derived from a systematic review of published peer-reviewed literature. METHODS AND MATERIALS: The PubMed electronic database was searched using combinations of the broad Haynes and Ingui filters along with a set of text words specific to cancer, radiomics (including texture analyses), reproducibility, and repeatability. This review has been reported in compliance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. From each full-text article, information was extracted regarding cancer type, class of radiomic feature examined, reporting quality of key processing steps, and statistical metric used to segregate stable features. RESULTS: Among 624 unique records, 41 full-text articles were subjected to review. The studies primarily addressed non-small cell lung cancer and oropharyngeal cancer. Only 7 studies addressed in detail every methodologic aspect related to image acquisition, preprocessing, and feature extraction. The repeatability and reproducibility of radiomic features are sensitive at various degrees to processing details such as image acquisition settings, image reconstruction algorithm, digital image preprocessing, and software used to extract radiomic features. First-order features were overall more reproducible than shape metrics and textural features. Entropy was consistently reported as one of the most stable first-order features. There was no emergent consensus regarding either shape metrics or textural features; however, coarseness and contrast appeared among the least reproducible. CONCLUSIONS: Investigations of feature repeatability and reproducibility are currently limited to a small number of cancer types. Reporting quality could be improved regarding details of feature extraction software, digital image manipulation (preprocessing), and the cutoff value used to distinguish stable features.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1)
Eisenhauer, E. A., Therasse, P., Bogaerts, J., Schwartz, L. H., Sargent, D., Ford, R., Dancey, J., Arbuck, S., Gwyther, S., Mooney, M., et al. (2009) · European Journal of Cancer · doi:10.1016/j.ejca.2008.10.026 · Cited by 29,534
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
New Guidelines to Evaluate the Response to Treatment in Solid Tumors
Therasse, P., Arbuck, S. G., Eisenhauer, E. A., Wanders, J., Kaplan, R. S., Rubinstein, L., Verweij, J., Van Glabbeke, M., van Oosterom, A. T., Christian, M. C., et al. (2000) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/92.3.205 · Cited by 15,660
A concordance correlation coefficient to evaluate reproducibility.
Lin, L. I.-K. (1989) · Biometrics · doi:10.2307/2532051 · Cited by 8,785 · cited 1×
“The CCC metric (70) assumed each observation was independent and was commonly reported in both repeatability and reproducibility studies.”
BackgroundRadiomics: extracting more information from medical images using advanced feature analysis
Lambin, P., Rios-Velazquez, E., Leijenaar, R., Carvalho, S., van Stiphout, R. G. P. M., Granton, P., Zegers, C. M. L., Gillies, R., Boellard, R., Dekker, A., et al. (2012) · European Journal of Cancer · doi:10.1016/j.ejca.2011.11.036 · Cited by 6,084
Self-citationDecoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach
Aerts, H. J. W. L., Velazquez, E. R., Leijenaar, R. T. H., Parmar, C., Grossmann, P., Carvalho, S., Bussink, J., Monshouwer, R., Haibe-Kains, B., Rietveld, D., et al. (2014) · Nature Communications · doi:10.1038/ncomms5006 · Cited by 5,132
Self-citationQUANTIFYING TEST-RETEST RELIABILITY USING THE INTRACLASS CORRELATION COEFFICIENT AND THE SEM
WEIR, J. P. (2005) · Journal of Strength and Conditioning Research · doi:10.1519/00124278-200502000-00038 · Cited by 4,842
Cited by
Zwanenburg, A., Vallières, M., Abdalah, M. A., Aerts, H. J. W. L., Andrearczyk, V., Apte, A., Ashrafinia, S., Bakas, S., Beukinga, R. J., Boellaard, R., et al. (2020) · Radiology · doi:10.1148/radiol.2020191145 · Cited by 4,177
“Because of this variability, features that can be reproduced from the same image using standardized radiomics software may nevertheless lack reproducibility in multicentric or multiscanner settings (14,19,33).”
BackgroundSelf-citationPredicting cancer outcomes with radiomics and artificial intelligence in radiology
Bera, K., Braman, N., Gupta, A., Velcheti, V., & Madabhushi, A. (2022) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-021-00560-7 · Cited by 819
Solving inverse problems using data-driven models
Arridge, S., Maass, P., Öktem, O., & Schönlieb, C.-B. (2019) · Acta Numerica · doi:10.1017/s0962492919000059 · Cited by 686
Kocak, B., Baessler, B., Bakas, S., Cuocolo, R., Fedorov, A., Maier-Hein, L., Mercaldo, N., Müller, H., Orlhac, F., Pinto dos Santos, D., et al. (2023) · Insights into Imaging · doi:10.1186/s13244-023-01415-8 · Cited by 471
Park, J. E., Park, S. Y., Kim, H. J., & Kim, H. S. (2019) · Korean Journal of Radiology · doi:10.3348/kjr.2018.0070 · Cited by 388
Radiomics in breast cancer classification and prediction
Conti, A., Duggento, A., Indovina, I., Guerrisi, M., & Toschi, N. (2021) · Seminars in Cancer Biology · doi:10.1016/j.semcancer.2020.04.002 · Cited by 386
Radiogenomics: bridging imaging and genomics
Bodalal, Z., Trebeschi, S., Nguyen-Kim, T. D. L., Schats, W., & Beets-Tan, R. (2019) · Abdominal Radiology · doi:10.1007/s00261-019-02028-w · Cited by 310
Zwanenburg, A. (2019) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-019-04391-8 · Cited by 306
Chetan, M. R., & Gleeson, F. V. (2021) · European Radiology · doi:10.1007/s00330-020-07141-9 · Cited by 265
Papadimitroulas, P., Brocki, L., Christopher Chung, N., Marchadour, W., Vermet, F., Gaubert, L., Eleftheriadis, V., Plachouris, D., Visvikis, D., Kagadis, G. C., et al. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.03.009 · Cited by 226
Understanding Sources of Variation to Improve the Reproducibility of Radiomics
Zhao, B. (2021) · Frontiers in Oncology · doi:10.3389/fonc.2021.633176 · Cited by 158
Radiomics feature reproducibility under inter-rater variability in segmentations of CT images
Haarburger, C., Müller-Franzes, G., Weninger, L., Kuhl, C., Truhn, D., & Merhof, D. (2020) · Scientific Reports · doi:10.1038/s41598-020-69534-6 · Cited by 139
Kaissis, G., Ziegelmayer, S., Lohöfer, F., Algül, H., Eiber, M., Weichert, W., Schmid, R., Friess, H., Rummeny, E., Ankerst, D., et al. (2019) · European Radiology Experimental · doi:10.1186/s41747-019-0119-0 · Cited by 79
Kaissis, G., Ziegelmayer, S., Lohöfer, F., Steiger, K., Algül, H., Muckenhuber, A., Yen, H.-Y., Rummeny, E., Friess, H., Schmid, R., et al. (2019) · PLOS ONE · doi:10.1371/journal.pone.0218642 · Cited by 76
Preuss, K., Thach, N., Liang, X., Baine, M., Chen, J., Zhang, C., Du, H., Yu, H., Lin, C., Hollingsworth, M. A., et al. (2022) · Cancers · doi:10.3390/cancers14071654 · Cited by 69
“Thus far, studies have been conducted to assess these sources of uncertainties and their impacts and to develop mitigation strategies, although more such studies are needed [91,127].”
BackgroundOncologic Imaging and Radiomics: A Walkthrough Review of Methodological Challenges
Stanzione, A., Cuocolo, R., Ugga, L., Verde, F., Romeo, V., Brunetti, A., & Maurea, S. (2022) · Cancers · doi:10.3390/cancers14194871 · Cited by 64
Zhang, Y., Lobo-Mueller, E. M., Karanicolas, P., Gallinger, S., Haider, M. A., & Khalvati, F. (2021) · Scientific Reports · doi:10.1038/s41598-021-80998-y · Cited by 62
Fedorov, A., Longabaugh, W. J. R., Pot, D., Clunie, D. A., Pieper, S. D., Gibbs, D. L., Bridge, C., Herrmann, M. D., Homeyer, A., Lewis, R., et al. (2023) · Radiographics · doi:10.1148/rg.230180 · Cited by 35
Gregory, J., Welliver, S., & Chong, J. (2020) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.27035 · Cited by 27
Minimising multi-centre radiomics variability through image normalisation: a pilot study
Campello, V. M., Martín-Isla, C., Izquierdo, C., Guala, A., Palomares, J. F. R., Viladés, D., Descalzo, M. L., Karakas, M., Çavuş, E., Raisi-Estabragh, Z., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-16375-0 · Cited by 24
Metadata
- Authors
- Traverso, A., Wee, L., Dekker, A., & Gillies, R.
- Venue
- International Journal of Radiation Oncology*Biology*Physics
- Publisher
- Elsevier BV
- Published
- 2018-11
- Volume
- 102
- Issue
- 4
- Pages
- 1143-1158
- ISSN
- 0360-3016 (ISSN-L)
- OpenAlex
- W2806986802
- Cited by
- 903
- References
- 78
- FWCI
- 16.72
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.