Building reliable radiomic models using image perturbation
Open accessAbstract
Radiomic model reliability is a central premise for its clinical translation. Presently, it is assessed using test–retest or external data, which, unfortunately, is often scarce in reality. Therefore, we aimed to develop a novel image perturbation-based method (IPBM) for the first of its kind toward building a reliable radiomic model. We first developed a radiomic prognostic model for head-and-neck cancer patients on a training (70%) and evaluated on a testing (30%) cohort using C-index. Subsequently, we applied the IPBM to CT images of both cohorts (Perturbed-Train and Perturbed-Test cohort) to generate 60 additional samples for both cohorts. Model reliability was assessed using intra-class correlation coefficient (ICC) to quantify consistency of the C-index among the 60 samples in the Perturbed-Train and Perturbed-Test cohorts. Besides, we re-trained the radiomic model using reliable RFs exclusively (ICC > 0.75) to validate the IPBM. Results showed moderate model reliability in Perturbed-Train (ICC: 0.565, 95%CI 0.518–0.615) and Perturbed-Test (ICC: 0.596, 95%CI 0.527–0.670) cohorts. An enhanced reliability of the re-trained model was observed in Perturbed-Train (ICC: 0.782, 95%CI 0.759–0.815) and Perturbed-Test (ICC: 0.825, 95%CI 0.782–0.867) cohorts, indicating validity of the IPBM. To conclude, we demonstrated capability of the IPBM toward building reliable radiomic models, providing community with a novel model reliability assessment strategy prior to prospective evaluation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Some studies reported that use of pre-processing image filters improved predictive performance of models.
“Previously, some studies reported the use of pre-processing image filters was shown to improve predictive performance of models [47, 48].”
From Radiomics-based prediction of FIGO grade for placenta accreta spectrum (2023) · cites this paper as [48]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 29,639 · see citing works
Radiomics: Images Are More than Pictures, They Are Data
Gillies, R. J., Kinahan, P. E., & Hricak, H. (2016) · Radiology · doi:10.1148/radiol.2015151169 · Cited by 8,421 · see citing works
Computational Radiomics System to Decode the Radiographic Phenotype
(2017) · doi:10.1158/0008-5472.can-17-0339 · Cited by 6,593 · see citing works
Radiomics: the bridge between medical imaging and personalized medicine
Lambin, P., Leijenaar, R. T. H., Deist, T. M., Peerlings, J., de Jong, E. E. C., van Timmeren, J., Sanduleanu, S., Larue, R. T. H. M., Even, A. J. G., Jochems, A., et al. (2017) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2017.141 · Cited by 5,688 · see citing works
Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach
Aerts, H. J. W. L., Velazquez, E. R., Leijenaar, R. T. H., Parmar, C., Grossmann, P., Carvalho, S., Bussink, J., Monshouwer, R., Haibe-Kains, B., Rietveld, D., et al. (2014) · Nature Communications · doi:10.1038/ncomms5006 · Cited by 5,132 · see citing works
The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository
Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., et al. (2013) · Journal of Digital Imaging · doi:10.1007/s10278-013-9622-7 · Cited by 4,599 · see citing works
Zwanenburg, A., Vallières, M., Abdalah, M. A., Aerts, H. J. W. L., Andrearczyk, V., Apte, A., Ashrafinia, S., Bakas, S., Beukinga, R. J., Boellaard, R., et al. (2020) · Radiology · doi:10.1148/radiol.2020191145 · Cited by 4,177 · see citing works
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. (2015) · BMC Medicine · doi:10.1186/s12916-014-0241-z · Cited by 1,927 · see citing works
Radiomics: the facts and the challenges of image analysis
Rizzo, S., Botta, F., Raimondi, S., Origgi, D., Fanciullo, C., MORGANTI, A. G., & Bellomi, M. (2018) · European Radiology Experimental · doi:10.1186/s41747-018-0068-z · Cited by 1,048 · see citing works
Machine Learning methods for Quantitative Radiomic Biomarkers
Parmar, C., Großmann, P., Bussink, J., Lambin, P., & Aerts, H. J. W. L. (2015) · Scientific Reports · doi:10.1038/srep13087 · Cited by 1,024 · see citing works
CT-based radiomic signature predicts distant metastasis in lung adenocarcinoma
Coroller, T. P., Grossmann, P., Hou, Y., Rios Velazquez, E., Leijenaar, R. T. H., Hermann, G., Lambin, P., Haibe-Kains, B., Mak, R. H., & Aerts, H. J. W. L. (2015) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2015.02.015 · Cited by 695 · see citing works
Intrinsic dependencies of CT radiomic features on voxel size and number of gray levels
Shafiq‐ul‐Hassan, M., Zhang, G. G., Latifi, K., Ullah, G., Hunt, D., Balagurunathan, Y., Abdalah, M. A., Schabath, M. B., Goldgof, D. B., Mackin, D., et al. (2017) · Medical Physics · doi:10.1002/mp.12123 · Cited by 587 · see citing works
Radiomics strategies for risk assessment of tumour failure in head-and-neck cancer
Vallières, M., Kay‐Rivest, E., Perrin, L. J., Liem, X., Furstoss, C., Aerts, H. J. W. L., Khaouam, N., Nguyen‐Tan, P. F., Wang, C.-S., Sultanem, K., et al. (2017) · Scientific Reports · doi:10.1038/s41598-017-10371-5 · Cited by 543 · see citing works
Yaniv, Z. R., Lowekamp, B., Johnson, H. J., & Beare, R. J. (2017) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-017-0037-8 · Cited by 504 · see citing works
Zhao, B., James, L. P., Moskowitz, C. S., Guo, P., Ginsberg, M. S., Lefkowitz, R. A., Qin, Y., Riely, G. J., Kris, M. G., & Schwartz, L. H. (2009) · Radiology · doi:10.1148/radiol.2522081593 · Cited by 337 · see citing works
Ibrahim, A., Primakov, S. P., Beuque, M. P. L., Woodruff, H. C., Halilaj, I., Wu, G., Refaee, T., Granzier, R. W. Y., Widaatalla, Y., Hustinx, R., et al. (2020) · Methods · doi:10.1016/j.ymeth.2020.05.022 · Cited by 266 · see citing works
Assessing robustness of radiomic features by image perturbation
Zwanenburg, A., Leger, S., Agolli, L., Pilz, K., Troost, E. G. C., Richter, C., & Löck, S. (2019) · Scientific Reports · doi:10.1038/s41598-018-36938-4 · Cited by 258 · see citing works
Self-citationHow can we combat multicenter variability in MR radiomics? Validation of a correction procedure
Orlhac, F., Lecler, A., Savatovsky, J., Goya-Outi, J., Nioche, C., Charbonneau, F., Ayache, N., Frouin, F., Duron, L., & Buvat, I. (2020) · European Radiology · doi:10.1007/s00330-020-07284-9 · Cited by 210 · see citing works
Influence of inter-observer delineation variability on radiomics stability in different tumor sites
Pavic, M., Bogowicz, M., Würms, X., Glatz, S., Finazzi, T., Riesterer, O., Roesch, J., Rudofsky, L., Friess, M., Veit‐Haibach, P., et al. (2018) · Acta Oncologica · doi:10.1080/0284186x.2018.1445283 · Cited by 202 · see citing works
Deep learning in head & neck cancer outcome prediction
Diamant, A., Chatterjee, A., Vallières, M., Shenouda, G., & Seuntjens, J. P. (2019) · Scientific Reports · doi:10.1038/s41598-019-39206-1 · Cited by 202 · see citing works
Fornacon-Wood, I., Mistry, H. B., Ackermann, C. J., Blackhall, F. H., McPartlin, A. J., Finn, C. F., Price, G. J., & O’Connor, J. P. B. (2020) · European Radiology · doi:10.1007/s00330-020-06957-9 · Cited by 198 · see citing works
Test–Retest Data for Radiomics Feature Stability Analysis: Generalizable or Study-Specific?
van Timmeren, J. E., Leijenaar, R. T. H., van Elmpt, W., Wang, J., Zhang, Z., Dekker, A. L. A. J., & Lambin, P. (2016) · Tomography · doi:10.18383/j.tom.2016.00208 · Cited by 191 · see citing works
Moradmand, H., Aghamiri, S. M. R., & Ghaderi, R. (2020) · Journal of Applied Clinical Medical Physics · doi:10.1002/acm2.12795 · Cited by 173 · see citing works
Ligero, M., Jordi-Ollero, O., Bernatowicz, K., García-Ruiz, A., Delgado-Muñoz, E., Leiva, D., Mast, R., Suárez, C., Sala-Llonch, R., Calvo, N., et al. (2020) · European Radiology · doi:10.1007/s00330-020-07174-0 · Cited by 168 · see citing works
Radiomics assessment of bladder cancer grade using texture features from diffusion‐weighted imaging
Zhang, X., Xu, X., Tian, Q., Baojuan Li, A., Wu, Y., Yang, Z., Liang, Z., Liu, Y., Cui, G., & Lu, H. (2017) · Journal of Magnetic Resonance Imaging · doi:10.1002/jmri.25669 · Cited by 163 · see citing works
Understanding Sources of Variation to Improve the Reproducibility of Radiomics
Zhao, B. (2021) · Frontiers in Oncology · doi:10.3389/fonc.2021.633176 · Cited by 158 · see citing works
Fave, X. J., Zhang, L., Yang, J., Mackin, D., Balter, P. A., Gómez, D. D. L. R., Followill, D. S., Jones, A. K., Stingo, F. C., & Court, L. E. (2016) · Translational Cancer Research · doi:10.21037/tcr.2016.07.11 · Cited by 121 · see citing works
Li, Y., Ammari, S., Balleyguier, C., Lassau, N., & Chouzenoux, É. (2021) · Cancers · doi:10.3390/cancers13123000 · Cited by 101 · see citing works
Radiomics feature robustness as measured using an MRI phantom
Lee, J., Steinmann, A., Ding, Y., Lee, H. J., Owens, C. A., Wang, J., Yang, J., Followill, D. S., Ger, R. B., Mackin, D., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-83593-3 · Cited by 100 · see citing works
Reiazi, R., Abbas, E., Famiyeh, P., Rezaie, A., Kwan, J. Y. Y., Patel, T., Bratman, S. V., Tadic, T., Liu, F., & Haibe‐Kains, B. (2021) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2021.104400 · Cited by 85 · see citing works
Fan, M., Zhang, P., Wang, Y., et al. (2019) · European Radiology · doi:10.1007/s00330-018-5891-3 · Cited by 84 · see citing works
Qiu, Q., Duan, J., Duan, Z., Meng, X., Ma, C., Zhu, J., Lu, J., Liu, T., & Yin, Y. (2019) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims.2019.03.02 · Cited by 67 · see citing works
Bian, Y., Jiang, H., Ma, C., Wang, L., Zheng, J., Jin, G., & Lu, J. (2020) · American Journal of Roentgenology · doi:10.2214/ajr.19.22123 · Cited by 65 · see citing works
Suter, Y., Knecht, U., Alão, M. N. R. F., Valenzuela, W., Hewer, E., Schucht, P., Wiest, R., & Reyes, M. (2020) · Cancer Imaging · doi:10.1186/s40644-020-00329-8 · Cited by 59 · see citing works
A Radiomics Model for Predicting the Response to Bevacizumab in Brain Necrosis after Radiotherapy
Cai, J., Zheng, J., Shen, J., et al. (2020) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-20-1264 · Cited by 54 · see citing works
Self-citationPark, S.-H., Lim, H. J., Bae, B. K., Hahm, M. H., Chong, G. O., Jeong, S. Y., & Kim, J. (2021) · Cancer Imaging · doi:10.1186/s40644-021-00388-5 · Cited by 48 · see citing works
Tang, F. H., Chu, C. Y. W., & Cheung, E. Y. (2021) · BJR|Open · doi:10.1259/bjro.20200073 · Cited by 48 · see citing works
Redundant feature elimination for multi-class problems
Appice, A., Ceci, M., Rawles, S., et al. (2004) · doi:10.1145/1015330.1015397 · Cited by 47 · see citing works
Lafata, K. J., Cai, J., Wang, C., Hong, J. C., Kelsey, C. R., & Yin, F.-F. (2018) · Physics in Medicine and Biology · doi:10.1088/1361-6560/aae56a · Cited by 42 · see citing works
Self-citationVallières, M., Kay-Rivest, E., Perrin, L., et al. (2017) · The Cancer Imaging Archive · doi:10.7937/k9/tcia.2017.8oje5q00 · Cited by 40 · see citing works
Blazis, S. P., Dieckens, D., Linsen, P. V. M., et al. (2021) · European Journal of Radiology · doi:10.1016/j.ejrad.2021.109526 · Cited by 32 · see citing works
Distant metastasis time to event analysis with CNNs in independent head and neck cancer cohorts
Lombardo, E., Kurz, C., Marschner, S. N., Avanzo, M., Gagliardi, V., Fanetti, G., Franchin, G., Stancanello, J., Corradini, S., Niyazi, M., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-85671-y · Cited by 30 · see citing works
Guerrisi, A., Loi, E., Ungania, S., Russillo, M., Bruzzaniti, V., Elia, F., Desiderio, F., Marconi, R., Solivetti, F., & Strigari, L. (2020) · Cancer Medicine · doi:10.1002/cam4.2709 · Cited by 24 · see citing works
Mottola, M., Ursprung, S., Rundo, L., Sánchez, L. E., Klatte, T., Mendichovszky, I. A., Stewart, G. D., Sala, E., & Bevilacqua, A. (2021) · Scientific Reports · doi:10.1038/s41598-021-90985-y · Cited by 23 · see citing works
Foy, J. J., Al‐Hallaq, H., Grekoski, V., Tran, T. H., Guruvadoo, K., Armato, S. G., & Sensakovic, W. F. (2020) · Physics in Medicine and Biology · doi:10.1088/1361-6560/abb172 · Cited by 21 · see citing works
Perfusion CT radiomics as potential prognostic biomarker in head and neck squamous cell carcinoma
Bogowicz, M., Tanadini‐Lang, S., Veit‐Haibach, P., et al. (2019) · Acta Oncologica · doi:10.1080/0284186x.2019.1629013 · Cited by 18 · see citing works
A downsampling strategy to assess the predictive value of radiomic features
Dirand, A.-S., Frouin, F., & Buvat, I. (2019) · Scientific Reports · doi:10.1038/s41598-019-54190-2 · Cited by 8 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Radiomics-based prediction of FIGO grade for placenta accreta spectrum
Bartels, H. C., O’Doherty, J., Wolsztynski, E., Brophy, D. P., MacDermott, R., Atallah, D., Saliba, S., Young, C., Downey, P., Donnelly, J., et al. (2023) · European Radiology Experimental · doi:10.1186/s41747-023-00369-2 · Cited by 19 · see citing works
- Authors
- Teng, X., Zhang, J., Zwanenburg, A., Sun, J., Huang, Y., Lam, S., Zhang, Y., Li, B., Zhou, T., Xiao, H., Liu, C., Li, W., Han, X., Ma, Z., Li, T., & Cai, J.
- Affiliations
- Hong Kong Polytechnic University (HK)
- German Cancer Research Center (DE)
- Corresponding author
- Cai, Jing
- ORCID
- Venue
- Scientific Reports
- Published
- 2022-06-16
- Volume
- 12
- Issue
- 1
- Pages
- 10035-10035
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC9203573
- OpenAlex
- W4282960446
- Cited by
- 40
- References
- 47
- Citations by year
- 2026: 9 · 2025: 4 · 2024: 13 · 2023: 9 · 2022: 5
- FWCI
- 4.40
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Scientific Reports (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PolyU Institutional Research Archive (Hong Kong Polytechnic University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Innovation and Technology Fund (ITS/080/19)
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Medical Imaging Techniques and Applications
- Keywords
- radiomics, model reliability, image perturbation, head and neck cancer, prognostic model, CT images, intraclass correlation coefficient, concordance index, synthetic data generation, radiomic feature selection
- MeSH terms
- Humans, Prognosis, Reproducibility of Results, Cohort Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
