Radiomics feature reproducibility under inter-rater variability in segmentations of CT images
Open accessAbstract
Identifying image features that are robust with respect to segmentation variability is a tough challenge in radiomics. So far, this problem has mainly been tackled in test–retest analyses. In this work we analyse radiomics feature reproducibility in two phases: first with manual segmentations provided by four expert readers and second with probabilistic automated segmentations using a recently developed neural network (PHiseg). We test feature reproducibility on three publicly available datasets of lung, kidney and liver lesions. We find consistent results both over manual and automated segmentations in all three datasets and show that there are subsets of radiomic features which are robust against segmentation variability and other radiomic features which are prone to poor reproducibility under differing segmentations. By providing a detailed analysis of robustness of the most common radiomics features across several datasets, we envision that more reliable and reproducible radiomic models can be built in the future based on this work.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Statistical features are affected by confounders such as inter-reader variability in segmentations and acquisition settings of scanners, limiting their applicability in diverse settings.
“Moreover, these statistical features are affected by several confounders, such as inter-reader variability in segmentations26 and acquisition settings of the scanners27, limiting their applicability in diverse settings.”
From Foundation Model for Cancer Imaging Biomarkers (2024) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
U-Net: Convolutional Networks for Biomedical Image Segmentation
Ronneberger, O., Fischer, P., & Brox, T. (2015) · Lecture notes in computer science · doi:10.1007/978-3-319-24574-4_28 · Cited by 99,404 · see citing works
A guideline of selecting and reporting intraclass correlation coefficients for reliability research
Koo, T. K., & Li, M. Y. (2016) · Journal of Chiropractic Medicine · doi:10.1016/j.jcm.2016.02.012 · Cited by 29,639 · see citing works
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 23,433 · see citing works
Advances in neural information processing systems 7
Cios, K. J., & Shields, M. E. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,089 · see citing works
Cicchetti, D. V. (1994) · Psychological Assessment · doi:10.1037/1040-3590.6.4.284 · Cited by 9,349 · cited 1× · see citing works
“Possible choices are the often-used interpretation of defining excellent agreement as ICC > 0.75 36 or the interpretation proposed by Koo and Li 37 , stating that ICC > 0.9 corresponds to excellent agreement.”
MethodInfluentialComputational Radiomics System to Decode the Radiographic Phenotype
(2017) · doi:10.1158/0008-5472.can-17-0339 · Cited by 6,593 · see citing works
Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach
Aerts, H. J. W. L., Velazquez, E. R., Leijenaar, R. T. H., Parmar, C., Grossmann, P., Carvalho, S., Bussink, J., Monshouwer, R., Haibe-Kains, B., Rietveld, D., et al. (2014) · Nature Communications · doi:10.1038/ncomms5006 · Cited by 5,132 · see citing works
The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository
Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., et al. (2013) · Journal of Digital Imaging · doi:10.1007/s10278-013-9622-7 · Cited by 4,599 · see citing works
Zwanenburg, A., Vallières, M., Abdalah, M. A., Aerts, H. J. W. L., Andrearczyk, V., Apte, A., Ashrafinia, S., Bakas, S., Beukinga, R. J., Boellaard, R., et al. (2020) · Radiology · doi:10.1148/radiol.2020191145 · Cited by 4,177 · see citing works
Armato, S. G., McLennan, G., Bidaut, L., McNitt‐Gray, M. F., Meyer, C. R., Reeves, A. P., Zhao, B., Aberle, D. R., Henschke, C. I., Hoffman, E. A., et al. (2011) · Medical Physics · doi:10.1118/1.3528204 · Cited by 2,887 · see citing works
The Liver Tumor Segmentation Benchmark (LiTS)
Bilic, P., Christ, P., Li, H. B., Vorontsov, E., Ben-Cohen, A., Kaissis, G., Szeskin, A., Jacobs, C., Mamani, G. E. H., Chartrand, G., et al. (2023) · Medical Image Analysis · doi:10.1016/j.media.2022.102680 · Cited by 1,412 · see citing works
Self-citationRepeatability and Reproducibility of Radiomic Features: A Systematic Review
Traverso, A., Wee, L., Dekker, A. L. A. J., & Gillies, R. J. (2018) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2018.05.053 · Cited by 915 · see citing works
Intraclass correlation – A discussion and demonstration of basic features
Liljequist, D., Elfving, B., & Roaldsen, K. S. (2019) · PLoS ONE · doi:10.1371/journal.pone.0219854 · Cited by 910 · see citing works
Berenguer, R., Pastor-Juan, M. D. R., Canales‐Vázquez, J., Castro‐García, M., Villas, M. V., Legorburo, F. M., & Sabater, S. (2018) · Radiology · doi:10.1148/radiol.2018172361 · Cited by 622 · see citing works
Robust Radiomics Feature Quantification Using Semiautomatic Volumetric Segmentation
Parmar, C., Velazquez, E. R., Leijenaar, R. T. H., Jermoumi, M., Carvalho, S., Mak, R. H., Mitra, S., Shankar, B. U., Kikinis, R., Haibe‐Kains, B., et al. (2014) · PLoS ONE · doi:10.1371/journal.pone.0102107 · Cited by 611 · see citing works
Kickingereder, P., Burth, S., Wick, A., GOETZ, M. A., Eidel, O., Schlemmer, H.-P., Maier-Hein, K. H., Wick, W., Bendszus, M., Radbruch, A., et al. (2016) · Radiology · doi:10.1148/radiol.2016160845 · Cited by 449 · see citing works
A probabilistic u-net for segmentation of ambiguous images
Kohl, S. A. A., Romera-Paredes, B., Meyer, C., De Fauw, J., Ledsam, J. R., Maier-Hein, K. H., Eslami, S. M. A., Rezende, D. J., & Ronneberger, O. (2018) · arXiv · doi:10.48550/arxiv.1806.05034 · Cited by 439 · see citing works
Heller, N. E., Sathianathen, N. J., Kalapara, A. A., Walczak, E. G., Moore, K., Kaluzniak, H., Rosenberg, J. E., Blake, P., Rengel, Z., Oestreich, M., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.00445 · Cited by 438 · see citing works
Reproducibility and Prognosis of Quantitative Features Extracted from CT Images
Balagurunathan, Y., Gu, Y., Wang, H., Kumar, V., Grove, O., Hawkins, S. K., Kim, J., Goldgof, D. B., Hall, L., Gatenby, R. A., et al. (2014) · Translational Oncology · doi:10.1593/tlo.13844 · Cited by 305 · see citing works
Choe, J., Lee, S. M., Do, K., Lee, G., Lee, J., Lee, S. M., & Seo, J. B. (2019) · Radiology · doi:10.1148/radiol.2019181960 · Cited by 296 · see citing works
Inter-observer variability of manual contour delineation of structures in CT
Joskowicz, L., Cohen, D., Caplan, N., & Sosna, J. (2018) · European Radiology · doi:10.1007/s00330-018-5695-5 · Cited by 271 · see citing works
Peerlings, J., Woodruff, H. C., Winfield, J. M., Ibrahim, A., BEERS, B. E. V., Heerschap, A., Jackson, A., Wildberger, J. E., Mottaghy, F. M., deSouza, N. M., et al. (2019) · Scientific Reports · doi:10.1038/s41598-019-41344-5 · Cited by 267 · see citing works
Assessing robustness of radiomic features by image perturbation
Zwanenburg, A., Leger, S., Agolli, L., Pilz, K., Troost, E. G. C., Richter, C., & Löck, S. (2019) · Scientific Reports · doi:10.1038/s41598-018-36938-4 · Cited by 258 · see citing works
Meyer, M., Ronald, J., Vernuccio, F., Nelson, R. C., Ramirez-Giraldo, J. C., Solomon, J., Patel, B. N., Samei, E., & Marin, D. (2019) · Radiology · doi:10.1148/radiol.2019190928 · Cited by 258 · see citing works
Influence of inter-observer delineation variability on radiomics stability in different tumor sites
Pavic, M., Bogowicz, M., Würms, X., Glatz, S., Finazzi, T., Riesterer, O., Roesch, J., Rudofsky, L., Friess, M., Veit‐Haibach, P., et al. (2018) · Acta Oncologica · doi:10.1080/0284186x.2018.1445283 · Cited by 202 · see citing works
Armato, S. G., III, McLennan, G., Bidaut, L., McNitt-Gray, M. F., Meyer, C. R., Reeves, A. P., Zhao, B., Aberle, D. R., Henschke, C. I., Hoffman, E. A., et al. (2015) · The Cancer Imaging Archive · doi:10.7937/k9/tcia.2015.lo9ql9sx · Cited by 150 · see citing works
Kim, H., Park, C. M., Lee, M., Park, S. J., Song, Y., Lee, J. H., Hwang, E. J., & Goo, J. M. (2016) · PLoS ONE · doi:10.1371/journal.pone.0164924 · Cited by 143 · see citing works
Kalpathy–Cramer, J., Mamomov, A., Zhao, B., Lü, L., Cherezov, D., Napel, S. A., Echegaray, S., Rubin, D. L., McNitt‐Gray, M. F., Lo, P., et al. (2016) · Tomography · doi:10.18383/j.tom.2016.00235 · Cited by 127 · see citing works
Supervised uncertainty quantification for segmentation with multiple annotations
Hu, S., Worrall, D., Knegt, S., Veeling, B., Huisman, H., & Welling, M. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32245-8_16 · Cited by 105 · see citing works
Kuhl, C. K., Alparslan, Y., Schmoee, J., Sequeira, B., Keulers, A., Brümmendorf, T. H., & Keil, S. (2018) · Radiology · doi:10.1148/radiol.2018180648 · Cited by 88 · see citing works
Self-citationYamashita, R., Perrin, T., Chakraborty, J., Chou, J. F., Horvat, N., Koszalka, M. A., Midya, A., Gonen, M., Allen, P., Jarnagin, W. R., et al. (2020) · European Radiology · doi:10.1007/s00330-019-06381-8 · Cited by 76 · see citing works
Tunali, I., Hall, L., Napel, S., Cherezov, D., Guvenis, A., Gillies, R., & Schabath, M. (2019) · Medical Physics (Lancaster) · doi:10.1002/mp.13808 · Cited by 72 · see citing works
Qiu, Q., Duan, J., Duan, Z., Meng, X., Ma, C., Zhu, J., Lu, J., Liu, T., & Yin, Y. (2019) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims.2019.03.02 · Cited by 67 · see citing works
A Hierarchical Probabilistic U-Net for Modeling Multi-Scale AmbiguitiesPreprint
Köhl, S., Romera‐Paredes, B., Maier-Hein, K. H., Rezende, D. J., Eslami, S. M. A., Kohli, P., Zisserman, A., & Ronneberger, O. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.13077 · Cited by 58 · see citing works
Reliability of tumor segmentation in glioblastoma: Impact on the robustness of MRI‐radiomic features
Tixier, F., Um, H., Young, R. J., & Veeraraghavan, H. (2019) · Medical Physics · doi:10.1002/mp.13624 · Cited by 51 · see citing works
PHiSeg: Capturing Uncertainty in Medical Image Segmentation
Baumgartner, C. F., Tezcan, K. C., Chaitanya, K., Hötker, A. M., Muehlematter, U. J., Schawkat, K., Becker, A. S., Donati, O., & Konukoglu, E. (2019) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-32245-8_14 · Cited by 34 · see citing works
Can Artificial Intelligence Fix the Reproducibility Problem of Radiomics?
Park, C. M. (2019) · Radiology · doi:10.1148/radiol.2019191154 · Cited by 21 · see citing works
Radiomic Feature Stability Analysis Based on Probabilistic Segmentations
Haarburger, C., Schock, J., Truhn, D., Weitz, P., Müller‐Franzes, G. A., Weninger, L., & Merhof, D. (2020) · IEEE International Symposium on Biomedical Imaging (ISBI) · doi:10.1109/isbi45749.2020.9098674 · Cited by 9 · see citing works
Self-citation
Flagged first, then by citations
Cited by
Sorted by citations
A review in radiomics: Making personalized medicine a reality via routine imaging
Guiot, J., Vaidyanathan, A., Deprez, L., Zerka, F., Danthine, D., Frix, A.-N., Lambin, P., Bottari, F., Tsoutzidis, N., Miraglio, B., et al. (2021) · Medicinal Research Reviews · doi:10.1002/med.21846 · Cited by 344 · see citing works
Radiomics and artificial intelligence for precision medicine in lung cancer treatment
Chen, M., Copley, S. J., Viola, P., Lu, H., & Aboagye, E. O. (2023) · Seminars in Cancer Biology · doi:10.1016/j.semcancer.2023.05.004 · Cited by 213 · see citing works
Zhang, Y., Zhang, X., Cheng, Y., Li, B., Teng, X.-Z., Zhang, J., Lam, S., Zhou, T., Ma, Z., Sheng, J., et al. (2023) · Military Medical Research · doi:10.1186/s40779-023-00458-8 · Cited by 200 · see citing works
Foundation Model for Cancer Imaging Biomarkers
Pai, S., Bontempi, D., Hadzic, I., Prudente, V., Sokač, M., Chaunzwa, T. L., Bernatz, S., Hosny, A., Mak, R. H., Birkbak, N. J., et al. (2024) · Nature Machine Intelligence · doi:10.1038/s42256-024-00807-9 · Cited by 199 · see citing works
Radiomics feature reliability assessed by intraclass correlation coefficient: a systematic review
Xue, C., Yuan, J., Lo, G. G., Chang, A. T. Y., Poon, D. M. C., Wong, O. L., Zhou, Y., & Chu, C. W. (2021) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims-21-86 · Cited by 178 · see citing works
Pfaehler, E. A. G., Zhovannik, I., Wei, L., Boellaard, R., Dekker, A. L. A. J., Monshouwer, R., Naqa, I. E., Bussink, J., Gillies, R. J., Wee, L., et al. (2021) · Physics and Imaging in Radiation Oncology · doi:10.1016/j.phro.2021.10.007 · Cited by 104 · see citing works
Radiomics: a primer on high-throughput image phenotyping
Lafata, K. J., Wang, Y., Konkel, B., Yin, F.-F., & Bashir, M. R. (2021) · Abdominal Radiology · doi:10.1007/s00261-021-03254-x · Cited by 92 · see citing works
Feature selection methods and predictive models in CT lung cancer radiomics
Ge, G. R., & Zhang, J. (2022) · Journal of Applied Clinical Medical Physics · doi:10.1002/acm2.13869 · Cited by 77 · see citing works
Setu, M. A. K., Horstmann, J., Schmidt, S., Stern, M. E., & Steven, P. (2021) · Scientific Reports · doi:10.1038/s41598-021-87314-8 · Cited by 71 · see citing works
Challenges for machine learning in clinical translation of big data imaging studies
Dinsdale, N. K., Bluemke, E., Sundaresan, V., Jenkinson, M., Smith, S. M., & Namburete, A. I. L. (2022) · Neuron · doi:10.1016/j.neuron.2022.09.012 · Cited by 69 · see citing works
Salmanpour, M. R., Hosseinzadeh, M., Rezaeijo, S. M., & Rahmim, A. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107714 · Cited by 69 · see citing works
Lambin, P., Woodruff, H. C., Mali, S. A., Zhong, X., Kuang, S., Lavrova, E., Khan, H., Lekadir, K., Zwanenburg, A., Deasy, J. O., et al. (2025) · Nature Reviews Clinical Oncology · doi:10.1038/s41571-025-01067-1 · Cited by 69 · see citing works
Oncologic Imaging and Radiomics: A Walkthrough Review of Methodological Challenges
Stanzione, A., Cuocolo, R., Ugga, L., Verde, F., Romeo, V., Brunetti, A., & Maurea, S. (2022) · Cancers · doi:10.3390/cancers14194871 · Cited by 64 · see citing works
Dunn, B., Pierobon, M., & Wei, Q. (2023) · Bioengineering · doi:10.3390/bioengineering10060690 · Cited by 64 · see citing works
Radiomics in Oncology, Part 1: Technical Principles and Gastrointestinal Application in CT and MRI
Caruso, D., Polici, M., Zerunian, M., Pucciarelli, F., Guido, G., Polidori, T., Landolfi, F., Nicolai, M., Lucertini, E., Tarallo, M., et al. (2021) · Cancers · doi:10.3390/cancers13112522 · Cited by 57 · see citing works
Wong, J., Baine, M. J., Wisnoskie, S., Bennion, N. R., Zheng, D., Lei, Y., Dalal, V., Hollingsworth, M. A., Lin, C. H., & Zheng, D. (2021) · Scientific Reports · doi:10.1038/s41598-021-95152-x · Cited by 49 · see citing works
Ferrante, M., Rinaldi, L. M., Botta, F., Hu, X., Dolp, A., Minotti, M., Piano, F. D., Funicelli, G., Volpe, S., Bellerba, F., et al. (2022) · Journal of Clinical Medicine · doi:10.3390/jcm11247334 · Cited by 43 · see citing works
Automated Tumor Segmentation in Radiotherapy
Savjani, R. R., Lauria, M., Bose, S., Deng, J., Yuan, Y., & Andrearczyk, V. (2022) · Seminars in Radiation Oncology · doi:10.1016/j.semradonc.2022.06.002 · Cited by 39 · see citing works
Re, M. D., Cucchiara, F., Rofi, E., Fontanelli, L., Petrini, I., Gri, N., Pasquini, G., Rizzo, M., Gabelloni, M., Belluomini, L., et al. (2020) · Cancer Immunology and Immunotherapy · doi:10.1007/s00262-020-02810-6 · Cited by 37 · see citing works
Multiparametric MRI and Radiomics in Prostate Cancer: A Review of the Current Literature
Midiri, F., Vernuccio, F., Purpura, P., Alongi, P., & Bartolotta, T. V. (2021) · Diagnostics · doi:10.3390/diagnostics11101829 · Cited by 37 · see citing works
Teng, X., Zhang, J., Ma, Z., Zhang, Y., Lam, S., Li, W., Xiao, H., Li, T., Li, B., Zhou, T., et al. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.974467 · Cited by 37 · see citing works
Majumder, S., Katz, S., Kontos, D., & Roshkovan, L. (2023) · BJR|Open · doi:10.1093/bjro/tzad004 · Cited by 42 · see citing works
Dutta, K., Roy, S., Whitehead, T., Luo, J.-Q. R., Jha, A. K., Li, S.-Q., Quirk, J., & Shoghi, K. (2021) · Cancers · doi:10.3390/cancers13153795 · Cited by 33 · see citing works
Staal, F. C., Taghavi, M., van der Reijd, D. J., Gómez, F. M., Imani, F., Klompenhouwer, E. G., Meek, D. B., Roberti, S., de Boer, M., Lambregts, D. M. J., et al. (2021) · European Journal of Radiology · doi:10.1016/j.ejrad.2021.109773 · Cited by 34 · see citing works
Granata, V., Fusco, R., Muzio, F. D., Brunese, M., Setola, S., Ottaiano, A., Cardone, C., Avallone, A., Patrone, R., Pradella, S., et al. (2023) · La radiologia medica · doi:10.1007/s11547-023-01710-w · Cited by 32 · see citing works
Haniff, N. S. M., Karim, M. K. A., Osman, N. H., Saripan, M. I., Isa, I. N. C., & Ibahim, M. J. (2021) · Diagnostics · doi:10.3390/diagnostics11091573 · Cited by 32 · see citing works
Lee, S. B., Cho, Y. J., Hong, Y., Jeong, D., Lee, J., Kim, S., Lee, S., & Choi, Y. H. (2021) · Investigative Radiology · doi:10.1097/rli.0000000000000839 · Cited by 32 · see citing works
Impact of image quality on radiomics applications
Cui, Y., & Yin, F.-F. (2022) · Physics in Medicine and Biology · doi:10.1088/1361-6560/ac7fd7 · Cited by 30 · see citing works
Endo, Y., Alaimo, L., Lima, H. A., Moazzam, Z., Ratti, F., Marques, H. P., Soubrane, O., Lam, V. W. T., Kitago, M., Poultsides, G. A., et al. (2022) · Annals of Surgical Oncology · doi:10.1245/s10434-022-12494-0 · Cited by 31 · see citing works
Teng, X., Wang, Y., Nicol, A. J., Ching, J. C. F., Wong, E. K. Y., Lam, K. T. C., Zhang, J., Lee, S. W. Y., & Cai, J. (2024) · Diagnostics · doi:10.3390/diagnostics14161835 · Cited by 30 · see citing works
Yousefi, B., Lariviere, M. J., Cohen, E. A., Buckingham, T. H., Yee, S. S., Black, T. A., Chien, A. L., Noël, P. B., Hwang, W., Katz, S. I., et al. (2021) · Scientific Reports · doi:10.1038/s41598-021-88239-y · Cited by 28 · see citing works
Park, T., Yoon, M. A., Cho, Y. C., Ham, S. J., Ko, Y., Kim, S., Jeong, H., & Lee, J. (2022) · Scientific Reports · doi:10.1038/s41598-022-10807-7 · Cited by 29 · see citing works
Comprehensive Analysis of Tumour Sub-Volumes for Radiomic Risk Modelling in Locally Advanced HNSCC
Leger, S., Zwanenburg, A., Leger, K., Lohaus, F., Linge, A., Schreiber, A., Kalinauskaitė, G., Tinhofer, I., Guberina, N., Guberina, M., et al. (2020) · Cancers · doi:10.3390/cancers12103047 · Cited by 29 · see citing works
Zaridis, D. I., Mylona, E., Tachos, N. S., Pezoulas, V. C., Grigoriadis, G., Tsiknakis, N., Kostas Marias, A., Tsiknakis, M., & Fotiadis, D. I. (2023) · Scientific Reports · doi:10.1038/s41598-023-27671-8 · Cited by 28 · see citing works
Shankar, L., Schöder, H., Sharon, E., Wolchok, J., Knopp, M., Wahl, R., Ellingson, B., Hall, N., Yaffe, M., Towbin, A., et al. (2023) · The Lancet Oncology · doi:10.1016/s1470-2045(22)00742-2 · Cited by 19 · see citing works
Naseri, H., Skamene, S., Tolba, M., Faye, M. D., Ramia, P., Khriguian, J., Patrick, H. M., Andrade, A. X., David, M., & Kildea, J. (2022) · Scientific Reports · doi:10.1038/s41598-022-13379-8 · Cited by 27 · see citing works
Spatial assessments in texture analysis: what the radiologist needs to know
Varghese, B. A., Fields, B. K. K., Hwang, D., Duddalwar, V. A., Matcuk, G. R., & Cen, S. (2023) · Frontiers in Radiology · doi:10.3389/fradi.2023.1240544 · Cited by 26 · see citing works
Ferrari, R., Trinci, M., Casinelli, A., Treballi, F., Leone, E., Caruso, D., Polici, M., Faggioni, L., Neri, E., & Galluzzo, M. (2024) · La radiologia medica · doi:10.1007/s11547-024-01904-w · Cited by 24 · see citing works
Challenges in Glioblastoma Radiomics and the Path to Clinical Implementation
Martin, P., Holloway, L. C., Metcalfe, P. E., Koh, E., & Brighi, C. (2022) · Cancers · doi:10.3390/cancers14163897 · Cited by 24 · see citing works
Kothari, G., Woon, B., Patrick, C., Korte, J., Wee, L., Hanna, G., Kron, T., Hardcastle, N., & Siva, S. (2022) · Scientific Reports · doi:10.1038/s41598-022-16520-9 · Cited by 27 · see citing works
Reiazi, R., Arrowsmith, C., Welch, M. L., Aghababazadeh, F. A., Eeles, C. B., Tadic, T., Hope, A. J., Bratman, S. V., & Haibe‐Kains, B. (2021) · Cancers · doi:10.3390/cancers13092269 · Cited by 22 · see citing works
Mottola, M., Ursprung, S., Rundo, L., Sánchez, L. E., Klatte, T., Mendichovszky, I. A., Stewart, G. D., Sala, E., & Bevilacqua, A. (2021) · Scientific Reports · doi:10.1038/s41598-021-90985-y · Cited by 23 · see citing works
Radiomics feature reproducibility: The elephant in the room
Klontzas, M. E. (2024) · European Journal of Radiology · doi:10.1016/j.ejrad.2024.111430 · Cited by 22 · see citing works
Singh, A., Horng, H., Roshkovan, L., Weeks, J. K., Hershman, M., Noël, P., Luna, J., Cohen, E., Pantalone, L., Shinohara, R., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-14160-7 · Cited by 17 · see citing works
Priya, S., Aggarwal, T., Ward, C. E., Bathla, G., Jacob, M., Gerke, A. K., Hoffman, E. A., & Nagpal, P. (2021) · Scientific Reports · doi:10.1038/s41598-021-92155-6 · Cited by 21 · see citing works
Siow, T. Y., Yeh, C., Lin, G., Lin, C., Wang, H., Liao, C., Toh, C., Chan, S.-C., Lin, C.-P., & Ng, S. (2022) · Cancers · doi:10.3390/cancers14246119 · Cited by 20 · see citing works
Pasini, G., Russo, G. I., Mantarro, C., Bini, F., Richiusa, S., Morgante, L., Comelli, A., Russo, G. I., SABINI, M. G., Cosentino, S., et al. (2023) · Diagnostics · doi:10.3390/diagnostics13243640 · Cited by 20 · see citing works
Mukherjee, S., Korfiatis, P. D., Patnam, N. G., Trivedi, K. H., Karbhari, A., Suman, G., Fletcher, J. G., & Goenka, A. H. (2024) · Abdominal Radiology · doi:10.1007/s00261-023-04127-1 · Cited by 20 · see citing works
Berta, L., Rizzetto, F., Mattia, C. D., Lizio, D., Felisi, M. M. J., Colombo, P. E., Carrazza, S., Gelmini, S., Bianchi, L. M. G., Artioli, D., et al. (2021) · Physica Medica · doi:10.1016/j.ejmp.2021.06.001 · Cited by 19 · see citing works
Granata, V., Fusco, R., Setola, S., Galdiero, R., Maggialetti, N., Patrone, R., Ottaiano, A., Nasti, G., Silvestro, L., Cassata, A., et al. (2023) · Infectious Agents and Cancer · doi:10.1186/s13027-023-00495-x · Cited by 17 · see citing works
Ren, Q.-Q., Zhu, P., Li, C.-D., Yan, M.-J., Liu, S., Zheng, C., & Xia, X. (2022) · Frontiers in Bioengineering and Biotechnology · doi:10.3389/fbioe.2022.872044 · Cited by 18 · see citing works
Radiomics in veterinary medicine: Overview, methods, and applications
Basran, P. S., & Porter, I. R. (2022) · Veterinary Radiology & Ultrasound · doi:10.1111/vru.13156 · Cited by 17 · see citing works
Gottlich, H. C., Gregory, A., Sharma, V., Khanna, A., Moustafa, A. U., Lohse, C. M., Potretzke, T. A., Korfiatis, P. D., Potretzke, A. M., Đenić, A., et al. (2023) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-023-00804-1 · Cited by 17 · see citing works
Joo, L., Jung, S. C., Lee, H., Park, S. Y., Kim, M., Park, J. E., & Choi, K. M. (2021) · Scientific Reports · doi:10.1038/s41598-021-96621-z · Cited by 17 · see citing works
Du, D., Lv, W., Lv, J., Chen, X., Wu, H., Rahmim, A., & Lu, L. (2022) · European Radiology · doi:10.1007/s00330-022-09229-w · Cited by 16 · see citing works
Variability Matters: Evaluating Inter-Rater Variability in Histopathology for Robust Cell Detection
Kang, C. M., Lee, C., Song, H., Ma, M., & Pereira, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-25082-8_37 · Cited by 14 · see citing works
HeLLePhant: A phantom mimicking non-small cell lung cancer for texture analysis in CT images
Rinaldi, L. M., Pezzotta, F., Santaniello, T., Marco, P. D., Bianchini, L., Origgi, D., Cremonesi, M., Milani, P., Mariani, M., & Botta, F. (2022) · Physica Medica · doi:10.1016/j.ejmp.2022.03.010 · Cited by 14 · see citing works
Robust Radiomic Signatures of Intervertebral Disc Degeneration From MRI
McSweeney, T. P., Tiulpin, A., Kowlagi, N., Määttä, J. H., Karppinen, J. I., & Saarakkala, S. S. (2025) · Spine · doi:10.1097/brs.0000000000005435 · Cited by 13 · see citing works
Mohammadi-Sadr, M., Cheki, M., Moslehi, M., Zarasvandnia, M., & Salamat, M. R. (2024) · Academic Radiology · doi:10.1016/j.acra.2024.10.049 · Cited by 13 · see citing works
Translating radiomics into clinical practice: A step-by-step guide to study design and evaluation
Chen, M., Linton‐Reid, K., Aboagye, E. O., & Copley, S. J. (2025) · Clinical Radiology · doi:10.1016/j.crad.2025.107053 · Cited by 15 · see citing works
PRISM: A Promptable and Robust Interactive Segmentation Model with Visual Prompts
Li, H., Liu, H., Hu, D., Wang, J., & Oguz, I. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-72384-1_37 · Cited by 12 · see citing works
Jahanshahi, A., Soleymani, Y., Ghaziyani, M. F., & Khezerloo, D. (2023) · The Egyptian Journal of Radiology and Nuclear Medicine · doi:10.1186/s43055-023-01029-6 · Cited by 12 · see citing works
Alsoof, D., McDonald, C. L., Durand, W. M., Diebo, B. G., Kuris, E. O., & Daniels, A. H. (2023) · The International Journal of Spine Surgery · doi:10.14444/8501 · Cited by 11 · see citing works
From pixels to prognosis: radiomics and AI in Alzheimer’s disease management
Peng, D., Huang, W., Liu, R. P., & Zhong, W. (2025) · Frontiers in Neurology · doi:10.3389/fneur.2025.1536463 · Cited by 10 · see citing works
Predictive performance of radiomic models based on features extracted from pretrained deep networks
Demircioğlu, A. (2022) · Insights into Imaging · doi:10.1186/s13244-022-01328-y · Cited by 10 · see citing works
Bianconi, F., Fravolini, M. L., Palumbo, I., Pascoletti, G., Nuvoli, S. M. F., Rondini, M., Spanu, A., & Palumbo, B. (2021) · Diagnostics · doi:10.3390/diagnostics11071224 · Cited by 10 · see citing works
Soleymani, Y., Jahanshahi, A., Pourfarshid, A., & Khezerloo, D. (2022) · Journal of medical imaging and radiation sciences · doi:10.1016/j.jmir.2022.09.018 · Cited by 10 · see citing works
Bakker, M. A. G., Orvalho, M. L., Matela, N., & Mota, A. M. (2024) · Journal of Imaging · doi:10.3390/jimaging10090218 · Cited by 9 · see citing works
Foundation Models for Quantitative Biomarker Discovery in Cancer ImagingPreprint
Pai, S., Bontempi, D., Hadžić, I., Prudente, V., Sokač, M., Chaunzwa, T. L., Bernatz, S., Hosny, A., Mak, R. H., Birkbak, N. J., et al. (2023) · medRxiv · doi:10.1101/2023.09.04.23294952 · Cited by 9 · see citing works
Zeng, D., Song, Z., Liu, Q., Huang, J., Wang, X. W., & Tang, Z. (2025) · BMC Medical Imaging · doi:10.1186/s12880-025-01664-7 · Cited by 8 · see citing works
Whybra, P., & Spezi, E. (2023) · Scientific Reports · doi:10.1038/s41598-023-41475-w · Cited by 8 · see citing works
Robustness and performance of radiomic features in diagnosing cystic renal masses
Könik, A., Miskin, N. P., Guo, Y., Shinagare, A. B., & Qin, L. (2021) · Abdominal Radiology · doi:10.1007/s00261-021-03241-2 · Cited by 8 · see citing works
Bartholomeus, G. A., van Amsterdam, W. A. C., den Harder, A. M., Willemink, M. J., van Hamersvelt, R. W., de Jong, P. A., & Leiner, T. (2023) · European Radiology · doi:10.1007/s00330-023-09643-8 · Cited by 8 · see citing works
Stability of Radiomic Models and Strategies to Enhance Reproducibility
Chaddad, A., & Liang, X. (2024) · IEEE Transactions on Radiation and Plasma Medical Sciences · doi:10.1109/trpms.2024.3365778 · Cited by 7 · see citing works
Ma, J., Yang, H., Chou, Y., Yoon, J. H., Allison, T., Komandur, R., McDunn, J., Tasneem, A., Do, R. K., Schwartz, L. H., et al. (2024) · Medical Physics · doi:10.1002/mp.17504 · Cited by 6 · see citing works
Dong, Y., Wang, P., Geng, H., Liu, Y., & Wang, E. (2025) · Journal of X-Ray Science and Technology · doi:10.1177/08953996241304988 · Cited by 6 · see citing works
Müller‐Franzes, G. A., Nebelung, S., Schock, J., Haarburger, C., Khader, F., Pedersoli, F., Schulze-Hagen, M. F., Kuhl, C. K., & Truhn, D. (2022) · Diagnostics · doi:10.3390/diagnostics12020247 · Cited by 6 · see citing works
Self-citationGe, G. R., Siddique, A., & Zhang, J. (2023) · Physics in Medicine and Biology · doi:10.1088/1361-6560/acce1c · Cited by 6 · see citing works
Widaatalla, Y., Wolswijk, T., Khan, M. D., Halilaj, I., Mosterd, K., Woodruff, H. C., & Lambin, P. (2025) · Cancers · doi:10.3390/cancers17050768 · Cited by 5 · see citing works
Haniff, N. S. M., Ng, K. H., Kamal, I., Zain, N. M., & Karim, M. K. A. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e36313 · Cited by 5 · see citing works
Chirra, P. V., Giriprakash, P. P., Rizk, A. G., Kurowski, J. A., Viswanath, S. E., & Gandhi, N. S. (2024) · Journal of Imaging Informatics in Medicine · doi:10.1007/s10278-024-01303-7 · Cited by 5 · see citing works
Xu, F., Zhang, Y., Ma, Q., Hu, L., Li, Y. H., Gao, C., Guo, P., Yang, X., Zhou, Y., Zhang, J., et al. (2025) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-025-07508-0 · Cited by 5 · see citing works
Qian, W.-L., Chen, Q., Jb, Z., Xu, J.-M., & Hu, C.-H. (2023) · Clinical Radiology · doi:10.1016/j.crad.2023.03.001 · Cited by 5 · see citing works
Chen, Y., Zhao, B., Zhu, C., Bie, C., He, X., Zheng, Z., & Song, X. (2023) · Magnetic Resonance Imaging · doi:10.1016/j.mri.2023.07.003 · Cited by 5 · see citing works
The impact of Inter-observation variation on radiomic features of pulmonary nodules
Zhu, W., Xu, F., Lou, K., Qiu, X., Huang, D., Huang, S., Xie, D., & Hu, H. (2025) · Frontiers in Oncology · doi:10.3389/fonc.2025.1567028 · Cited by 4 · see citing works
Arisar, F. A. Q., Salinas-Miranda, E., Ali, H. A., Lajkosz, K., Chen, C., Azhie, A., Healy, G. M., Deniffel, D., Haider, M. A., & Bhat, M. (2023) · Transplant International · doi:10.3389/ti.2023.11149 · Cited by 4 · see citing works
Stability of Multi-Parametric Prostate MRI Radiomic Features to Variations in Segmentation
Seetha, S. T., Garanzini, E. M., Tenconi, C., Marenghi, C., Avuzzi, B., Catanzaro, M. A., Stagni, S., Villa, S., Chiorda, B. N., Badenchini, F., et al. (2023) · Journal of Personalized Medicine · doi:10.3390/jpm13071172 · Cited by 4 · see citing works
Artificial Intelligence Applications for Imaging Metabolic Bone Diseases
Isaac, A., Akdoğan, A. I., Dalili, D., Saber, N., Drobny, D., Guglielmi, G., Marc Modat, A., & Bazzocchi, A. (2024) · Seminars in Musculoskeletal Radiology · doi:10.1055/s-0044-1789218 · Cited by 4 · see citing works
Chest CT in COVID-19 Pneumonia: Potentials and Limitations of Radiomics and Artificial Intelligence
Vernuccio, F., Cutaia, G., Cannella, R., Vernuccio, L., Lagalla, R., & Midiri, M. (2021) · Studies in computational intelligence · doi:10.1007/978-3-030-74761-9_3 · Cited by 4 · see citing works
Ramlee, S., Delgado-Ortet, M., Sánchez, L. E., Aloj, L., & Manavaki, R. (2025) · PLoS ONE · doi:10.1371/journal.pone.0335219 · Cited by 3 · see citing works
Pediatric Personalized Deep Learning Models for Segmentation of Hepatoblastoma at CT and MRI
Modanwal, G., Kumar, S., Viswanathan, V. S., Morin, C. E., Rees, M. A., Squires, J. E., Tang, E., Katzenstein, H. M., Towbin, A. J., Madabhushi, A., et al. (2026) · Radiology Imaging Cancer · doi:10.1148/rycan.250041 · Cited by 3 · see citing works
Radiomics-Based Classification of Pathological Patterns in Common Carotid Artery Wall
Jadoon, M., Poli, F., Boutouyrie, P., Khettab, H., Joywin-Melitus, G., Sam, D.-O., Bianchini, E., Faita, F., Jouven, X., Empana, J. P., et al. (2026) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2026.02.017 · Cited by 3 · see citing works
Soleymani, Y., Sheikhzadeh, P., Mohammadzadeh, M., & Khezerloo, D. (2023) · Journal of Biomedical Physics and Engineering · doi:10.31661/jbpe.v0i0.2208-1525 · Cited by 3 · see citing works
Mirza‐Aghazadeh‐Attari, M., Afyouni, S., Zandieh, G., Nia, I. Y., Mohseni, A., Borhani, A., Madani, S. P., Shahbazian, H., Ansari, G., Kim, A., et al. (2023) · Journal of Computer Assisted Tomography · doi:10.1097/rct.0000000000001557 · Cited by 3 · see citing works
Martin, P., Holloway, L. C., Metcalfe, P. E., Koh, E., Aly, F., Chan, E., & Brighi, C. (2025) · Physical and Engineering Sciences in Medicine · doi:10.1007/s13246-025-01613-2 · Cited by 2 · see citing works
From diagnosis to treatment: navigating the course for pancreatic neuroendocrine neoplasms
Mastnak, L., & Badovinac, D. (2025) · Annals of Medicine · doi:10.1080/07853890.2025.2569992 · Cited by 2 · see citing works
Chupetlovska, K., Lipman, K. B. W. G., Bodalal, Z., Aricò, F. M., Topff, L., Maas, M., & Trebeschi, S. (2025) · European Journal of Radiology Artificial Intelligence · doi:10.1016/j.ejrai.2025.100053 · Cited by 2 · see citing works
Sümer, E., Danyeli, D. A. E., Yakicier, M. C., Pamir, M. N., Özduman, K., Di̇nçer, A., & Öztürk-Işık, E. (2026) · Frontiers in Oncology · doi:10.3389/fonc.2026.1775958 · Cited by 2 · see citing works
Mori, M., Alborghetti, L., Palumbo, D., Broggi, S., Raspanti, D., Rovere, P. Q., Vecchio, A. D., COBELLI, F. D., & Fiorino, C. (2022) · Physica Medica · doi:10.1016/j.ejmp.2022.06.018 · Cited by 2 · see citing works
A Deep Learning Framework for Analysis of the Eustachian Tube and the Internal Carotid Artery
Amanian, A., Jain, A., Xiao, Y., Kim, C., Ding, A. S., Sahu, M., Taylor, R. H., Unberath, M., Ward, B. K., Galaiya, D. J., et al. (2024) · Otolaryngology · doi:10.1002/ohn.789 · Cited by 2 · see citing works
- Authors
- Haarburger, C., Müller-Franzes, G., Weninger, L., Kuhl, C., Truhn, D., & Merhof, D.
- Affiliations
- RWTH Aachen University (DE)
- Universitätsklinikum Aachen (DE)
- Fraunhofer Institute for Digital Medicine (DE)
- Corresponding author
- Haarburger, Christoph
- Venue
- Scientific Reports
- Published
- 2020-07-29
- Volume
- 10
- Issue
- 1
- Pages
- 12688-12688
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC7391354
- OpenAlex
- W3046015102
- Cited by
- 139
- References
- 38
- Citations by year
- 2026: 24 · 2025: 20 · 2024: 17 · 2023: 28 · 2022: 26 · 2021: 22 · 2020: 2
- FWCI
- 9.49
- Type
- journal article
- Language
- en
- Update notices
- correction (2021-11-16)
- Open access
- gold
- doi.org
- Locations
- Scientific Reports (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Publikationsdatenbank der Fraunhofer-Gesellschaft (Fraunhofer-Gesellschaft) (submittedVersion, other-oa)
- Fraunhofer-Publica (Fraunhofer-Gesellschaft) (submittedVersion)
- RWTH Publications (RWTH Aachen) (submittedVersion, other-oa)
- Funders
- RWTH Aachen University
- Interreg
- Topics
- Health Sciences › Medicine › Radiology, Nuclear Medicine and Imaging › Radiomics and Machine Learning in Medical Imaging
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Medical Image Segmentation Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Image Retrieval and Classification Techniques
- Keywords
- radiomics, feature reproducibility, interobserver variability, segmentation variability, CT images, manual segmentation, automatic segmentation, lung lesions, renal lesions, liver lesions, feature robustness
- MeSH terms
- Automation, Humans, Neoplasms, Radiographic Image Interpretation, Computer-Assisted, Tomography, X-Ray Computed, Observer Variation, Databases, Factual, Neural Networks, Computer
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
