False Discovery Rates in PET and CT Studies with Texture Features: A Systematic Review
Open accessThe DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
Abstract
PURPOSE: A number of recent publications have proposed that a family of image-derived indices, called texture features, can predict clinical outcome in patients with cancer. However, the investigation of multiple indices on a single data set can lead to significant inflation of type-I errors. We report a systematic review of the type-I error inflation in such studies and review the evidence regarding associations between patient outcome and texture features derived from positron emission tomography (PET) or computed tomography (CT) images. METHODS: For study identification PubMed and Scopus were searched (1/2000-9/2013) using combinations of the keywords texture, prognostic, predictive and cancer. Studies were divided into three categories according to the sources of the type-I error inflation and the use or not of an independent validation dataset. For each study, the true type-I error probability and the adjusted level of significance were estimated using the optimum cut-off approach correction, and the Benjamini-Hochberg method. To demonstrate explicitly the variable selection bias in these studies, we re-analyzed data from one of the published studies, but using 100 random variables substituted for the original image-derived indices. The significance of the random variables as potential predictors of outcome was examined using the analysis methods used in the identified studies. RESULTS: Fifteen studies were identified. After applying appropriate statistical corrections, an average type-I error probability of 76% (range: 34-99%) was estimated with the majority of published results not reaching statistical significance. Only 3/15 studies used a validation dataset. For the 100 random variables examined, 10% proved to be significant predictors of survival when subjected to ROC and multiple hypothesis testing analysis. CONCLUSIONS: We found insufficient evidence to support a relationship between PET or CT texture features and patient survival. Further fit for purpose validation of these image-derived biomarkers should be supported by appropriate biological and statistical evidence before their association with patient outcome is investigated in prospective studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164 · cited 1×
“Fig 1 describes the study flow diagram according to the PRISMA guidelines for reporting systematic reviews [45].”
MethodDermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Textural Features for Image Classification
Haralick, R. M., Shanmugam, K., & Dinstein, I. (1973) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1973.4309314 · Cited by 22,886
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
A simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655 · cited 1×
“It has been proposed that for linear models, such as multiple regression, a minimum of 10 to 15 observations per predictor variable will produce reasonably stable estimates [52,53].”
BackgroundIntratumor Heterogeneity and Branched Evolution Revealed by Multiregion Sequencing
Gerlinger, M., Rowan, A. J., Horswell, S., Larkin, J., Endesfelder, D., Gronroos, E., Martinez, P., Matthews, N., Stewart, A., Tarpey, P., et al. (2012) · New England Journal of Medicine · doi:10.1056/nejmoa1113205 · Cited by 7,774 · cited 3×
“Cancer heterogeneity is a phenomenon associated with clonal branch evolution (genetic variability) and regional differences in the tumour microenvironment (non-genetic variability) [4,5].”
BackgroundNowell, P. C. (1976) · Science · doi:10.1126/science.959840 · Cited by 6,471
Statistical and structural approaches to texture
Haralick, R. M. (1979) · Proceedings of the IEEE · doi:10.1109/proc.1979.11328 · Cited by 5,759
Cited by
Radiomics in medical imaging—“how-to” guide and critical reflection
van Timmeren, J. E., Cester, D., Tanadini-Lang, S., Alkadhi, H., & Baessler, B. (2020) · Insights into Imaging · doi:10.1186/s13244-020-00887-2 · Cited by 1,316
Applications and limitations of radiomics
Yip, S. S. F., & Aerts, H. J. W. L. (2016) · Physics in Medicine and Biology · doi:10.1088/0031-9155/61/13/r150 · Cited by 1,210
Nioche, C., Orlhac, F., Boughdad, S., Reuzé, S., Goya-Outi, J., Robert, C., Pellot-Barakat, C., Soussan, M., Frouin, F., & Buvat, I. (2018) · Cancer Research · doi:10.1158/0008-5472.can-18-0125 · Cited by 1,167
Repeatability and Reproducibility of Radiomic Features: A Systematic Review
Traverso, A., Wee, L., Dekker, A., & Gillies, R. (2018) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2018.05.053 · Cited by 903
CT Texture Analysis: Definitions, Applications, Biologic Correlates, and Challenges
Lubner, M. G., Smith, A. D., Sandrasegaran, K., Sahani, D. V., & Pickhardt, P. J. (2017) · RadioGraphics · doi:10.1148/rg.2017170056 · Cited by 773
Deep learning in medical imaging and radiation therapy
Sahiner, B., Pezeshk, A., Hadjiiski, L. M., Wang, X., Drukker, K., Cha, K. H., Summers, R. M., & Giger, M. L. (2019) · Medical Physics · doi:10.1002/mp.13264 · Cited by 740
Machine and deep learning methods for radiomics
Avanzo, M., Wei, L., Stancanello, J., Vallières, M., Rao, A., Morin, O., Mattonen, S. A., & El Naqa, I. (2020) · Medical Physics · doi:10.1002/mp.13678 · Cited by 605
Zhang, B., Tian, J., Dong, D., Gu, D., Dong, Y., Zhang, L., Lian, Z., Liu, J., Luo, X., Pei, S., et al. (2017) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-16-2910 · Cited by 494
Characterization of PET/CT images using texture analysis: the past, the present… any future?
Hatt, M., Tixier, F., Pierce, L., Kinahan, P. E., Le Rest, C. C., & Visvikis, D. (2017) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-016-3427-0 · Cited by 475
Di Leo, G., & Sardanelli, F. (2020) · European Radiology Experimental · doi:10.1186/s41747-020-0145-y · Cited by 455
A Postreconstruction Harmonization Method for Multicenter Radiomic Studies in PET
Orlhac, F., Boughdad, S., Philippe, C., Stalla-Bourdillon, H., Nioche, C., Champion, L., Soussan, M., Frouin, F., Frouin, V., & Buvat, I. (2018) · Journal of Nuclear Medicine · doi:10.2967/jnumed.117.199935 · Cited by 393
Sala, E., Mema, E., Himoto, Y., Veeraraghavan, H., Brenton, J. D., Snyder, A., Weigelt, B., & Vargas, H. A. (2017) · Clinical Radiology · doi:10.1016/j.crad.2016.09.013 · Cited by 369
Delta-radiomics features for the prediction of patient outcomes in non–small cell lung cancer
Fave, X., Zhang, L., Yang, J., Mackin, D., Balter, P., Gomez, D., Followill, D., Jones, A. K., Stingo, F., Liao, Z., et al. (2017) · Scientific Reports · doi:10.1038/s41598-017-00665-z · Cited by 353
Artificial intelligence and machine learning in cancer imaging
Koh, D.-M., Papanikolaou, N., Bick, U., Illing, R., Kahn, C. E., Kalpathi-Cramer, J., Matos, C., Martí-Bonmatí, L., Miles, A., Mun, S. K., et al. (2022) · Communications Medicine · doi:10.1038/s43856-022-00199-0 · Cited by 311
Zwanenburg, A. (2019) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-019-04391-8 · Cited by 306
Chetan, M. R., & Gleeson, F. V. (2021) · European Radiology · doi:10.1007/s00330-020-07141-9 · Cited by 265
Lucia, F., Visvikis, D., Desseroit, M.-C., Miranda, O., Malhaire, J.-P., Robin, P., Pradier, O., Hatt, M., & Schick, U. (2018) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-017-3898-7 · Cited by 245
“high risk of false-positive rate, as recently highlighted [8, 21, 28].”
BackgroundMethodRepeatability of Multiparametric Prostate MRI Radiomics Features
Schwier, M., van Griethuysen, J., Vangel, M. G., Pieper, S., Peled, S., Tempany, C., Aerts, H. J. W. L., Kikinis, R., Fennessy, F. M., & Fedorov, A. (2019) · Scientific Reports · doi:10.1038/s41598-019-45766-z · Cited by 237
Hagiwara, A., Fujita, S., Ohno, Y., & Aoki, S. (2020) · Investigative Radiology · doi:10.1097/rli.0000000000000666 · Cited by 183
“It is largely unknown how the repeatability and reproducibility of each feature are propagated to the final result of radiomics, and therefore, validation of a radiomics algorithm against another independent dataset is considered to be crucial.(215) There are several multi-institutional efforts to standardize and incr…”
Huang, P., Park, S., Yan, R., Lee, J., Chu, L. C., Lin, C. T., Hussien, A., Rathmell, J., Thomas, B., Chen, C., et al. (2018) · Radiology · doi:10.1148/radiol.2017162725 · Cited by 173
Ho, L. M., Samei, E., Mazurowski, M. A., Zheng, Y., Allen, B. C., Nelson, R. C., & Marin, D. (2019) · American Journal of Roentgenology · doi:10.2214/ajr.18.20097 · Cited by 65
Lekadir, K., Osuala, R., Gallin, C., Lazrak, N., Kushibar, K., Tsakou, G., Aussó, S., Alberich, L. C., Marias, K., Tsiknakis, M., et al. (2021) · arXiv · doi:10.48550/arxiv.2109.09658 · Cited by 58
“…hundreds or even thousands [44] of radiomics features extracted from each patient to construct the radiomics signatures possibly us-ing the same dataset after splitting to address multiple target variables, ignoring essential concepts like multiple comparisons corrections and type I errors [26].”
Background
Metadata
- Authors
- Chalkidou, A., O’Doherty, M. J., & Marsden, P. K.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-05-04
- Volume
- 10
- Issue
- 5
- Pages
- e0124165
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W1892105223
- Cited by
- 344
- References
- 61
- FWCI
- 5.09
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.