The DOI matches its Crossref record; 61 of the 61 references it cites were found in a registry or an index.
False Discovery Rates in PET and CT Studies with Texture Features: A Systematic Review
Open accessAbstract
PURPOSE: A number of recent publications have proposed that a family of image-derived indices, called texture features, can predict clinical outcome in patients with cancer. However, the investigation of multiple indices on a single data set can lead to significant inflation of type-I errors. We report a systematic review of the type-I error inflation in such studies and review the evidence regarding associations between patient outcome and texture features derived from positron emission tomography (PET) or computed tomography (CT) images. METHODS: For study identification PubMed and Scopus were searched (1/2000-9/2013) using combinations of the keywords texture, prognostic, predictive and cancer. Studies were divided into three categories according to the sources of the type-I error inflation and the use or not of an independent validation dataset. For each study, the true type-I error probability and the adjusted level of significance were estimated using the optimum cut-off approach correction, and the Benjamini-Hochberg method. To demonstrate explicitly the variable selection bias in these studies, we re-analyzed data from one of the published studies, but using 100 random variables substituted for the original image-derived indices. The significance of the random variables as potential predictors of outcome was examined using the analysis methods used in the identified studies. RESULTS: Fifteen studies were identified. After applying appropriate statistical corrections, an average type-I error probability of 76% (range: 34-99%) was estimated with the majority of published results not reaching statistical significance. Only 3/15 studies used a validation dataset. For the 100 random variables examined, 10% proved to be significant predictors of survival when subjected to ROC and multiple hypothesis testing analysis. CONCLUSIONS: We found insufficient evidence to support a relationship between PET or CT texture features and patient survival. Further fit for purpose validation of these image-derived biomarkers should be supported by appropriate biological and statistical evidence before their association with patient outcome is investigated in prospective studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Textural Features for Image Classification
Haralick, R. M., Shanmugam, K., & Dinstein, I. (1973) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/tsmc.1973.4309314 · Cited by 22,886
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
A simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655
Intratumor Heterogeneity and Branched Evolution Revealed by Multiregion Sequencing
Gerlinger, M., Rowan, A. J., Horswell, S., Larkin, J., Endesfelder, D., Gronroos, E., Martinez, P., Matthews, N., Stewart, A., Tarpey, P., et al. (2012) · New England Journal of Medicine · doi:10.1056/nejmoa1113205 · Cited by 7,774 · cited 3×
“Cancer heterogeneity is a phenomenon associated with clonal branch evolution (genetic variability) and regional differences in the tumour microenvironment (non-genetic variability) [4,5].”
BackgroundNowell, P. C. (1976) · Science · doi:10.1126/science.959840 · Cited by 6,471
Statistical and structural approaches to texture
Haralick, R. M. (1979) · Proceedings of the IEEE · doi:10.1109/proc.1979.11328 · Cited by 5,759
Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach
Aerts, H. J. W. L., Velazquez, E. R., Leijenaar, R. T. H., Parmar, C., Grossmann, P., Carvalho, S., Bussink, J., Monshouwer, R., Haibe-Kains, B., Rietveld, D., et al. (2014) · Nature Communications · doi:10.1038/ncomms5006 · Cited by 5,132
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“A number of contributing factors that in general add to the probability of a research finding being false are listed in [24].”
BackgroundFrom RECIST to PERCIST: Evolving Considerations for PET Response Criteria in Solid Tumors
Wahl, R. L., Jacene, H., Kasamon, Y., & Lodge, M. A. (2009) · Journal of Nuclear Medicine · doi:10.2967/jnumed.108.057307 · Cited by 3,777
Babyak, M. A. (2004) · Psychosomatic Medicine · doi:10.1097/01.psy.0000127692.23278.a9 · Cited by 2,061
Textons, the elements of texture perception, and their interactions
Julesz, B. (1981) · Nature · doi:10.1038/290091a0 · Cited by 1,703
Increasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597
The patterns and dynamics of genomic instability in metastatic pancreatic cancer
Campbell, P. J., Yachida, S., Mudie, L. J., Stephens, P. J., Pleasance, E. D., Stebbings, L. A., Morsberger, L. A., Latimer, C., McLaren, S., Lin, M.-L., et al. (2010) · Nature · doi:10.1038/nature09460 · Cited by 1,317 · cited 1×
“It is widely accepted that biological tumour heterogeneity is associated with poor prognosis in cancer patients as it can contribute to treatment failure and drug resistance, and this has important consequences for personalized-medicine [4,46,47].”
Background(1996) · PubMed · Cited by 1,188
Textural features corresponding to textural properties
Amadasun, M., & King, R. (1989) · IEEE Transactions on Systems, Man, and Cybernetics · doi:10.1109/21.44046 · Cited by 1,148
Dangers of Using "Optimal" Cutpoints in the Evaluation of Prognostic Factors
Altman, D. G., Lausen, B., Sauerbrei, W., & Schumacher, M. (1994) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/86.11.829 · Cited by 1,143
Model uncertainty, data mining and statistical inference
Chatfield, C. (1995) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2983440 · Cited by 1,103
Tumor hypoxia adversely affects the prognosis of carcinoma of the head and neck
Brizel, D. M., Sibley, G. S., Prosnitz, L. R., Scher, R. L., & Dewhirst, M. W. (1997) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/s0360-3016(97)00101-6 · Cited by 1,068
Tixier, F., Le Rest, C. C., Hatt, M., Albarghach, N., Pradier, O., Metges, J.-P., Corcos, L., & Visvikis, D. (2011) · Journal of Nuclear Medicine · doi:10.2967/jnumed.110.082404 · Cited by 696
Bivariate median splits and spurious statistical significance.
Maxwell, S. E., & Delaney, H. D. (1993) · Psychological Bulletin · doi:10.1037//0033-2909.113.1.181 · Cited by 665
Berghmans, T., Dusart, M., Paesmans, M., Hossein-Foucher, C., Buvat, I., Castaigne, C., Scherpereel, A., Mascaux, C., Moreau, M., Roelandts, M., et al. (2008) · Journal of Thoracic Oncology · doi:10.1097/jto.0b013e31815e6d6b · Cited by 524
Quantitative Imaging in Cancer Evolution and Ecology
Gatenby, R. A., Grove, O., & Gillies, R. J. (2013) · Radiology · doi:10.1148/radiol.13122697 · Cited by 515
Ganeshan, B., Panayiotou, E., Burnand, K., Dizdarevic, S., & Miles, K. (2012) · European Radiology · doi:10.1007/s00330-011-2319-8 · Cited by 505
Exploring feature-based approaches in PET images for predicting cancer treatment outcomes
El Naqa, I., Grigsby, P. W., Apte, A., Kidd, E., Donnelly, E., Khullar, D., Chaudhari, S., Yang, D., Schmitt, M., Laforest, R., et al. (2009) · Pattern Recognition · doi:10.1016/j.patcog.2008.08.011 · Cited by 480
Ng, F., Ganeshan, B., Kozarski, R., Miles, K. A., & Goh, V. (2013) · Radiology · doi:10.1148/radiol.12120254 · Cited by 432
Leijenaar, R. T. H., Carvalho, S., Velazquez, E. R., van Elmpt, W. J. C., Parmar, C., Hoekstra, O. S., Hoekstra, C. J., Boellaard, R., Dekker, A. L. A. J., Gillies, R. J., et al. (2013) · Acta Oncologica · doi:10.3109/0284186x.2013.812798 · Cited by 427
Spatial and Temporal Heterogeneity in High-Grade Serous Ovarian Cancer: A Phylogenetic Analysis
Schwarz, R. F., Ng, C. K. Y., Cooke, S. L., Newman, S., Temple, J., Piskorz, A. M., Gale, D., Sayal, K., Murtaza, M., Baldwin, P. J., et al. (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001789 · Cited by 400
Cook, G. J. R., Yip, C., Siddique, M., Goh, V., Chicklore, S., Roy, A., Marsden, P., Ahmad, S., & Landau, D. (2013) · Journal of Nuclear Medicine · doi:10.2967/jnumed.112.107375 · Cited by 394
Self-citationGoh, V., Ganeshan, B., Nathan, P., Juttla, J. K., Vinayan, A., & Miles, K. A. (2011) · Radiology · doi:10.1148/radiol.11110264 · Cited by 367
Orlhac, F., Soussan, M., Maisonobe, J.-A., Garcia, C. A., Vanderlinden, B., & Buvat, I. (2014) · Journal of Nuclear Medicine · doi:10.2967/jnumed.113.129858 · Cited by 356
Galavis, P. E., Hollensen, C., Jallow, N., Paliwal, B., & Jeraj, R. (2010) · Acta Oncologica · doi:10.3109/0284186x.2010.498437 · Cited by 344
Ganeshan, B., Skogen, K., Pressney, I., Coutroubis, D., & Miles, K. (2012) · Clinical Radiology · doi:10.1016/j.crad.2011.08.012 · Cited by 310
Reproducibility and Prognosis of Quantitative Features Extracted from CT Images
Balagurunathan, Y., Gu, Y., Wang, H., Kumar, V., Grove, O., Hawkins, S., Kim, J., Goldgof, D. B., Hall, L. O., Gatenby, R. A., et al. (2014) · Translational Oncology · doi:10.1593/tlo.13844 · Cited by 306
Miles, K. A., Ganeshan, B., Griffiths, M. R., Young, R. C. D., & Chatwin, C. R. (2009) · Radiology · doi:10.1148/radiol.2502071879 · Cited by 253
Three-dimensional solid texture analysis in biomedical imaging: Review and opportunities
Depeursinge, A., Foncubierta-Rodriguez, A., Van De Ville, D., & Müller, H. (2014) · Medical Image Analysis · doi:10.1016/j.media.2013.10.005 · Cited by 244
Metastatic progression of breast cancer: insights from 50 years of autopsies
Cummings, M. C., Simpson, P. T., Reid, L. E., Jayanthan, J., Skerman, J., Song, S., McCart Reed, A. E., Kutasovic, J. R., Morey, A. L., Marquart, L., et al. (2014) · The Journal of Pathology · doi:10.1002/path.4288 · Cited by 210
Hatt, M., Tixier, F., Cheze Le Rest, C., Pradier, O., & Visvikis, D. (2013) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-013-2486-8 · Cited by 208
Validation of Biomarker-Based Risk Prediction Models
Taylor, J. M. G., Ankerst, D. P., & Andridge, R. R. (2008) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-07-4534 · Cited by 207
Zhang, H., Graham, C. M., Elci, O., Griswold, M. E., Zhang, X., Khan, M. A., Pitman, K., Caudell, J. J., Hamilton, R. D., Ganeshan, B., et al. (2013) · Radiology · doi:10.1148/radiol.13130110 · Cited by 207
Yip, C., Landau, D., Kozarski, R., Ganeshan, B., Thomas, R., Michaelidou, A., & Goh, V. (2014) · Radiology · doi:10.1148/radiol.13122869 · Cited by 204
Win, T., Miles, K. A., Janes, S. M., Ganeshan, B., Shastry, M., Endozo, R., Meagher, M., Shortman, R. I., Wan, S., Kayani, I., et al. (2013) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-12-1307 · Cited by 203
Combined PET/CT image characteristics for radiotherapy tumor response in lung cancer
Vaidya, M., Creach, K. M., Frye, J., Dehdashti, F., Bradley, J. D., & El Naqa, I. (2012) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2011.10.014 · Cited by 201
The Effect of Small Tumor Volumes on Studies of Intratumoral Heterogeneity of Tracer Uptake
Brooks, F. J., & Grigsby, P. W. (2013) · Journal of Nuclear Medicine · doi:10.2967/jnumed.112.116715 · Cited by 170
Tan, S., Kligerman, S. J., Chen, W., Lu, M., Kim, G., Feigenberg, S. J., D'Souza, W. D., Suntharalingam, M., & Lü, W. (2012) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2012.10.017 · Cited by 155
Cheng, N.-M., Fang, Y.-H. D., Chang, J. T., Huang, C., Tsan, D., Ng, S., Wang, H., Lin, C., Liao, C.-T., & Yen, T. (2013) · Journal of Nuclear Medicine · doi:10.2967/jnumed.112.119289 · Cited by 143
Austin, P. C., Mamdani, M., Juurlink, D. N., & Hux, J. E. (2006) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.01.012 · Cited by 142
Why do so many prognostic factors fail to pan out?
Hilsenbeck, S. G., Clark, G. M., & McGuire, W. L. (1992) · Breast Cancer Research and Treatment · doi:10.1007/bf01840833 · Cited by 138
Ravanelli, M., Farina, D., Morassi, M., Roca, E., Cavalleri, G., Tassi, G., & Maroldi, R. (2013) · European Radiology · doi:10.1007/s00330-013-2965-0 · Cited by 122
PRACTICAL p-VALUE ADJUSTMENT FOR OPTIMALLY SELECTED CUTPOINTS
Hilsenbeck, S. G., & Clark, G. M. (1996) · Statistics in Medicine · Cited by 107
Interpretation of microarray data in cancer
Michiels, S., Koscielny, S., & Hill, C. (2007) · British Journal of Cancer · doi:10.1038/sj.bjc.6603673 · Cited by 103
A comparison of various methods for multivariate regression with highly collinear variables
Kiers, H. A. L., & Smilde, A. K. (2006) · Statistical Methods & Applications · doi:10.1007/s10260-006-0025-5 · Cited by 99
Zugazagoitia, J., Enguita, A. B., Nuñez, J. A., Iglesias, L., & Ponce, S. (2014) · Europe PMC (PubMed Central) · doi:10.3978/j.issn.2072-1439.2014.01.27 · Cited by 95
Kalpathy–Cramer, J., Freymann, J., Kirby, J., Kinahan, P. E., & Prior, F. (2014) · Translational Oncology · doi:10.1593/tlo.13862 · Cited by 92
Willaime, J. M. Y., Turkheimer, F., Kenny, L. M., & Aboagye, E. O. (2012) · Physics in Medicine and Biology · doi:10.1088/0031-9155/58/2/187 · Cited by 89
Predicting Future Morphological Changes of Lesions from Radiotracer Uptake in 18F-FDG-PET Images
Bağcı, U., Yao, J., Miller-Jaster, K., Chen, X., & Mollura, D. J. (2013) · PLoS ONE · doi:10.1371/journal.pone.0057105 · Cited by 38
Gensure, R. H., Foran, D. J., Lee, V. M., et al. (2012) · Academic Radiology · doi:10.1016/j.acra.2012.04.015 · Cited by 20
What do we mean by validating a prognostic model?
(2000) · doi:10.1002/(sici)1097-0258(20000229)19:4<453::aid-sim350>3.0.co;2-5
Cited by
Radiomics in medical imaging—“how-to” guide and critical reflection
van Timmeren, J. E., Cester, D., Tanadini-Lang, S., Alkadhi, H., & Baessler, B. (2020) · Insights into Imaging · doi:10.1186/s13244-020-00887-2 · Cited by 1,316
Applications and limitations of radiomics
Yip, S. S. F., & Aerts, H. J. W. L. (2016) · Physics in Medicine and Biology · doi:10.1088/0031-9155/61/13/r150 · Cited by 1,210
Nioche, C., Orlhac, F., Boughdad, S., Reuzé, S., Goya-Outi, J., Robert, C., Pellot-Barakat, C., Soussan, M., Frouin, F., & Buvat, I. (2018) · Cancer Research · doi:10.1158/0008-5472.can-18-0125 · Cited by 1,167
Repeatability and Reproducibility of Radiomic Features: A Systematic Review
Traverso, A., Wee, L., Dekker, A., & Gillies, R. (2018) · International Journal of Radiation Oncology*Biology*Physics · doi:10.1016/j.ijrobp.2018.05.053 · Cited by 903
CT Texture Analysis: Definitions, Applications, Biologic Correlates, and Challenges
Lubner, M. G., Smith, A. D., Sandrasegaran, K., Sahani, D. V., & Pickhardt, P. J. (2017) · RadioGraphics · doi:10.1148/rg.2017170056 · Cited by 773
Metadata
- Authors
- Chalkidou, A., O’Doherty, M. J., & Marsden, P. K.
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-05-04
- Volume
- 10
- Issue
- 5
- Pages
- e0124165
- ISSN
- 1932-6203 (ISSN-L)
- OpenAlex
- W1892105223
- Cited by
- 344
- References
- 61
- FWCI
- 5.09
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.