The DOI matches its Crossref record; 125 of the 126 references it cites were found in a registry or an index.
Methodological conduct of prognostic prediction models developed using machine learning in oncology: a systematic review
Open accessAbstract
BACKGROUND: Describe and evaluate the methodological conduct of prognostic prediction models developed using machine learning methods in oncology. METHODS: We conducted a systematic review in MEDLINE and Embase between 01/01/2019 and 05/09/2019, for studies developing a prognostic prediction model using machine learning methods in oncology. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement, Prediction model Risk Of Bias ASsessment Tool (PROBAST) and CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) to assess the methodological conduct of included publications. Results were summarised by modelling type: regression-, non-regression-based and ensemble machine learning models. RESULTS: Sixty-two publications met inclusion criteria developing 152 models across all publications. Forty-two models were regression-based, 71 were non-regression-based and 39 were ensemble models. A median of 647 individuals (IQR: 203 to 4059) and 195 events (IQR: 38 to 1269) were used for model development, and 553 individuals (IQR: 69 to 3069) and 50 events (IQR: 17.5 to 326.5) for model validation. A higher number of events per predictor was used for developing regression-based models (median: 8, IQR: 7.1 to 23.5), compared to alternative machine learning (median: 3.4, IQR: 1.1 to 19.1) and ensemble models (median: 1.7, IQR: 1.1 to 6). Sample size was rarely justified (n = 5/62; 8%). Some or all continuous predictors were categorised before modelling in 24 studies (39%). 46% (n = 24/62) of models reporting predictor selection before modelling used univariable analyses, and common method across all modelling types. Ten out of 24 models for time-to-event outcomes accounted for censoring (42%). A split sample approach was the most popular method for internal validation (n = 25/62, 40%). Calibration was reported in 11 studies. Less than half of models were reported or made available. CONCLUSIONS: The methodological conduct of machine learning based clinical prediction models is poor. Guidance is urgently needed, with increased awareness and education of minimum prediction modelling standards. Particular focus is needed on sample size estimation, development and validation analysis methods, and ensuring the model is available for independent validation, to improve quality of machine learning based clinical prediction models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Unfortunately, calibration measures are still under-reported.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [94]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
Alcantud, J. C. R., Varela, G., Santos-Buitrago, B., Santos‐García, G., & Jiménez, M. F. (2019) · PLoS ONE · doi:10.1371/journal.pone.0218283 · Cited by 41
RetractedLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2008.08.010 · Cited by 51,384
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,762
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955
Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., Wood, A. M., & Carpenter, J. R. (2009) · BMJ · doi:10.1136/bmj.b2393 · Cited by 7,299
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878
Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project
Conroy, R. (2003) · European Heart Journal · doi:10.1016/s0195-668x(03)00114-3 · Cited by 5,290
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationStatistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340
The APACHE III Prognostic System
Knaus, W. A., Wagner, D. P., Draper, E. A., Zimmerman, J. E., Bergner, M., BASTOS, P. G., Sirio, C., Murphy, D. J., Lotring, T., Damiano, A., et al. (1991) · CHEST Journal · doi:10.1378/chest.100.6.1619 · Cited by 3,539
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
PROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group† (2019) · Annals of Internal Medicine · doi:10.7326/m18-1376 · Cited by 2,984
Self-citationNashef, S. A. M., Roques, F., Sharples, L. D., Nilsson, J., Smith, C., Goldstone, A. R., & Lockowandt, U. (2012) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezs043 · Cited by 2,971
Calculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917 · cited 1×
“Use of smaller datasets for non-regression and ensemble machine learning models is problematic and increases their risk of overfitting further due to increased flexibility and categorisation of prediction inherent to many machine learning methods [116, 117].”
BackgroundChristodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.02.004 · Cited by 2,119
Self-citationMoons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001744 · Cited by 2,018
Self-citationHippisley-Cox, J., Coupland, C., & Brindle, P. (2017) · BMJ · doi:10.1136/bmj.j2099 · Cited by 1,703
Moons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
Self-citationA Preoperative Nomogram for Disease Recurrence Following Radical Prostatectomy for Prostate Cancer
Kattan, M. W., Eastham, J. A., Stapleton, A. M. F., Wheeler, T. M., & Scardino, P. T. (1998) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/90.10.766 · Cited by 1,243
Machine Learning and Prediction in Medicine — Beyond the Peak of Inflated Expectations
Chen, J. H., & Asch, S. M. (2017) · New England Journal of Medicine · doi:10.1056/nejmp1702071 · Cited by 1,171
Nagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152
Prediction models for cardiovascular disease risk in the general population: systematic review
Damen, J. A. A. G., Hooft, L., Schuit, E., Debray, T. P. A., Collins, G. S., Tzoulaki, I., Lassale, C. M., Siontis, G. C. M., Chiocchia, V., Roberts, C., et al. (2016) · BMJ · doi:10.1136/bmj.i2416 · Cited by 935
Inappropriate use of bivariable analysis to screen risk factors for use in multivariable analysis
Sun, G.-W., Shook, T. L., & Kay, G. L. (1996) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(96)00025-x · Cited by 933
Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints
van der Ploeg, T., Austin, P. C., & Steyerberg, E. W. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-137 · Cited by 809
Reporting of artificial intelligence prediction models
Collins, G. S., & Moons, K. G. M. (2019) · The Lancet · doi:10.1016/s0140-6736(19)30037-6 · Cited by 701
No rationale for 1 variable per 10 events criterion for binary logistic regression analysis
van Smeden, M., de Groot, J. A. H., Moons, K. G. M., Collins, P. G. S., Altman, D. G., Eijkemans, M. J. C., & Reitsma, J. B. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0267-3 · Cited by 613
Reporting and Methods in Clinical Prediction Research: A Systematic Review
Bouwmeester, W., Zuithoff, N. P. A., Mallett, S., Geerlings, M. I., Vergouwe, Y., Steyerberg, E. W., Altman, D. G., & Moons, K. G. M. (2012) · PLoS Medicine · doi:10.1371/journal.pmed.1001221 · Cited by 600 · cited 2×
“However, compared to the number of prediction models that are developed, very few are used in clinical practice and many models contribute to research waste [14–17].”
BackgroundGroenwold, R. H. H., White, I. R., Donders, A. R. T., Carpenter, J. R., Altman, D. G., & Moons, K. G. M. (2012) · Canadian Medical Association Journal · doi:10.1503/cmaj.110977 · Cited by 538
Self-citationCollins, G. S., Mallett, S., Omar, O., & Yu, L.-M. (2011) · BMC Medicine · doi:10.1186/1741-7015-9-103 · Cited by 484
Andaur Navarro, C. L., Damen, J. A. A., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., Moons, K. G. M., et al. (2021) · BMJ · doi:10.1136/bmj.n2281 · Cited by 415
Deep learning-based survival prediction of oral cancer patients
Kim, D. W., Lee, S., Kwon, S., Nam, W., Cha, I., & Kim, H. J. (2019) · Scientific Reports · doi:10.1038/s41598-019-43372-7 · Cited by 324
Song, X., Liu, X., Liu, F., & Wang, C. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104484 · Cited by 301
Balki, I., Amirabadi, A., Levman, J., Martel, A. L., Emersic, Z., Meden, B., Garcia-Pedrero, A., Ramirez, S. C., Kong, D., Moody, A. R., et al. (2019) · Canadian Association of Radiologists Journal · doi:10.1016/j.carj.2019.06.002 · Cited by 299
Use of machine learning to analyse routinely collected intensive care unit data: a systematic review
Shillan, D., Sterne, J. A. C., Champneys, A., & Gibbison, B. (2019) · Critical Care · doi:10.1186/s13054-019-2564-9 · Cited by 251
Prognostic models for breast cancer: a systematic review
Phung, M. T., Tin, S. T., & Elwood, M. (2019) · BMC Cancer · doi:10.1186/s12885-019-5442-6 · Cited by 214
Reporting methods in studies developing prognostic models in cancer: a review
Mallett, S., Royston, P., Dutton, S., Waters, R., & Altman, D. G. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-20 · Cited by 206
Wang, W., Kiik, M., Peek, N. B., Ćurčin, V., Marshall, I. J., Rudd, A. G., Wang, Y., Douiri, A., Wolfe, C. D. A., & Bray, B. (2020) · PLoS ONE · doi:10.1371/journal.pone.0234722 · Cited by 205
Collins, G. S., Omar, O., Shanyinde, M., & Yu, L.-M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.06.020 · Cited by 192
Knol, M. J., Janssen, K. J. M., Donders, A. R. T., Egberts, T. C. G., Heerdink, E. R., Grobbee, D. E., Moons, K. G. M., & Geerlings, M. I. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.08.028 · Cited by 190
Collins, G. S., Ogundimu, E. O., Cook, J. A., Manach, Y. L., & Altman, D. G. (2016) · Statistics in Medicine · doi:10.1002/sim.6986 · Cited by 185
Survival outcome prediction in cervical cancer: Cox models vs deep-learning model
Matsuo, K., Purushotham, S., Jiang, B., Mandelbaum, R. S., Takiuchi, T., Liu, Y., & Roman, L. D. (2018) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2018.12.030 · Cited by 183
Risk Prediction Models for Colorectal Cancer: A Systematic Review
Usher‐Smith, J. A., Walter, F. M., Emery, J. D., Win, A. K., & Griffin, S. J. (2015) · Cancer Prevention Research · doi:10.1158/1940-6207.capr-15-0274 · Cited by 182
Karhade, A., Thio, Q., Ogink, P., Bono, C. M., Ferrone, M., Oh, K., Saylor, P., Schoenfeld, A. J., Shin, J. H., Harris, M. B., et al. (2019) · Neurosurgery · doi:10.1093/neuros/nyz070 · Cited by 179
Mushkudiani, N. A., Hukkelhoven, C. W. P. M., Hernández, A. V., Murray, G. D., Choi, S. C., Maas, A. I. R., & Steyerberg, E. W. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.06.011 · Cited by 177
Hippisley‐Cox, J., & Coupland, C. A. C. (2015) · BMJ Open · doi:10.1136/bmjopen-2015-007825 · Cited by 177
Karhade, A., Thio, Q., Ogink, P., Shah, A. A., Bono, C. M., Oh, K., Saylor, P., Schoenfeld, A. J., Shin, J. H., Harris, M. B., et al. (2018) · Neurosurgery · doi:10.1093/neuros/nyy469 · Cited by 151
Predicting Breast Cancer Recurrence Using Machine Learning Techniques
Abreu, P. H., Santos, M. S., Abreu, M. H., Andrade, B., & Silva, D. C. (2016) · ACM Computing Surveys · doi:10.1145/2988544 · Cited by 142
Thamer, M., Kaufman, J. S., Zhang, Y., Zhang, Q., Cotter, D. J., & Bang, H. (2015) · American Journal of Kidney Diseases · doi:10.1053/j.ajkd.2015.05.014 · Cited by 140
Sauerbrei, W., Boulesteix, A., & Binder, H. (2011) · Journal of Biopharmaceutical Statistics · doi:10.1080/10543406.2011.629890 · Cited by 134
Merath, K., Hyer, J. M., Mehta, R., Farooq, A., Bagante, F., Sahara, K., Tsilimigras, D. I., Beal, E., Paredes, A. Z., Wu, L., et al. (2020) · Journal of Gastrointestinal Surgery · doi:10.1007/s11605-019-04338-2 · Cited by 131
Predicting breast cancer 5-year survival using machine learning: A systematic review
Li, J., Zhou, Z., Dong, J., Fu, Y., Li, Y., Luan, Z., & Peng, X. (2021) · PLoS ONE · doi:10.1371/journal.pone.0250370 · Cited by 128
Tseng, Y., Huang, C.-E., Wen, C.-N., Lai, P.-Y., Wu, M., Sun, Y.-C., Wang, H., & Lu, J. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.05.003 · Cited by 121
Karadaghy, O. A., Shew, M. A., New, J., & Bur, A. M. (2019) · JAMA Otolaryngology–Head & Neck Surgery · doi:10.1001/jamaoto.2019.0981 · Cited by 120
Taninaga, J., Nishiyama, Y., Fujibayashi, K., Gunji, T., Sasabe, N., Iijima, K., & Naito, T. (2019) · Scientific Reports · doi:10.1038/s41598-019-48769-y · Cited by 120
Dhiman, P., Ma, J., Navarro, C. L. A., Speich, B., Bullock, G. S., Damen, J. A., Kirtley, S., Hooft, L., Riley, R. D., CALSTER, B. V., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.06.024 · Cited by 113
Alabi, R. O., Elmusrati, M. S., Sawazaki‐Calone, Í., Kowalski, L. P., Haglund, C. H., Coletta, R. D., Mäkitie, A. A., Salo, T. A., Leivo, I. V. V., & Almangush, A. (2019) · Archiv für Pathologische Anatomie und Physiologie und für Klinische Medicin · doi:10.1007/s00428-019-02642-5 · Cited by 109
Heus, P., Damen, J. A., Pajouheshnia, R., Scholten, R. J. P. M., Reitsma, J. B., Collins, P. G. S., Altman, D. G., Moons, K. G. M., & Hooft, L. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-025611 · Cited by 108
Fahey, M., Crayton, E., Wolfe, C. D. A., & Douiri, A. (2018) · PLoS ONE · doi:10.1371/journal.pone.0185402 · Cited by 102
Use of machine learning to predict early biochemical recurrence after robot‐assisted prostatectomy
Wong, N. C., Lam, C. J., Patterson, L., & Shayegan, B. (2018) · British Journal of Urology · doi:10.1111/bju.14477 · Cited by 101
Luna, J. M., Chao, H., Diffenderfer, E. S., Valdés, G., Chinniah, C., Grace, X., Cengel, K. A., Solberg, T. D., Berman, A. T., & Simone, C. B. (2019) · Radiotherapy and Oncology · doi:10.1016/j.radonc.2019.01.003 · Cited by 101
Extent of Resection in Meningioma: Predictive Factors and Clinical Implications
D, J.-M. L. M., Corniola, M. V., Broi, M. D., Joswig, H., Scheie, D., Schaller, K., Helseth, E., & Meling, T. R. (2019) · Scientific Reports · doi:10.1038/s41598-019-42451-z · Cited by 96
Muhlestein, W. E., Akagi, D. S., Davies, J. M., & Chambless, L. B. (2018) · Neurosurgery · doi:10.1093/neuros/nyy343 · Cited by 89
A risk score for predicting perioperative blood transfusion in liver surgery
Pulitanò, C., Arru, M., Bellio, L., Rossini, S., Ferla, G., & Aldrighetti, L. A. (2007) · British journal of surgery · doi:10.1002/bjs.5731 · Cited by 86
Banerjee, A., Chen, S., Fatemifar, G., Zeina, M., Lumbers, R. T., Mielke, J., Gill, S. K., Kotecha, D., Freitag, D. F., Denaxas, S., et al. (2021) · BMC Medicine · doi:10.1186/s12916-021-01940-7 · Cited by 80
Shung, D. L., Simonov, M., Gentry, M., Au, B., & Laine, L. (2019) · Digestive Diseases and Sciences · doi:10.1007/s10620-019-05645-z · Cited by 79
Sahle, B. W., Owen, A. J., Chin, K. L., & Reid, C. M. (2017) · Journal of Cardiac Failure · doi:10.1016/j.cardfail.2017.03.005 · Cited by 78
Paik, E. S., Lee, J., Park, J., Kim, J.-H., Kim, M., Kim, T., Choi, C. H., Kim, B., Bae, D., & Seo, S. W. (2019) · Journal of Gynecologic Oncology · doi:10.3802/jgo.2019.30.e65 · Cited by 75
Wang, H.-H., Wang, Y.-H., Liang, C.-W., & Li, Y. (2019) · JAMA Dermatology · doi:10.1001/jamadermatol.2019.2335 · Cited by 75
Markaki, M., Tsamardinos, I., Langhammer, A., Lagani, V., Hveem, K., & Røe, O. D. (2018) · EBioMedicine · doi:10.1016/j.ebiom.2018.03.027 · Cited by 71
Fong, Y., Evans, J., Brook, D., Kenkre, J. E., Jarvis, P., & Gower-Thomas, K. (2015) · Annals of The Royal College of Surgeons of England · doi:10.1308/003588414x14055925060514 · Cited by 66
Wang, X., Zhang, Y., Hao, S., Zheng, L., Liao, J., Ye, C., Xia, M., Wang, O., Liu, M., Weng, C. H., et al. (2019) · Journal of Medical Internet Research · doi:10.2196/13260 · Cited by 62
Auffenberg, G. B., Ghani, K. R., Ramani, S., Usoro, E., Denton, B. T., ROGERS, C. G., Stockton, B. R., Miller, D. C., & Singh, K. (2018) · European Urology · doi:10.1016/j.eururo.2018.09.050 · Cited by 61
Khalaf, M. H., H, V. S. M. P., Abdelrazek, M. A., Shah, R. P., Khosla, A., Jackson, K., Desai, M., & Kothary, N. (2018) · Radiology · doi:10.1148/radiol.2018180257 · Cited by 58
Use of the Gail Model and Breast Cancer Preventive Therapy Among Three Primary Care Specialties
Corbelli, J. A., Borrero, S. B., Bonnema, R. A., McNamara, M., Kraemer, K. L., Rubio, D. M., Karpov, I. O., & McNeil, M. (2014) · Journal of Women s Health · doi:10.1089/jwh.2014.4742 · Cited by 57
Lebrett, M. B., Balata, H., Evison, M., Colligan, D., Duerden, R., Elton, P., Greaves, M., Howells, J., Irion, K., Karunaratne, D., et al. (2020) · Thorax · doi:10.1136/thoraxjnl-2020-214626 · Cited by 57
A novel prediction method for lymph node involvement in endometrial cancer: machine learning
Günakan, E., Suat, A. T. A. N., Haberal, N., Küçükyıldız, İ. A., Gökçe, E., & Ayhan, A. (2019) · International Journal of Gynecological Cancer · doi:10.1136/ijgc-2018-000033 · Cited by 51
Oyaga-Iriarte, E., Insausti, A., Sayar, O., & Aldaz, A. (2019) · Journal of Pharmacological Sciences · doi:10.1016/j.jphs.2019.03.004 · Cited by 51
van Niftrik, C. H. B., van der Wouden, F., Staartjes, V. E., Fierstra, J., Stienen, M. N., Akeret, K., Sebök, M., Fedele, T., Sarnthein, J., Bozinov, O., et al. (2019) · Neurosurgery · doi:10.1093/neuros/nyz145 · Cited by 50
Nartowt, B. J., Hart, G. R., Roffman, D. A., Llor, X., Ali, I. M., Muhammad, W., Liang, Y., & Deng, J. (2019) · PLoS ONE · doi:10.1371/journal.pone.0221421 · Cited by 50
Xu, Y., Kong, S., Cheung, W. Y., Bouchard‐Fortier, A., Dort, J. C., Quan, H., Buie, E. M., McKinnon, G., & Quan, M. L. (2019) · BMC Cancer · doi:10.1186/s12885-019-5432-8 · Cited by 46
Artificial neural network models to predict nodal status in clinically node-negative breast cancer
Dihge, L., Ohlsson, M., Edén, P., Bendahl, P., & Rydén, L. (2019) · BMC Cancer · doi:10.1186/s12885-019-5827-6 · Cited by 44
Development of Deep Learning Algorithm for Detection of Colorectal Cancer in EHR Data
Yu-Hsiang, W., Phung-Anh, N., Mohaimenul, I. M., Li, Y., & Yang, H. (2019) · Studies in health technology and informatics · doi:10.3233/shti190259 · Cited by 41
Semi-supervised learning to improve generalizability of risk prediction models
Chi, S., Li, X., Tian, Y., Li, J., Kong, X., Ding, K., Weng, C., & Li, J. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103117 · Cited by 40
Yan, P., Huang, R., Hu, P., Liu, F., Zhu, X., Hu, P., Yin, H., Zhang, J., Meng, T., & Huang, Z. (2019) · Journal of Neuro-Oncology · doi:10.1007/s11060-019-03181-4 · Cited by 40
Ryu, S. M., Lee, S.-H., Kim, E., & Eoh, W. (2018) · World Neurosurgery · doi:10.1016/j.wneu.2018.12.091 · Cited by 39
Creating Prognostic Systems for Well-Differentiated Thyroid Cancer Using Machine Learning
Yang, C., Gardiner, L. A., Wang, H., Hueman, M. T., & Chen, D. (2019) · Frontiers in Endocrinology · doi:10.3389/fendo.2019.00288 · Cited by 39
Kaviarasi, R., & Gandhi Raj, R. (2019) · Journal of Medical Systems · doi:10.1007/s10916-019-1297-2 · Cited by 39
Elarre, P. S., Oyaga-Iriarte, E., Yu, K. H., Baudin, V., Moreno, L. A., Carranza, O. E., Ortega, A. C., Ponz‐Sarvisé, M., Sosa, L., Rotellar, F., et al. (2019) · Cancers · doi:10.3390/cancers11050606 · Cited by 38
Bridge, J., Blakey, J., & Bonnett, L. J. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-0913-7 · Cited by 38
Fuse, K., Uemura, S., Tamura, S., Suwabe, T., Katagiri, T., Tanaka, T. N., Ushiki, T., Shibasaki, Y., Sato, N., Yano, T., et al. (2019) · Cancer Medicine · doi:10.1002/cam4.2401 · Cited by 34
Tian, Z., Yen, A., Zhou, Z., Shen, C., Albuquerque, K., & Hrycushko, B. A. (2019) · Brachytherapy · doi:10.1016/j.brachy.2019.04.004 · Cited by 32
Collins, S. D., Peek, N. B., Riley, R. D., & Martin, G. P. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.12.011 · Cited by 32
Greene, M. Z., Hughes, T. L., Hanlon, A. L., Huang, L., Sommers, M. S., & Meghani, S. H. (2018) · Preventive Medicine Reports · doi:10.1016/j.pmedr.2018.11.007 · Cited by 30
A prospective study examining cachexia predictors in patients with incurable cancer
Vagnildhaug, O. M., Brunelli, C., Hjermstad, M. J., Strasser, F., Baracos, V. E., Wilcock, A., Nabal, M., Kaasa, S., Laird, B. J. A., & Solheim, T. S. (2019) · BMC Palliative Care · doi:10.1186/s12904-019-0429-2 · Cited by 30
Facciorusso, A., Del Prete, V., Antonino, M., Buccino, V. R., & Muscatiello, N. (2019) · Pancreatology · doi:10.1016/j.pan.2019.07.038 · Cited by 30
Egger, M. E., Stevenson, M. A., Bhutiani, N., Jordan, A. C., Scoggins, C. R., Philips, P., Martin, R. C. G., & McMasters, K. M. (2019) · Annals of Surgical Oncology · doi:10.1245/s10434-019-07690-4 · Cited by 28
Survivability prediction of colon cancer patients using neural networks
Al-Bahrani, R., Agrawal, A., & Choudhary, A. (2017) · Health Informatics Journal · doi:10.1177/1460458217720395 · Cited by 26
Gait speed and survival of older surgical patient with cancer: Prediction after machine learning
Sasani, K., Catanese, H. N., Ghods, A., Rokni, S. A., Ghasemzadeh, H. G., Downey, R. J., & Shahrokni, A. (2018) · Journal of Geriatric Oncology · doi:10.1016/j.jgo.2018.06.012 · Cited by 25
Corniola, M. V., D, J.-M. L. M., Broi, M. D., Joswig, H., Schaller, K., Helseth, E., & Meling, T. R. (2019) · Acta Neurochirurgica · doi:10.1007/s00701-019-03862-z · Cited by 25
Feng, S., Li, H.-B., Fan, F., Li, J., Cao, H., Xia, Z., Yang, K., Zhu, X., Cheng, T., & Cheng, Q. (2019) · Scientific Reports · doi:10.1038/s41598-019-47211-7 · Cited by 25
Can machine learning predict resecability of a peritoneal carcinomatosis?
Maubert, A., Birtwisle, L., Bernard, J., Benizri, E. I., & Bereder, J. M. (2019) · Surgical Oncology · doi:10.1016/j.suronc.2019.04.008 · Cited by 24
Predicting Overall Survival in Patients with Metastatic Rectal Cancer: a Machine Learning Approach
Zhao, B., Gabriel, R. A., Vaida, F., Lopez, N., Eisenstein, S. G., & Clary, B. M. (2019) · Journal of Gastrointestinal Surgery · doi:10.1007/s11605-019-04373-z · Cited by 24
Grigore, B., Lewis, R. A., Peters, J., Robinson, S., & Hyde, C. (2020) · BMC Cancer · doi:10.1186/s12885-020-07572-z · Cited by 23
Bradley, A., Meer, R. V. D., & McKay, C. J. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-027192 · Cited by 19
Li, M., Zhan, C., Sui, X., Jiang, W., Shi, Y., Yang, X., Feng, M., Wang, J., & Wang, Q. (2019) · Frontiers in Oncology · doi:10.3389/fonc.2019.00771 · Cited by 16
Tighe, D. F., Lewis-Morris, T., & Freitas, A. A. (2019) · British Journal of Oral and Maxillofacial Surgery · doi:10.1016/j.bjoms.2019.05.026 · Cited by 16
Velázquez, N., Press, B. H., Renson, A., Wysock, J. S., Taneja, S. S., Huang, W. C. W., & Bjurlin, M. A. (2018) · Clinical Genitourinary Cancer · doi:10.1016/j.clgc.2018.09.018 · Cited by 15
Beachler, D. C., de Luise, C., Yin, R., Gangemi, K., Cochetti, P. T., & Lanes, S. (2018) · Pharmacoepidemiology and Drug Safety · doi:10.1002/pds.4681 · Cited by 15
Obrzut, B., Kusy, M., Semczuk, A., Obrzut, M., & Kluska, J. (2019) · Journal of Cancer · doi:10.7150/jca.33945 · Cited by 14
Hammer, J., Geinitz, H., Nieder, C., Track, C., Thames, H. D., Seewald, D. H., Petzer, A. L., Helfgott, R., Spiegl, K. J., Heck, D., et al. (2018) · Clinical Breast Cancer · doi:10.1016/j.clbc.2018.08.001 · Cited by 13
Lindsay, W. D., Ahern, C. A., Tobias, J. S., Berlind, C. G., Chinniah, C., Gabriel, P. E., Gee, J. C., & Simone, C. B. (2018) · Medical Physics · doi:10.1002/mp.13314 · Cited by 11
Yang, X., Wang, F., Feng, J., Hu, Y., Lun, D., Hua, K., Liu, Y.-H., Zhang, H., & Yang, L. (2019) · World Neurosurgery · doi:10.1016/j.wneu.2019.02.183 · Cited by 9
Zheng, B., Lin, J., Li, Y., Zhuo, X., Huang, X., Shen, Q., & Tang, Y. (2019) · Supportive Care in Cancer · doi:10.1007/s00520-019-04699-z · Cited by 7
Iraji, M. S. (2019) · Journal of Applied Biomedicine · doi:10.32725/jab.2018.007 · Cited by 7
Thapa, S., Fischbach, L. A., Delongchamp, R. R., Faramawi, M. F., & Orloff, M. S. (2019) · Gastroenterology Research and Practice · doi:10.1155/2019/8321942 · Cited by 7
Mahmoudian, M., Seyednasrollah, F. S., Koivu, L., Hirvonen, O., Jyrkkiö, S. M., & Elo, L. L. (2019) · F1000Research · doi:10.12688/f1000research.8192.2 · Cited by 5
Zhou, H.-F., Lü, J., Zhu, H., Guo, J., Huang, M. X., Ji, J., Lv, W., Li, Y., Xu, H., Chen, L., et al. (2019) · CardioVascular and Interventional Radiology · doi:10.1007/s00270-019-02331-5 · Cited by 5
Sim, J.-A., & Yun, Y. H. (2019) · Studies in health technology and informatics · doi:10.3233/shti190548 · Cited by 5
Mahmoudian, M., Seyednasrollah, F. S., Koivu, L., Hirvonen, O., Jyrkkiö, S. M., & Elo, L. L. (2016) · F1000Research · doi:10.12688/f1000research.8192.1 · Cited by 5
Xu, Y., Kong, S., Cheung, W. Y., Quan, M. L., Nakoneshny, S. C., & Dort, J. C. (2019) · Head & Neck · doi:10.1002/hed.25682 · Cited by 4
Research works printed but not linked: 4
- The Endnote Team. Endnote. Philadelphia: Clarivate Analytics; 2013.
- (4) Cited 2020 Feb 12. Available from: https://bmjopen.bmj.com/content/9/4/e025611.
- StataCorp. Stata Statistical Software: Release 15. College Station: StataCorp LLC; 2017.
- (2) Cited 2021 Mar 5. Available from: https://ijgc.bmj.com/content/29/2/320.
Cited by
Sorted by citations
Collins, G. S., Moons, K. G. M., Dhiman, P., Riley, R. D., Beam, A. L., Van Calster, B., Ghassemi, M., Liu, X., Reitsma, J. B., van Smeden, M., et al. (2024) · BMJ · doi:10.1136/bmj-2023-078378 · Cited by 4,718
“…models have shown that they are often poorly conducted (including deficiencies in study design or data collection 37 38 ); use poor methodology 37 38 ; are incompletely reported with key details missing 39-54 ; are consequently at high risk of bias 41 49 55-57 ; rarely adhere to open science…”
Method(2024) · BMJ · doi:10.1136/bmj.q902 · Cited by 416
“…models have shown that they are often poorly conducted (including deficiencies in study design or data collection 37 38 ); use poor methodology 37 38 ; are incompletely reported with key details missing 39-54 ; are consequently at high risk of bias 41 49 55-57 ; rarely adhere to open science…”
Stability of clinical prediction models developed using statistical or machine learning methods
Riley, R. D., & Collins, G. S. (2023) · Biometrical Journal · doi:10.1002/bimj.202200302 · Cited by 211
APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support
Kwong, J. C. C., Khondker, A., Lajkosz, K., McDermott, M. B. A., Borrat, X., McCradden, M. D., Mamdani, M., Kulkarni, G. S., & Johnson, A. E. W. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.35377 · Cited by 159
Evidence of questionable research practices in clinical prediction models
White, N. M., Parsons, R., Collins, P. G. S., & Barnett, A. G. (2023) · BMC Medicine · doi:10.1186/s12916-023-03048-6 · Cited by 149
Metadata
- Authors
- Dhiman, P., Ma, J., Andaur Navarro, C. L., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Van Calster, B., Moons, K. G. M., & Collins, G. S.
- Affiliations
- University of Oxford (GB)
- University of Basel (CH)
- Utrecht University (NL)
- Keele University (GB)
- Corresponding author
- Dhiman, Paula
- ORCID
- Venue
- BMC Medical Research Methodology
- Published
- 2022-04-08
- Volume
- 22
- Issue
- 1
- Pages
- 101-101
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC8991704
- OpenAlex
- W4223540626
- Cited by
- 140
- References
- 126
- Citations by year
- 2026: 37 · 2025: 34 · 2024: 42 · 2023: 23 · 2022: 4
- FWCI
- 3.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- Lirias (KU Leuven) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
- Funders
- Cancer Research UK (27294)
- National Institute for Health and Care Research
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- prognostic model, machine learning, oncology, methodological quality, tripod, PROBAST, charm, regression models, ensemble learning, sample size justification, predictor selection, time-to-event outcomes, internal validation, calibration, cancer prognosis, cancer survival prediction, cancer risk prediction, tumor response prediction, tumor recurrence
- MeSH terms
- Machine Learning, Humans, Medical Oncology, Prognosis, Research Design, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
