Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints
Open accessAbstract
BACKGROUND: Modern modelling techniques may potentially provide more accurate predictions of binary outcomes than classical techniques. We aimed to study the predictive performance of different modelling techniques in relation to the effective sample size ("data hungriness"). METHODS: We performed simulation studies based on three clinical cohorts: 1282 patients with head and neck cancer (with 46.9% 5 year survival), 1731 patients with traumatic brain injury (22.3% 6 month mortality) and 3181 patients with minor head injury (7.6% with CT scan abnormalities). We compared three relatively modern modelling techniques: support vector machines (SVM), neural nets (NN), and random forests (RF) and two classical techniques: logistic regression (LR) and classification and regression trees (CART). We created three large artificial databases with 20 fold, 10 fold and 6 fold replication of subjects, where we generated dichotomous outcomes according to different underlying models. We applied each modelling technique to increasingly larger development parts (100 repetitions). The area under the ROC-curve (AUC) indicated the performance of each model in the development part and in an independent validation part. Data hungriness was defined by plateauing of AUC and small optimism (difference between the mean apparent AUC and the mean validated AUC <0.01). RESULTS: We found that a stable AUC was reached by LR at approximately 20 to 50 events per variable, followed by CART, SVM, NN and RF models. Optimism decreased with increasing sample sizes and the same ranking of techniques. The RF, SVM and NN models showed instability and a high optimism even with >200 events per variable. CONCLUSIONS: Modern modelling techniques such as SVM, NN and RF may need over 10 times as many events per variable to achieve a stable AUC and a small optimism than classical modelling techniques such as LR. This implies that such modern techniques should only be used in medical prediction problems if very large data sets are available.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The sample size must be several times larger if we want to use machine learning models.
“The sample size must be several times larger if we want to use machine learning models.53”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [53]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,706 · see citing works
Classification and Regression Trees.
Gordon, A. D., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1984) · Biometrics · doi:10.2307/2530946 · Cited by 23,834 · see citing works
Classification and Regression Trees.
Ryzin, J. V., Breiman, L., Friedman, J. H., Olshen, R. A., & Stone, C. J. (1986) · Journal of the American Statistical Association · doi:10.2307/2288003 · Cited by 21,028 · see citing works
A simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655 · see citing works
Regression Modeling Strategies
Harrell, F. E. (2001) · Springer series in statistics · doi:10.1007/978-1-4757-3462-1 · Cited by 9,270 · see citing works
Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating
Steyerberg, E. W. (2019) · Statistics for Biology and Health · doi:10.1007/978-3-030-16399-0 · Cited by 3,281 · see citing works
Self-citationInternal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746 · see citing works
Self-citationPrognosis Research Strategy (PROGRESS) 3: Prognostic Model Research
Steyerberg, E. W., Moons, K. G. M., van der Windt, D. A., Hayden, J. A., Perel, P., Schroter, S., Riley, R. D., Hemingway, H., Altman, D. G., & for the PROGRESS Group (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001381 · Cited by 1,508 · see citing works
Self-citationSteyerberg, E. W. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-77244-8 · Cited by 1,491 · see citing works
Self-citationSteyerberg, E. W., Eijkemans, M. J. C., Harrell, F. E., & Habbema, J. D. F. (2000) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(20000430)19:8 · Cited by 784 · see citing works
Self-citationPlague: Past, Present, and Future
Stenseth, N. C., Atshabar, B. B., Begon, M. E., Belmain, S. R., Bertherat, E., Carniel, É., Gage, K. L., Leirs, H., & Rahalison, L. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050003 · Cited by 554 · see citing works
Classification and Regression Trees
Jones, E., Harden, S., & Crawley, M. J. (2022) · doi:10.1002/9781119634461.ch20 · Cited by 505 · see citing works
Data Mining and Statistics for Decision Making
Tufféry, S. (2011) · Wiley eBooks · doi:10.1002/9780470979174 · Cited by 384 · see citing works
Efficient progressive sampling
Provost, F. J., Jensen, D. D., & Oates, T. (1999) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/312129.312188 · Cited by 361 · see citing works
Assessment of Claims of Improved Prediction Beyond the Framingham Risk Score
Tzoulaki, I. (2009) · JAMA · doi:10.1001/jama.2009.1757 · Cited by 276 · see citing works
IMPACT Database of Traumatic Brain Injury: Design And Description
Marmarou, A., Lu, J., Butcher, I., McHugh, G. S., Mushkudiani, N. A., Murray, G. D., Steyerberg, E. W., & Maas, A. I. R. (2007) · Journal of Neurotrauma · doi:10.1089/neu.2006.0036 · Cited by 236 · see citing works
Self-citationData Mining and Statistics for Decision Making
Tuffry, S. (2011) · Medical Entomology and Zoology · Cited by 187 · see citing works
Static versus dynamic sampling for data mining
John, G. H., & Langley, P. (1996) · Cited by 171 · see citing works
Impact of comorbidity on short‐term mortality and overall survival of head and neck cancer patients
Datema, F. R., Ferrier, M. B., van der Schroeff, M. P., & de Jong, R. J. B. (2009) · Head & Neck · doi:10.1002/hed.21245 · Cited by 168 · see citing works
Harrison, R. F., & Kennedy, R. L. (2005) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2004.09.012 · Cited by 111 · see citing works
Selker, H. P., Griffith, J. L., Patil, S., Long, W. J., & D’Agostino, R. B. (1995) · PubMed · Cited by 101 · see citing works
Austin, P. C., Lee, D. S., Steyerberg, E. W., & Tu, J. V. (2012) · Biometrical Journal · doi:10.1002/bimj.201100251 · Cited by 79 · see citing works
Self-citationIncidence and prediction of major cardiovascular complications in head and neck surgery
Datema, F. R., Poldermans, D., & de Jong, R. J. B. (2010) · Head & Neck · doi:10.1002/hed.21351 · Cited by 69 · see citing works
Breast cancer survival and chemotherapy: A support vector machine analysis
Lee, Y.-J., Mangasarian, O. L., & Wolberg, W. H. (2000) · DIMACS series in discrete mathematics and theoretical computer science · doi:10.1090/dimacs/055/01 · Cited by 53 · see citing works
Omar, R. (2010) · Biometrics · doi:10.1111/j.1541-0420.2010.01431.x · Cited by 51 · see citing works
SAMPLE SIZE AND MODELING ACCURACY OF DECISION TREE BASED DATA MINING TOOLS
Morgan, J., Daugherty, R. A., Hilchie, A., & Carey, B. (2003) · Cited by 45 · see citing works
Requiem for large-scale models
Lee, D. B. (1975) · ACM SIGSIM Simulation Digest · doi:10.1145/1102945.1102950 · Cited by 26 · see citing works
Prediction of Survival with Alternative Modeling Techniques Using Pseudo Values
van der Ploeg, T., Datema, F. R., de Jong, R. J. B., & Steyerberg, E. W. (2014) · PLoS ONE · doi:10.1371/journal.pone.0100234 · Cited by 22 · see citing works
Self-citationDeveloping a Prognostic Model for Traumatic Brain Injury—A Missed Opportunity?
Young, N. H., & Andrews, P. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050168 · Cited by 21 · see citing works
Prediction of intracranial findings on CT-scans by alternative modelling techniques
van der Ploeg, T., Smits, M., Dippel, D. W. J., Hunink, M. G. M., & Steyerberg, E. W. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-143 · Cited by 17 · see citing works
Self-citationHarrell, F., Jr. (1996) · doi:10.1002/(sici)1097-0258(19960229)15:4
Ennis (1998) · doi:10.1002/(sici)1097-0258(19981115)17:21
Flagged first, then by citations
2 of the 34 references couldn’t be matched to a known paper.
Research works printed but not linked: 1
- The pre-publication history for this paper can be accessed here: http://www.biomedcentral.com/1471-2288/14/137/prepub
Cited by
Sorted by citations
Calculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917 · see citing works
Christodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.02.004 · Cited by 2,119 · see citing works
Self-citationMoons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669 · see citing works
Riley, R. D., Snell, K. I., Ensor, J., Burke, D. L., Harrell Jr, F. E., Moons, K. G., & Collins, G. S. (2019) · Statistics in Medicine · doi:10.1002/sim.7992 · Cited by 1,409 · see citing works
Moons, K. G. M., Damen, J. A. A., Kaul, T., Hooft, L., Andaur Navarro, C., Dhiman, P., Beam, A. L., Van Calster, B., Celi, L. A., Denaxas, S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-082505 · Cited by 1,069 · see citing works
Churpek, M. M., Yuen, T. C., Winslow, C., Meltzer, D. O., Kattan, M. W., & Edelson, D. P. (2016) · Critical Care Medicine · doi:10.1097/ccm.0000000000001571 · Cited by 638 · see citing works
Logistic regression was as good as machine learning for predicting major chronic diseases
Nusinovici, S., Tham, Y. C., Yan, M. Y. C., Ting, D. S. W., Li, J., Sabanayagam, C., Wong, T. Y., & Cheng, C. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.03.002 · Cited by 569 · see citing works
Goldstein, B. A., Návar, A. M., & Carter, R. E. (2016) · European Heart Journal · doi:10.1093/eurheartj/ehw302 · Cited by 551 · see citing works
Developing clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544 · see citing works
Pavlou, M., Ambler, G., Seaman, S. R., Iorio, M. D., & Omar, R. (2015) · Statistics in Medicine · doi:10.1002/sim.6782 · Cited by 421 · see citing works
Andaur Navarro, C. L., Damen, J. A. A., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., Moons, K. G. M., et al. (2021) · BMJ · doi:10.1136/bmj.n2281 · Cited by 415 · see citing works
van den Goorbergh, R., van Smeden, M., Timmerman, D., & CALSTER, B. V. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac093 · Cited by 387 · see citing works
Machine Learning Applications for Mass Spectrometry-Based Metabolomics
Liebal, U. W., An, P. N. T., Sudhakar, M., Raman, K., & Blank, L. M. (2020) · Metabolites · doi:10.3390/metabo10060243 · Cited by 371 · see citing works
Machine learning and deep learning applications in microbiome research
Medina, R. H., Kutuzova, S., Nielsen, K. N., Johansen, J., Hansen, L. H., Nielsen, M. E., & Rasmussen, S. (2022) · ISME Communications · doi:10.1038/s43705-022-00182-9 · Cited by 356 · see citing works
Predictive modeling of biomass gasification with machine learning-based regression methods
Elmaz, F., Yücel, Ö., & Mutlu, A. Y. (2019) · Energy · doi:10.1016/j.energy.2019.116541 · Cited by 282 · see citing works
Comparison of variable selection methods for clinical predictive modeling
Sanchez-Pinto, L. N., Venable, L. R., Fahrenbach, J. P., & Churpek, M. M. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.05.006 · Cited by 281 · see citing works
Using Resistin, glucose, age and BMI to predict the presence of breast cancer
Patrício, M., Pereira, J., Crisóstomo, J., Matafome, P. N., Gomes, M., Seiça, R. M. F., & Caramelo, F. J. (2018) · BMC Cancer · doi:10.1186/s12885-017-3877-1 · Cited by 277 · see citing works
Internet of Health Things: Toward intelligent vital signs monitoring in hospital wards
da Costa, C. A., Pasluosta, C. F., Eskofier, B. M., da Silva, D. B., & Righi, R. D. R. (2018) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2018.05.005 · Cited by 278 · see citing works
Statistical Primer: developing and validating a risk prediction model†
Grant, S. W., Collins, P. G. S., & Nashef, S. A. M. (2018) · European Journal of Cardio-Thoracic Surgery · doi:10.1093/ejcts/ezy180 · Cited by 254 · see citing works
Bailly, A., Blanc, C., Francis, É., Guillotin, T., Jamal, F., Wakim, B., & Roy, P. (2022) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106504 · Cited by 252 · see citing works
Use of machine learning to analyse routinely collected intensive care unit data: a systematic review
Shillan, D., Sterne, J. A. C., Champneys, A., & Gibbison, B. (2019) · Critical Care · doi:10.1186/s13054-019-2564-9 · Cited by 251 · see citing works
Machine Learning in Epidemiology and Health Outcomes Research
Wiemken, T. L., & Kelley, R. R. (2019) · Annual Review of Public Health · doi:10.1146/annurev-publhealth-040119-094437 · Cited by 228 · see citing works
Multivariable Modeling Strategies
Harrell, F. E. (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7_4 · Cited by 221 · see citing works
Meehan, A. J., Lewis, S. J., Fazel, S., Fusar-Poli, P., Steyerberg, E. W., Stahl, D., & Danese, A. (2022) · Molecular Psychiatry · doi:10.1038/s41380-022-01528-4 · Cited by 219 · see citing works
Self-citationGravesteijn, B. Y., Nieboer, D., Ercole, A., Lingsma, H. F., Nelson, D., CALSTER, B. V., Steyerberg, E. W., Åkerlund, C. A. I., Amrein, K., Anđelić, N., et al. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.03.005 · Cited by 184 · see citing works
Self-citationPoor performance of clinical prediction models: the harm of commonly applied methods
Steyerberg, E. W., Uno, H., Ioannidis, J. P. A., CALSTER, B. V., Ukaegbu, C. I., Dhingra, T. G., Syngal, S., & Kastrinos, F. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.11.013 · Cited by 166 · see citing works
Self-citationCommunication and computation efficiency in Federated Learning: A survey
Almanifi, O. R. A., Chow, C., Tham, M.-L., Chuah, J. H., & Kanesan, J. (2023) · Internet of Things · doi:10.1016/j.iot.2023.100742 · Cited by 169 · see citing works
Richter, A. N., & Khoshgoftaar, T. M. (2018) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2018.06.002 · Cited by 162 · see citing works
Gibson, P. B., Chapman, W. E., Altınok, A., Monache, L. D., DeFlorio, M. J., & Waliser, D. E. (2021) · Communications Earth & Environment · doi:10.1038/s43247-021-00225-4 · Cited by 165 · see citing works
Introducing stacking machine learning approaches for the prediction of rock deformation
Koopialipoor, M., Asteris, P. G., Mohammed, A. S., Alexakis, D. Ε., Mamou, A., & Armaghani, D. J. (2022) · Transportation Geotechnics · doi:10.1016/j.trgeo.2022.100756 · Cited by 160 · see citing works
Machine Learning and Psychological Research: The Unexplored Effect of Measurement
Jacobucci, R., & Grimm, K. J. (2020) · Perspectives on Psychological Science · doi:10.1177/1745691620902467 · Cited by 160 · see citing works
General Aspects of Fitting Regression Models
Harrell, F. E. (2015) · Springer series in statistics · doi:10.1007/978-3-319-19425-7_2 · Cited by 153 · see citing works
APPRAISE-AI Tool for Quantitative Evaluation of AI Studies for Clinical Decision Support
Kwong, J. C. C., Khondker, A., Lajkosz, K., McDermott, M. B. A., Borrat, X., McCradden, M. D., Mamdani, M., Kulkarni, G. S., & Johnson, A. E. W. (2023) · JAMA Network Open · doi:10.1001/jamanetworkopen.2023.35377 · Cited by 151 · see citing works
Lynam, A. L., Dennis, J. M., Owen, K. R., Oram, R. A., Jones, A. G., Shields, B. M., & Ferrat, L. A. (2020) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00075-2 · Cited by 142 · see citing works
Benedetto, U., Dimagli, A., Sinha, S., Cocomello, L., Gibbison, B., Caputo, M., Gaunt, T. R., Lyon, M., Holmes, C., & Angelini, G. D. (2020) · Journal of Thoracic and Cardiovascular Surgery · doi:10.1016/j.jtcvs.2020.07.105 · Cited by 142 · see citing works
Sufriyana, H., Husnayain, A., Chen, Y.-L., Kuo, C.-Y., Singh, O., Yeh, T.-Y., Wu, Y., & Su, E. C. (2020) · JMIR Medical Informatics · doi:10.2196/16503 · Cited by 145 · see citing works
Voice patterns in schizophrenia: A systematic review and Bayesian meta-analysis
Parola, A., Simonsen, A., Bliksted, V., & Fusaroli, R. (2019) · Schizophrenia Research · doi:10.1016/j.schres.2019.11.031 · Cited by 141 · see citing works
Dhiman, P., Ma, J., Andaur Navarro, C. L., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Van Calster, B., et al. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01577-x · Cited by 140 · see citing works
Schmidt, A., Cappucciati, M., Raduà, J., Rutigliano, G., Rocchetti, M., Dell’Osso, L., Politi, P., Borgwardt, S., Reilly, T. J., Valmaggia, L., et al. (2016) · Schizophrenia Bulletin · doi:10.1093/schbul/sbw098 · Cited by 131 · see citing works
Machine Learning in Pharmacometrics: Opportunities and Challenges
McComb, M., Bies, R., & Ramanathan, M. (2022) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.14801 · Cited by 128 · see citing works
Allen, M. M., Pareek, A., Krych, A. J., Hewett, T. E., Levy, B. A., Stuart, M. J., & Dahm, D. L. (2016) · The American Journal of Sports Medicine · doi:10.1177/0363546516648439 · Cited by 127 · see citing works
Seto, H., Oyama, A., Kitora, S., Toki, H., Yamamoto, R., Kotoku, J., Haga, A., Shinzawa, M., Yamakawa, M., Fukui, S., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-20149-z · Cited by 142 · see citing works
Embryo selection with artificial intelligence: how to evaluate and compare methods?
Kragh, M. F., & Karstoft, H. (2021) · Journal of Assisted Reproduction and Genetics · doi:10.1007/s10815-021-02254-6 · Cited by 130 · see citing works
A scoping review of artificial intelligence-based methods for diabetes risk prediction
Mohsen, F., Al-Absi, H. R. H., Yousri, N. A., El Hajj, N., & Shah, Z. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00933-5 · Cited by 125 · see citing works
Andaur Navarro, C. L., Damen, J. A. A., van Smeden, M., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., et al. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.11.015 · Cited by 121 · see citing works
Systematic misestimation of machine learning performance in neuroimaging studies of depression
Flint, C., Cearns, M., Opel, N., Redlich, R., Mehler, D. M. A., Emden, D., Winter, N. R., Leenings, R., Eickhoff, S. B., Kircher, T., et al. (2021) · Neuropsychopharmacology · doi:10.1038/s41386-021-01020-7 · Cited by 120 · see citing works
Collins, L., Bennett, A. F., Leonard, S., & Penman, T. D. (2019) · Global Change Biology · doi:10.1111/gcb.14735 · Cited by 120 · see citing works
Predicting breast cancer 5-year survival using machine learning: A systematic review
Li, J., Zhou, Z., Dong, J., Fu, Y., Li, Y., Luan, Z., & Peng, X. (2021) · PLoS ONE · doi:10.1371/journal.pone.0250370 · Cited by 123 · see citing works
Finger, R. P., Schmitz-Valckenberg, S., Schmid, M. C., Rubin, G. S., Dunbar, H., Tufail, A., Crabb, D. P., Binns, A. M., Sánchez, C. I., Margaron, P., et al. (2018) · Ophthalmologica · doi:10.1159/000491402 · Cited by 111 · see citing works
Bishara, A. M., Chiu, C., Whitlock, E. L., Douglas, V. C., Lee, S., Butte, A. J., Leung, J. M., & Donovan, A. L. (2022) · BMC Anesthesiology · doi:10.1186/s12871-021-01543-y · Cited by 112 · see citing works
Senanayake, S. J., White, N. M., Graves, N., Healy, H. G., Baboolal, K., & Kularatna, S. (2019) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2019.103957 · Cited by 108 · see citing works
Evaluating the effect of dataset size on predictive model using supervised learning technique
Ajiboye, A. R., Abdullah-Arshah, R., Qin, H., & Isah-Kebbe, H. (2015) · International Journal of Computer Systems & Software Engineering · doi:10.15282/ijsecs.1.2015.6.0006 · Cited by 104 · see citing works
Predicting hypertension using machine learning: Findings from Qatar Biobank Study
AlKaabi, L. A., Ahmed, L. S., Attiyah, M. F. A., & Abdel‐Rahman, M. E. (2020) · PLoS ONE · doi:10.1371/journal.pone.0240370 · Cited by 109 · see citing works
Artificial intelligence-enabled decision support in nephrology
Loftus, T. J., Shickel, B., Ozrazgat‐Baslanti, T., Ren, Y., Glicksberg, B. S., Cao, J., Singh, K., Chan, L., Nadkarni, G. N., & Bihorac, A. (2022) · Nature Reviews Nephrology · doi:10.1038/s41581-022-00562-3 · Cited by 109 · see citing works
Predicting population health with machine learning: a scoping review
Morgenstern, J., Buajitti, E., O’Neill, M., Piggott, T., Goel, V., Fridman, D., Kornas, K., & Rosella, L. C. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-037860 · Cited by 98 · see citing works
Tan, K. R., Seng, J. J. B., Kwan, Y. H., Chen, Y. J., Zainudin, S. B., Loh, D. H. F., Liu, N., & Low, L. L. (2021) · Journal of Diabetes Science and Technology · doi:10.1177/19322968211056917 · Cited by 103 · see citing works
An introduction to machine learning for classification and prediction
Black, J. E., Kueper, J. K., & Williamson, T. S. (2022) · Family Practice · doi:10.1093/fampra/cmac104 · Cited by 103 · see citing works
Kosolwattana, T., Liu, C., Hu, R., Han, S., Chen, H., & Lin, Y. (2023) · BioData Mining · doi:10.1186/s13040-023-00330-4 · Cited by 106 · see citing works
Prognostic models will be victims of their own success, unless…
Lenert, M. C., Matheny, M. E., & Walsh, C. G. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz145 · Cited by 90 · see citing works
Studerus, E., Ramyead, A., & Riecher‐Rössler, A. (2017) · Psychological Medicine · doi:10.1017/s0033291716003494 · Cited by 91 · see citing works
Sufriyana, H., Wu, Y., & Su, E. C. (2020) · EBioMedicine · doi:10.1016/j.ebiom.2020.102710 · Cited by 100 · see citing works
Clift, A. K., Dodwell, D., Lord, S., Petrou, S., Brady, M., Collins, G. S., & Hippisley-Cox, J. (2023) · BMJ · doi:10.1136/bmj-2022-073800 · Cited by 97 · see citing works
Artificial intelligence in nutrition research: perspectives on current and future applications
Côté, M., & Lamarche, B. (2021) · Applied Physiology, Nutrition, and Metabolism · doi:10.1139/apnm-2021-0448 · Cited by 94 · see citing works
Importance of sample size on the quality and utility of AI-based prediction models for healthcare
Riley, R. D., Ensor, J., Snell, K. I. E., Archer, L., Whittle, R., Dhiman, P., Alderman, J. E., Liu, X., Kirton, L., Manson-Whitton, J., et al. (2025) · The Lancet Digital Health · doi:10.1016/j.landig.2025.01.013 · Cited by 93 · see citing works
Kusuma, K., Larsen, M. E., Quiroz, J. C., Gillies, M. B., Burnett, A., Qian, J., & Torok, M. H. (2022) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2022.09.050 · Cited by 90 · see citing works
Arina, P., Kaczorek, M. R., Hofmaenner, D. A., Pisciotta, W., Refinetti, P., Singer, M., Mazomenos, E. B., & Whittle, J. (2024) · Anesthesiology · doi:10.1097/aln.0000000000004764 · Cited by 81 · see citing works
Hoogland, J., IntHout, J., Belias, M., Rovers, M. M., Riley, R. D., Harrell, F. E., Moons, K. G. M., Debray, T. P. A., & Reitsma, J. B. (2021) · Statistics in Medicine · doi:10.1002/sim.9154 · Cited by 79 · see citing works
Risk of bias of prognostic models developed using machine learning: a systematic review in oncology
Dhiman, P., Ma, J., Andaur Navarro, C. L., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Van Calster, B., et al. (2022) · Diagnostic and Prognostic Research · doi:10.1186/s41512-022-00126-w · Cited by 79 · see citing works
Resting state EEG power spectrum and functional connectivity in autism: a cross-sectional analysis
Garcés, P., Baumeister, S., Mason, L., Chatham, C. H., Holiga, Š., Dukart, J., Jones, E. J. H., Banaschewski, T., Baron‐Cohen, S., Bölte, S., et al. (2022) · Molecular Autism · doi:10.1186/s13229-022-00500-x · Cited by 78 · see citing works
Bullock, G. S., Mylott, J. A., Hughes, T., Nicholson, K. F., Riley, R. D., & Collins, P. G. S. (2022) · Sports Medicine · doi:10.1007/s40279-022-01698-9 · Cited by 88 · see citing works
Machine learning in predicting antimicrobial resistance: a systematic review and meta-analysis
Tang, R., Luo, R., Tang, S., Song, H., & Chen, X. (2022) · International Journal of Antimicrobial Agents · doi:10.1016/j.ijantimicag.2022.106684 · Cited by 82 · see citing works
Austin, P. C., Harrell, F. E., & Steyerberg, E. W. (2021) · Statistical Methods in Medical Research · doi:10.1177/09622802211002867 · Cited by 74 · see citing works
Self-citationPredicting relapse or recurrence of depression: systematic review of prognostic models
Moriarty, A. S., Meader, N., Snell, K. I. E., Riley, R. D., Paton, L. W., Dawson, S., Hendon, J., Chew-Graham, C. A., Gilbody, S., Churchill, R., et al. (2022) · The British Journal of Psychiatry · doi:10.1192/bjp.2021.218 · Cited by 71 · see citing works
Early PREdiction of sepsis using leukocyte surface biomarkers: the ExPRES-sepsis cohort study
Shankar-Hari, M., Datta, D., Wilson, J., Assi, V., Stephen, J., Weir, C. J., Rennie, J., Antonelli, J., Bateman, A., Felton, J. M., et al. (2018) · Intensive Care Medicine · doi:10.1007/s00134-018-5389-0 · Cited by 74 · see citing works
Giummarra, M. J., Cameron, P. A., Ponsford, J., Ioannou, L., Gibson, S. J., Jennings, P. A., & Georgiou-Karistianis, N. (2017) · Journal of Occupational Rehabilitation · doi:10.1007/s10926-016-9642-5 · Cited by 68 · see citing works
Speiser, J. L., Callahan, K. E., Houston, D. K., Fanning, J. T., Gill, T. M., Guralnik, J. M., Newman, A. B., Pahor, M., Rejeski, W. J., & Miller, M. E. (2020) · The Journals of Gerontology Series A · doi:10.1093/gerona/glaa138 · Cited by 68 · see citing works
Wang, X., Ren, H., Ren, J., Song, W., Qiao, Y., Ren, Z., Zhao, Y., Linghu, L., Cui, Y., Zhao, Z., et al. (2023) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2023.107340 · Cited by 66 · see citing works
Strength of Association between Pentosan Polysulfate and a Novel Maculopathy
Hanif, A. M., Shah, R., Yan, J., Varghese, J. S., Patel, S. A., Cribbs, B. E., O’Keefe, G. A. D., Hendrick, A. M., Shantha, J. G., Hubbard, G. B., et al. (2019) · Ophthalmology · doi:10.1016/j.ophtha.2019.04.024 · Cited by 64 · see citing works
Machine Learning Improves Cardiovascular Risk Definition for Young, Asymptomatic Individuals
Sánchez‐Cabo, F., Rosselló, X., Fuster, V., Benito, F., Manzano, J. P., Silla-Castro, J. C., Fernández-Alvira, J. M., Oliva, B., Fernández‐Friera, L., López‐Melgar, B., et al. (2020) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2020.08.017 · Cited by 65 · see citing works
Big Data in Head and Neck Cancer
Resteghini, C., Trama, A., Borgonovi, E., Hosni, H., Corrao, G., Orlandi, E. M., Calareso, G., Cecco, L. D., Piazza, C., Mainardi, L. T., et al. (2018) · Current Treatment Options in Oncology · doi:10.1007/s11864-018-0585-2 · Cited by 62 · see citing works
Grabman, J. H., Dobolyi, D. G., Berelovich, N. L., & Dodson, C. S. (2019) · Journal of Applied Research in Memory and Cognition · doi:10.1037/h0101835 · Cited by 64 · see citing works
Pedersen, C. F., Andersen, M. Ø., Carreon, L. Y., & Eiskjær, S. P. (2020) · Global Spine Journal · doi:10.1177/2192568220967643 · Cited by 63 · see citing works
Lai, Q., Spoletini, G., Mennini, G., Laureiro, Z. L., Tsilimigras, D. I., Pawlik, T. M., & Rossi, M. (2020) · World Journal of Gastroenterology · doi:10.3748/wjg.v26.i42.6679 · Cited by 63 · see citing works
Gazis, I.-Z., Schoening, T., Alevizos, E., & Greinert, J. (2018) · Biogeosciences · doi:10.5194/bg-15-7347-2018 · Cited by 62 · see citing works
Fold-stratified cross-validation for unbiased and privacy-preserving federated learning
Bey, R., Goussault, R., Grolleau, F., Benchoufi, M., & Porcher, R. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa096 · Cited by 60 · see citing works
Xiong, S., Chen, W., Jia, X., Jia, Y., & Liu, C. (2023) · BMC Pulmonary Medicine · doi:10.1186/s12890-023-02570-w · Cited by 60 · see citing works
van der Ploeg, T., Nieboer, D., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.03.002 · Cited by 59 · see citing works
Self-citationJiang, Y., Zhang, X., Ma, R., Wang, X., Liu, J., Keerman, M., Yan, Y., Ma, J., Song, Y., Zhang, J., et al. (2021) · Clinical Epidemiology · doi:10.2147/clep.s313343 · Cited by 59 · see citing works
Silvey, S. G., & Liu, J. (2024) · Journal of Medical Internet Research · doi:10.2196/60231 · Cited by 59 · see citing works
Evidence of Inflated Prediction Performance: A Commentary on Machine Learning and Suicide Research
Jacobucci, R., Littlefield, A. K., Millner, A. J., Kleiman, E. M., & Steinley, D. (2021) · Clinical Psychological Science · doi:10.1177/2167702620954216 · Cited by 56 · see citing works
Juwara, L., Arora, N., Gornitsky, M., Saha‐Chaudhuri, P., & Velly, A. M. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2020.104170 · Cited by 56 · see citing works
McNamara, M. E., Zisser, M., Beevers, C. G., & Shumake, J. (2022) · Behaviour Research and Therapy · doi:10.1016/j.brat.2022.104086 · Cited by 56 · see citing works
Shi, C., Faris, P., McNeil, D. A., Patterson, S. K., Potestio, M. L., Thawer, S., & McLaren, L. (2018) · BMC Oral Health · doi:10.1186/s12903-017-0444-8 · Cited by 56 · see citing works
Strijker, M., Chen, J., Mungroop, T. H., Jamieson, N. B., van Eijck, C. H. J., Steyerberg, E. W., Wilmink, J. W., Koerkamp, B. G., van Laarhoven, H. W. M., & Besselink, M. G. (2019) · British journal of surgery · doi:10.1002/bjs.11111 · Cited by 55 · see citing works
Self-citationPua, Y., Kang, H., Thumboo, J., Clark, R., Chew, E., Poon, C. L.-L., Chong, H., & Yeo, S. (2019) · Knee Surgery, Sports Traumatology, Arthroscopy · doi:10.1007/s00167-019-05822-7 · Cited by 49 · see citing works
Interpretable machine learning for materials design
Dean, J., Scheffler, M., Purcell, T. A. R., Barabash, S. V., Bhowmik, R., & Bazhirov, T. (2023) · Journal of materials research/Pratt's guide to venture capital sources · doi:10.1557/s43578-023-01164-w · Cited by 54 · see citing works
Predicting lung cancer survival based on clinical data using machine learning: A review
Altuhaifa, F. A., Win, K. T., & Su, G. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.107338 · Cited by 53 · see citing works
A Novel Risk Calculator Predicts 90-Day Readmission Following Total Joint Arthroplasty
Goltz, D. E., Ryan, S. P., Hopkins, T. J., Howell, C. B., Attarian, D. E., Bolognesi, M. P., & Seyler, T. M. (2019) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.18.00843 · Cited by 52 · see citing works
Oosterhoff, J. H. F., Gravesteijn, B. Y., Karhade, A. V., Jaarsma, R. L., Kerkhoffs, G. M. M. J., Ring, D., Schwab, J. H., Steyerberg, E. W., & Doornberg, J. N. (2021) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.21.00341 · Cited by 52 · see citing works
Self-citationHudec, M., Mináriková, E., Mesiar, R., Saranti, A., & Holzinger, A. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106916 · Cited by 51 · see citing works
- Authors
- van der Ploeg, T., Austin, P. C., & Steyerberg, E. W.
- Affiliations
- University of Toronto (CA)
- Erasmus MC (NL)
- Venue
- BMC Medical Research Methodology
- Published
- 2014-12
- Volume
- 14
- Issue
- 1
- Pages
- 137-137
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC4289553
- OpenAlex
- W1602160603
- Cited by
- 809
- References
- 34
- Citations by year
- 2026: 118 · 2025: 107 · 2024: 91 · 2023: 117 · 2022: 114 · 2021: 107 · 2020: 59 · 2019: 39 · 2018: 30 · 2017: 13 · 2016: 8 · 2015: 6
- FWCI
- 9.51
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (publishedVersion)
- TSpace (University of Toronto) (submittedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Keywords
- effective sample size, predictive performance, binary outcome prediction, support vector machine, artificial neural networks, random forest, logistic regression, classification and regression trees, optimism, traumatic brain injury mortality, mild traumatic brain injury, CT abnormalities, model stability
- MeSH terms
- Brain Injuries, Data Interpretation, Statistical, Head and Neck Neoplasms, Humans, ROC Curve, Models, Statistical, Databases, Factual, Head Injuries, Closed, Treatment Outcome, Endpoint Determination, Support Vector Machine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review