Sample size and predictive performance of machine learning methods with survival data: A simulation study
Open accessAbstract
Prediction models are increasingly developed and used in diagnostic and prognostic studies, where the use of machine learning (ML) methods is becoming more and more popular over traditional regression techniques. For survival outcomes the Cox proportional hazards model is generally used and it has been proven to achieve good prediction performances with few strong covariates. The possibility to improve the model performance by including nonlinearities, covariate interactions and time‐varying effects while controlling for overfitting must be carefully considered during the model building phase. On the other hand, ML techniques are able to learn complexities from data at the cost of hyper‐parameter tuning and interpretability. One aspect of special interest is the sample size needed for developing a survival prediction model. While there is guidance when using traditional statistical models, the same does not apply when using ML techniques. This work develops a time‐to‐event simulation framework to evaluate performances of Cox regression compared, among others, to tuned random survival forest, gradient boosting, and neural networks at varying sample sizes. Simulations were based on replications of subjects from publicly available databases, where event times were simulated according to a Cox model with nonlinearities on continuous variables and time‐varying effects and on the SEER registry data.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Sample size calculations for machine learning models are considerably more complex and might require simulations.
“Sample size calculations for such models are considerably more complex and might require simulations.54”
From Developing clinical prediction models: a step-by-step guide (2024) · cites this paper as [54]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Elements of Statistical Learning
Hastie, T., Friedman, J., & Tibshirani, R. (2001) · Springer Series in Statistics · doi:10.1007/978-0-387-21606-5 · Cited by 24,162 · see citing works
A logical calculus of the ideas immanent in nervous activity
McCulloch, W. S., & Pitts, W. (1943) · The Bulletin of Mathematical Biophysics · doi:10.1007/bf02478259 · Cited by 18,566 · see citing works
Modeling Survival Data: Extending the Cox Model
Therneau, T. M., & Grambsch, P. M. (2000) · Statistics for Biology and Health · doi:10.1007/978-1-4757-3294-8 · Cited by 8,674 · see citing works
Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340 · see citing works
Calculating the sample size required for developing a clinical prediction model
Riley, R. D., Ensor, J., Snell, K. I. E., Harrell, F. E., Martin, G. P., Reitsma, J. B., Moons, K. G. M., Collins, G., & van Smeden, M. (2020) · BMJ · doi:10.1136/bmj.m441 · Cited by 2,917 · see citing works
Ishwaran, H., Kogalur, U. B., Blackstone, E. H., & Lauer, M. S. (2008) · The Annals of Applied Statistics · doi:10.1214/08-aoas169 · Cited by 2,714 · see citing works
Big Data and Machine Learning in Health Care
Beam, A. L., & Kohane, I. S. (2018) · JAMA · doi:10.1001/jama.2017.18391 · Cited by 2,506 · see citing works
Levamisole and Fluorouracil for Adjuvant Therapy of Resected Colon Carcinoma
Moertel, C. G., Fleming, T. R., Macdonald, J. S., Haller, D. G., Laurie, J. A., Goodman, P. J., Ungerleider, J. S., Emerson, W. A., Tormey, D. C., Glick, J. H., et al. (1990) · New England Journal of Medicine · doi:10.1056/nejm199002083220602 · Cited by 2,251 · see citing works
Peduzzi, P., Concato, J., Feinstein, A. R., & Holford, T. R. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(95)00048-8 · Cited by 2,154 · see citing works
Royston, P. J., & Parmar, M. (2002) · Statistics in Medicine · doi:10.1002/sim.1203 · Cited by 1,537 · see citing works
Riley, R. D., Snell, K. I., Ensor, J., Burke, D. L., Harrell Jr, F. E., Moons, K. G., & Collins, G. S. (2019) · Statistics in Medicine · doi:10.1002/sim.7992 · Cited by 1,409 · see citing works
Using simulation studies to evaluate statistical methods
Morris, T. P., White, I. R., & Crowther, M. J. (2019) · Statistics in Medicine · doi:10.1002/sim.8086 · Cited by 1,354 · see citing works
Prognosis and prognostic research: what, why, and how?
Moons, K. G. M., Royston, P., Vergouwe, Y., Grobbee, D. E., & Altman, D. G. (2009) · BMJ · doi:10.1136/bmj.b375 · Cited by 1,337 · see citing works
Ishwaran, H., Kogalur, U. B., Blackstone, E. H., & Lauer, M. S. (2008) · Cited by 1,310 · see citing works
Using simulation studies to evaluate statistical methods
Morris, T., White, I. R., & Crowther, M. (2019) · UCL Discovery (University College London) · Cited by 1,011 · see citing works
Moertel, C. G., Fleming, T. R., Macdonald, J. S., Haller, D. G., Laurie, J. A., Tangen, C. M., Ungerleider, J. S., Emerson, W. A., Tormey, D. C., Glick, J. H., et al. (1995) · Annals of Internal Medicine · doi:10.7326/0003-4819-122-5-199503010-00001 · Cited by 924 · see citing works
External validation of a Cox prognostic model: principles and methods
Royston, P. J., & Altman, D. G. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-33 · Cited by 911 · see citing works
Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints
van der Ploeg, T., Austin, P. C., & Steyerberg, E. W. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-137 · Cited by 809 · see citing works
Boosting Algorithms: Regularization, Prediction and Model Fitting
Bühlmann, P., & Hothorn, T. (2007) · Statistical Science · doi:10.1214/07-sts242 · Cited by 741 · see citing works
Royston, P. J., & Sauerbrei, W. (2008) · Wiley series in probability and statistics · doi:10.1002/9780470770771 · Cited by 719 · see citing works
BOOSTING ALGORITHMS: REGULARIZATION, PREDICTION AND MODEL FITTINGPreprint
Bühlmann, P., & Hothorn, T. (2007) · Cited by 704 · see citing works
Predicting sample size required for classification performance
Figueroa, R. L., Zeng‐Treitler, Q., Kandula, S., & Ngo, L. H. (2012) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-12-8 · Cited by 526 · see citing works
Balki, I., Amirabadi, A., Levman, J., Martel, A. L., Emersic, Z., Meden, B., Garcia-Pedrero, A., Ramirez, S. C., Kong, D., Moody, A. R., et al. (2019) · Canadian Association of Radiologists Journal · doi:10.1016/j.carj.2019.06.002 · Cited by 299 · see citing works
Lee, C., Yoon, J., & van der Schaar, M. (2019) · IEEE Transactions on Biomedical Engineering · doi:10.1109/tbme.2019.2909027 · Cited by 297 · see citing works
Brand, R., Rosenman, R. H., Sholtz, R. I., et al. (1976) · Circulation · doi:10.1161/01.cir.53.2.348 · Cited by 250 · see citing works
Schumacher, M. C., Bastert, G., Bojar, H., Hübner, K., Olschewski, M., Sauerbrei, W., Schmoor, C., Beyerle, C., Neumann, R. L. A., & Rauschecker, H. (1994) · Journal of Clinical Oncology · doi:10.1200/jco.1994.12.10.2086 · Cited by 241 · see citing works
A scalable discrete-time survival model for neural networks
Gensheimer, M. F., & Narasimhan, B. (2019) · PeerJ · doi:10.7717/peerj.6257 · Cited by 232 · see citing works
Simulating biologically plausible complex survival data
Crowther, M. J., & Lambert, P. C. (2013) · Statistics in Medicine · doi:10.1002/sim.5823 · Cited by 164 · see citing works
Role of Isolated Locoregional Recurrence of Breast Cancer: Results of Four Prospective Studies
Schmoor, C., Sauerbrei, W., Bastert, G., et al. (2000) · Journal of Clinical Oncology · doi:10.1200/jco.2000.18.8.1696 · Cited by 161 · see citing works
The index of prediction accuracy: an intuitive measure useful for evaluating risk prediction models
Kattan, M. W., & Gerds, T. A. (2018) · Diagnostic and Prognostic Research · doi:10.1186/s41512-018-0029-2 · Cited by 149 · see citing works
Ravdin, P. M., & Clark, G. M. (1992) · Breast Cancer Research and Treatment · doi:10.1007/bf01840841 · Cited by 147 · see citing works
A Plea for Neutral Comparison Studies in Computational Sciences
Boulesteix, A., Lauer, S., & Eugster, M. J. A. (2013) · PLoS ONE · doi:10.1371/journal.pone.0061562 · Cited by 142 · see citing works
Efron‐Type Measures of Prediction Error for Survival Analysis
Gerds, T. A., & Schumacher, M. C. (2007) · Biometrics · doi:10.1111/j.1541-0420.2007.00832.x · Cited by 134 · see citing works
Riley, R. D., Collins, P. G. S., Ensor, J., Archer, L., Booth, S., Mozumder, S. I., Rutherford, M. J., van Smeden, M., Lambert, P. C., & Snell, K. I. E. (2021) · Statistics in Medicine · doi:10.1002/sim.9275 · Cited by 116 · see citing works
Survival analysis and neural nets
Liestbl, K., Andersen, P. K., & Andersen, U. (1994) · Statistics in Medicine · doi:10.1002/sim.4780131202 · Cited by 114 · see citing works
Rahnenführer, J., de Bin, R., Benner, A., Ambrogi, F., Lusa, L., Boulesteix, A., Migliavacca, E., Binder, H., Michiels, S., Sauerbrei, W., et al. (2023) · BMC Medicine · doi:10.1186/s12916-023-02858-y · Cited by 103 · see citing works
Self-citationA general framework for neural network models on censored survival data
Biganzoli, E. M., Boracchi, P., & Marubini, E. (2002) · Neural Networks · doi:10.1016/s0893-6080(01)00131-9 · Cited by 85 · see citing works
Deep Neural Networks for Survival Analysis Using Pseudo Values
Zhao, L., & Feng, D. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.2980204 · Cited by 80 · see citing works
Flexible boosting of accelerated failure time models
Schmid, M. C., & Hothorn, T. (2008) · BMC Bioinformatics · doi:10.1186/1471-2105-9-269 · Cited by 74 · see citing works
Austin, P. C., Harrell, F. E., & Steyerberg, E. W. (2021) · Statistical Methods in Medical Research · doi:10.1177/09622802211002867 · Cited by 74 · see citing works
Gerds, T. A., & Kattan, M. W. (2021) · Chapman & Hall/CRC eBooks · doi:10.1201/9781138384484 · Cited by 71 · see citing works
Simulating Survival Data Using the simsurv R Package
Brilleman, S. L., Wolfe, R. S., Moreno‐Betancur, M., et al. (2021) · Journal of Statistical Software · doi:10.18637/jss.v097.i03 · Cited by 63 · see citing works
Continuous and discrete-time survival prediction with neural networks
Kvamme, H., & Borgan, Ø. (2021) · Lifetime Data Analysis · doi:10.1007/s10985-021-09532-6 · Cited by 43 · see citing works
Modeling Variables With a Spike at Zero: Examples and Practical Recommendations
Lorenz, E., Jenkner, C., Sauerbrei, W., & Becher, H. (2017) · American Journal of Epidemiology · doi:10.1093/aje/kww122 · Cited by 37 · see citing works
Joffe, M. M., Hoover, D. R., Jacobson, L. P., Kingsley, L., Chmiel, J. S., Visscher, B. R., & Robins, J. M. (1998) · Statistics in Medicine · doi:10.1002/(sici)1097-0258(19980530)17:10 · Cited by 34 · see citing works
Smith, H., Sweeting, M. J., Morris, T. P., & Crowther, M. J. (2022) · Diagnostic and Prognostic Research · doi:10.1186/s41512-022-00124-y · Cited by 23 · see citing works
Selection of artificial neural network models for survival analysis with Genetic Algorithms
Ambrogi, F., Lama, N., Boracchi, P., & Biganzoli, E. M. (2007) · Computational Statistics & Data Analysis · doi:10.1016/j.csda.2007.05.001 · Cited by 15 · see citing works
Self-citationMeira‐Machado, L., & Sestelo, M. (2016) · The R Journal · doi:10.32614/rj-2016-059 · Cited by 13 · see citing works
Covariate-Adjusted Restricted Mean Survival Times and Curves
Eng, K. H., & Seagle, B.-L. L. (2016) · Journal of Clinical Oncology · doi:10.1200/jco.2016.67.2279 · Cited by 10 · see citing works
The roles of predictors in cardiovascular risk models - a question of modeling culture?
Wallisch, C., Agibetov, A., Dunkler, D., Haller, M. C., Matthias Samwald, A., Dorffner, G., & Heinze, G. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01487-4 · Cited by 8 · see citing works
Fornili, M., Boracchi, P., Ambrogi, F., et al. (2018) · BMC Bioinformatics · doi:10.1186/s12859-018-2179-1 · Cited by 1 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 1
- Biganzoli E, Boracchi P, Mariani L, Marubini E. Feed forward neural networks for the analysis of censored survival data: a partial logistic regression approach. Stat Med. 1998;17(10):1169-1186.
Cited by
Sorted by citations
Developing clinical prediction models: a step-by-step guide
Efthimiou, O., Seo, M., Chalkou, K., Debray, T., Egger, M., & Salanti, G. (2024) · BMJ · doi:10.1136/bmj-2023-078276 · Cited by 544 · see citing works
- Authors
- Infante, G., Miceli, R., & Ambrogi, F.
- Affiliations
- University of Milan (IT)
- Fondazione IRCCS Istituto Nazionale dei Tumori (IT)
- Corresponding author
- Ambrogi, Federico
- Venue
- Statistics in Medicine
- Publisher
- Wiley
- Published
- 2023-11-10
- Volume
- 42
- Issue
- 30
- Pages
- 5657-5675
- ISSN
- 0277-6715 (ISSN-L)
- 1097-0258
- OpenAlex
- W4388569132
- Cited by
- 33
- References
- 51
- Citations by year
- 2026: 6 · 2025: 16 · 2024: 11
- FWCI
- 6.70
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Statistics in Medicine (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Archivio Istituzionale della Ricerca (Universita Degli Studi Di Milano) (submittedVersion, other-oa)
- Funders
- Ministero dell’Istruzione, dell’Università e della Ricerca (PRIN 2017, prot. 20178S4EK9_004)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- sample size, survival prediction model, machine learning, Cox proportional hazards model, random survival forest, gradient boosting, artificial neural networks, nonlinear covariate effects, time-dependent effects, overfitting, hyperparameter optimization, prognostic model, SEER database
- MeSH terms
- Machine Learning, Computer Simulation, Humans, Proportional Hazards Models, Neural Networks, Computer, Sample Size
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
