The DOI matches its Crossref record; 125 of the 126 references it cites were found in a registry or an index.
Methodological conduct of prognostic prediction models developed using machine learning in oncology: a systematic review
Open accessAbstract
BACKGROUND: Describe and evaluate the methodological conduct of prognostic prediction models developed using machine learning methods in oncology. METHODS: We conducted a systematic review in MEDLINE and Embase between 01/01/2019 and 05/09/2019, for studies developing a prognostic prediction model using machine learning methods in oncology. We used the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement, Prediction model Risk Of Bias ASsessment Tool (PROBAST) and CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) to assess the methodological conduct of included publications. Results were summarised by modelling type: regression-, non-regression-based and ensemble machine learning models. RESULTS: Sixty-two publications met inclusion criteria developing 152 models across all publications. Forty-two models were regression-based, 71 were non-regression-based and 39 were ensemble models. A median of 647 individuals (IQR: 203 to 4059) and 195 events (IQR: 38 to 1269) were used for model development, and 553 individuals (IQR: 69 to 3069) and 50 events (IQR: 17.5 to 326.5) for model validation. A higher number of events per predictor was used for developing regression-based models (median: 8, IQR: 7.1 to 23.5), compared to alternative machine learning (median: 3.4, IQR: 1.1 to 19.1) and ensemble models (median: 1.7, IQR: 1.1 to 6). Sample size was rarely justified (n = 5/62; 8%). Some or all continuous predictors were categorised before modelling in 24 studies (39%). 46% (n = 24/62) of models reporting predictor selection before modelling used univariable analyses, and common method across all modelling types. Ten out of 24 models for time-to-event outcomes accounted for censoring (42%). A split sample approach was the most popular method for internal validation (n = 25/62, 40%). Calibration was reported in 11 studies. Less than half of models were reported or made available. CONCLUSIONS: The methodological conduct of machine learning based clinical prediction models is poor. Guidance is urgently needed, with increased awareness and education of minimum prediction modelling standards. Particular focus is needed on sample size estimation, development and validation analysis methods, and ensuring the model is available for independent validation, to improve quality of machine learning based clinical prediction models.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Unfortunately, calibration measures are still under-reported.”
From Evaluation of performance measures in predictive artificial intelligence models to support medical decisions: overview and guidance (2025) · cites this paper as [94]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
1 retracted
Alcantud, J. C. R., Varela, G., Santos-Buitrago, B., Santos‐García, G., & Jiménez, M. F. (2019) · PLoS ONE · doi:10.1371/journal.pone.0218283 · Cited by 41
RetractedLeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Harris, P. A., Taylor, R., Thielke, R., Payne, J., Gonzalez, N., & Conde, J. G. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2008.08.010 · Cited by 51,384
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,762
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955
Sterne, J. A. C., White, I. R., Carlin, J. B., Spratt, M., Royston, P., Kenward, M. G., Wood, A. M., & Carpenter, J. R. (2009) · BMJ · doi:10.1136/bmj.b2393 · Cited by 7,299
Harrell (2015) · Springer Series in Statistics · doi:10.1007/978-3-319-19425-7 · Cited by 6,878
Research works printed but not linked: 4
- The Endnote Team. Endnote. Philadelphia: Clarivate Analytics; 2013.
- (4) Cited 2020 Feb 12. Available from: https://bmjopen.bmj.com/content/9/4/e025611.
- StataCorp. Stata Statistical Software: Release 15. College Station: StataCorp LLC; 2017.
- (2) Cited 2021 Mar 5. Available from: https://ijgc.bmj.com/content/29/2/320.
Cited by
Sorted by citations
Items 101–141 of 141
The Ethics of AI in Medical Research: A Call for Open and Honest Discussion
Tahseen, S., Memon, M. A., Kamran, S. M., & Pathan, K. T. (2024) · Liaquat Medical Research Journal · doi:10.38106/lmrj.2024.6.2-07 · Cited by 1
Deep weighted survival neural networks to survival risk prediction
Hui, Y., Wang, Q., Zhou, X., Gu, L., & Zhao, Z. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01670-2 · Cited by 2
Gammall, J., & Lai, A. G. (2025) · Discover Oncology · doi:10.1007/s12672-025-02523-1 · Cited by 1
Lazare, S., Gu, Z., Earland, N. J., Harmon, A., Pacula, M., Rivera, M., Tellis, A., Whitfield, D., Benson, A., Gerndt, S. P., et al. (2024) · medRxiv · doi:10.1101/2024.09.27.24314491 · Cited by 1
Talimtzi, P., Ntolkeras, A., Kostopoulos, G., Bougioukas, K. I., Pagkalidou, E., Ouranidis, A., Pataka, A., & Haidich, A. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4621569
White, N. M., Parsons, R., Borg, D. N., Collins, P. G. S., & Barnett, A. G. (2023) · doi:10.31219/osf.io/nh9sx
Clinical Prediction Models for Treatment Outcomes in Newly-diagnosed EpilepsyPreprint
Ratcliffe, C., Pradeep, V., Marson, A. G., Keller, S. S., & Bonnett, L. J. (2024) · medRxiv · doi:10.1101/2024.01.12.24301215
Harrison‐Brown, M., Scholes, C. J., Ebrahimi, M., Bell, C., & Kirwan, G. W. (2024) · Clinical Rehabilitation · doi:10.1177/02692155241233348
Bate, S. G., Stokes, V., Greenlee, H., Goh, K. Y., Whiting, G., Kitchen, G. B., Martin, G. P., Parker, A. J., Wilson, A. J., & on behalf of the MRI Critical Care Data Group (2024) · Critical Care Explorations · doi:10.1097/cce.0000000000001067
Lusa, L., Kappenberg, F., Collins, P. G. S., Schmid, M., Sauerbrei, W., & Rahnenführer, J. (2024) · medRxiv · doi:10.1101/2024.08.09.24311759
Mohammed, Z. M. S., Ibrahim, M. A., Mohammed, S. A., Hamad, S. H. M., Central Bureau of Statistics, K., Khamis, G. S. M., & Elmorsy, E. M. (2025) · International Journal of ADVANCED AND APPLIED SCIENCES · doi:10.21833/ijaas.2025.07.002
Sun, Y., Wang, Y., Luu, H. P. K., Liu, A., Zhang, C., Muluneh, B., Zheng, W., Yao, H., Basch, E., & Zhou, J. (2025) · medRxiv · doi:10.1101/2025.10.02.25337154
Guolla, L., Mbuagbaw, L. C. E., Ma, J., Thabane, L., Gupta, S., & Nathan, P. C. (2025) · Health Open Research · doi:10.12688/healthopenres.13924.1
Additional file 1 of Evidence of questionable research practices in clinical prediction models
White, N. M., Parsons, R., Collins, G., & Barnett, A. G. (2024) · Figshare · doi:10.6084/m9.figshare.26615050
Additional file 6 of Evidence of questionable research practices in clinical prediction models
White, N. M., Parsons, R., Collins, G., & Barnett, A. G. (2024) · Figshare · doi:10.6084/m9.figshare.26615065
Additional file 4 of Evidence of questionable research practices in clinical prediction models
White, N. M., Parsons, R., Collins, G., & Barnett, A. G. (2024) · Figshare · doi:10.6084/m9.figshare.26615059
Additional file 7 of Evidence of questionable research practices in clinical prediction models
White, N. M., Parsons, R., Collins, G., & Barnett, A. G. (2024) · Figshare · doi:10.6084/m9.figshare.26615068
Dhiman, P., Ma, J., Qi, C., Bullock, G., Sergeant, J. C., Riley, R. D., & Collins, G. (2023) · Figshare · doi:10.6084/m9.figshare.23994599
Zhuang, Q., Zhang, A. Y., Cong, R. S. T. Y., Yang, G. M., Neo, P. S. H., Tan, D. S., Chua, M. L. K., Tan, I. B., Wong, F. Y., Eng Hock Ong, M., et al. (2024) · Figshare · doi:10.6084/m9.figshare.25865063
Additional file 2 of Evidence of questionable research practices in clinical prediction models
White, N. M., Parsons, R., Collins, G., & Barnett, A. G. (2024) · Figshare · doi:10.6084/m9.figshare.26615053
Additional file 3 of Evidence of questionable research practices in clinical prediction models
White, N. M., Parsons, R., Collins, G., & Barnett, A. G. (2024) · Figshare · doi:10.6084/m9.figshare.26615056
Additional file 5 of Evidence of questionable research practices in clinical prediction models
White, N. M., Parsons, R., Collins, G., & Barnett, A. G. (2024) · Figshare · doi:10.6084/m9.figshare.26615062
Final Model Reporting in Oncology Prediction Model Studies
Gao, Y., Wu, M., Yi, H., Xu, B., Gan, T., & Wu, I. X. (2025) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.70102
Ai-Driven Risk-Based Monitoring in Oncology Clinical Trials
Dandu, H., & Singh, S. (2026) · International Conference on Communication, Computing and Electronics Systems (ICCCES) · doi:10.1109/iccces62661.2026.11436612
Piyavechvirat, N., Huang, Y.-J., & Haq, Q. M. U. (2026) · Scientific Reports · doi:10.1038/s41598-026-47362-4
Huet‐Dastarac, M., Dankar, F. K., Liu, D., Kababji, S. E., Pilgram, L., & Emam, K. E. (2026) · Journal of Medical Internet Research · doi:10.2196/88678
Risk factors and nomogram prediction model for prognosis in sepsis with acute kidney injury
Lai, J.-M., Zheng, Y., Liang, T.-Y., Huang, J., & Quan, Y.-Y. (2026) · BMC Nephrology · doi:10.1186/s12882-026-05039-0
Tan, L., Lu, D., Huang, H., Lin, D., & Zheng, D. (2026) · Frontiers in Molecular Biosciences · doi:10.3389/fmolb.2026.1811387
Abdelhay, O. (2026) · Intelligence-Based Medicine · doi:10.1016/j.ibmed.2026.100422 · Cited by 1
Pan-Cancer Gene Expression Predicts Per-Cancer Radiation Therapy Response
Glickman, H., & Enger, S. A. (2026) · Advances in Radiation Oncology · doi:10.1016/j.adro.2026.102140
Ahmed, E. N., Ahmed, M. S., & Mushtaq, A. H. (2026) · Precision Oncology · doi:10.3390/precisoncol1030011
Yuan, D.-F., Liu, Y.-Y., Liu, S.-S., Peng, N.-N., Zhu, X.-Y., Qian, Q.-J., Wang, B., & Yin, Y.-Q. (2026) · Frontiers in Immunology · doi:10.3389/fimmu.2026.1846559
Machine learning-enabled prediction of ART pregnancy outcomes: a systematic review and meta-analysis
Li, B., Liu, H., Yu, F., Xiong, M., Tang, T., Wu, R.-H., Zhao, S.-M.-Z., Shi, C.-L., Zhang, T.-W., Yao, B., et al. (2026) · Journal of Assisted Reproduction and Genetics · doi:10.1007/s10815-026-03988-x
Zhou, Y., Sun, B., Cai, X., & Qi, J. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1841719
Abidin, N. Z., Shariff, N. M., Zamri, E. N., & Shamsuddin, S. (2026) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2026.103506
Riaz, S., Khuda, F., Nasim, A., Qazi, R., Alnemari, R. M., & Alsubaie, N. (2026) · Psychiatry Research · doi:10.1016/j.psychres.2026.117432
Prakash, G. H., Tejaswini, B. D., Ganesh, B., & Kumar, D. S. (2026) · Measurement and Evaluations in Cancer Care · doi:10.1016/j.ymecc.2026.100041
Lin, Y., Sun, S., Yang, S., & Sun, J. (2026) · Frontiers in Immunology · doi:10.3389/fimmu.2026.1891348
Wang, L., Wang, M., Chen, Z., Chuanlin, C., Ma, Y., Sui, M., & 应晓华 (2026) · Research Square · doi:10.21203/rs.3.rs-10892305/v1
Almansour, N. M. (2026) · Biomedicines · doi:10.3390/biomedicines14102298
Zhang, S., Peng, M., Liu, G., & 吴拥军 (2026) · Research Square · doi:10.21203/rs.3.rs-11021999/v1
Metadata
- Authors
- Dhiman, P., Ma, J., Andaur Navarro, C. L., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Van Calster, B., Moons, K. G. M., & Collins, G. S.
- Affiliations
- University of Oxford (GB)
- University of Basel (CH)
- Utrecht University (NL)
- Keele University (GB)
- Corresponding author
- Dhiman, Paula
- ORCID
- Venue
- BMC Medical Research Methodology
- Published
- 2022-04-08
- Volume
- 22
- Issue
- 1
- Pages
- 101-101
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC8991704
- OpenAlex
- W4223540626
- Cited by
- 140
- References
- 126
- Citations by year
- 2026: 37 · 2025: 34 · 2024: 42 · 2023: 23 · 2022: 4
- FWCI
- 3.72
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Keele Research Repository (Keele University) (submittedVersion, cc-by)
- Lirias (KU Leuven) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Utrecht University Repository (Utrecht University) (submittedVersion, other-oa)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, other-oa)
- Funders
- Cancer Research UK (27294)
- National Institute for Health and Care Research
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- prognostic model, machine learning, oncology, methodological quality, tripod, PROBAST, charm, regression models, ensemble learning, sample size justification, predictor selection, time-to-event outcomes, internal validation, calibration, cancer prognosis, cancer survival prediction, cancer risk prediction, tumor response prediction, tumor recurrence
- MeSH terms
- Machine Learning, Humans, Medical Oncology, Prognosis, Research Design, Bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
