The DOI matches its Crossref record; 156 of the 157 references it cites were found in a registry or an index.
Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies: The CHARMS Checklist
Open accessAbstract
Carl Moons and colleagues provide a checklist and background explanation for critically appraising and extracting data from systematic reviews of prognostic and diagnostic prediction modelling studies. Please see later in the article for the Editors' Summary.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Previous guidance for systematic reviews of prediction model studies mainly addressed formulation of the review aim, searching,12 critical appraisal (CHARMS)13 and data extraction of relevant studies. There is, however, little guidance on how to quantitatively synthesize the results of external validation studies.”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [13]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Data extraction was conducted independently by 4 reviewers (R.Q., F.G, Y.A., and S.M.), following the recommendations in the CHARMS checklist.23”
From Linguistic markers for identifying post-traumatic stress disorder and associated symptoms: a systematic literature review (2025) · cites this paper as [23]
The cited source says what the claim attributes to it.
In the citing paper
“Currently there is little information available to guide the development of a well-defined review question however the Quality in Prognosis Studies (QUIPS) tool [34] and the Checklist for critical appraisal and data extraction for systematic reviews of prediction modelling studies (CHARMS Checklist) [38] have been developed to assist in this process (see Table 1).”
From What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences (2018) · cites this paper as [38]
The cited source says what the claim attributes to it.
In the citing paper
“The Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) was employed to define the objectives of this systematic review and meta-analysis (29).”
From Machine learning-based risk predictive models for diabetic kidney disease in type 2 diabetes mellitus patients: a systematic review and meta-analysis (2025) · cites this paper as [29]
The cited source says what the claim attributes to it.
In the citing paper
“We employed the CHARMS checklist and the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guidelines to develop standardized forms for data extraction (29, 32).”
From Machine learning-based risk predictive models for diabetic kidney disease in type 2 diabetes mellitus patients: a systematic review and meta-analysis (2025) · cites this paper as [29]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Self-citationDermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
Self-citationvon Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-8-200710160-00010 · Cited by 17,796
Self-citationQUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Self-citation2013 ACCF/AHA Guideline for the Management of Heart Failure
Yancy, C. W., Jessup, M., Bozkurt, B., Butler, J., Casey, D. E., Drazner, M. H., Fonarow, G. C., Geraci, S. A., Horwich, T., Januzzi, J. L., et al. (2013) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.05.019 · Cited by 12,536
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349
Self-citationMultiple imputation using chained equations: Issues and guidance for practice
White, I. R., Royston, P., & Wood, A. M. (2011) · Statistics in Medicine · doi:10.1002/sim.4067 · Cited by 9,792 · cited 1×
“Numerous recommendations for reporting missing data in medical research have been proposed [103,104,114,115].”
BackgroundA simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655 · cited 1×
“The number of events-pervariable (EPV) is commonly used to calculate the sample size, where attaining a sample size with an EPV of ten or more is frequently recommended to avoid overfitting [97–101].”
Method
Cited by
Items 501–600 of 1,004
Grüter, B. E., Sun, W., Fierstra, J., Regli, L., & Germans, M. R. (2021) · Neurosurgical Review · doi:10.1007/s10143-020-01464-3 · Cited by 23
Morin, M. P., Dubuc, J., Freycon, P., & Buczinski, S. (2021) · Journal of Dairy Science · doi:10.3168/jds.2020-19475 · Cited by 31
Predictors of COVID-19 Vaccine Uptake in Healthcare Workers: A Cross-Sectional Study in Greece
Galanis, P., Moisoglou, I., Vraka, I. S., Siskou, O. C., Konstantakopoulou, O., Katsiroumpa, A., & Καϊτελίδου, Δ. (2021) · Journal of Occupational and Environmental Medicine · doi:10.1097/jom.0000000000002463 · Cited by 25
“Twenty-three predictor variables were used and a minimum number of 230 vaccinated HCWs were included in our study so as to perform a valid multivariate regression analysis.(13) Finally, a sample size substantial increase was in order to minimize random error.”
MethodAtkins, D., Makridis, C. A., Alterovitz, G., Ramoni, R. B., & Clancy, C. M. (2022) · Annual Review of Biomedical Data Science · doi:10.1146/annurev-biodatasci-122220-110053 · Cited by 25
Overton, M., Swain, N., Falling, C., Gwynne‐Jones, D., Fillingim, R. B., & Mani, R. (2023) · Frontiers in Pain Research · doi:10.3389/fpain.2022.1082252 · Cited by 31
van Ravenhorst, C. G., Schluep, M., Endeman, H., Stolker, R.-J., & Hoeks, S. E. (2023) · Critical Care · doi:10.1186/s13054-023-04306-y · Cited by 23
“Strengths and limitations This study contains a comprehensive search and extensive analysis using current guidelines for reviewing and assessing bias of prediction model studies [7, 17].”
MethodEvaluating the performance of the National Early Warning Score in different diagnostic groups
Price, C., Prytherch, D. R., Kostakis, I., & Briggs, J. (2023) · Resuscitation · doi:10.1016/j.resuscitation.2023.110032 · Cited by 26
Volkers, E. J., Algra, A., Kappelle, L. J., & Greving, J. P. (2017) · European Stroke Journal · doi:10.1177/2396987317739122 · Cited by 23
Aluvaala, J., Collins, P. G. S., Maina, M., Berkley, J. A., & English, M. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0649-6 · Cited by 28
“(DOCX 17 kb) Abbreviations AUROC: Area under the receiver operating characteristic curve; CHARMS: CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies; CRIB: Clinical risk index for babies; GRADE: Grading of Recommendations Assessment, Development, and Evaluation;…”
BackgroundMethodInfluentialAn internally validated prognostic model for success in revision stapes surgery for otosclerosis
Wegner, I., Vincent, R., Derks, L. S. M., Rauh, S. P., Heymans, M. W., Stegeman, I., & Grolman, W. (2018) · The Laryngoscope · doi:10.1002/lary.27132 · Cited by 23
“Selection of predictors based on univariable analysis carries a high risk of predictor selection bias.(39) Predictor selection bias occurs when predictors are selected that have a large but invalid association with the outcome.”
BackgroundLi, Y., Jin, N., Zhan, Q., Huang, Y., Sun, A., Yin, F., Li, Z., Hu, J., & Liu, Z. (2025) · Frontiers in Endocrinology · doi:10.3389/fendo.2025.1495306 · Cited by 35
Yoong, S. Q., Porock, D. C., Whitty, D., Tam, W., & Zhang, H. (2023) · Palliative Medicine · doi:10.1177/02692163231180657 · Cited by 24
Appel, K. S., Geisler, R., Maier, D., Miljukov, O., Pütz, S. M., & Vehreschild, J. J. (2023) · Clinical Infectious Diseases · doi:10.1093/cid/ciad618 · Cited by 21
“In future pandemics, data and resource sharing alongside the application of recommended model development and reporting guidelines [6, 7, 15, 16] would improve score quality and visibility, leading to better implementation for the benefit of the patients.”
BackgroundMethodInfluentialPrediction of Morbidity and Mortality After Esophagectomy: A Systematic Review
Amerongen, M. P. V. N., de Grooth, H., Veerman, G., Ziesemer, K. A., Henegouwen, M. I. V. B., & Tuinman, P. R. (2024) · Annals of Surgical Oncology · doi:10.1245/s10434-024-14997-4 · Cited by 32
Deprez, J., Ohde, N., Eilegård, A., Bååth, C., Hommel, A., Hultin, L., Josefson, A., Kottner, J., & Beeckman, D. (2024) · International Wound Journal · doi:10.1111/iwj.14962 · Cited by 21
“Key terms, defined through the PICOTS framework, 17 – 19 including incontinence associated dermatitis and prognostic factors using various combinations, were used.”
MethodZhu, Y., Du, M., Li, P., Lu, H., Li, A., & Xu, S. (2025) · International Journal of Implant Dentistry · doi:10.1186/s40729-025-00590-1 · Cited by 29
Risk prediction models for frailty in older adults: A systematic review and critical appraisal
Kong, L., Li, Y., Lyu, Q., Liu, D., & Yang, J. (2025) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2025.105068 · Cited by 22
Nayebirad, S., Hassanzadeh, A., Vahdani, A. M., Mohamadi, A., Forghani, S., Shafiee, A., & Masoudkabir, F. (2025) · BMC Cardiovascular Disorders · doi:10.1186/s12872-025-04746-0 · Cited by 26
“Critical appraisal and data extraction for systematic reviews of prediction modeling studies (CHARMS) checklist [14] and prediction model risk of bias assessment tool (PROBAST) [15] were used by two independent reviewers (S.N and A.”
Meisner, A., Kerr, K. F., Thiessen‐Philbrook, H., Coca, S. G., & Parikh, C. R. (2015) · Kidney International · doi:10.1038/ki.2015.283 · Cited by 24
Engelhardt, E. G., Révész, D., Tamminga, H. J., Punt, C. J. A., Koopman, M., Onwuteaka-Philipsen, B. D., Steyerberg, E. W., Jansma, I. P., de Vet, H. C. W., & Coupé, V. M. H. (2017) · Clinical Colorectal Cancer · doi:10.1016/j.clcc.2017.06.007 · Cited by 22
Heestermans, T., Payne, B. A., Kayode, G. A., Amoakoh‐Coleman, M., Schuit, E., Rijken, M. J., Klipstein‐Grobusch, K., Bloemenkamp, K. W. M., Grobbee, D. E., & Browne, J. L. (2019) · BMJ Global Health · doi:10.1136/bmjgh-2019-001759 · Cited by 27
“data collection process and data items Data extraction was performed by one author (TH) through a piloted standardised form, which was based on the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) checklist (online supplementary data 4).(16) General s…”
MethodFINDRISC in Latin America: a systematic review of diagnosis and prognosis models
Carrillo-Larco, R. M., Aparcana-Granda, D. J., Mejia, J. R., & Bernabé‐Ortiz, A. (2020) · BMJ Open Diabetes Research & Care · doi:10.1136/bmjdrc-2019-001169 · Cited by 27
Systematic review of machine learning models for personalised dosing of heparin
Falconer, N., Abdel‐Hafez, A., Scott, I. A., Marxen, S., Canaris, S., & Barras, M. A. (2021) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.14852 · Cited by 20
“14 Data preparation Collection of tasks used to prepare the data for analysis and modelling, including data cleansing, handling of missing values, data transformation and normalisation, and addressing data imbalances.”
BackgroundMethodInfluentialPrediction Models for Prognosis of Cervical Cancer: Systematic Review and Critical Appraisal
He, B., Chen, W., Liu, L., Hou, Z., Zhu, H., Cheng, H., Zhang, Y., Zhan, S., & Wang, S. (2021) · Frontiers in Public Health · doi:10.3389/fpubh.2021.654454 · Cited by 25
“The assessment of ROB in PROBAST involved four domains (participants, predictors, outcome, and analysis), and 20 signaling questions covered the key aspects of prediction model studies.”
MethodInfluentialNies, K., Smits, L. J. M., Kassem, M., Nederkoorn, P. J., van Oostenbrugge, R. J., & Kooi, M. E. (2021) · Frontiers in Neurology · doi:10.3389/fneur.2021.718438 · Cited by 22
Risk prediction models for esophageal cancer: A systematic review and critical appraisal
Li, H., Sun, D., Cao, M., He, S., Zheng, Y., Yu, X., Wu, Z. L., Lei, L., Peng, J., Li, J., et al. (2021) · Cancer Medicine · doi:10.1002/cam4.4226 · Cited by 23
Deawjaroen, K., Sillabutra, J., Poolsup, N., Stewart, D. C., & Suksomboon, N. (2021) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.15104 · Cited by 26
“Limited reporting of these measures may be due to the requirement of an optimal predefined threshold.(52) It is a challenge to set threshold balancing sensitivity and specificity values, as a model with higher sensitivity will report lower specificity.”
BackgroundMethodTran, A., Fernando, S. M., Rochwerg, B., Cook, D. J., Crowther, M. A., Fowler, R. A., Alhazzani, W. A., Siegal, D. M., Castellucci, L. A., Zarychanski, R., et al. (2021) · Critical Care Medicine · doi:10.1097/ccm.0000000000005382 · Cited by 29
“We performed extraction using a tool modified from the CHARMS checklist for prognostic factors (25).”
MethodAlharbi, E., Nadeem, F., & Cherif, A. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-021-01704-6 · Cited by 22
“Moons KGM, de Groot JAH, Bouwmeester W, Vergouwe Y, Mallett S, Altman DG, Reitsma JB, Collins GS. Critical appraisal and data extraction for systematic reviews of prediction modelling studies: the CHARMS checklist.”
MethodInfluentialWESSELS, F., Kuntz, S., KRIEGHOFF-HENNING, E., SCHMITT, M., BRAUN, V., Worst, T. S., NEUBERGER, M., STEEG, M., Gaiser, T., FRÖHLING, S., et al. (2022) · Minerva Urology and Nephrology · doi:10.23736/s2724-6051.22.04758-9 · Cited by 25
Reproducibility of prediction models in health services research
Belbasis, L., & Panagiotou, O. A. (2022) · BMC Research Notes · doi:10.1186/s13104-022-06082-4 · Cited by 24
“To support the conduct of systematic reviews of prediction models, there is a validated search algorithm for prediction modelling studies in PubMed [34], and a guidance on how to construct a data extraction form [35].”
MethodLi, Y., Xie, F., Xiong, Q., Lei, H., & Feng, P. (2022) · Frontiers in Oncology · doi:10.3389/fonc.2022.946038 · Cited by 25
“The list of extraction items was designed based on the modi fi ed version of Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) (77).”
MethodOtokiti, A. U., Ozoude, M. M., Williams, K. S., Sadiq-Onilenla, R. A., Ojo, S. A., Warsame, L., Walsh, S., & Edomwande, M. (2022) · JMIR Research Protocols · doi:10.2196/37685 · Cited by 30
“Data sources for model development are critical for the predictive accuracy, applicability, and reproducibility of any algorithm [11,12,43,50,51].”
InfluentialUtility of risk prediction models to detect atrial fibrillation in screened participants
Poorthuis, M. H. F., Jones, N. R., Sherliker, P., Clack, R. M., de Borst, G. J., Clarke, R., Lewington, S., Halliday, A. W., & Bulbulia, R. A. (2020) · European Journal of Preventive Cardiology · doi:10.1093/eurjpc/zwaa082 · Cited by 25
Prognostic factors and prediction models for acute aortic dissection: a systematic review
Ren, Y., Huang, S., Li, Q., Liu, C., Li, L., Tan, J., Zou, K., & Sun, X. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-042435 · Cited by 23
Hu, P., Xu, Y., Liu, Y., Li, Y., Ye, L., Zhang, S., Zhu, X., Qi, Y., Zhang, H., Sun, Q., et al. (2021) · Frontiers in Neurology · doi:10.3389/fneur.2021.683051 · Cited by 22
Muttalib, F., Clavel, V., Yaeger, L., Shah, V. S., & Adhikari, N. K. J. (2020) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2020.05.016 · Cited by 29
Zhang, W., Tang, Y., Liu, H., Li, Y., Wang, C. C., Chen, S. F., Huang, J., & Xiao, X. Y. (2021) · PLoS ONE · doi:10.1371/journal.pone.0257768 · Cited by 21
“After confirming the inclusion of the literature, we extracted the data by using the CHARMS checklist (the checklist for critical appraisal and data extraction for systematic reviews of prediction modeling studies) [11], which mainly includes the author(s) and publication year, country or region considered, study desi…”
MethodPrediction of delirium using data mining: A systematic review
Chua, S. J., Wrigley, S., Hair, C., & Sahathevan, R. (2021) · Journal of Clinical Neuroscience · doi:10.1016/j.jocn.2021.07.029 · Cited by 22
Risk Prediction Models for Inadvertent Intraoperative Hypothermia: A Systematic Review
Yan, L., Yao, L., Zhao, Q., Xiao, M., Li, Y., & Min, S. (2021) · Journal of PeriAnesthesia Nursing · doi:10.1016/j.jopan.2021.02.011 · Cited by 21
Vannucci, M., Laracca, G. G., Mercantini, P., Perretta, S., Padoy, N., Dallemagne, B., & Mascagni, P. (2021) · Surgery · doi:10.1016/j.surg.2021.10.001 · Cited by 31
Lozanovski, V. J., Ramouz, A., Aminizadeh, E., Ali‐Hasan‐Al‐Saegh, S., Khajeh, E., Probst, H., Picardi, S., Rupp, C., Chang, D., Probst, P., et al. (2021) · BJS Open · doi:10.1093/bjsopen/zrab130 · Cited by 22
Poénou, G., Toledano, E., Helfer, H., Plaisance, L., Happe, F., Versini, E., Diab, N., Djennaoui, S., & Mahé, I. (2022) · Cancers · doi:10.3390/cancers14081937 · Cited by 22
“Data extraction was performed according to the CHARMS checklist by one researcher (GP) using a predesigned data extraction form and an assessment of risk of bias was performed with the PROBAST list [24] (Appendix A).”
MethodHueting, T. A., van Maaren, M. C., Hendriks, M. P., Koffijberg, H., & Siesling, S. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.10.016 · Cited by 25
van der Ende, N. A. M., Kremers, F. C. C., van der Steen, W., Venema, E., Kappelhof, M., Majoie, C. B. L. M., Postma, A. A., Boiten, J., van den Wijngaard, I. R., van der Lugt, A., et al. (2023) · Stroke · doi:10.1161/strokeaha.122.040065 · Cited by 21
Philip, M. M., Welch, A. E., McKiddie, F. I., & Nath, M. (2023) · Cancer Medicine · doi:10.1002/cam4.6278 · Cited by 24
Mertens, E., Barrenechea-Pulache, A., Sagastume, D., Vasquez, M. S., Vandevijvere, S., & Peñalvo, J. L. (2023) · BMC Cancer · doi:10.1186/s12885-023-11174-w · Cited by 29
“Data extraction and synthesis Data extraction for each risk prediction model was performed in duplicate using a standardised electronic excel template based on the framework of the CHARMS (critical appraisal and data extraction for systematic reviews of prediction modelling studies) checklist [23].”
MethodCollins, P. G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Diabetic Medicine · doi:10.1111/dme.12654 · Cited by 25
Self-citationEliciting regret improves decision making at the end of life
Djulbegović, B., Tsalatsanis, A., Mhaskar, R. S., Hozo, I., Miladinović, B., & Tuch, H. (2016) · European Journal of Cancer · doi:10.1016/j.ejca.2016.08.027 · Cited by 21
“Because of the estimated high event rate (death), we assumed that between 170 and 200 patients would provide adequate power for the main goals of the analysis [25].”
BackgroundTreatment use in prognostic model research: a systematic review of cardiovascular prognostic studies
Pajouheshnia, R., Damen, J. A., Groenwold, R. H. H., Moons, K. G. M., & Peelen, L. M. (2017) · Diagnostic and Prognostic Research · doi:10.1186/s41512-017-0015-0 · Cited by 28
“Directed by the CHARMS checklist [13], a list of key items (Additional file 2) for extraction was derived for the current review by one author (RP) and updated after group consideration (RP, LMP, RHHG, JAAGD, KGMM).”
MethodSelf-citationSystematic review of risk prediction scores for venous thromboembolism following joint replacement
Kunutsor, S. K., Beswick, A. D., Whitehouse, M. R., & Blom, A. W. (2018) · Thrombosis Research · doi:10.1016/j.thromres.2018.06.024 · Cited by 22
“We conducted this review in accordance with the CHARMS checklist[14] and PRISMA guidelines,[15] (Appendix A) and using a predefined protocol, which has been registered in the PROSPERO prospective register of systematic reviews (CRD42018088712).”
MethodA Systematic Review of Clinical Prediction Rules for the Diagnosis of Chronic Heart Failure
Gallagher, J., McCormack, D., Zhou, S., Ryan, F., Watson, C. J., McDonald, K., & Ledwidge, M. T. (2019) · ESC Heart Failure · doi:10.1002/ehf2.12426 · Cited by 23
“The systematic review was undertaken according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)(7) guidelines and the CHARMS checklist for systematic reviews of prediction models.(8)”
MethodWang, T., Lü, R., Lai, S., Schiller, J. H., Zhou, F. L., Ci, B., Wang, S., Gao, X., Yao, B., Gerber, D. E., et al. (2019) · Cancer Informatics · doi:10.1177/1176935119837547 · Cited by 21
Veirman, E., Ryckeghem, D. M. L. V., Paepe, A. L. D., Kirtley, O. J., & Crombez, G. (2019) · PAIN Reports · doi:10.1097/pr9.0000000000000775 · Cited by 22
“They were judged as appropriately handled when multiple imputation was used.(74) For example, Karran et al.”
BackgroundMethodResultInfluentialCooray, S. D., Wijeyaratne, L. A., Soldatos, G., Allotey, J., Boyle, J. A., & Teede, H. J. (2020) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph17093048 · Cited by 27
“Model Development Two studies were inadequately powered with less than 10 events per predictor (EPP) [23,25] (Table 4 and Figure 3), generating significant risks for overfitting and consequently biased predictions [19].”
BackgroundMethodInfluentialRadiomics in Carotid Plaque: A Systematic Review and Radiomics Quality Score Assessment
Hou, C., Liu, X., Du, Y., Cheng, L., Liu, L., Nie, F., Zhang, W., & He, W. (2023) · Ultrasound in Medicine & Biology · doi:10.1016/j.ultrasmedbio.2023.06.008 · Cited by 20
Ho, L. T. F., Pugh, C., Seth, S., Arakelyan, S., Lone, N., Lyall, M. J., Anand, A., Fleuriot, J., Galdi, P., & Guthrie, B. (2024) · The Lancet Healthy Longevity · doi:10.1016/s2666-7568(23)00264-7 · Cited by 22
Malik, S., Tenorio, B. G., Moond, V., Dahiya, D. S., Vora, R. S., & Dbouk, N. (2024) · Journal of Gastroenterology and Hepatology · doi:10.1111/jgh.16645 · Cited by 20
“…data as per the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modeling Studies (the CHARMS checklist), 14 covering study type, author details, publication year, participant demographics, data sources, the etiology of liver disease, methods used to…”
Sun, Y., Wen, X., Zhang, Y., Jin, L., Yang, C., Zhang, Q., Jiang, M. C., Xu, Z., Guo, W., Su, J., et al. (2025) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2025.108870 · Cited by 33
Prediction models for self-harm and suicide: a systematic review and critical appraisal
Seyedsalehi, A., Bailey, J. P. M., Ogonah, M. G. T., Fanshawe, T. R., & Fazel, S. (2025) · BMC Medicine · doi:10.1186/s12916-025-04367-6 · Cited by 28
Framework for Using Risk Stratification to Improve Clinical Preventive Service Guidelines
Lin, J. S., Evans, C. V., Grossman, D. C., Tseng, C., & Krist, A. H. (2017) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2017.07.023 · Cited by 22
“In general, consideration of a risk-stratified approach may be important if one or more of the following conditions exist: (1) There are known clinically important differences in the distribution of disease (or disease risk factors), or variation in prognosis across populations; (2) in updating existing guidelines, pr…”
BackgroundMethodEvidence for Residual Immunity to Smallpox After Vaccination and Implications for Re-emergence
Kunasekaran, M. P., Chen, X., Costantino, V., Chughtai, A. A., & MacIntyre, C. R. (2019) · Military Medicine · doi:10.1093/milmed/usz181 · Cited by 27
Risk prediction models for acute kidney injury in adults: An overview of systematic reviews
Acker, P. V., Biesen, W. V., Nagler, E. V., Koobasi, M., Veys, N., & Vanmassenhove, J. (2021) · PLoS ONE · doi:10.1371/journal.pone.0248899 · Cited by 20
“We recommend future SRs in the field to use the CHARMS (Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies) checklist [7] to ensure all relevant data from the individual studies are extracted; apply the PROBAST tool for individual risk of bias assessment for each development…”
MethodLocquet, M., Diep, A. N., Beaudart, C., Dardenne, N., Brabant, C., Bruyère, O., & Donneau, A. (2021) · Archives of Public Health · doi:10.1186/s13690-021-00630-3 · Cited by 20
“ALC: Absolute Lymphocyte Count; ANC: Absolute Neutrophil Count; AST: ASpartate aminoTransferase; AUROC: Area Under the ROC curve; BUN: Blood Urea Nitrogen; CAD: Coronary Artery Disease; CHARMS: CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies; CI: Confidence I…”
MethodInfluentialAppleyard, T., Thomas, M. J., Antcliff, D., & Peat, G. (2022) · Arthritis Care & Research · doi:10.1002/acr.25035 · Cited by 25
“Data extraction of eligible studies was performed by paired reviewers (TA and MJT, DA and GP), using a shared Microsoft Excel (17) worksheet incorporating items for extraction as outlined by Cochrane (18) and the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies checklist (19…”
MethodPredictive models in emergency medicine and their missing data strategies: a systematic review
Arnaud, É., Elbattah, M., Ammirati, C., Dequen, G., & Ghazali, D. A. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00770-6 · Cited by 24
“The data collection process Data in the included publications (publication references, the relevant CHARMS items, missing data strategies, the dataset split, and best AUROC value for training and validation stages) were entered on an Excel ® (Microsoft Corp., Cupertino, USA) spread-sheet by one investigator (E.A.) and…”
ResultInfluentialPrediction model for cognitive frailty in older adults: A systematic review and critical appraisal
Huang, J., Zeng, X., Hu, M., Ning, H., Wu, S., Peng, R., & Feng, H. (2023) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2023.1119194 · Cited by 21
“KEYWORDS cognitive frailty, prediction models, older adults, systematic review, critical appraisal Abbreviations: AUC, area under the receiver operating curve; CF, cognitive frailty; CHARMS, Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies; EPV, event per variable; IAGG, In…”
MethodThe Prognostic Utility of Anticholinergic Burden Scales: An Integrative Review and Gap Analysis
Michael, H. U., Enechukwu, O. H., Brouillette, M., Tamblyn, R. M., Fellows, L. K., & Mayo, N. E. (2023) · Drugs & Aging · doi:10.1007/s40266-023-01050-4 · Cited by 30
Machine learning prediction models for diabetic kidney disease: systematic review and meta-analysis
Chen, L., Shao, X., & Yu, P. (2023) · Endocrine · doi:10.1007/s12020-023-03637-8 · Cited by 27
“To assess the reported prediction models, reviewers used a standardized data extraction form based on the CHARMS checklist [16] (checklist for critical appraisal and data extraction for systematic reviews of prediction modeling studies).”
Mudrik, A., Nadkarni, G. N., Efros, O., Glicksberg, B. S., Klang, E., & Soffer, S. (2024) · British Journal of Haematology · doi:10.1111/bjh.19738 · Cited by 27
Byrne, C. J., Toarta, C., Backus, B. E., & Holt, T. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0816-4 · Cited by 23
Personalized Pancreatic Cancer Management
Bradley, A., Meer, R. V. D., & McKay, C. J. (2019) · Pancreas · doi:10.1097/mpa.0000000000001312 · Cited by 20
Predictive models of diabetes complications: protocol for a scoping review
Ndjaboué, R., Farhat, I., Ferlatte, C.-A., Ngueta, G., Guay, D., Delorme, S., Ivers, N. M., Shah, B. R., Straus, S. E., Yu, C. H. Y., et al. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01391-w · Cited by 22
Gastric perforation in a case of gastric volvulus.
Farag, S., Fiallo, V. M., Nash, S. R., & Navab, F. (1996) · PubMed · Cited by 20
Risk Factors for Falls Among Hospitalized Medical Patients – A Systematic Review and Meta-analysis
Heinzmann, J., Rossen, M. L., Efthimiou, O., Baumgartner, C., Wertli, M. M., Rodondi, N., Aubert, C. E., & Liechti, F. D. (2024) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2024.06.015 · Cited by 27
Burke, L., Hinkson, A., Haghnejad, V., Jones, R. L., Parker, R., & Rowe, I. A. C. (2024) · JHEP Reports · doi:10.1016/j.jhepr.2024.101227 · Cited by 29
“One reviewer (LB) extracted the relevant data from eligible studies using a standardised proforma derived from the CHARMS checklist 24 and a sample was checked by a second reviewer (IR).”
Galanty, M., Luitse, D. M. R., Noteboom, S. H., Croon, P. M., Vlaar, A. P. J., Poell, T., Sánchez, C. I., Blanke, T., & Išgum, I. (2024) · Scientific Reports · doi:10.1038/s41598-024-83218-5 · Cited by 26
“This tool specifically draws from existing dataset documentation tools and guidelines, including Datasheets for Datasets 15 , BIAS 18 , and the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist 10 .”
InfluentialTeles, A. S., de Moura, I. R., Silva, F., Roberts, A., & Stahl, D. R. (2025) · Information Fusion · doi:10.1016/j.inffus.2025.102981 · Cited by 27
Dixon, G., Thould, H., Wells, M., Tsaneva-Atanasova, K. T., Scotton, C. J., Gibbons, M., Barratt, S. L., & Rodrigues, J. C. L. (2025) · European Respiratory Review · doi:10.1183/16000617.0194-2024 · Cited by 32
Nies, K., Smits, L. J. M., van Kuijk, S. M. J., Hosseini, A. A., van Dam-Nolen, D. H. K., Kwee, R. M., Kurosaki, Y., Rupert, I., Nederkoorn, P. J., de Jong, P. A., et al. (2025) · Stroke · doi:10.1161/strokeaha.124.050020 · Cited by 19
Tortosa, F. G., Perré, F., Neumann, I., Ragusa, M. A., Lossetti, L., Guaresti, G., & Izcovich, A. (2026) · BMJ Open · doi:10.1136/bmjopen-2024-096313 · Cited by 22
Bradley, A., Meer, R. V. D., & McKay, C. J. (2019) · BMJ Open · doi:10.1136/bmjopen-2018-027192 · Cited by 19
“The use of a small p value has the benefit generating a model from fewer predictors but carries the risk of missing potentially important variables while the use of larger p values potentiates inclusion of predictors of less importance.(15) One study reported using multivariable analysis for predictor selection determ…”
BackgroundMethodInfluentialAstarita, A., Mingrone, G., Airale, L., Vallelonga, F., Covella, M., Catarinella, C., Cesareo, M., Bruno, G., Leone, D., Giordana, C., et al. (2021) · Cancers · doi:10.3390/cancers13071631 · Cited by 22
“The score was internally validated by bootstrapping according to the CHARMS Recommendations [15]: 1000 random samplings with replacement from the original sample were performed in order to assess the discrimination and the calibration of the score.”
MethodInfluentialMonahan, A. C., & Feldman, S. S. (2021) · JMIR Medical Informatics · doi:10.2196/30022 · Cited by 23
“Furthermore, all studies corresponded to the outcome definition of the systematic review question, which reduced the potential for bias from different outcome definitions and measurement methods that can lead to differences in study results and would be a source of heterogeneity across studies [34].”
BackgroundMethodResultInfluentialPoorthuis, M. H. F., Herings, R. A. R., Dansey, K. D., Damen, J. A., Greving, J. P., Schermerhorn, M. L., & de Borst, G. J. (2021) · Stroke · doi:10.1161/strokeaha.120.032527 · Cited by 20
Klunder, J. H., Panneman, S. L., Wallace, E., de Vries, R., Joling, K. J., Maarsingh, O. R., & van Hout, H. P. J. (2022) · PLoS ONE · doi:10.1371/journal.pone.0275116 · Cited by 23
Stojanov, T., Audigé, L., Modler, L., Aghlmandi, S., Appenzeller‐Herzog, C., Loucas, R., Loucas, M., & Müller, A. M. (2022) · JSES International · doi:10.1016/j.jseint.2022.09.003 · Cited by 20
Risk prediction models for breast cancer-related lymphedema: A systematic review and meta-analysis
Shen, A., Wei, X., Zhu, F., Sun, M., Ke, S., Qiang, W., & Lü, Q. (2023) · European Journal of Oncology Nursing · doi:10.1016/j.ejon.2023.102326 · Cited by 23
Vagliano, I., Dormosh, N., Rios, M., Luik, T. T., Buonocore, T. M., Elbers, P., Dongelmans, D. A., Schut, M. C., & Abu–Hanna, A. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104504 · Cited by 22
Using microRNAs as Novel Predictors of Urologic Cancer Survival: An Integrated Analysis
Chen, Z., Zhan, Y., Chi, J., Guo, S., Zhong, X., He, A., Zheng, J., Gong, Y., Li, X., & Zhou, L. (2018) · EBioMedicine · doi:10.1016/j.ebiom.2018.07.014 · Cited by 19
“A standardized table was developed by two authors (Guo and Zhong) according to the CHARMS checklist [34].”
BackgroundMethodThe added value of health indicators to mortality predictions in old age: A systematic review
Kusumastuti, S., Rozing, M. P., Lund, R., Mortensen, E. L., & Westendorp, R. G. J. (2018) · European Journal of Internal Medicine · doi:10.1016/j.ejim.2018.06.019 · Cited by 20
Cai, R., Wu, X., Li, C., & Chao, J. (2020) · Journal of Hypertension · doi:10.1097/hjh.0000000000002442 · Cited by 20
“We conducted this systematic review following the guidance from the Cochrane Prognosis Methods Group and the CHARMS checklist [10] and present this review in accordance with the Preferred Reporting Items for Systematic Reviews and Meta Analyses (PRISMA) [11] (Supplemental 1, http://links.”
MethodRoheger, M., Folkerts, A., Krohm, F., Skoetz, N., & Kalbe, E. (2020) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-0071-8 · Cited by 22
Models to predict relapse in psychosis: A systematic review
Sullivan, S. A., Northstone, K., Gadd, C., Walker, J. S., Margelyte, R., Richards, A., & Whiting, P. F. (2017) · PLoS ONE · doi:10.1371/journal.pone.0183998 · Cited by 25
Liu, X., Liu, X., Jin, C., Luo, Y., Yang, L., Ning, X., Zhuo, C., & Xiao, F. (2024) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2024.07.005 · Cited by 20
Guidelines on cerclage placement: a comparative systematic review
Mudrik, A., Levy, R., Petrecca, A., Gulersen, M., Chauhan, S. P., Erez, O., & Rottenstreich, M. (2025) · American Journal of Obstetrics & Gynecology MFM · doi:10.1016/j.ajogmf.2025.101727 · Cited by 18
Predictors of recurrent venous thromboembolism and bleeding in patients with cancer: a meta-analysis
Khan, F. A., Tritschler, T., Marx, C. E., Lanting, V. R., Rochwerg, B. N., Tran, A., Fernando, S. M., Lorenzetti, D., Wunsch, H., Holodinsky, J. K., et al. (2025) · European Heart Journal · doi:10.1093/eurheartj/ehaf453 · Cited by 23
Cooray, S. D., Boyle, J. A., Soldatos, G., Wijeyaratne, L. A., & Teede, H. J. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1151-0 · Cited by 21
Through the quality kaleidoscope: reflections on research in dentomaxillofacial imaging
Rohlin, M., Horner, K., Lindh, C., & Wenzel, A. (2020) · Dentomaxillofacial Radiology · doi:10.1259/dmfr.20190484 · Cited by 19
Risk prediction models for emergence delirium in paediatric general anaesthesia: a systematic review
Petre, M.-A., Saha, B., Kasuya, S., Englesakis, M. F., Gai, N., Peliowski, A., & Aoyama, K. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-043968 · Cited by 18
Metadata
- Authors
- Moons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S.
- Affiliations
- University Medical Center Utrecht (NL)
- University of Oxford (GB)
- Corresponding author
- Moons, Karel G. M.
- Venue
- PLoS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-10-14
- Volume
- 11
- Issue
- 10
- Pages
- e1001744
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC4196729
- OpenAlex
- W2077663753
- Semantic Scholar
- c79bfa9a67c8745d7620fcfe8cf582b9d0407073
- Cited by
- 2,018
- References
- 157
- Citations by year
- 2026: 319 · 2025: 348 · 2024: 253 · 2023: 229 · 2022: 214 · 2021: 219 · 2020: 144 · 2019: 93 · 2018: 74 · 2017: 57 · 2016: 35 · 2015: 28 · 2014: 1
- FWCI
- 11.19
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, cc-by)
- Funders
- Cancer Research UK (16895)
- National Institute for Health and Care Research (NF-SI-0513-10131)
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (project 918.10.615, 918.10.615)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- critical appraisal, data extraction, predictive modeling, prognostic model, diagnostic model, systematic reviews, reporting quality
- MeSH terms
- Decision Support Techniques, Humans, Models, Theoretical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.