The DOI matches its Crossref record; 156 of the 157 references it cites were found in a registry or an index.
Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies: The CHARMS Checklist
Open accessAbstract
Carl Moons and colleagues provide a checklist and background explanation for critically appraising and extracting data from systematic reviews of prognostic and diagnostic prediction modelling studies. Please see later in the article for the Editors' Summary.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Previous guidance for systematic reviews of prediction model studies mainly addressed formulation of the review aim, searching,12 critical appraisal (CHARMS)13 and data extraction of relevant studies. There is, however, little guidance on how to quantitatively synthesize the results of external validation studies.”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [13]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Data extraction was conducted independently by 4 reviewers (R.Q., F.G, Y.A., and S.M.), following the recommendations in the CHARMS checklist.23”
From Linguistic markers for identifying post-traumatic stress disorder and associated symptoms: a systematic literature review (2025) · cites this paper as [23]
The cited source says what the claim attributes to it.
In the citing paper
“Currently there is little information available to guide the development of a well-defined review question however the Quality in Prognosis Studies (QUIPS) tool [34] and the Checklist for critical appraisal and data extraction for systematic reviews of prediction modelling studies (CHARMS Checklist) [38] have been developed to assist in this process (see Table 1).”
From What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences (2018) · cites this paper as [38]
The cited source says what the claim attributes to it.
In the citing paper
“The Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) was employed to define the objectives of this systematic review and meta-analysis (29).”
From Machine learning-based risk predictive models for diabetic kidney disease in type 2 diabetes mellitus patients: a systematic review and meta-analysis (2025) · cites this paper as [29]
The cited source says what the claim attributes to it.
In the citing paper
“We employed the CHARMS checklist and the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guidelines to develop standardized forms for data extraction (29, 32).”
From Machine learning-based risk predictive models for diabetic kidney disease in type 2 diabetes mellitus patients: a systematic review and meta-analysis (2025) · cites this paper as [29]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Self-citationDermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
Self-citationvon Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-8-200710160-00010 · Cited by 17,796
Self-citationQUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Self-citation2013 ACCF/AHA Guideline for the Management of Heart Failure
Yancy, C. W., Jessup, M., Bozkurt, B., Butler, J., Casey, D. E., Drazner, M. H., Fonarow, G. C., Geraci, S. A., Horwich, T., Januzzi, J. L., et al. (2013) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.05.019 · Cited by 12,536
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349
Self-citationMultiple imputation using chained equations: Issues and guidance for practice
White, I. R., Royston, P., & Wood, A. M. (2011) · Statistics in Medicine · doi:10.1002/sim.4067 · Cited by 9,792 · cited 1×
“Numerous recommendations for reporting missing data in medical research have been proposed [103,104,114,115].”
BackgroundA simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655 · cited 1×
“The number of events-pervariable (EPV) is commonly used to calculate the sample size, where attaining a sample size with an EPV of ten or more is frequently recommended to avoid overfitting [97–101].”
Method
Cited by
Items 101–200 of 1,004
Which Models Can I Use to Predict Adult ICU Length of Stay? A Systematic Review*
Verburg, I. W. M., Atashi, A., Eslami, S., Holman, R., Abu–Hanna, A., de Jonge, E., Peek, N. B., & de Keizer, N. F. (2016) · Critical Care Medicine · doi:10.1097/ccm.0000000000002054 · Cited by 98
Predicting population health with machine learning: a scoping review
Morgenstern, J., Buajitti, E., O’Neill, M., Piggott, T., Goel, V., Fridman, D., Kornas, K., & Rosella, L. C. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-037860 · Cited by 122
Tan, K. R., Seng, J. J. B., Kwan, Y. H., Chen, Y. J., Zainudin, S. B., Loh, D. H. F., Liu, N., & Low, L. L. (2021) · Journal of Diabetes Science and Technology · doi:10.1177/19322968211056917 · Cited by 117
“27 The “Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies” (CHARMS) was used to frame this review’s objective.”
MethodFahey, M., Crayton, E., Wolfe, C. D. A., & Douiri, A. (2018) · PLoS ONE · doi:10.1371/journal.pone.0185402 · Cited by 109
“We followed the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) Checklist to guide the framing of the review aim, search strategy, and study inclusion and exclusion criteria (S1 and S5 Tables)[14].”
MethodInfluentialHimmelreich, J. C. L., Veelers, L., Lucassen, W. A. M., Schnabel, R. B., Rienstra, M., van Weert, H. C. P. M., & Harskamp, R. E. (2020) · EP Europace · doi:10.1093/europace/euaa005 · Cited by 110
“5 or 10 years) since this is an important methodological considerations when wanting to translate summary risk model performance to clinical settings.(15)”
BackgroundMethodSiddiqi, T. J., Usman, M., Shahid, I., Ahmed, J., Khan, S. U., Ya’qoub, L., Rihal, C. S., & Alkhouli, M. A. (2021) · European Journal of Preventive Cardiology · doi:10.1093/eurjpc/zwab018 · Cited by 102
“Data extraction was conducted in accordance to the CHARM (Critical Appraisal and Data Extraction for Systematic Reviews of the Prediction Modelling Studies) checklist.(7) The data underlying this article are available in the article and in its Supplementary material online.”
MethodMerlo, J., Mulinari, S., Wemrell, M., Subramanian, S. V., & Hedblad, B. (2017) · SSM - Population Health · doi:10.1016/j.ssmph.2017.08.005 · Cited by 125
A systematic review of the quality of clinical prediction models in in vitro fertilisation.
Ratna, M. B., Bhattacharya, S., Abdulrahim, B., & Mclernon, D. (2020) · Human Reproduction · doi:10.1093/humrep/dez258 · Cited by 67
Haniffa, R., Isaam, I., Silva, A. P. D., Dondorp, A. M., & de Keizer, N. F. (2018) · Critical Care · doi:10.1186/s13054-017-1930-8 · Cited by 100
“Quality assessment Study quality was assessed in accordance with the CHARMS guidelines [23] and is presented as Additional file 2.”
MethodBryant, A., Hiu, S. K. W., Kunonga, P., Gajjar, K. B., Craig, D., Vale, L. D., Winter-Roach, B. A., Elattar, A., & Naik, R. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd015048.pub2 · Cited by 126
O’Caoimh, R., Cornally, N., Weathers, E., O'Sullivan, R., Fitzgerald, C., Orfila, F., Clarnette, R., Paúl, C., & Molloy, D. W. (2015) · Maturitas · doi:10.1016/j.maturitas.2015.03.009 · Cited by 89
Hendriksen, J. M. T., Geersing, G., Lucassen, W. A. M., Erkens, P. M. G., Stoffers, J., van Weert, H. C. P. M., Buller, H. R., Hoes, A. W., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.h4438 · Cited by 107
“21 Next, the diagnostic models easily applicable in primary care were validated in the AMUSE-2 dataset—that is, a large independent prospectively constructed cohort of patients presenting to their general practitioner with complaints suggestive of pulmonary embolism.”
Self-citationSystematic review of prediction models for gestational hypertension and preeclampsia
Antwi, E., Amoakoh‐Coleman, M., Vieira, D. L., Madhavaram, S., Koram, K. A., Grobbee, D. E., Agyepong, I. A., & Klipstein‐Grobusch, K. (2020) · PLoS ONE · doi:10.1371/journal.pone.0230955 · Cited by 103
“The specific objectives of this study were to identify prediction models for gestational hypertension and preeclampsia in the literature, assess the methodological quality of the prediction modeling studies by applying the CHARMS checklist and identify prediction models that can be applied in low and middle income cou…”
BackgroundMethodInfluentialMurphy, R. M., Klopotowska, J. E., de Keizer, N. F., Jager, K. J., Leopold, J. H., Dongelmans, D. A., Abu–Hanna, A., & Schut, M. C. (2023) · PLoS ONE · doi:10.1371/journal.pone.0279842 · Cited by 115
A systematic review of dengue outbreak prediction models: Current scenario and future directions
Leung, X. Y., Islam, R. M., Adhami, M., Ilić, D., McDonald, L., Palawaththa, S., Diug, B. O., Munshi, S. U., & Karim, M. N. (2023) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0010631 · Cited by 113
“This systematic review’s aim, search strategy and study selection process were devised in accordance with 80 the seven items in the CHARMS framework.(23) The review followed the Preferred Reporting Items for 81 Systematic Review and Meta-Analysis (‘PRISMA’) guidelines,(24) and was registered in PROSPERO 82 (CRD4201810…”
MethodChildhood asthma prediction models: a systematic review
Smit, H. A., Pinart, M., Antó, J. M., Keil, T. O. P., Bousquet, J. J., Carlsen, K. H., Moons, K. G. M., Hooft, L., & Carlsen, K. C. L. (2015) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(15)00428-2 · Cited by 94
Self-citationStuderus, E., Ramyead, A., & Riecher‐Rössler, A. (2017) · Psychological Medicine · doi:10.1017/s0033291716003494 · Cited by 102
“This might be because up until recently, no guidance existed to help form a well-defined review question and determine which details to extract and critically appraise from prediction modeling studies (Moons et al. 2014).”
BackgroundMethodInfluentialPregnancy outcome predictors in antiphospholipid syndrome: A systematic review and meta-analysis
Walter, I. J., Haneveld, M. J. K., Lely, A. T., Bloemenkamp, K. W. M., Limper, M., & Kooiman, J. (2021) · Autoimmunity Reviews · doi:10.1016/j.autrev.2021.102901 · Cited by 105
Meertens, L. J. E., van Montfort, P., Scheepers, H. C. J., van Kuijk, S. M. J., Aardenburg, R., Langenveld, J., van Dooren, I. M. A., Zwaan, I. M., Spaanderman, M. E. A., & Smits, L. J. M. (2018) · Acta Obstetricia Et Gynecologica Scandinavica · doi:10.1111/aogs.13358 · Cited by 102
“Risk of bias was assessed according to the CHARMS guidelines.”
MethodInfluentialLamain-de Ruiter, M., Kwee, A., Naaktgeboren, C. A., de Groot, I., Evers, I. M., Groenendaal, F., Hering, Y. R., Huisjes, A. J. M., Kirpestein, C., Monincx, W. M., et al. (2016) · BMJ · doi:10.1136/bmj.i4338 · Cited by 123
Self-citationVernooij, L. M., van Klei, W. A., Moons, K. G. M., Takada, T., van Waes, J. A. R., & Damen, J. A. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013139.pub2 · Cited by 113
Self-citationPrognostic prediction models for chronic postsurgical pain in adults: a systematic review
Papadomanolakis‐Pakis, N., Uhrbrand, P., Haroutounian, S., & Nikolajsen, L. (2021) · Pain · doi:10.1097/j.pain.0000000000002261 · Cited by 99
Sun, J., Lam, C., Christie, L., Blair, C., Li, X., Werdiger, F., Yang, Q. F., Bivard, A., Lin, L., & Parsons, M. W. (2023) · Frontiers in Neurology · doi:10.3389/fneur.2023.1079205 · Cited by 125
Echevarria, C., Steer, J., Heslop-Marshall, K., Stenton, S., Hickey, P. M., Hughes, R. J., Wijesinghe, M., Harrison, R. N., Steen, N. D., Simpson, A. J., et al. (2017) · Thorax · doi:10.1136/thoraxjnl-2016-209298 · Cited by 95
Lee, A. S. Y., Mu, J., Joynt, G. M., Chiu, C. H., Lai, V. K. W., Gin, T., & Underwood, M. J. (2017) · British Journal of Anaesthesia · doi:10.1093/bja/aew476 · Cited by 93
Face Mask Use in the Community for Reducing the Spread of COVID-19: A Systematic Review
Coclite, D., Napoletano, A., Gianola, S., del Monaco, A., D’Angelo, D., Fauci, A. J., Iacorossi, L., Latina, R., Torre, G. L., Mastroianni, C. M., et al. (2021) · Frontiers in Medicine · doi:10.3389/fmed.2020.594269 · Cited by 95
“In addition, for prediction modeling studies, the Checklist for critical Appraisal and data extraction for systematic Reviews of prediction Modeling Studies (CHARMS) was utilized (25).”
MethodNavarro, C. A., Damen, J., Takada, T., Nijman, S., Dhiman, P., Ma, J., Collins, G., Bajpai, R., Riley, R., Moons, K., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2020-038832 · Cited by 49
Self-citationSystematic review of prediction models for pulmonary tuberculosis treatment outcomes in adults
Peetluk, L. S., Ridolfi, F. M., Rebeiro, P. F., Liu, D., Rolla, V. C., & Sterling, T. R. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-044687 · Cited by 99
Zengerling, F., Kunath, F., Jensen, K., Ruf, C. G., Schmidt, S., & Spek, A. (2017) · Urologic Oncology Seminars and Original Investigations · doi:10.1016/j.urolonc.2017.06.047 · Cited by 83
“For data extraction, we used an adapted version of the CHARMS Checklist [16].”
MethodSanfelici, R., Dwyer, D. B., Antonucci, L. A., & Koutsouleris, N. (2020) · Biological Psychiatry · doi:10.1016/j.biopsych.2020.02.009 · Cited by 82
“While ML’s lack of homogeneous thresholds is mainly handled via CV schemes averaging performances across folds and repetitions, the use of p values or data-derived thresholds without a proper training–test separation might have inflated Cox regression models’ performance (63).”
InfluentialClark, J. R., Nijs, J., Yeowell, G., & Goodwin, P. C. (2017) · Pain Physician · doi:10.36076/ppj.2017.20.5.487 · Cited by 82
Prognostic models for intracerebral hemorrhage: systematic review and meta-analysis
Gregório, T., Pipa, S., Cavaleiro, P., Atanásio, G., Albuquerque, I., Chaves, P. C., & Azevedo, L. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0613-8 · Cited by 71
Harnan, S., Tappenden, P., Cooper, K. L., Stevens, J., Bessey, A., Rafia, R., Ward, S. E., Wong, R., Stein, R. C., & Brown, J. E. (2019) · Health Technology Assessment · doi:10.3310/hta23300 · Cited by 83
“Hassan 2015,(139) Hassan 2015(140) UK ER+, HER2– (assumed) LN0 (assumed) Recommendation Decision 26 9 2 7 8 10 (38) 15 (58) 9 (35) –6 (–23)”
Soffer, S., Morgenthau, A. S., Shimon, O., Barash, Y., Konen, E., Glicksberg, B. S., & Klang, E. (2021) · Academic Radiology · doi:10.1016/j.acra.2021.05.014 · Cited by 92
Kee, O. T., Harun, H., Mustafa, N., Abdul Murad, N. A., Chin, S. F., Jaafar, R., & Abdullah, N. (2023) · Cardiovascular Diabetology · doi:10.1186/s12933-023-01741-7 · Cited by 82
“…or may not require external validation, thus, the three study types that are, prediction models development studies without external validation, prediction model development studies with external validation, and external model validation studies with or without model updating were included [23].”
BackgroundFayad, A. A., Ansari, M., Yang, H., Ruddy, T. D., & Wells, G. A. (2016) · Anesthesiology · doi:10.1097/aln.0000000000001132 · Cited by 95
Warfarin dosing algorithms: A systematic review
Asiimwe, I. G., Zhang, E. J., Osanlou, R., Jorgensen, A., & Pirmohamed, M. (2020) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.14608 · Cited by 93
Machine learning in the prediction of medical inpatient length of stay
Bacchi, S., Tan, Y., Oakden‐Rayner, L., Jannes, J., Kleinig, T. J., & Koblar, S. A. (2020) · Internal Medicine Journal · doi:10.1111/imj.14962 · Cited by 106
Nickson, D., Meyer, C., Walasek, L., & Toro, C. T. (2023) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-023-02341-x · Cited by 108
Use of unstructured text in prognostic clinical prediction models: a systematic review
Seinen, T. M., Fridgeirsson, E. A., Ioannou, S., Jeannetot, D. Y. B., John, L. H., Kors, J. A., Markus, A. F., Pera, V., Rekkas, A., Williams, R. D., et al. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac058 · Cited by 102
“Some items are based on clinical prediction item sets from the critical appraisal and data extraction for systematic reviews of prediction modeling studies (CHARMS) checklist 17 and the TRIPOD statement.”
MethodTran, A., Fernando, S. M., Rochwerg, B. N., Barbaro, R. P., Hodgson, C. L., Munshi, L., MacLaren, G., Ramanathan, K. R., Hough, C. L., Brochard, L. J., et al. (2022) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(22)00296-x · Cited by 88
Brennan, N. A., Shamp, W., Maynes, E., Cheng, X., & Bullimore, M. A. (2024) · Optometry and Vision Science · doi:10.1097/opx.0000000000002176 · Cited by 80
Cnossen, M. C., Scholten, A. C., Lingsma, H. F., Synnot, A., Haagsma, J., Steyerberg, P. E. W., & Polinder, S. (2017) · The Journal of Neuropsychiatry and Clinical Neurosciences · doi:10.1176/appi.neuropsych.16090165 · Cited by 79
Shung, D. L., Simonov, M., Gentry, M., Au, B., & Laine, L. (2019) · Digestive Diseases and Sciences · doi:10.1007/s10620-019-05645-z · Cited by 82
Risk of bias of prognostic models developed using machine learning: a systematic review in oncology
Dhiman, P., Ma, J., Andaur Navarro, C. L., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Van Calster, B., et al. (2022) · Diagnostic and Prognostic Research · doi:10.1186/s41512-022-00126-w · Cited by 79
“The data extraction form was developed using PROBAST, in combination with the CHARMS checklist [32], and applied to each model based on information within the model development article (including any external validation if reported within the development article).”
MethodSelf-citationGerry, S., Birks, J., Bonnici, T., Watkinson, P., Kirtley, S., & Collins, P. G. S. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-019268 · Cited by 92
An intersectional analysis providing more precise information on inequities in self-rated health
Wemrell, M., Karlsson, N., Vicente, R. P., & Merlo, J. (2021) · International Journal for Equity in Health · doi:10.1186/s12939-020-01368-0 · Cited by 92
Banerjee, A., Chen, S., Fatemifar, G., Zeina, M., Lumbers, R. T., Mielke, J., Gill, S. K., Kotecha, D., Freitag, D. F., Denaxas, S., et al. (2021) · BMC Medicine · doi:10.1186/s12916-021-01940-7 · Cited by 85
“…for extraction were adapted and developed with consensus among co-authors from published frameworks for new risk markers (AHA [6]), diagnostic accuracy (QUADAS-2 [25]), prognostic tools (CHARMS [26], PROGRESS [7], TRIPOD [27]) and ML (AI-TREE [20], Christodoulou [23]) (Additional file_Web table 2).”
MethodMa, J., Dhiman, P., Qi, C., Bullock, G. S., van Smeden, M., Riley, R. D., & Collins, P. G. S. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.07.017 · Cited by 84
“The data extraction form was developed based on the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guideline [27] and the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modeling Studies [28].”
Systematic review of predictive risk models for adverse drug events in hospitalized patients
Falconer, N., Barras, M. A., & Cottrell, N. (2018) · British Journal of Clinical Pharmacology · doi:10.1111/bcp.13514 · Cited by 87
Prognostication after intracerebral hemorrhage: a review
Witsch, J., Siegerink, B., Nolte, C. H., Sprügel, M. I., Steiner, T., Endres, M., & Huttner, H. B. (2021) · Neurological Research and Practice · doi:10.1186/s42466-021-00120-5 · Cited by 82
“Studies were selected according to the CHARMS checklist [8].”
MethodBullock, G. S., Mylott, J. A., Hughes, T., Nicholson, K. F., Riley, R. D., & Collins, P. G. S. (2022) · Sports Medicine · doi:10.1007/s40279-022-01698-9 · Cited by 107
“The Critical Appraisal and Data Extraction for System‑ atic Reviews of Prediction Modelling Studies Checklist (CHARMS) [31] was used to frame the systematic review and develop the data extraction protocol.”
InfluentialHilkens, N. A., van Asch, C. J. J., Werring, D. J., Wilson, D., Rinkel, G. J. E., Algra, A., Velthuis, B. K., de Kort, G. A. P., Witkamp, T. D., van Nieuwenhuizen, K. M., et al. (2018) · Journal of Neurology Neurosurgery & Psychiatry · doi:10.1136/jnnp-2017-317262 · Cited by 77
Thinking About the Future: A Review of Prognostic Scales Used in Acute Stroke
Drozdowska, B. A., Singh, S., & Quinn, T. J. (2019) · Frontiers in Neurology · doi:10.3389/fneur.2019.00274 · Cited by 83
Schultebraucks, K., & Galatzer‐Levy, I. R. (2019) · Journal of Traumatic Stress · doi:10.1002/jts.22384 · Cited by 82
Yang, C. B., Kors, J. A., Ioannou, S., John, L. H., Markus, A. F., Rekkas, A., de Ridder, M. A. J., Seinen, T. M., Williams, R. D., & Rijnbeek, P. R. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac002 · Cited by 83
“Data extraction was based on the Checklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) Checklist,10 and the TRIPOD Statement.7 The data extraction form contained items from these 2 checklists together with some additional items (eg, the reporting of code l…”
MethodWiezer, M., Hage‐Fransen, M. A. H., Otto, A., Wieffer-Platvoet, M., Slotman, M. H., der Sanden, M. W. G. N., & Pool‐Goudzwaard, A. (2020) · Musculoskeletal Science and Practice · doi:10.1016/j.msksp.2020.102154 · Cited by 113
Miles, J., Turner, J., Jacques, R., Williams, J., & Mason, S. M. (2020) · Diagnostic and Prognostic Research · doi:10.1186/s41512-020-00084-1 · Cited by 91
“Data was extracted using the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist [21].”
MethodLeonhardt, C.-S., Gustorff, C., Klaiber, U., Blanc, S. L., Stamm, T. A., Verbeke, C. S., Prager, G. W., & Strobel, O. (2024) · Gastroenterology · doi:10.1053/j.gastro.2024.05.028 · Cited by 89
Pregnancy outcome predictors in systemic lupus erythematosus: a systematic review and meta-analysis
Wind, M., Fierro, J. J., Bloemenkamp, K. W. M., de Leeuw, K., Lely, A. T., Limper, M., Sueters, M., Teng, Y. K. O., Walter, I. J., & Kooiman, J. (2024) · The Lancet Rheumatology · doi:10.1016/s2665-9913(24)00160-7 · Cited by 86
Cognitive screening instruments to identify vascular cognitive impairment: A systematic review
Ghafar, M. Z. A. A., Miptah, H. N., & Ó’Caoimh, R. (2019) · International Journal of Geriatric Psychiatry · doi:10.1002/gps.5136 · Cited by 80
Predicting 5-Year Risk of RRT in Stage 3 or 4 CKD: Development and External Validation
Schroeder, E. B., Yang, X., Thorp, M. L., Arnold, B. M., Tabano, D. C., Petrik, A. F., Smith, D. H., Platt, R. W., & Johnson, E. S. (2016) · Clinical Journal of the American Society of Nephrology · doi:10.2215/cjn.01290216 · Cited by 79
Tran, A., Fernando, S. M., Rochwerg, B. N., Vaillancourt, C., Inaba, K., Kyeremanteng, K., Nolan, J. P., McCREDIE, V. A., Petrosoniak, A., Hicks, C. M., et al. (2020) · Resuscitation · doi:10.1016/j.resuscitation.2020.05.052 · Cited by 82
Uncertainty-aware deep learning in healthcare: A scoping review
Loftus, T. J., Shickel, B., Ruppert, M. M., Balch, J. A., Ozrazgat-Baslanti, T., Tighe, P. J., Efron, P. A., Hogan, W. R., Rashidi, P., Upchurch, G. R., et al. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000085 · Cited by 73
“Data extraction was performed according to a modification of CHARMS criteria, which included methods for measuring uncertainty in deep learning predictions [10].”
MethodMcGeoch, L. J., Saunders, C. L., Griffin, S. J., Emery, J. D., Walter, F. M., Thompson, D. J., Antoniou, A. C., & Usher‐Smith, J. A. (2019) · Cancer Epidemiology Biomarkers & Prevention · doi:10.1158/1055-9965.epi-19-0059 · Cited by 74
Wemrell, M., Bennet, L., & Merlo, J. (2019) · BMJ Open Diabetes Research & Care · doi:10.1136/bmjdrc-2019-000749 · Cited by 77
Bektaş, M., Tuynman, J. B., Pereira, J. C., Burchell, G. L., & van der Peet, D. L. (2022) · World Journal of Surgery · doi:10.1007/s00268-022-06728-1 · Cited by 83
“An inventory of data items was formed according to the Cochrane guidance for data collection, and the CHARMS checklist [20].”
MethodRhon, D. I., Molloy, J. M., Monnier, A., Hando, B., & Newman, P. (2021) · European Journal of Sport Science · doi:10.1080/17461391.2021.1931464 · Cited by 75
“We used the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies (CHARMS) checklist to guide our approach (Moons et al., 2014).”
BackgroundMethodCarvalho, L. D. F., Sette, C. P., & Ferrari, B. (2018) · Cyberpsychology Journal of Psychosocial Research on Cyberspace · doi:10.5817/cp2018-3-5 · Cited by 82
“…of Observational studies in Epidemiology (STROBE; von Elm et al., 2007); for bias risk: Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies (CHARMS; Moons et al., 2014) guidelines and some keys were created according to the methods of the selected studies.”
MethodInfluentialvan Bussel, E. F., Hoevenaar‐Blom, M. P., Poortvliet, R. K. E., Gussekloo, J., van Dalen, J. W., van Gool, W. A., Richard, E., & van Charante, E. P. M. (2020) · Preventive Medicine · doi:10.1016/j.ypmed.2020.105986 · Cited by 88
Jarvis, H., O'Keefe, H., Craig, D., Stow, D., Hanratty, B., & Anstee, Q. M. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-049767 · Cited by 71
Galli, C., Donos, N., & Calciolari, E. (2024) · Information · doi:10.3390/info15020068 · Cited by 88
Allen, D., Ma, B., Leung, K. C., Graham, M., Pannu, N., Traboulsi, M., Goodhart, D., Knudtson, M., & James, M. (2017) · Canadian Journal of Cardiology · doi:10.1016/j.cjca.2017.01.018 · Cited by 70
Bellou, V., Tzoulaki, I., van Smeden, M., Moons, K. G. M., Εvangelou, Ε., & Belbasis, L. (2021) · European Respiratory Journal · doi:10.1183/13993003.02964-2020 · Cited by 76
“Table 1 shows a detailed description of the PICOTS (Population, Index prognostic factor, Comparator prognostic factor, Outcome, Timing, Setting) system [5, 9, 13].”
Self-citationShelmerdine, S. C., Arthurs, O. J., Denniston, A. K., & Sebire, N. J. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2021-100385 · Cited by 79
Impact of sex on outcomes after cardiac surgery: A systematic review and meta-analysis
Dixon, L. K., Tommaso, E. D., Dimagli, A., Sinha, S., Sandhu, M. S., Benedetto, U., & Angelini, G. D. (2021) · International Journal of Cardiology · doi:10.1016/j.ijcard.2021.09.011 · Cited by 67
Prognostic variables and scores identifying the end of life in COPD: a systematic review
Smith, L., Moore, E., Ali, I., Smeeth, L. L., Stone, P. C., & Quint, J. K. (2017) · International Journal of COPD · doi:10.2147/copd.s137868 · Cited by 67
“…to multivariate models based on the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist 26 such as modeling method, handling of predictors, method for selection of predictors, shrinkage of predictor weights, univariate and multivariate…”
Holland, M., & Kellett, J. G. (2021) · European Journal of Internal Medicine · doi:10.1016/j.ejim.2021.12.024 · Cited by 70
“The CHARM checklist for systematic reviews of predictive models [37] was not adhered to as the aim of this review was to determine from all the available literature, biased or otherwise, the range of discrimination for mortality and the mortality rates associated with different levels of NEWS at different prediction w…”
MethodFazel, S., Burghart, M., Fanshawe, T. R., Gil, S. D., Monahan, J. T., & Yu, R. (2022) · Journal of Criminal Justice · doi:10.1016/j.jcrimjus.2022.101902 · Cited by 82
Wu, S., Zeng, N., Sun, F., Zhou, J., Wu, X., Sun, Y., Wang, B., Zhan, S., Kong, Y., Jia, J., et al. (2021) · Clinical Gastroenterology and Hepatology · doi:10.1016/j.cgh.2021.02.040 · Cited by 71
Machine learning techniques for mortality prediction in emergency departments: a systematic review
Naemi, A., Schmidt, T., Mansourvar, M., Naghavi‐Behzad, M., Ebrahimi, A., & Wiil, U. K. (2021) · BMJ Open · doi:10.1136/bmjopen-2021-052663 · Cited by 82
Alabi, R. O., Elmusrati, M. S., Leivo, I. V. V., Almangush, A., & Mäkitie, A. A. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105464 · Cited by 75
Bai, A. D., Agarwal, A., Steinberg, M., Showler, A., Burry, L. D., TOMLINSON, G. A., Bell, C. M., & Morris, A. M. (2017) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2017.04.025 · Cited by 71
Translating AI to Clinical Practice: Overcoming Data Shift with Explainability
Choi, Y., Yu, W., Nagarajan, M. B., Teng, P., Goldin, J. G., Raman, S. S., Enzmann, D. R., Kim, G. H. J., & Brown, M. S. (2023) · RadioGraphics · doi:10.1148/rg.220105 · Cited by 65
Slieker, R. C., van der Heijden, A. A. W. A., Siddiqui, M. K., Langendoen‐Gort, M., Nijpels, G., Herings, R. M. C., Feenstra, T., Moons, K. G. M., Bell, S., Elders, P. J. M., et al. (2021) · BMJ · doi:10.1136/bmj.n2134 · Cited by 69
Self-citationPredicting falls in community-dwelling older adults: a systematic review of prognostic models
Gade, G. V., Jørgensen, M. G., Ryg, J., Riis, J., Thomsen, K., Masud, T., & Andersen, S. (2021) · BMJ Open · doi:10.1136/bmjopen-2020-044170 · Cited by 74
Prognostic models for mortality risk in patients requiring ECMO
Pladet, L. C. A., Barten, J. M. M., Vernooij, L. M., Kraemer, C. V. E., Bunge, J. J. H., Scholten, E., Montenij, L. J., Kuijpers, M., Donker, D. W., Cremer, O. L., et al. (2023) · Intensive Care Medicine · doi:10.1007/s00134-022-06947-z · Cited by 66
“Items of the data extraction form were based on the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modeling Studies (CHARMS) which addresses key information on the source of data, study design, participants, outcomes, candidate predictors, prediction horizon, sample size, mis…”
MethodMachine Learning: A New Approach for Dose Individualization
Li, Q., Tang, B., Wu, Y., Yao, B., Zhang, W., Zheng, Y., Zhou, Y., van den Anker, J. N., Hao, G., & Zhao, W. (2023) · Clinical Pharmacology & Therapeutics · doi:10.1002/cpt.3049 · Cited by 70
“9,10 Extracted data included the study drugs, study population, data source, sample size, predictive target, predictor, details of the ML model development (e.g., ML algorithms and feature selection), validation type, and predictive performance measurement.”
Anderson, K., Bird, M., MacPherson, S., & Blair, A. C. (2016) · International Psychogeriatrics · doi:10.1017/s1041610216000570 · Cited by 79
Prediction models for the risk of gestational diabetes: a systematic review
Ruiter, M. L.-D., Kwee, A., Naaktgeboren, C. A., Franx, A., Moons, K. G. M., & Koster, M. P. H. (2017) · Diagnostic and Prognostic Research · doi:10.1186/s41512-016-0005-7 · Cited by 74
“For critical appraisal of the individual studies, we used the recently published CHARMS checklist [16].”
MethodInfluentialSelf-citationPrediction of Chemotherapeutic Efficacy in Non–Small Cell Lung Cancer by Serum Metabolomic Profiling
Tian, Y., Wang, Z., Liu, X., Duan, J., Feng, G., Yin, Y., Gu, J. F., Chen, Z., Gao, S., Bai, H., et al. (2018) · Clinical Cancer Research · doi:10.1158/1078-0432.ccr-17-2855 · Cited by 64
“This prediction modeling study was carefully designed by following the CHARMS checklists (27), including the Figure 3.”
MethodPrognostication with point-of-care echocardiography during cardiac arrest: A systematic review
Reynolds, J. C., Issa, M. S., Nicholson, T., Drennan, I. R., Berg, K. M., O’Neil, B. J., Welsford, M., Andersen, L. W., Böttiger, B. W., Callaway, C. W., et al. (2020) · Resuscitation · doi:10.1016/j.resuscitation.2020.05.004 · Cited by 70
Ray, S., Lahiri, A., Jha, S. S., Bhattacharya, S., & Chakraborty, A. (2020) · Indian Journal of Public Health · doi:10.4103/ijph.ijph_464_20 · Cited by 62
Kareemi, H., Vaillancourt, C., Rosenberg, H., Fournier, K., & Yadav, K. (2020) · Academic Emergency Medicine · doi:10.1111/acem.14190 · Cited by 66
“We also used previously published guidelines for prediction modeling research to direct our study development (Debray guide) and data extraction (CHARMS).(19,21) Our study has several implications for clinical practice and future research.”
Methodvan Royen, F. S.-A., Moons, K. G. M., Geersing, G., & van Smeden, M. (2022) · European Respiratory Journal · doi:10.1183/13993003.00250-2022 · Cited by 63
Self-citationPrediction models for depression risk among older adults: systematic review and critical appraisal
Tan, J., Ma, C., Zhu, C., Wang, Y., Zou, X., Li, H., Li, J., He, Y., & Wu, C. (2022) · Ageing Research Reviews · doi:10.1016/j.arr.2022.101803 · Cited by 69
(2017) · Europe PMC (PubMed Central) · Cited by 61
Vagliano, I., Chesnaye, N. C., Leopold, J. H., Jager, K. J., Abu–Hanna, A., & Schut, M. C. (2022) · Clinical Kidney Journal · doi:10.1093/ckj/sfac181 · Cited by 75
“We created a data-extraction form (Supplementary Table S1) based on the Transparent Reporting of aMultivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) and the Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklists [32, 33…”
MethodCai, S., Li, J., Gao, J., Pan, W., & Zhang, Y. (2022) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2022.104340 · Cited by 66
Klement, W., & Emam, K. E. (2023) · Journal of Medical Internet Research · doi:10.2196/48763 · Cited by 72
Metadata
- Authors
- Moons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S.
- Affiliations
- University Medical Center Utrecht (NL)
- University of Oxford (GB)
- Corresponding author
- Moons, Karel G. M.
- Venue
- PLoS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-10-14
- Volume
- 11
- Issue
- 10
- Pages
- e1001744
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC4196729
- OpenAlex
- W2077663753
- Semantic Scholar
- c79bfa9a67c8745d7620fcfe8cf582b9d0407073
- Cited by
- 2,018
- References
- 157
- Citations by year
- 2026: 319 · 2025: 348 · 2024: 253 · 2023: 229 · 2022: 214 · 2021: 219 · 2020: 144 · 2019: 93 · 2018: 74 · 2017: 57 · 2016: 35 · 2015: 28 · 2014: 1
- FWCI
- 11.19
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, cc-by)
- Funders
- Cancer Research UK (16895)
- National Institute for Health and Care Research (NF-SI-0513-10131)
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (project 918.10.615, 918.10.615)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- critical appraisal, data extraction, predictive modeling, prognostic model, diagnostic model, systematic reviews, reporting quality
- MeSH terms
- Decision Support Techniques, Humans, Models, Theoretical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.