Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies: The CHARMS Checklist
Open accessThe DOI matches its Crossref record; 156 of the 157 references it cites were found in a registry or an index.
Abstract
Carl Moons and colleagues provide a checklist and background explanation for critically appraising and extracting data from systematic reviews of prognostic and diagnostic prediction modelling studies. Please see later in the article for the Editors' Summary.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Previous guidance for systematic reviews of prediction model studies mainly addressed formulation of the review aim, searching,12 critical appraisal (CHARMS)13 and data extraction of relevant studies. There is, however, little guidance on how to quantitatively synthesize the results of external validation studies.”
From A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes (2019) · cites this paper as [13]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Data extraction was conducted independently by 4 reviewers (R.Q., F.G, Y.A., and S.M.), following the recommendations in the CHARMS checklist.23”
From Linguistic markers for identifying post-traumatic stress disorder and associated symptoms: a systematic literature review (2025) · cites this paper as [23]
The cited source says what the claim attributes to it.
In the citing paper
“Currently there is little information available to guide the development of a well-defined review question however the Quality in Prognosis Studies (QUIPS) tool [34] and the Checklist for critical appraisal and data extraction for systematic reviews of prediction modelling studies (CHARMS Checklist) [38] have been developed to assist in this process (see Table 1).”
From What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences (2018) · cites this paper as [38]
The cited source says what the claim attributes to it.
In the citing paper
“The Checklist for Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) was employed to define the objectives of this systematic review and meta-analysis (29).”
From Machine learning-based risk predictive models for diabetic kidney disease in type 2 diabetes mellitus patients: a systematic review and meta-analysis (2025) · cites this paper as [29]
The cited source says what the claim attributes to it.
In the citing paper
“We employed the CHARMS checklist and the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) guidelines to develop standardized forms for data extraction (29, 32).”
From Machine learning-based risk predictive models for diabetic kidney disease in type 2 diabetes mellitus patients: a systematic review and meta-analysis (2025) · cites this paper as [29]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Self-citationLiberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Self-citationDermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020
Self-citationvon Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-8-200710160-00010 · Cited by 17,796
Self-citationQUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Self-citation2013 ACCF/AHA Guideline for the Management of Heart Failure
Yancy, C. W., Jessup, M., Bozkurt, B., Butler, J., Casey, D. E., Drazner, M. H., Fonarow, G. C., Geraci, S. A., Horwich, T., Januzzi, J. L., et al. (2013) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.05.019 · Cited by 12,536
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349
Self-citationMultiple imputation using chained equations: Issues and guidance for practice
White, I. R., Royston, P., & Wood, A. M. (2011) · Statistics in Medicine · doi:10.1002/sim.4067 · Cited by 9,792 · cited 1×
“Numerous recommendations for reporting missing data in medical research have been proposed [103,104,114,115].”
BackgroundA simulation study of the number of events per variable in logistic regression analysis
Peduzzi, P., Concato, J., Kemper, E., Holford, T. R., & Feinstein, A. R. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00236-3 · Cited by 9,655 · cited 1×
“The number of events-pervariable (EPV) is commonly used to calculate the sample size, where attaining a sample size with an EPV of ten or more is frequently recommended to avoid overfitting [97–101].”
Method
Cited by
Moons, K. G. M., Altman, D. G., Reitsma, J. B., Ioannidis, J. P. A., Macaskill, P., Steyerberg, E. W., Vickers, A. J., Ransohoff, D. F., & Collins, G. S. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0698 · Cited by 5,147
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Cancer · doi:10.1038/bjc.2014.639 · Cited by 3,293
“The TRIPOD Statement does not prescribe how studies developing, validating, or updating prediction models should be undertaken, nor should it be used as a tool for explicitly assessing quality or quantifying risk of bias in such studies (52).”
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
“The TRIPOD Statement does not prescribe how studies developing, validating, or updating prediction models should be undertaken, nor should it be used as a tool for explicitly assessing quality or quantifying risk of bias in such studies [52].”
MethodSelf-citationWynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070
Self-citationPROBAST: A Tool to Assess the Risk of Bias and Applicability of Prediction Model Studies
Wolff, R. F., Moons, K. G. M., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., Mallett, S., & for the PROBAST Group† (2019) · Annals of Internal Medicine · doi:10.7326/m18-1376 · Cited by 2,984
“Assessors are first asked to report their systematic review question based on the guidance given in the CHARMS checklist.(41)”
InfluentialSelf-citationChristodoulou, E., Ma, J., Collins, G. S., Steyerberg, E. W., Verbakel, J. Y., & Van Calster, B. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.02.004 · Cited by 2,119
“Definitions of what constitutes ML and the differences with statistical modeling have been discussed at length in the literature [8], yet the distinction is not clear-cut [9].”
BackgroundMethodLiu, X., Faes, L., Kale, A. U., Wagner, S. K., Fu, D. J., Bruynseels, A., Mahendiran, T., Moraes, G., Shamdas, M., Kern, C., et al. (2019) · The Lancet Digital Health · doi:10.1016/s2589-7500(19)30123-2 · Cited by 1,883
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. (2015) · BMC Medicine · doi:10.1186/s12916-014-0241-z · Cited by 1,927
Self-citationMoons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
Self-citationMunn, Z., Stern, C., Aromataris, E., Lockwood, C., & Jordan, Z. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-017-0468-4 · Cited by 1,164
“Checklist) [38] have been developed to assist in this process (see Table 1).”
BackgroundMoons, K. G. M., Damen, J. A. A., Kaul, T., Hooft, L., Andaur Navarro, C., Dhiman, P., Beam, A. L., Van Calster, B., Celi, L. A., Denaxas, S., et al. (2025) · BMJ · doi:10.1136/bmj-2024-082505 · Cited by 1,069
“…intended use of the prediction model) criteria as provided by the guidance of the Cochrane Prognosis Methods group (https://methods.cochrane.org/ prognosis/) and described in CHARMS (checklist for critical appraisal and data extraction for systematic reviews of prediction modelling studies 61 ).”
Self-citationPrediction models for cardiovascular disease risk in the general population: systematic review
Damen, J. A. A. G., Hooft, L., Schuit, E., Debray, T. P. A., Collins, G. S., Tzoulaki, I., Lassale, C. M., Siontis, G. C. M., Chiocchia, V., Roberts, C., et al. (2016) · BMJ · doi:10.1136/bmj.i2416 · Cited by 935
Self-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · British Journal of Surgery · doi:10.1002/bjs.9736 · Cited by 889
Self-citationSample size for binary logistic prediction models: Beyond events per variable criteria
van Smeden, M., Moons, K. G., de Groot, J. A., Collins, G. S., Altman, D. G., Eijkemans, M. J., & Reitsma, J. B. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218784726 · Cited by 791
“This minimal sample size criterion has also generally been accepted as a methodological quality item in appraising published prediction modeling studies.(2,14,16) However, some authors have expressed concerns that that the EPV 10 rule-of-thumb is not based on convincing scientific reasoning.”
BackgroundMethodSelf-citationCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Circulation · doi:10.1161/circulationaha.114.014508 · Cited by 793
“The TRIPOD Statement does not prescribe how studies developing, validating, or updating prediction models should be undertaken, nor should it be used as a tool for explicitly assessing quality or quantifying risk of bias in such studies (52).”
Self-citationA guide to systematic review and meta-analysis of prognostic factor studies
Riley, R. D., Moons, K. G. M., Snell, K. I. E., Ensor, J., Hooft, L., Altman, D. G., Hayden, J., Collins, G. S., & Debray, T. P. A. (2019) · BMJ · doi:10.1136/bmj.k4597 · Cited by 742
Self-citationIzcovich, A., Ragusa, M. A., Tortosa, F., Lavena Marzio, M. A., Agnoletti, C., Bengolea, A., Ceirano, A., Espinosa, F., Saavedra, E., Sanguine, V., et al. (2020) · PLOS ONE · doi:10.1371/journal.pone.0241955 · Cited by 660
A guide to systematic review and meta-analysis of prediction model performance
Debray, T. P. A., Damen, J. A. A. G., Snell, K. I. E., Ensor, J., Hooft, L., Reitsma, J. B., Riley, R. D., & Moons, K. G. M. (2017) · BMJ · doi:10.1136/bmj.i6460 · Cited by 657
Self-citationNo rationale for 1 variable per 10 events criterion for binary logistic regression analysis
van Smeden, M., de Groot, J. A. H., Moons, K. G. M., Collins, G. S., Altman, D. G., Eijkemans, M. J. C., & Reitsma, J. B. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0267-3 · Cited by 595
“EPV, which is thought to be a key determinant of the performance of logistic regression models, is frequently used in sample size considerations and as a methodological quality item for critically appraising published studies [9–11].”
Self-citationFenton, J. J., Weyrich, M. S., Durbin, S., Liu, Y., Bang, H., & Melnikow, J. (2018) · JAMA · doi:10.1001/jama.2018.3712 · Cited by 555
Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2017) · doi:10.23970/ahrqepcmethguide2 · Cited by 448
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · European Urology · doi:10.1016/j.eururo.2014.11.025 · Cited by 439
Self-citationGerry, S., Bonnici, T., Birks, J., Kirtley, S., Virdee, P. S., Watkinson, P. J., & Collins, G. S. (2020) · BMJ · doi:10.1136/bmj.m1501 · Cited by 383
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.010 · Cited by 324
Self-citationELAPSS score for prediction of risk of growth of unruptured intracranial aneurysms
Backes, D., Rinkel, G. J. E., Greving, J. P., Velthuis, B. K., Murayama, Y., Takao, H., Ishibashi, T., Igase, M., terBrugge, K. G., Agid, R., et al. (2017) · Neurology · doi:10.1212/wnl.0000000000003865 · Cited by 313
“, 90 outcomes), 900 patients would be sufficient for such analyses.(19,20)”
BackgroundDevelopment of type 2 diabetes mellitus in people with intermediate hyperglycaemia
Richter, B., Hemmingsen, B., Metzendorf, M.-I., & Takwoingi, Y. (2018) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012661.pub2 · Cited by 283
A framework for meta-analysis of prediction model studies with binary and time-to-event outcomes
Debray, T. P., Damen, J. A., Riley, R. D., Snell, K., Reitsma, J. B., Hooft, L., Collins, G. S., & Moons, K. G. (2019) · Statistical Methods in Medical Research · doi:10.1177/0962280218785504 · Cited by 278
Self-citationCowley, L. E., Farewell, D. M., Maguire, S., & Kemp, A. M. (2019) · Diagnostic and Prognostic Research · doi:10.1186/s41512-019-0060-y · Cited by 263
Smith, E. E., Kent, D. M., Bulsara, K. R., Leung, L. Y., Lichtman, J. H., Reeves, M. J., Towfighi, A., Whiteley, W. N., & Zahuranec, D. B. (2018) · Stroke · doi:10.1161/str.0000000000000160 · Cited by 261
Synthetic and External Controls in Clinical Trials – A Primer for Researchers
Thorlund, K., Dron, L., Park, J. J., & Mills, E. J. (2020) · Clinical Epidemiology · doi:10.2147/clep.s242097 · Cited by 257
Fundamentals of Clinical Data Science
(2019) · Springer eBooks · doi:10.1007/978-3-319-99713-1 · Cited by 235
Damen, J. A., Pajouheshnia, R., Heus, P., Moons, K. G. M., Reitsma, J. B., Scholten, R. J. P. M., Hooft, L., & Debray, T. P. A. (2019) · BMC Medicine · doi:10.1186/s12916-019-1340-7 · Cited by 250
Self-citationRecommendations for assessing the risk of bias in systematic reviews of health-care interventions
Viswanathan, M., Patnode, C. D., Berkman, N. D., Bass, E. B., Chang, S., Hartling, L., Murad, M. H., Treadwell, J. R., & Kane, R. L. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.12.004 · Cited by 221
Salazar de Pablo, G., Studerus, E., Vaquerizo-Serrano, J., Irving, J., Catalan, A., Oliver, D., Baldwin, H., Danese, A., Fazel, S., Steyerberg, E. W., et al. (2021) · Schizophrenia Bulletin · doi:10.1093/schbul/sbaa120 · Cited by 219
Meehan, A. J., Lewis, S. J., Fazel, S., Fusar-Poli, P., Steyerberg, E. W., Stahl, D., & Danese, A. (2022) · Molecular Psychiatry · doi:10.1038/s41380-022-01528-4 · Cited by 219
“Informed by the CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modeling Studies (CHARMS) [22] and Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement [2], one reviewer (AJM) completed extraction while a second (SJ…”
MethodWatt, J., Tricco, A. C., Talbot-Hamon, C., Pham, B., Rios, P., Grudniewicz, A., Wong, C., Sinclair, D., & Straus, S. E. (2018) · BMC Medicine · doi:10.1186/s12916-017-0986-2 · Cited by 220
“Additionally, this study was initiated prior to the introduction of the CHARMS checklist, which means that biases introduced in model development, validation, and evaluation of our included studies are less well described; however, we feel that we were able to identify important sources of selection bias, measurement…”
MethodLin, J. S., Evans, C. V., Johnson, E., Redmond, N., Coppola, E. L., & Smith, N. (2018) · JAMA · doi:10.1001/jama.2018.4242 · Cited by 216
Prognostic models for breast cancer: a systematic review
Phung, M. T., Tin Tin, S., & Elwood, J. M. (2019) · BMC Cancer · doi:10.1186/s12885-019-5442-6 · Cited by 209
Wu, S., Yuan, R., Wang, Y., Wei, C., Zhang, S., Yang, X., Wu, B., & Liu, M. (2018) · Stroke · doi:10.1161/strokeaha.118.022001 · Cited by 203
Snell, K. I. E., Levis, B., Damen, J. A. A., Dhiman, P., Debray, T. P. A., Hooft, L., Reitsma, J. B., Moons, K. G. M., Collins, G. S., & Riley, R. D. (2023) · BMJ · doi:10.1136/bmj-2022-073538 · Cited by 190
Self-citationSystematic review of prediction models for delirium in the older adult inpatient
Lindroth, H., Bratzke, L., Purvis, S., Brown, R., Coburn, M., Mrkobrada, M., Chan, M. T. V., Davis, D. H. J., Pandharipande, P., Carlsson, C. M., et al. (2018) · BMJ Open · doi:10.1136/bmjopen-2017-019223 · Cited by 198
“Data collection overlapped with the initial diagnosis of delirium in the highest performing model as well as in two other included studies, likely exaggerating model performance.(15) 27 32 35 Low EPV combined with limited application of internal validation techniques contributed to an increased risk of bias and likely…”
InfluentialBellou, V., Belbasis, L., Konstantinidis, A. K., Tzoulaki, I., & Evangelou, E. (2019) · BMJ · doi:10.1136/bmj.l5358 · Cited by 197
Stirland, L. E., González-Saavedra, L., Mullin, D. S., Ritchie, C. W., Muniz-Terrera, G., & Russ, T. C. (2020) · BMJ · doi:10.1136/bmj.m160 · Cited by 189
Prognostic models in obstetrics: available, but far from applicable
Kleinrouweler, C. E., Cheong-See, F. M., Collins, G. S., Kwee, A., Thangaratinam, S., Khan, K. S., Mol, B. W. J., Pajkrt, E., Moons, K. G. M., & Schuit, E. (2016) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2015.06.013 · Cited by 193
Self-citationPinna, T., & Edwards, D. J. (2020) · Frontiers in Psychology · doi:10.3389/fpsyg.2020.01792 · Cited by 191
“This is a tool developed by the Cochrane Collaboration Prognosis Review Method Group (Moons et al., 2014) to critically apprise the quality of reporting and bias.”
MethodAchten, N. B., Klingenberg, C., Benitz, W. E., Stocker, M., Schlapbach, L. J., Giannoni, E., Bokelaar, R., Driessen, G. J. A., Brodin, P., Uthaya, S., et al. (2019) · JAMA Pediatrics · doi:10.1001/jamapediatrics.2019.2825 · Cited by 187
A conceptual framework for prognostic research
Kent, P., Cancelliere, C., Boyle, E., Cassidy, J. D., & Kongsted, A. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01050-7 · Cited by 185
“Whereas PROGRESS, TRIPOD and CHARMS focus on descriptive epidemiology and prediction of outcome, others have emphasised the importance of differentiating between this and research aimed at establishing causal relationships [22, 33].”
MethodInfluentialAssociation between vision impairment and mortality: a systematic review and meta-analysis
Ehrlich, J. R., Ramke, J., Macleod, D., Burn, H., Lee, C. N., Zhang, J. H., Waldock, W., Swenor, B. K., Gordon, I., Congdon, N., et al. (2021) · The Lancet Global Health · doi:10.1016/s2214-109x(20)30549-0 · Cited by 182
Shin, S., Austin, P. C., Ross, H. J., Abdel-Qadir, H., Freitas, C., Tomlinson, G., Chicco, D., Mahendiran, M., Lawler, P. R., Billia, F., et al. (2021) · ESC Heart Failure · doi:10.1002/ehf2.13073 · Cited by 180
“11 , 12 Other available approaches to assessing quality of prognostic studies, such as the TRIPOD, were considered, but its aim is to increase transparency of reporting, and from an operational standpoint, it was not ideally suited to evaluate prognostic studies using ML. 49 The modi fi ed CHARMS…”
BackgroundHage‐Fransen, M. A. H., Wiezer, M., Otto, A., Wieffer‐Platvoet, M. S., Slotman, M. H., Nijhuis‐van der Sanden, M. W. G., & Pool‐Goudzwaard, A. L. (2021) · Acta Obstetricia et Gynecologica Scandinavica · doi:10.1111/aogs.14027 · Cited by 177
Liu, T., Krentz, A., Lu, L., & Curcin, V. (2025) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztae080 · Cited by 177
To buy or not to buy—evaluating commercial AI solutions in radiology (the ECLAIR guidelines)
Omoumi, P., Ducarouge, A., Tournier, A., Harvey, H., Kahn, C. E., Louvet-de Verchère, F., Pinto Dos Santos, D., Kober, T., & Richiardi, J. (2021) · European Radiology · doi:10.1007/s00330-020-07684-x · Cited by 167
“In general, one may refer to detailed guidelines [3, 21, 24], but several points are of particular importance:”
BackgroundNickols, W. A., Kuntz, T. M., Shen, J., Maharjan, S., Mallick, H., Franzosa, E. A., Thompson, K. N., Nearing, J. T., & Huttenhower, C. (2026) · Nature Methods · doi:10.1038/s41592-025-02923-9 · Cited by 245
“This is in contrast to MaAsLin 2, which total-sum scales the feature abundance profiles, sets zeros to half the minimum observed abundance, log transforms all abundances (including imputed values) and performs a linear regression on the transformed values.”
Dijkland, S. A., Foks, K. A., Polinder, S., Dippel, D. W. J., Maas, A. I. R., Lingsma, H. F., & Steyerberg, E. W. (2020) · Journal of Neurotrauma · doi:10.1089/neu.2019.6401 · Cited by 162
Machine learning for genetic prediction of psychiatric disorders: a systematic review
Bracher-Smith, M., Crawford, K., & Escott-Price, V. (2021) · Molecular Psychiatry · doi:10.1038/s41380-020-0825-2 · Cited by 157
“elling studies (CHARMS) checklist [20] were included as-is or modified.”
MethodEfficace, F., Collins, P. G. S., Cottone, F., Giesinger, J. M., Sommer, K., Anota, A., Schlüssel, M. M., Fazi, P., & Vignetti, M. (2021) · Value in Health · doi:10.1016/j.jval.2020.10.017 · Cited by 160
Karter, A. J., Warton, E. M., Lipska, K. J., Ralston, J. D., Moffet, H. H., Jackson, G. G., Huang, E. S., & Miller, D. R. (2017) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2017.3844 · Cited by 150
Fernando, S. M., Tran, A., Cheng, W., Rochwerg, B. N., Taljaard, M., Vaillancourt, C., Rowan, K., Harrison, D. A., Nolan, J. P., Kyeremanteng, K., et al. (2019) · BMJ · doi:10.1136/bmj.l6373 · Cited by 168
Validation of the DECAF score to predict hospital mortality in acute exacerbations of COPD
Echevarria, C., Steer, J., Heslop-Marshall, K., Stenton, S., Hickey, P. M., Hughes, R. J., Wijesinghe, M., Harrison, R., Steen, N. D., Simpson, A. J., et al. (2016) · Thorax · doi:10.1136/thoraxjnl-2015-207775 · Cited by 149
Dretzke, J., Chuchu, N., Agarwal, R., Herd, C. P., Chua, W., Fabritz, L., Bayliss, S. E., Kotecha, D., Deeks, J. J., Kirchhof, P. F., et al. (2020) · EP Europace · doi:10.1093/europace/euaa041 · Cited by 160
Clinician checklist for assessing suitability of machine learning applications in healthcare
Scott, I., Carter, S., & Coiera, E. (2021) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2020-100251 · Cited by 143
Benedetto, U., Dimagli, A., Sinha, S., Cocomello, L., Gibbison, B., Caputo, M., Gaunt, T. R., Lyon, M., Holmes, C., & Angelini, G. D. (2020) · Journal of Thoracic and Cardiovascular Surgery · doi:10.1016/j.jtcvs.2020.07.105 · Cited by 149
Sufriyana, H., Husnayain, A., Chen, Y.-L., Kuo, C.-Y., Singh, O., Yeh, T.-Y., Wu, Y., & Su, E. C. (2020) · JMIR Medical Informatics · doi:10.2196/16503 · Cited by 169
“Key items of population, index, comparator, outcomes, timing, setting (PICOTS) [19] and additional items [15,20] composed the eligibility criteria.”
MethodMachine learning in vascular surgery: a systematic review and critical appraisal
Li, B., Feridooni, T., Cuen-Ojeda, C., Kishibe, T., de Mestral, C., Mamdani, M., & Al-Omran, M. (2022) · npj Digital Medicine · doi:10.1038/s41746-021-00552-y · Cited by 137
Dhiman, P., Ma, J., Andaur Navarro, C. L., Speich, B., Bullock, G., Damen, J. A. A., Hooft, L., Kirtley, S., Riley, R. D., Van Calster, B., et al. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01577-x · Cited by 140
Self-citationFarrington, K., Covic, A. C., Aucella, F., Clyne, N., de Vos, L., Findlay, A., Fouque, D. P., Grodzicki, T. K., Iyasere, O. U., Jager, K. J., et al. (2016) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfw356 · Cited by 171
Mahar, A. L., Compton, C. C., Halabi, S., Hess, K. R., Weiser, M. R., & Groome, P. A. (2017) · Journal of Surgical Oncology · doi:10.1002/jso.24774 · Cited by 151
“The final criteria include all key elements described by the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) guidelines(20,21) and the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist.(22) Clinical…”
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.13244 · Cited by 207
Self-citationPredicting risk of postpartum haemorrhage: a systematic review
Neary, C., Naheed, S., McLernon, D. J., & Black, M. E. (2020) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.16379 · Cited by 163
Huang, Y., Talwar, A., Chatterjee, S., & Aparasu, R. R. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01284-z · Cited by 178
Henderson, J. T., Thompson, J. H., Burda, B. U., & Cantor, A. (2017) · JAMA · doi:10.1001/jama.2016.18315 · Cited by 130
Debray, T. P. A., Riley, R. D., Rovers, M. M., Reitsma, J. B., Moons, K. G. M., & Cochrane IPD Meta-analysis Methods group (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001886 · Cited by 129
“The CHARMS checklist is a very recent guideline and checklist for data extraction and critical appraisal of primary prediction model studies in systematic reviews [35].”
MethodSelf-citationXu, H.-L., Gong, T., Liu, F.-H., Chen, H.-Y., Xiao, Q., Hou, Y., Huang, Y., Sun, H., Shi, Y., Gao, S., et al. (2022) · EClinicalMedicine · doi:10.1016/j.eclinm.2022.101662 · Cited by 150
Campagnini, S., Arienti, C., Patrini, M., Liuzzi, P., Mannini, A., & Carrozza, M. C. (2022) · Journal of NeuroEngineering and Rehabilitation · doi:10.1186/s12984-022-01032-4 · Cited by 155
“Data collection The CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) was used [16].”
MethodA proposed framework for developing quality assessment tools
Whiting, P., Wolff, R., Mallett, S., Simera, I., & Savović, J. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0604-6 · Cited by 121
Self-citationMortality prediction models in the general trauma population: A systematic review
de Munter, L., Polinder, S., Lansink, K. W. W., Cnossen, M. C., Steyerberg, E. W., & de Jongh, M. A. C. (2016) · Injury · doi:10.1016/j.injury.2016.12.009 · Cited by 137
Machine Learning Models for Predicting Neonatal Mortality: A Systematic Review
Mangold, C., Zoretic, S., Thallapureddy, K., Moreira, A., Chorath, K. T., & Moreira, A. G. (2021) · Neonatology · doi:10.1159/000516891 · Cited by 144
Fernandez-Felix, B. M., López‐Alcalde, J., Figuls, M. R. I., Muriel, A., & Zamora, J. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-023-01849-0 · Cited by 137
“The CHARMS checklist (CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies) provides guidance for both formulating the review question, and for extracting data the primary studies reporting prediction models [5].”
MethodLamain-de Ruiter, M., Kwee, A., Naaktgeboren, C. A., de Groot, I., Evers, I. M., Groenendaal, F., Hering, Y. R., Huisjes, A. J. M., Kirpestein, C., Monincx, W. M., et al. (2016) · BMJ · doi:10.1136/bmj.i4338 · Cited by 123
Self-citationCai, Y., Cai, Y.-Q., Tang, L.-Y., Wang, Y.-H., Gong, M., Jing, T.-C., Li, H.-J., Li-Ling, J., Hu, W., Yin, Z., et al. (2024) · BMC Medicine · doi:10.1186/s12916-024-03273-7 · Cited by 122
Andaur Navarro, C. L., Damen, J. A. A., van Smeden, M., Takada, T., Nijman, S. W. J., Dhiman, P., Ma, J., Collins, G. S., Bajpai, R., Riley, R. D., et al. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.11.015 · Cited by 121
Self-citationLaureano-Phillips, J., Robinson, R. D., Aryal, S., Blair, S. E., Wilson, D., Boyd, K., Schrader, C. D., Zenarosa, N. R., & Wang, H. (2019) · Annals of Emergency Medicine · doi:10.1016/j.annemergmed.2018.12.010 · Cited by 131
Anteby, R., Horesh, N., Soffer, S., Zager, Y., Barash, Y., Amiel, I., Rosin, D., Gutman, M., & Klang, E. (2021) · Surgical Endoscopy · doi:10.1007/s00464-020-08168-1 · Cited by 127
Predicting breast cancer 5-year survival using machine learning: A systematic review
Li, J., Zhou, Z., Dong, J., Fu, Y., Li, Y., Luan, Z., & Peng, X. (2021) · PLoS ONE · doi:10.1371/journal.pone.0250370 · Cited by 139
“…for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS) was also used for data extraction [21], the extracted data include the following: • Basic literature information: first author, year, country of research, published type, disease…”
MethodSmith, L. A., Oakden-Rayner, L., Bird, A., Zeng, M., To, M.-S., Mukherjee, S., & Palmer, L. J. (2023) · The Lancet Digital Health · doi:10.1016/s2589-7500(23)00177-2 · Cited by 116
van Asch, C. J. J., Velthuis, B. K., Rinkel, G. J. E., Algra, A., de Kort, G. A. P., Witkamp, T. D., de Ridder, J. C. M., van Nieuwenhuizen, K. M., de Leeuw, F., Schonewille, W. J., et al. (2015) · BMJ · doi:10.1136/bmj.h5762 · Cited by 121
How to conduct a systematic review and meta-analysis of prognostic model studies
Damen, J. A., Moons, K. G. M., van Smeden, M., & Hooft, L. (2022) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2022.07.019 · Cited by 124
“tematic Reviews of Prediction Modelling Studies [23], Population, Index model,”
InfluentialSelf-citationSerghiou, S., Kyriakopoulou, A., & Ioannidis, J. P. A. (2016) · Molecular Cancer · doi:10.1186/s12943-016-0535-1 · Cited by 115
“The following data were extracted for all articles following the CHARMS checklist [34]: title; authors; year of publication; journal of publication; groupings (i.”
MethodManothummetha, K., Chuleerarux, N., Sanguankeo, A., Kates, O. S., Hirankarn, N., Thongkam, A., Dioverti, M. V., Torvorapanit, P., Langsiri, N., Worasilchai, N., et al. (2022) · JAMA Network Open · doi:10.1001/jamanetworkopen.2022.6822 · Cited by 120
“8 (8-81) days for the BNT162b2 vaccine, and 25.”
Silva, K. D., Lee, W. K., Forbes, A., Demmer, R. T., Barton, C., & Enticott, J. (2020) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2020.104268 · Cited by 108
Paolucci, S., Iosa, M., Coiro, P., Venturiero, V., Savo, A., Angelis, D. D., & Morone, G. (2019) · Frontiers in Neurology · doi:10.3389/fneur.2019.00926 · Cited by 119
Chen, Y., Qian, X., Zhang, Y., Su, W., Huang, Y., Wang, X., Chen, X., Zhao, E., Han, L., & Ma, Y. (2022) · Frontiers in Aging Neuroscience · doi:10.3389/fnagi.2022.840386 · Cited by 133
“The quality of the included studies was evaluated independently by two investigators, using the CHARMS (Moons et al., 2014).”
Tornero-Costa, R., Martinez-Millana, A., Azzopardi-Muscat, N., Lazeri, L., Traver, V., & Novillo-Ortiz, D. (2023) · JMIR Mental Health · doi:10.2196/42045 · Cited by 103
Biomarkers of Vascular Inflammation for Cardiovascular Risk Prognostication
Antonopoulos, A. S., Angelopoulos, A., Papanikolaou, P., Simantiris, S., Oikonomou, E. K., Vamvakaris, K., Koumpoura, A., Farmaki, M., Trivella, M., Vlachopoulos, C. V., et al. (2021) · JACC. Cardiovascular imaging · doi:10.1016/j.jcmg.2021.09.014 · Cited by 115
Towards the best kidney failure prediction tool: a systematic review and selection aid
Ramspek, C. L., de Jong, Y., Dekker, F. W., & van Diepen, M. (2019) · Nephrology Dialysis Transplantation · doi:10.1093/ndt/gfz018 · Cited by 112
Huang, Y., Li, J., Li, M., & Aparasu, R. R. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-023-02078-1 · Cited by 134
“To guide data extraction for ML prediction models, two checklists, including the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies (CHARMS) checklist and Machine Learning Methods in Health Economics and Outcomes Research Checklist were utilized [22, 23].”
MethodDigra, M., Dhir, R., & Sharma, N. (2022) · Arabian Journal of Geosciences · doi:10.1007/s12517-022-10246-8 · Cited by 118
“…the research paper related to the literature and to achieve the objectives of our research based upon Pre-ferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) (McInnes et al 2018) and the recommendation for systematic review for prediction model (CHARM model)(Moons et al 2014).”
MethodZheng, Q., Yang, L., Zeng, B., Li, J., Guo, K., Liang, Y., & Liao, G. (2020) · EClinicalMedicine · doi:10.1016/j.eclinm.2020.100669 · Cited by 108
“The research question was formulated according to previously published recommendations for systematic reviews of prediction models (CHARMS checklist) [28].”
Hodgson, L. E., Sarnowski, A., Roderick, P. J., Dimitrov, B. D., Venn, R. M., & Forni, L. G. (2017) · BMJ Open · doi:10.1136/bmjopen-2017-016591 · Cited by 100
“For example, one study used a case– control design, which is inappropriate for developing a prediction model as it does not enable calculation of absolute risks and thus yields incorrect estimates of model intercept or baseline hazard.(20) A wide variety of predictors were considered with use of univariate analysis to…”
Method
Metadata
- Authors
- Moons, K. G. M., de Groot, J. A. H., Bouwmeester, W., Vergouwe, Y., Mallett, S., Altman, D. G., Reitsma, J. B., & Collins, G. S.
- Affiliations
- University Medical Center Utrecht (NL)
- University of Oxford (GB)
- Corresponding author
- Moons, Karel G. M.
- Venue
- PLoS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2014-10-14
- Volume
- 11
- Issue
- 10
- Pages
- e1001744
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC4196729
- OpenAlex
- W2077663753
- Semantic Scholar
- c79bfa9a67c8745d7620fcfe8cf582b9d0407073
- Cited by
- 2,018
- References
- 157
- Citations by year
- 2026: 319 · 2025: 348 · 2024: 253 · 2023: 229 · 2022: 214 · 2021: 219 · 2020: 144 · 2019: 93 · 2018: 74 · 2017: 57 · 2016: 35 · 2015: 28 · 2014: 1
- FWCI
- 11.19
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- University of Birmingham Research Portal (University of Birmingham) (submittedVersion, cc-by)
- Funders
- Cancer Research UK (16895)
- National Institute for Health and Care Research (NF-SI-0513-10131)
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (project 918.10.615, 918.10.615)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- critical appraisal, data extraction, predictive modeling, prognostic model, diagnostic model, systematic reviews, reporting quality
- MeSH terms
- Decision Support Techniques, Humans, Models, Theoretical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.