A systematic evaluation of the quality of meta-analyses in the critical care literature
Open accessThe DOI matches its Crossref record; 7 of the 7 references it cites were found in a registry or an index.
Abstract
Introduction: Meta-analyses have been suggested to be the highest form of evidence available to clinicians to guide clinical practice in critical care. The purpose of this study was to systematically evaluate the quality of meta-analyses that address topics pertinent to critical care. Methods: To identify potentially eligible meta-analyses for inclusion, a systematic search of Medline, EMBASE and the Cochrane Database of Systematic Reviews was undertaken, using broad search terms relevant to intensive care, including: intensive care, critical care, shock, resuscitation, inotropes and mechanical ventilation. Predetermined inclusion criteria were applied to each identified meta-analysis independently by two authors. To assess report quality, the included meta-analyses were assessed using the component and overall scores from the Overview Quality Assessment Questionnaire (OQAQ). The quality of reports published before and after the publication of the QUOROM statement was compared. Results: A total of 139 reports of meta-analyses were included (kappa = 0.93). The overall quality of reports of meta-analyses was found to be poor, with an estimated mean overall OQAQ score of 3.3 (95% CI; 3.0–3.6). Only 43 (30.9%) were scored as having minimal or minor flaws (>5). We noted problems with the reporting of key characteristics of meta-analyses, such as performing a thorough literature search, avoidance of bias in the inclusion of studies and appropriately referring to the validity of the included studies. After the release of the QUOROM statement, however, an improvement in the overall quality of published meta-analyses was noted. Conclusion: The overall quality of the reports of meta-analyses available to critical care physicians is poor. Physicians should critically evaluate these studies prior to considering applying the results of these studies in their clinical practice.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Moher, D., Jones, A., Lepage, L., & for the CONSORT Group (2001) · JAMA · doi:10.1001/jama.285.15.1992 · Cited by 1,027 · cited 1×
“A similar improvement in the quality of reports has been found with regards to the quality of reports of RCTs following the publication of the Consolidated Standards of Reporting Trials (CONSORT) statement [24].”
BackgroundThe Medical Review Article: State of the Science
MULROW, C. D. (1987) · Annals of Internal Medicine · doi:10.7326/0003-4819-106-3-485 · Cited by 856 · cited 1×
“Traditional sources of information such as narrative review articles, medical textbooks and the clinical opinion of experts are often at odds with the best current available evidence [1,2].”
BackgroundMeta-analysis: Potentials and promise
Egger, M., & Smith, G. D. (1997) · BMJ · doi:10.1136/bmj.315.7119.1371 · Cited by 646
Large Trials vs Meta-analysis of Smaller Trials: How Do Their Results Compare?
Cappelleri, J. C. (1996) · JAMA · doi:10.1001/jama.1996.03540160054033 · Cited by 336
Evaluating Meta-analyses in the General Surgical Literature: A Critical Appraisal
Dixon, E., Hameed, M., Sutherland, F., Cook, D. J., & Doig, C. (2005) · Annals of Surgery · doi:10.1097/01.sla.0000154258.30305.df · Cited by 129 · cited 3×
“Data from this study were compared with the data published in previous reports from the emergency medicine [17], anaesthesia [19] and general surgery [18] literature.”
BackgroundResultSelf-citationAntman, E. M. (1992) · JAMA · doi:10.1001/jama.1992.03490020088036 · Cited by 11 · cited 1×
“Traditional sources of information such as narrative review articles, medical textbooks and the clinical opinion of experts are often at odds with the best current available evidence [1,2].”
Background
Research works printed but not linked: 19
- Cook DJ, Meade MO, Fink MP: How to keep up with the critical care literature and avoid being buried alive. Crit Care Med 1996, 24: 1757-1768. 10.1097/00003246-199610000-00026 · DOI
- Guyatt GH, Haynes RB, Jaeschke RZ, Cook DJ, Green L, Naylor CD, Wilson MC, Richardson WS: Users' Guides to the Medical Literature: XXV. Evidence-based medicine: principles for applying the Users' Guides to patient care. Evidence-Based Medicine Working Group. JAMA 2000, 284: 1290-1296. 10.1001/jama.284.10.1290 · DOI · Paper page
- McIntyre LA, Fergusson DA, Hebert PC, Moher D, Hutchison JS: Prolonged therapeutic hypothermia after traumatic brain injury in adults: a systematic review. JAMA 2003, 289: 2992-2999. 10.1001/jama.289.22.2992 · DOI · Paper page
- Jadad AR, Cook DJ, Browman GP: A guide to interpreting discordant systematic reviews. CMAJ 1997, 156: 1411-1416.
- Henderson WR, Dhingra VK, Chittock DR, Fenwick JC, Ronco JJ: Hypothermia in the management of traumatic brain injury. A systematic review and meta-analysis. Intensive Care Med 2003, 29: 1637-1644. 10.1007/s00134-003-1848-2 · DOI · Paper page
- Finfer S, Bellomo R, Boyce N, French J, Myburgh J, Norton R: A comparison of albumin and saline for fluid resuscitation in the intensive care unit. N Engl J Med 2004, 350: 2247-2256. 10.1056/NEJMoa040232 · DOI · Paper page
- Cook DJ, Reeve BK, Guyatt GH, Heyland DK, Griffith LE, Buckingham L, Tryba M: Stress ulcer prophylaxis in critically ill patients. Resolving discordant meta-analyses. JAMA 1996, 275: 308-314. 10.1001/jama.275.4.308 · DOI · Paper page
- Cook D, Guyatt G, Marshall J, Leasa D, Fuller H, Hall R, Peters S, Rutledge F, Griffith L, McLellan A, et al.: A comparison of sucralfate and ranitidine for the prevention of upper gastrointestinal bleeding in patients requiring mechanical ventilation. Canadian Critical Care Trials Group. N Engl J Med 1998, 338: 791-797. 10.1056/NEJM199803193381203 · DOI · Paper page
- Cochrane Injuries Group Albumin Reviewers: Human albumin administration in critically ill patients: systematic review of randomised controlled trials. BMJ 1998, 317: 235-240. · DOI · Paper page
- LeLorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian F: Discrepancies between meta-analyses and subsequent large randomized, controlled trials. N Engl J Med 1997, 337: 536-542. 10.1056/NEJM199708213370806 · DOI · Paper page
- Flather MD, Farkouh ME, Pogue JM, Yusuf S: Strengths and limitations of meta-analysis: larger studies may be more reliable. Control Clin Trials 1997, 18: 568-579. 10.1016/S0197-2456(97)00024-X · DOI · Paper page
- Celermajer DS: Evidence-based medicine: how good is the evidence? Med J Aust 2001, 174: 293-295. · doi:10.5694/j.1326-5377.2001.tb143274.x
- Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchison BG, Milner RA, Streiner DL: Agreement among reviewers of review articles. J Clin Epidemiol 1991, 44: 91-98. 10.1016/0895-4356(91)90205-N · DOI · Paper page
- Oxman AD, Guyatt GH: Validation of an index of the quality of review articles. J Clin Epidemiol 1991, 44: 1271-1278. 10.1016/0895-4356(91)90160-B · DOI · Paper page
- Kelly KD, Travers A, Dorgan M, Slater L, Rowe BH: Evaluating the quality of systematic reviews in the emergency medicine literature. Ann Emerg Med 2001, 38: 518-526. 10.1067/mem.2001.115881 · DOI
- Choi PT, Halpern SH, Malik N, Jadad AR, Tramer MR, Walder B: Examining the evidence in anesthesia literature: a critical appraisal of systematic reviews. Anesth Analg 2001, 92: 700-709. 10.1097/00000539-200103000-00029 · DOI
- Shojania KG, Bero LA: Taking advantage of the explosion of systematic reviews: an efficient MEDLINE search strategy. Eff Clin Pract 2001, 4: 157-162.
- Hunt DL, McKibbon KA: Locating and appraising systematic reviews. Ann Intern Med 1997, 126: 532-538. · DOI · Paper page
- Villar J, Carroli G, Belizan JM: Predictive ability of meta-analyses of randomised controlled trials. Lancet 1995, 345: 772-776. 10.1016/S0140-6736(95)90646-0 · DOI · Paper page
Cited by
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Tong, A., Sainsbury, P., & Craig, J. (2007) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzm042 · Cited by 42,509
“Subsequent research has shown that these checklists have improved the quality of reporting of study types relevant to each checklist [5, 25], and we believe that the effect of COREQ is likely to be similar.”
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.005 · Cited by 19,575
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gotzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · BMJ · doi:10.1136/bmj.b2700 · Cited by 17,648
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2010) · International Journal of Surgery · doi:10.1016/j.ijsu.2010.02.007 · Cited by 12,156
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.006 · Cited by 10,578
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00136 · Cited by 10,349
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · Italian Journal of Public Health · doi:10.2427/5768 · Cited by 3,374
Reprint—Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & and the PRISMA Group (2009) · Physical Therapy · doi:10.1093/ptj/89.9.873 · Cited by 2,486
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483
Urrútia, G., & Bonfill, X. (2010) · Medicina Clínica · doi:10.1016/j.medcli.2010.01.015 · Cited by 1,369
Principais itens para relatar Revisões sistemáticas e Meta-análises: A recomendação PRISMA
(2015) · Epidemiologia e Serviços de Saúde · doi:10.5123/s1679-49742015000200017 · Cited by 1,238
Epidemiology and reporting characteristics of systematic reviews.
Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M., & Altman, D. G. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040078 · Cited by 891
“The QUOROM Statement, which provides reporting guidance for meta-analysis of RCTs, was developed ten years ago in the hope of having a similar impact on meta-analysis [34], and there are some indications that its use is associated with improved quality of reporting [35].”
MethodMoher, D. (2009) · Journal of Chinese Integrative Medicine · doi:10.3736/jcim20090918 · Cited by 909
Page, M. J., McKenzie, J. E., Kirkham, J., Dwan, K., Kramer, S., Green, S., & Forbes, A. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000035.pub2 · Cited by 281
Moseley, A. M., Elkins, M. R., Herbert, R. D., Maher, C. G., & Sherrington, C. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.09.018 · Cited by 202
Delaney, A., Bagshaw, S. M., Ferland, A., Laupland, K., Manns, B., & Doig, C. (2007) · Critical Care Medicine · doi:10.1097/01.ccm.0000253394.15628.fd · Cited by 112
Self-citationTao, K.-M., Li, X.-Q., Zhou, Q.-H., Moher, D., Ling, C.-Q., & Yu, W.-F. (2011) · PLoS ONE · doi:10.1371/journal.pone.0027611 · Cited by 101
“Systematic reviews of this specialty have been considered to be higher quality than reviews published in other specialty medicine journals [28], so the conclusion of this study should be treated cautiously.”
BackgroundMa, B., Guo, J., Qi, G., Li, H., Peng, J., Zhang, Y., Ding, Y., & Yang, K. (2011) · PLoS ONE · doi:10.1371/journal.pone.0020185 · Cited by 56
“Examination of changes in reporting quality of MAs published in Medline, Embase and the Cochrane Database of Systematic Reviews, suggested that quality had improved after the release of the QUORUM reporting guidelines [18,19].”
BackgroundResults reporting in single case experiments and single case meta-analysis
Vannest, K. J., Peltier, C., & Haas, A. (2018) · Research in Developmental Disabilities · doi:10.1016/j.ridd.2018.04.029 · Cited by 51
Metadata
- Authors
- Delaney, A., Bagshaw, S. M., Ferland, A., Manns, B., Laupland, K. B., & Doig, C. J.
- Venue
- Critical Care
- Published
- 2005-09-09
- Volume
- 9
- Issue
- 5
- Pages
- R575-82
- ISSN
- 1364-8535 (ISSN-L)
- OpenAlex
- W1774591799
- Cited by
- 142
- References
- 7
- FWCI
- 8.11
- Type
- journal article
- Language
- en
- Open access
- gold
- Licence
- creativecommons.org/licenses/by/2.0/ (tdm)
- creativecommons.org/licenses/by/2.0/ (vor)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.