Seventy-Five Trials and Eleven Systematic Reviews a Day: How Will We Ever Keep Up?
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
When Archie Cochrane reproached the medical profession for not having critical summaries of all randomised controlled trials, about 14 reports of trials were being published per day. There are now 75 trials, and 11 systematic reviews of trials, per day and a plateau in growth has not yet been reached. Although trials, reviews, and health technology assessments have undoubtedly had major impacts, the staple of medical literature synthesis remains the non-systematic narrative review. Only a small minority of trial reports are being analysed in up-to-date systematic reviews. Given the constraints, Archie Cochrane's vision will not be achieved without some serious changes in course. To meet the needs of patients, clinicians, and policymakers, unnecessary trials need to be reduced, and systematic reviews need to be prioritised. Streamlining and innovation in methods of systematic reviewing are necessary to enable valid answers to be found for most patient questions. Finally, clinicians and patients require open access to these important resources.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, such reviews have become ever more complicated, due to the complexity of interventions being studied and the amount of evidence being published which needs to be incorporated [1].”
From Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR) (2018) · cites this paper as [1]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When the number of studies is an order of magnitude greater than that, the inability of narrative processes to adequately summarize the outcomes increases exponentially 2,3 .”
From Meta-analysis and the science of research synthesis (2018) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The number of RCTs published accelerates every year,3”
From Trialstreamer: A living, automatically updated database of clinical trial reports (2020) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Screening references is a time‐consuming step necessary for systematic reviews and guideline development.”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [4]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, systematic review methodology and the language used to express that methodology, has progressed significantly since their appearance in healthcare in the 1970’s and 80’s [7, 8].”
From What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences (2018) · cites this paper as [7]
Experimental — accuracy not yet measured
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Avoidable waste in the production and reporting of research evidence
Chalmers, I., & Glasziou, P. (2009) · The Lancet · doi:10.1016/s0140-6736(09)60329-9 · Cited by 2,208
Self-citationMeta-Analysis in Clinical Research
L'ABBÉ, K. A., DETSKY, A. S., & O'ROURKE, K. (1987) · Annals of Internal Medicine · doi:10.7326/0003-4819-107-2-224 · Cited by 1,267 · cited 1×
“By the mid-1980s, the need to minimise the likelihood of being misled by the effects of biases and the play of chance in reviews of research evidence was being made evident in articles [10–14] and textbooks [15].”
BackgroundMeta-Analyses of Randomized Controlled Trials
Sacks, H. S., Berrier, J., Reitman, D., Ancona-Berk, V. A., & Chalmers, T. C. (1987) · New England Journal of Medicine · doi:10.1056/nejm198702193160806 · Cited by 1,250
Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors
De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., et al. (2004) · New England Journal of Medicine · doi:10.1056/nejme048225 · Cited by 1,241
Publication bias in clinical trials due to statistical significance or direction of trial results
Hopewell, S., Loudon, K., Clarke, M. J., Oxman, A. D., & Dickersin, K. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000006.pub3 · Cited by 898
Epidemiology and reporting characteristics of systematic reviews.
Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M., & Altman, D. G. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040078 · Cited by 891 · cited 2×
“variously estimated at between six and 16 (Cochrane reviews now include an average of over 12 trials per review [27,28]; M Clarke, personal communication), but many reviews have covered much the same territory.”
BackgroundThe Medical Review Article: State of the Science
MULROW, C. D. (1987) · Annals of Internal Medicine · doi:10.7326/0003-4819-106-3-485 · Cited by 856 · cited 1×
“By the mid-1980s, the need to minimise the likelihood of being misled by the effects of biases and the play of chance in reviews of research evidence was being made evident in articles [10–14] and textbooks [15].”
BackgroundHow Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis
Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., & Moher, D. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-4-200708210-00179 · Cited by 838 · cited 1×
“now many years out of date [26].”
Avoidable Waste in the Production and Reporting of Research Evidence
Chalmers, I., & Glasziou, P. (2009) · Obstetrics & Gynecology · doi:10.1097/aog.0b013e3181c3020d · Cited by 600 · cited 1×
“duction and reporting of research evidence [29].”
Self-citationGuidelines for reading literature reviews.
Oxman, A. D., & Guyatt, G. H. (1988) · Europe PMC (PubMed Central) · Cited by 549
Epidemiology and reporting of randomised trials published in PubMed journals
Chan, A.-W., & Altman, D. G. (2005) · The Lancet · doi:10.1016/s0140-6736(05)71879-1 · Cited by 474 · cited 1×
“century, and in reports of systematic reviews since the 1980s—and a plateau in growth has not yet been reached. With a median of perhaps 80 participants per trial, the number of people being enrolled in trials is likely to be more than 2,000,000 per year”
BackgroundThe development of health technology assessment
Banta, D. (2003) · Health Policy · doi:10.1016/s0168-8510(02)00059-3 · Cited by 334 · cited 1×
“Government health technology assessment agencies were also established as policymakers sought to have more reliable evidence of the effects of other forms of health care interventions [6].”
BackgroundIssues in the Registration of Clinical Trials
Zarin, D. A., Ide, N. C., Tse, T., Harlan, W. R., West, J. C., & Lindberg, D. A. B. (2007) · JAMA · doi:10.1001/jama.297.19.2112 · Cited by 222 · cited 1×
“into force, more than 200 ongoing trials per week were being registered [23].”
BackgroundPublication of Clinical Trials Supporting Successful New Drug Applications: A Literature Analysis
Lee, K., Bacchetti, P., & Sim, I. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050191 · Cited by 215 · cited 1×
“For a variety of reasons, a large proportion of trials have remained unpublished [18,19].”
BackgroundDECREASED SURVIVAL RELATED TO IRRADIATION POSTOPERATIVELY IN EARLY OPERABLE BREAST CANCER
Stjernswärd, J. (1974) · The Lancet · doi:10.1016/s0140-6736(74)90142-1 · Cited by 181
Effects of ascorbic acid on the common cold
Chalmers, T. C. (1975) · The American Journal of Medicine · doi:10.1016/0002-9343(75)90127-8 · Cited by 155 · cited 1×
“Another challenged beliefs about vitamin C and the common cold [8].”
BackgroundLind, J. (2004) · doi:10.1590/s0042-96862004001000018 · Cited by 111 · cited 1×
“before the subject could be set in a clear and proper light, it was necessary to remove a great deal of rubbish’’ [2].”
BackgroundLind, J. (1980) · The Classics of Medicine Library eBooks · Cited by 109
The Value of “Traditional” Reviews in the Era of Systematic Reviewing
Dijkers, M. P. J. M. (2009) · American Journal of Physical Medicine & Rehabilitation · doi:10.1097/phm.0b013e31819c59c6 · Cited by 103 · cited 1×
“shows, this is far from the case. The growth has been even more remarkable in non-systematic (“narrative”) reviews and case reports. Journal publishing of non-systematic reviews, and the emergence of many journals whose sole product is non-systematic reviews, has far outstripped the growth of systematic reviews and HT…”
BackgroundThe origins, evolution, and future of The Cochrane Database of Systematic Reviews
Starr, M., Chalmers, I., Clarke, M., & Oxman, A. D. (2009) · International Journal of Technology Assessment in Health Care · doi:10.1017/s026646230909062x · Cited by 93
Self-citationSmith, G. T., & Wells, N. E. J. (1979) · Monographs · Cited by 82
Smith, G. T., & Wells, N. E. J. (1979) · Monographs · Cited by 82
Ritchie, M. (1996) · Project Muse (Johns Hopkins University) · Cited by 70
Updating Systematic Reviews: An International Survey
Garritty, C., Tsertsvadze, A., Tricco, A. C., Sampson, M., & Moher, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0009914 · Cited by 62 · cited 1×
“Using current methods, the Cochrane Collaboration has not been able to keep even half of its reviews up-todate [31], and other organisations are in a similar predicament [32].”
MethodTHE TYPICAL COCHRANE REVIEW: How Many Trials? How Many Participants?
Mallett, S., & Clarke, M. (2002) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462302000624 · Cited by 54
The National Library of Medicine
Horwitz, N. H. (2002) · Neurosurgery · doi:10.1097/00006123-200211000-00034 · Cited by 47
Information synthesis: a practical guide.
Pg, G. (1986) · Europe PMC (PubMed Central) · Cited by 41
(1983) · Clinical Research Practices and Drug Regulatory Affairs · doi:10.3109/10601338309029371 · Cited by 24
Coping With the Biomedical Literature
Warren, K. S. (1981) · Medical Entomology and Zoology · Cited by 23
Chalmers, L. (1991) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462300007029 · Cited by 14 · cited 1×
“In 1988, regularly updated electronic publication of systematic reviews and meta-analyses, along with bibliographies of randomised trials, began in the perinatal field [16,17].”
BackgroundBarron, B., & Bukantz, S. C. (1967) · Archives of Internal Medicine · doi:10.1001/archinte.1967.00290240069001 · Cited by 13 · cited 1×
“portant component of the evidence base [5].”
The National Library of Medicine
Cummings, M. (1976) · Journal of the American Podiatric Medical Association · doi:10.7547/87507315-66-9-712 · Cited by 5
Research works printed but not linked: 3
- Duncan A 1773 Introduction. Medical and Philosophical Commentaries. Volume First, Part I. London: J Murray 6 7
- Chalmers I 1979 Randomized controlled trials of fetal monitoring 1973–1977. Thalhammer O Baumgarten K Pollak A Perinatal Medicine Stuttgart Georg Thieme 260 265
- Koch GG 2006 No improvement – still less than half of the Cochrane reviews are up to date. XIV Cochrane Colloquium, Dublin, Ireland
Cited by
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
“They are becoming increasingly common; in 2010, 11 new reviews were estimated to be published daily [1].”
BackgroundMunn, Z., Peters, M. D. J., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0611-x · Cited by 11,385
Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., et al. (2017) · BMJ · doi:10.1136/bmj.j4008 · Cited by 10,636
Updated methodological guidance for the conduct of scoping reviews
Peters, M. D. J., Marnie, C., Tricco, A. C., Pollock, D., Munn, Z., Alexander, L., McInerney, P., Godfrey, C. M., & Khalil, H. (2020) · JBI Evidence Synthesis · doi:10.11124/jbies-20-00167 · Cited by 9,046
SANRA—a scale for the quality assessment of narrative review articles
Baethge, C., Goldbeck-Wood, S., & Mertens, S. (2019) · Research Integrity and Peer Review · doi:10.1186/s41073-019-0064-8 · Cited by 3,469
“found that they constitute the largest share of all text types in medicine and they concluded that they “remain the staple of medical literature” [1].”
Background
Metadata
- Authors
- Bastian, H., Glasziou, P., & Chalmers, I.
- Affiliations
- Institute for Quality and Efficiency in Health Care (DE)
- Bond University (AU)
- Corresponding author
- Bastian, Hilda
- Venue
- PLoS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2010-09-21
- Volume
- 7
- Issue
- 9
- Pages
- e1000326
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- PMC
- PMC2943439
- OpenAlex
- W2146668368
- Cited by
- 1,188
- References
- 32
- Citations by year
- 2026: 48 · 2025: 57 · 2024: 70 · 2023: 77 · 2022: 79 · 2021: 126 · 2020: 100 · 2019: 102 · 2018: 83 · 2017: 76 · 2016: 76 · 2015: 71 · 2014: 62 · 2013: 86 · 2012: 44
- FWCI
- 14.05
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Medicine (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- e-publications@bond (Bond University) (submittedVersion)
- Bond University Research Portal (Bond University) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Center for Clinical and Translational Research
- U.S. National Library of Medicine
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- systematic reviews, randomized controlled trials, health technology assessment, open access, evidence synthesis, Cochrane Collaboration
- MeSH terms
- Humans, Policy Making, Publishing, Review Literature as Topic, Randomized Controlled Trials as Topic, Evidence-Based Medicine, Biomedical Technology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.