The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Many randomized controlled trials (RCTs) are biased and difficult to reproduce due to methodological flaws and poor reporting. There is increasing attention for responsible research practices and implementation of reporting guidelines, but whether these efforts have improved the methodological quality of RCTs (e.g., lower risk of bias) is unknown. We, therefore, mapped risk-of-bias trends over time in RCT publications in relation to journal and author characteristics. Meta-information of 176,620 RCTs published between 1966 and 2018 was extracted. The risk-of-bias probability (random sequence generation, allocation concealment, blinding of patients/personnel, and blinding of outcome assessment) was assessed using a risk-of-bias machine learning tool. This tool was simultaneously validated using 63,327 human risk-of-bias assessments obtained from 17,394 RCTs evaluated in the Cochrane Database of Systematic Reviews (CDSR). Moreover, RCT registration and CONSORT Statement reporting were assessed using automated searches. Publication characteristics included the number of authors, journal impact factor (JIF), and medical discipline. The annual number of published RCTs substantially increased over 4 decades, accompanied by increases in authors (5.2 to 7.8) and institutions (2.9 to 4.8). The risk of bias remained present in most RCTs but decreased over time for allocation concealment (63% to 51%), random sequence generation (57% to 36%), and blinding of outcome assessment (58% to 52%). Trial registration (37% to 47%) and the use of the CONSORT Statement (1% to 20%) also rapidly increased. In journals with a higher impact factor (>10), the risk of bias was consistently lower with higher levels of RCT registration and the use of the CONSORT Statement. Automated risk-of-bias predictions had accuracies above 70% for allocation concealment (70.7%), random sequence generation (72.1%), and blinding of patients/personnel (79.8%), but not for blinding of outcome assessment (62.7%). In conclusion, the likelihood of bias in RCTs has generally decreased over the last decades. This optimistic trend may be driven by increased knowledge augmented by mandatory trial registration and more stringent reporting guidelines and journal requirements. Nevertheless, relatively high probabilities of bias remain, particularly in journals with lower impact factors. This emphasizes that further improvement of RCT registration, conduct, and reporting is still urgently needed.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
RoB 2: a revised tool for assessing risk of bias in randomised trials
Sterne, J. A. C., Savović, J., Page, M. J., Elbers, R. G., Blencowe, N. S., Boutron, I., Cates, C. J., Cheng, H.-Y., Corbett, M. S., Eldridge, S. M., et al. (2019) · BMJ · doi:10.1136/bmj.l4898 · Cited by 34,787
Improving the Quality of Reporting of Randomized Controlled Trials: The CONSORT Statement
Begg, C. (1996) · JAMA · doi:10.1001/jama.1996.03540080059030 · Cited by 3,127
The History and Meaning of the Journal Impact Factor
Garfield, E. (2006) · JAMA · doi:10.1001/jama.295.1.90 · Cited by 2,770 · cited 1×
“Even though the JIF (the average number of times its articles has been cited in other articles for 2 years) is not a very suitable indicator of journal quality [23], no unbiased alternatives exist.”
BackgroundWhy Most Discovered True Associations Are Inflated
Ioannidis, J. P. A. (2008) · Epidemiology · doi:10.1097/ede.0b013e31818131e7 · Cited by 1,495 · cited 1×
“Across RCTs, there is a major risk for inflated estimates and problems with randomization, allocation concealment, and blinding [2,3].”
BackgroundHow to increase value and reduce waste when research priorities are set
Chalmers, I., Bracken, M. B., Djulbegovic, B., Garattini, S., Grant, J., Gülmezoglu, A. M., Howells, D. W., Ioannidis, J. P. A., & Oliver, S. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62229-1 · Cited by 1,436
Biomedical research: increasing value, reducing waste.
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., Salman, R. A.-S., Chan, A.-W., & Glasziou, P. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62329-6 · Cited by 1,003
Self-citationTurner, L., Shamseer, L., Altman, D. G., Weeks, L., Peters, J., Kober, T., Dias, S., Schulz, K. F., Plint, A. C., & Moher, D. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000030.pub2 · Cited by 740 · cited 2×
“Fourth, even though the CONSORT Statement was introduced to improve RCT reporting [8], the rapid increase of RCTs that mention following the CONSORT guideline does not guarantee adherence, and reporting methodological quality can remain suboptimal [14].”
BackgroundSelf-citationHopewell, S., Clarke, M., Moher, D., Wager, E., Middleton, P., Altman, D. G., Schulz, K. F., & and the CONSORT Group (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050020 · Cited by 674
Self-citationRobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv044 · Cited by 317
Vinkers, C. H., Tijdink, J. K., & Otte, W. M. (2015) · BMJ · doi:10.1136/bmj.h6467 · Cited by 237
Self-citation
Cited by
Cro, S., Kahan, B. C., Rehal, S., Chis Ster, A., Carpenter, J. R., White, I. R., & Cornelius, V. R. (2022) · BMJ · doi:10.1136/bmj-2022-070146 · Cited by 52
Methodology reporting improved over time in 176,469 randomized controlled trials
Kilicoglu, H., Jiang, L., Hoang, L., Mayo-Wilson, E., Vinkers, C. H., & Otte, W. M. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.08.004 · Cited by 19
Self-citationDo it faster with PICOS: Generative AI-Assisted systematic review screening
Vallamchetla, S. K., Abdelkader, O., Elnaggar, A., Ramadan, D., Islam Shourav, M. M., Riaz, I. B., & Lin, M. P. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104860 · Cited by 13
Reproducibility failure in biomedical research: problems and solutions
Haven, T. L., & Ioannidis, J. P. A. (2026) · Annual Review of Medicine · doi:10.1146/annurev-med-050124-050859 · Cited by 9
Burley, J., & McAloon, J. (2024) · Journal of Contextual Behavioral Science · doi:10.1016/j.jcbs.2023.11.003 · Cited by 13
Metadata
- Authors
- Vinkers, C. H., Lamberink, H. J., Tijdink, J. K., Heus, P., Bouter, L., Glasziou, P., Moher, D., Damen, J. A., Hooft, L., & Otte, W. M.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2021-04-19
- Volume
- 19
- Issue
- 4
- Pages
- e3001162
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- OpenAlex
- W3155544647
- Cited by
- 103
- References
- 14
- FWCI
- 11.02
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement. https://app.cochat.ai/papers/10.1371/journal.pbio.3001162
- The paper
- Vinkers, C. H., Lamberink, H. J., Tijdink, J. K., Heus, P., Bouter, L., Glasziou, P., Moher, D., Damen, J. A., Hooft, L., & Otte, W. M. (2021). The methodological quality of 176,620 randomized controlled trials published between 1966 and 2018 reveals a positive trend but also an urgent need for improvement. PLOS Biology. https://doi.org/10.1371/journal.pbio.3001162
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.