The DOI matches its Crossref record; 55 of the 55 references it cites were found in a registry or an index.
Methodological quality of COVID-19 clinical research
Open accessPublished version of a preprint
Abstract
The COVID-19 pandemic began in early 2020 with major health consequences. While a need to disseminate information to the medical community and general public was paramount, concerns have been raised regarding the scientific rigor in published reports. We performed a systematic review to evaluate the methodological quality of currently available COVID-19 studies compared to historical controls. A total of 9895 titles and abstracts were screened and 686 COVID-19 articles were included in the final analysis. Comparative analysis of COVID-19 to historical articles reveals a shorter time to acceptance (13.0[IQR, 5.0–25.0] days vs. 110.0[IQR, 71.0–156.0] days in COVID-19 and control articles, respectively; p < 0.0001). Furthermore, methodological quality scores are lower in COVID-19 articles across all study designs. COVID-19 clinical studies have a shorter time to publication and have lower methodological quality scores than control studies in the same journal. These studies should be revisited with the emergence of stronger evidence.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



![Fig. 4 Differences in methodological quality scores in COVID-19 compared to historical control articles. A Time to acceptance was reduced in COVID-19 articles compared to control articles (13.0 [IQR, 5.0–25.0] days vs. 110.0 [IQR, 71.0–156.0] days, n = 347 for COVID-19 and n = 414 for controls; p < 0.0001). B When compared to historical control articles, COVID-19 articles were associated with lower case series score (n = 277 for COVID-19 and n = 277 for controls; p < 0.0001). C COVID-19 articles were associated with lower NOS cohort score compared to historical control articles (n = 174 for COVID-19 and n = 174 for controls; p < 0.0001). D COVID-19 articles were associated with lower NOS case–control score compared to historical control articles (n = 32 for COVID-19 and n = 32 for controls; p = 0.003). E COVID-19 articles were associated with higher diagnostic risk of bias (QUADAS-2) compared to historical control articles (n = 53 for COVID-19 and n = 53 for controls; p = 0.02). For panel A, boxplot captures 5, 25, 50, 75 and 95% from the first to last whisker. Orange represents COVID-19 articles and blue represents control articles. Two-sided Mann–Whitney U-test was conducted to evaluate differences in time to acceptance between COVID-19 and control articles. Differences in study quality scores were evaluated by two-sided Kruskal–Wallis test. Differences in diagnostic risk of bias were quantified by Chi-squares test. p < 0.05 was considered statistically significant.](https://figures.cochat.ai/paper-figures/4017af719be4496cad21569fa8e874d2/methodological-quality-of-covid-19-clinical-research-figure-4.jpg)
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Jung et al.6 measured the adherence to methodological tools using the Cochrane risk bias tool. They detected a 32% decline in the adherence rate to a high methodological quality.”
From Artificial Intelligence in Detecting Statistical Errors: Implications for Authors, Reviewers, and Editors (2025) · cites this paper as [6]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“At the same time, incomplete reporting or non-adherence to guidelines for RCTs [69] was common, most likely attributable to circumstances related to the pandemic itself, such as short time frames for study planning and conceptualization, difficulties in recruitment and implementation due to contact restrictions or accelerated peer-review of scientific journals [70].”
From Interventions to build resilience and to ameliorate negative psychosocial effects of the COVID-19 pandemic on children and adolescents: a systematic review and meta-analysis (2024) · cites this paper as [70]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Finally, poor reporting has been commonly investigated in literature 37 38 ; owing to the limited information of the data extraction process reported by review authors, we could not fully investigate the association between good practice recommendations and the likelihood of data extraction.”
From Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study (2022) · cites this paper as [37]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
3 retracted
Cardiovascular Disease, Drug Therapy, and Mortality in Covid-19
Mehra, M. R., Desai, S. S., Kuy, S., Henry, T. D., & Patel, A. N. (2020) · New England Journal of Medicine · doi:10.1056/nejmoa2007621 · Cited by 1,428
RetractedMehra, M. R., Desai, S. S., Ruschitzka, F., & Patel, A. N. (2020) · The Lancet · doi:10.1016/s0140-6736(20)31180-6 · Cited by 1,200
RetractedMehra, M. R., Desai, S. S., Kuy, S., Henry, T. D., & Patel, A. N. (2020) · New England Journal of Medicine · doi:10.1056/nejmc2021225 · Cited by 463
RetractedThe Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
Sterne, J. A., Hernán, M. A., Reeves, B. C., Savović, J., Berkman, N. D., Viswanathan, M., Henry, D., Altman, D. G., Ansari, M. T., Boutron, I., et al. (2016) · BMJ · doi:10.1136/bmj.i4919 · Cited by 20,329
(2014) · Medical Entomology and Zoology · Cited by 17,633
Stang, A. (2010) · European Journal of Epidemiology · doi:10.1007/s10654-010-9491-z · Cited by 17,396
QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies
Whiting, P. F., Rutjes, A. W. S., Westwood, M. E., Mallett, S., Deeks, J. J., Reitsma, J. B., Leeflang, M. M. G., Sterne, J. A. C., Bossuyt, P. M. M., & the QUADAS-2 Group* (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-8-201110180-00009 · Cited by 14,199
Dexamethasone in Hospitalized Patients with Covid-19
The RECOVERY Collaborative Group (2021) · New England Journal of Medicine · doi:10.1056/nejmoa2021436 · Cited by 11,477
Cited by
COVID-19 pandemic effects on health worker's mental health: systematic review and meta-analysis
Aymerich, C., Pedruzo, B., Pérez, J. L., Laborda, M., Herrero, J., Blanco, J., Mancebo, G., Andrés, L., Estévez, O., Fernandez, M., et al. (2022) · European Psychiatry · doi:10.1192/j.eurpsy.2022.1 · Cited by 234
“COVID-19 pandemic has stimulated the publication of many studies in a short time, some of themof limited quality [80].”
BackgroundLinardon, J., Messer, M., Rodgers, R. F., & Fuller‐Tyszkiewicz, M. (2022) · International Journal of Eating Disorders · doi:10.1002/eat.23640 · Cited by 169
“However, we also share optimism expressed by others (e.g., Jung et al., 2021) that higher quality research—which typically takes longer to produce—is likely to become available in the coming months to balance concerns with methodological difficulties of early COVID-19 research.”
BackgroundYe, J., He, L., & Beestrum, M. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00908-6 · Cited by 157
“Prior research has indicated that COVID-19 research ranked low in the hierarchy of scienti fi c evidence, with numerous articles published in lower impact factor journals and within a shorter timeframe 81 .”
Circadian disturbances, sleep difficulties and the COVID-19 pandemic
Salehinejad, M. A., Azarkolah, A., Ghanavati, E., & Nitsche, M. A. (2021) · Sleep Medicine · doi:10.1016/j.sleep.2021.07.011 · Cited by 96
“This is an important point to consider as there have been concerns over the methodological quality and validity of reported findings/perspectives related to the pandemic [12,13].”
Validity of data extraction in evidence synthesis practice of adverse events: reproducibility study
Xu, C., Yu, T., Furuya-Kanamori, L., Lin, L., Zorzela, L., Zhou, X., Dai, H., Loke, Y., & Vohra, S. (2022) · BMJ · doi:10.1136/bmj-2021-069155 · Cited by 82
Metadata
- Authors
- Jung, R. G., Di Santo, P., Clifford, C., Prosperi-Porta, G., Skanes, S., Hung, A., Parlow, S., Visintini, S., Ramirez, F. D., Simard, T., & Hibbert, B.
- Affiliations
- University of Ottawa (CA)
- Ottawa Hospital (CA)
- Ottawa Heart Institute (CA)
- Université de Bordeaux (FR)
- Mayo Clinic (US)
- Corresponding author
- Hibbert, Benjamin
- Venue
- Nature Communications
- Published
- 2021-02-11
- Volume
- 12
- Issue
- 1
- Pages
- 943-943
- ISSN
- 2041-1723 (ISSN-L)
- PMC
- PMC7878793
- OpenAlex
- W3128106047
- Cited by
- 189
- References
- 54
- Citations by year
- 2026: 4 · 2025: 35 · 2024: 20 · 2023: 41 · 2022: 50 · 2021: 37
- FWCI
- 7.21
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Nature Communications (publishedVersion, cc-by)
- medRxiv (acceptedVersion)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Royal College of Physicians and Surgeons of Canada
- Canadian Institutes of Health Research
- Topics
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Infectious Diseases › COVID-19 Clinical Research Studies
- Keywords
- COVID-19 pandemic, methodological quality, clinical research, scientific rigor, time to publication, peer review, historical controls
- MeSH terms
- Clinical Studies as Topic, Data Accuracy, COVID-19, Animals, Humans, Periodicals as Topic, Research Design, Time Factors, Peer Review, Research, Pandemics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Methodological quality of COVID-19 clinical research. https://app.cochat.ai/papers/10.1038/s41467-021-21220-5
- The paper
- Jung, R. G., Di Santo, P., Clifford, C., Prosperi-Porta, G., Skanes, S., Hung, A., Parlow, S., Visintini, S., Ramirez, F. D., Simard, T., & Hibbert, B. (2021). Methodological quality of COVID-19 clinical research. Nature Communications. https://doi.org/10.1038/s41467-021-21220-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.