Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
Open accessAbstract
1Quality | Ethics | Open Science | Translation (QUEST), Berlin Institute of Health, Berlin, Germany. 2Charité–Universitätsmedizin Berlin, Berlin, Germany. 3School of Information Sciences, University of Illinois at Urbana–Champaign, Champaign, IL, USA. 4University Grenoble Alpes, CNRS, Grenoble INP, LIG, Grenoble, France. 5Department of Neuroscience, University of California, San Diego, La Jolla, CA, USA. 6SciCrunch Inc., San Diego, CA, USA. 7Department of Cardiology, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands. 8Urban Vitality Center of Expertise, Amsterdam University of Applied Sciences, Amsterdam, the Netherlands. 9New South Wales Health Statewide Biobank, New South Wales Health Pathology, Sydney, New South Wales, Australia. 10Faculty of Medicine and Health, University of Sydney, Sydney, New South Wales, Australia. 11UMR 5505 IRIT, Université de Toulouse, UPS, Toulouse, France. 12CellBank Australia, Children’s Medical Research Institute and The University of Sydney, Westmead, NSW, Australia. 13University Grenoble Alpes, TIMC UMR5525, La Tronche, France. 14California Institute of Technology, Pasadena, CA, USA. 15NeuroCure Cluster of Excellence, Charité–Universitätsmedizin Berlin, corporate member of the Freie Universität Berlin, HumboldtUniversität zu Berlin and Berlin Institute of Health, Berlin, Germany.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 7 cited sources checked · 0 supported · 0 flagged
- Findingown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
7.3% of COVID-19 preprints included bar graphs of continuous data.
“7.3% of preprints included bar graphs of continuous data.”
Quote in the paperSubstantial claim - Findingown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
34.4% of COVID-19 preprints acknowledged at least one study limitation.
“Approximately one third (34.4%) of COVID-19 preprints acknowledged at least one study limitation.”
Quote in the paperNot a substantial claim - Findingown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
13.6% of screened COVID-19 preprints shared open data.
“13.6% of preprints shared open data and 14.3% shared open code, making it easier for others to reuse data or reproduce results.”
Quote in the paperSubstantial claim - Findingown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
Over 8,000 preprint screening reports were distributed through hypothes.is and Twitter.
“By this means, reports on more than 8,000 COVID preprints have been shared using the web annotation tool hypothes.is (RRID:SCR_000430) and have been tweeted out via @SciScoreReports (https://hypothes.is/users/sciscore).”
Quote in the paperNot a substantial claim - Findingown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
14.3% of screened COVID-19 preprints shared open code.
“13.6% of preprints shared open data and 14.3% shared open code, making it easier for others to reuse data or reproduce results.”
Quote in the paperSubstantial claim
- Findingown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
Of the 552 preprints containing cell lines, 7% reported authentication or contamination-free maintenance of those lines.
“Among the 552 preprints that included cell lines, 7% described how the cell lines were authenticated (e.g., short tandem repeat profiling) or were kept free of contamination (e.g., mycoplasma detection tests).”
Quote in the paperSubstantial claim - Findingown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
Blinding statements appeared in 2.7% of COVID-19 preprints.
“Statements regarding sample size calculations (1.4%), blinding (2.7%) and randomization (11.4%) were uncommon, even among studies that contained a human ethics statement (present in 2.4%, 5.4% and 12.6%, respectively).”
Quote in the paperSubstantial claim - Findingown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
Randomization statements appeared in 11.4% of COVID-19 preprints.
“Statements regarding sample size calculations (1.4%), blinding (2.7%) and randomization (11.4%) were uncommon, even among studies that contained a human ethics statement (present in 2.4%, 5.4% and 12.6%, respectively).”
Quote in the paperSubstantial claim - Findingown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
7.6% of preprints with color maps used rainbow colormaps.
“Among papers with color maps, 7.6% used rainbow colormaps, which are not colorblind safe and also create visual artifacts for viewers with normal vision7.”
Quote in the paperNot a substantial claim - Findingown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
Sample size calculation statements appeared in 1.4% of COVID-19 preprints.
“Statements regarding sample size calculations (1.4%), blinding (2.7%) and randomization (11.4%) were uncommon, even among studies that contained a human ethics statement (present in 2.4%, 5.4% and 12.6%, respectively).”
Quote in the paperSubstantial claim - Findingown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
Among COVID-19 preprints with a human ethics statement, 12.6% included randomization statements.
“Statements regarding sample size calculations (1.4%), blinding (2.7%) and randomization (11.4%) were uncommon, even among studies that contained a human ethics statement (present in 2.4%, 5.4% and 12.6%, respectively).”
Quote in the paperSubstantial claim - Findingown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
An ethics approval statement for human or animal research appeared in 1,775 preprints, or 27%.
“1,775 preprints (27%) contained an ethics approval statement for human or animal research.”
Quote in the paperNot a substantial claim - Findingown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
Among COVID-19 preprints with a human ethics statement, 2.4% included sample size calculation statements.
“Statements regarding sample size calculations (1.4%), blinding (2.7%) and randomization (11.4%) were uncommon, even among studies that contained a human ethics statement (present in 2.4%, 5.4% and 12.6%, respectively).”
Quote in the paperSubstantial claim - Findingown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
Sex is addressed as a biological variable in 38% of COVID-19 preprints that include an ethics approval statement.
“Although there are known sex differences in COVID-1910, only 20% of all COVID-19 preprints, and 38% of preprints with an ethics approval statement, address sex as a biological variable.”
Quote in the paperSubstantial claim - Findingown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
Among COVID-19 preprints with a human ethics statement, 5.4% included blinding statements.
“Statements regarding sample size calculations (1.4%), blinding (2.7%) and randomization (11.4%) were uncommon, even among studies that contained a human ethics statement (present in 2.4%, 5.4% and 12.6%, respectively).”
Quote in the paperSubstantial claim - Findingown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
Nonhuman organisms, mainly mice, were used in 6.1% of COVID-19 preprints.
“6.1% of preprints used nonhuman organisms, mainly mice.”
Quote in the paperNot a substantial claim - Findingown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
Sex is addressed as a biological variable in 20% of all COVID-19 preprints.
“Although there are known sex differences in COVID-1910, only 20% of all COVID-19 preprints, and 38% of preprints with an ethics approval statement, address sex as a biological variable.”
Quote in the paperSubstantial claim - Premisecitation§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
Many preprints do not report whether experiments were blinded or randomized.
“Examples include failure to state whether experiments were blinded or randomized2, failure to report the sex of participants2 and misuse of bar graphs to display continuous data3.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [2] US National Institutes of Health. https://grants.nih.gov/grants/guide/notice-files/not-od-15-103.html (2015).
Not checked in the free run.
- Premisecitation§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
Many preprints misuse bar graphs to display continuous data.
“Examples include failure to state whether experiments were blinded or randomized2, failure to report the sex of participants2 and misuse of bar graphs to display continuous data3.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [3] Weissgerber, T. L. et al. Circulation 140, 1506–1518 (2019). source
Not checked in the free run.
- Premisecitation§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
Many preprints do not report the sex of participants.
“Examples include failure to state whether experiments were blinded or randomized2, failure to report the sex of participants2 and misuse of bar graphs to display continuous data3.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [2] US National Institutes of Health. https://grants.nih.gov/grants/guide/notice-files/not-od-15-103.html (2015).
Not checked in the free run.
- Interpretationown data§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
Large-scale automated screening of preprints, with rapid feedback to authors and readers, is feasible.
“Our work shows that it is feasible to conduct large-scale automated screening of preprints and provide rapid feedback to authors and readers.”
Quote in the paperSubstantial claim - Interpretationreasoning§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
The remaining roughly 75% of COVID-19 preprints likely consist of secondary or tertiary analyses, modeling studies, or cell line studies, which do not need ethics approval.
“This suggests that nearly three quarters of COVID-19 preprints are secondary or tertiary analyses, modeling studies or cell line studies that do not require approval.”
Quote in the paperNot a substantial claim - Interpretationreasoning§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?
Because many COVID-19 preprints are modeling studies, criteria such as sample size, blinding, and randomization are not always applicable.
“Many COVID-19 preprints are modeling studies, however, and hence these criteria are not always relevant.”
Quote in the paperSubstantial claim
- Classified as probably not a substantial claim. 4
- May combine two claims. 5
Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.
Premisecitation§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?Most COVID-19 preprints are posted on medRxiv, with others on bioRxiv or other servers.
“Most of these appear on medRxiv; others appear on bioRxiv or other servers1.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [1] Callaway, E. Nature 582, 167–168 (2020). source
Not checked in the free run.
Classified as probably not a substantial claim. May combine two claims.
Interpretationnone given§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?Automated tools are imperfect: they err and cannot always judge whether a detected problem applies to a particular paper.
“Automated tools are not perfect—they make mistakes, and they cannot always determine whether a problem is relevant to a given paper.”
Quote in the paperSubstantial claimMay combine two claims.
Premisecitation§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?Rainbow colormaps are unsafe for colorblind readers and produce visual artifacts even for those with normal vision.
“Among papers with color maps, 7.6% used rainbow colormaps, which are not colorblind safe and also create visual artifacts for viewers with normal vision7.”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [7] Saladi, S. eLife https://elifesciences.org/labs/c2292989/jetfighter-towards-figure-accuracy-and-accessibility (2020).
Not checked in the free run.
May combine two claims.
Interpretationreasoning§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?Automated tools can rapidly identify possible issues and may serve as a supplement to peer review.
“Despite these limitations, automated tools can quickly flag potential problems and may complement peer reviews.”
Quote in the paperSubstantial claimMay combine two claims.
Premisecitation§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?Bar graphs of continuous data are problematic because different datasets can produce identical bar graphs, and the underlying data may support conclusions different from those suggested by summary statistics alone.
“This is problematic because many different datasets can lead to the same bar graph, and the actual data may suggest different conclusions from those implied by the summary statistics alone3.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [3] Weissgerber, T. L. et al. Circulation 140, 1506–1518 (2019). source
Not checked in the free run.
Classified as probably not a substantial claim.
Premisecitation§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?There are known sex differences in COVID-19.
“Although there are known sex differences in COVID-1910, only 20% of all COVID-19 preprints, and 38% of preprints with an ethics approval statement, address sex as a biological variable.”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [10] Wenham, C., Smith, J. & Morgan, R., the Gender and COVID-19 Working Group. Lancet 395, 846–848 (2020). source
Not checked in the free run.
Classified as probably not a substantial claim.
Interpretationnone given§ Automated screening of COVID-19 preprints: can we help authors to improve transparency and reproducibility?Certain problems are beyond the detection capacity of automated tools.
“Moreover, some problems are too complex for automated tools to detect.”
Quote in the paperSubstantial claim
The paper relies on this without support.
At the scale of COVID-19 preprints, routine quality assessment and flagging of common reporting problems are not feasible.
“Preprints are vetted before posting to confirm that they describe scientific studies and to prevent posting on topics that could damage public health; however, routine assessment of manuscript quality or flagging of common reporting problems is not feasible at this scale.”
The paper relies on this without support.
The volume of COVID-19 preprints published since February exceeds scientists' capacity to keep up.
“Unfortunately, it has been impossible for scientists to keep pace with the thousands of COVID-19 preprints published since February.”
The paper relies on this without support.
The absence of traditional peer review in preprints raises concerns about preprint quality.
“Although publishing results in preprints allows them to be posted rapidly, the absence of traditional peer review has raised concerns about preprint quality.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Automated screening based on natural language processing and machine learning could reduce the burden of manual checking, streamline the peer review process, and contribute to better reporting quality.
“Automated screening, based on natural language processing (NLP) and machine learning approaches, could reduce the burden of manual checking, streamline the peer review process, and contribute to better reporting quality18–20.”
From Text classification models for assessing the completeness of randomized controlled trial publications based on CONSORT reporting guidelines (2024) · cites this paper as [19]
In the main text (page not supplied), the paper says automated screening can rapidly flag reporting problems at a scale impractical for manual assessment, may complement peer review, and may help authors improve manuscripts. It does not identify NLP or machine learning as the approaches used or say that screening streamlines peer review; it also says further research is needed to determine whether reporting improves. (LLM, conf=0.83)
Experimental — accuracy not yet measuredPartly, automated screening and compliance checking are already done, but future LLMs could enhance these applications.
“Partly, this is already done [3, 4] but future LLMs could enhance these applications.”
The ScreenIT pipeline has been used to post public reports on more than 23,000 bioRxiv and medRxiv COVID-19 preprints.
“The ScreenIT pipeline, which includes a growing set of automated tools, has been used to post public reports on more than 23,000 bioRxiv and medRxiv COVID-19 preprints [17].”
From Is the future of peer review automated? (2022) · cites this paper as [17]
Existing tools cannot determine whether the data support the authors' conclusions.
“They cannot determine whether the methods used are suitable to answer the research question, or whether the data support the authors’ conclusions.”
From Is the future of peer review automated? (2022) · cites this paper as [17]
Existing tools are not capable of understanding or interpreting the research in the context of the scientific literature.
“Existing tools are not capable of understanding or interpreting the research in the context of the scientific literature.”
From Is the future of peer review automated? (2022) · cites this paper as [17]
Tools make mistakes.
“Tools make mistakes.”
From Is the future of peer review automated? (2022) · cites this paper as [17]
Automated tools have important limitations.
“Despite these advantages, automated tools have important limitations [17].”
From Is the future of peer review automated? (2022) · cites this paper as [17]
Existing tools cannot determine whether the methods used are suitable to answer the research question.
“They cannot determine whether the methods used are suitable to answer the research question, or whether the data support the authors’ conclusions.”
From Is the future of peer review automated? (2022) · cites this paper as [17]
Tools that assess reporting quality may not capture information that reflects the methodological quality of the experiment.
“Furthermore, tools that assess reporting quality may not capture information that reflects the methodological quality of the experiment.”
From Is the future of peer review automated? (2022) · cites this paper as [17]
Tools cannot always determine whether an item is relevant to a given paper, especially when reporting is poor.
“They cannot always determine whether an item is relevant to a given paper, especially when reporting is poor.”
From Is the future of peer review automated? (2022) · cites this paper as [17]
Large-scale automated screening of bioRxiv and medRxiv preprints for rigor and transparency criteria is feasible and could raise awareness about factors affecting transparency and reproducibility.
“Largescale automated screening of bioRxiv and medRxiv preprints for rigor and transparency criteria is feasible and could raise awareness about factors affecting transparency and reproducibility (87).”
From The devil is in the details: Reporting and transparent research practices in sports medicine and orthopedic clinical trials (2021) · cites this paper as [87]
Cites
COVID-19: the gendered impacts of the outbreak
Wenham, C., Smith, J., & Morgan, R. (2020) · The Lancet · doi:10.1016/s0140-6736(20)30526-2 · Cited by 1,701 · see citing works
Will the pandemic permanently alter scientific publishing?
Callaway, E. (2020) · Nature · doi:10.1038/d41586-020-01520-4 · Cited by 111 · see citing works
Reveal, Don’t Conceal: Transforming Data Visualization to Improve Transparency
Weissgerber, T. L., Winham, S. J., Heinzen, E. P., Milin-Lazovic, J. S., Garcia-Valencia, O., Bukumiric, Z., Savic, M. D., Garovic, V. D., & Milic, N. M. (2019) · Circulation · doi:10.1161/circulationaha.118.037777 · Cited by 101 · see citing works
Self-citationThe Rigor and Transparency Index Quality Metric for Assessing Biological and Medical Science Methods
Menke, J., Roelandse, M., Ozyurt, B., Martone, M., & Bandrowski, A. (2020) · iScience · doi:10.1016/j.isci.2020.101698 · Cited by 90 · see citing works
Self-citationLabbé, C., Grima, N., Gautier, T., Favier, B., & Byrne, J. A. (2019) · PLOS ONE · doi:10.1371/journal.pone.0213266 · Cited by 55 · see citing works
Self-citationODDPub – a Text-Mining Algorithm to Detect Data Sharing in Biomedical Publications
Riedel, N., Kip, M., & Bobrov, E. (2020) · Data Science Journal · doi:10.5334/dsj-2020-042 · Cited by 44 · see citing works
Self-citationRecombinant DNA research: proposed actions under guidelines.
Health, U. N. I. O. (1984) · PubMed · Cited by 35 · see citing works
Automatic recognition of self-acknowledged limitations in clinical research literature
Halil Kilicoglu, B., Rosemblat, G., Malički, M., & ter Riet, G. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy038 · Cited by 15 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- Saladi, S. eLife https://elifesciences.org/labs/c2292989/jetfighter-towards-figure-accuracy-and-accessibility (2020).
Cited by
Sorted by citations
Hosseini, M., & Horbach, S. P. J. M. (2023) · Research Integrity and Peer Review · doi:10.1186/s41073-023-00133-5 · Cited by 243 · see citing works
“, implication of automated screening[3] or automated systems to check compliance with journal policies and marks of quality[4]), the introduction of novel LLMs has the potential to move automation much further.”
BackgroundCabanac, G., Labbé, C., & Magazinov, A. (2021) · arXiv · doi:10.48550/arxiv.2107.06751 · Cited by 224 · see citing works
“Screening pipelines (e.g., Weissgerber et al., 2021) may include such detection software and run it ahead of peer review.”
Self-citationIs the future of peer review automated?
Schulz, R., Barnett, A., Bernard, R., Brown, N. J. L., Byrne, J. A., Eckmann, P., Gazda, M. A., Kilicoglu, H., Prager, E. M., Salholz-Hillel, M., et al. (2022) · BMC Research Notes · doi:10.1186/s13104-022-06080-6 · Cited by 80 · see citing works
“The ScreenIT pipeline, which includes a growing set of automated tools, has been used to post public reports on more than 23,000 bioRxiv and medRxiv COVID-19 preprints [17].”
BackgroundMethodSelf-citationHamilton, D. G., Hong, K., Fraser, H., Rowhani-Farid, A., Fidler, F., & Page, M. J. (2023) · BMJ · doi:10.1136/bmj-2023-075767 · Cited by 86 · see citing works
Schönbrodt, F. D., Gärtner, A., Frank, M., Gollwitzer, M., Ihle, M., Mischkowski, D., Phan, L. V., Schmitt, M., Scheel, A. M., Schubert, A., et al. (2022) · doi:10.31234/osf.io/rgh5b · Cited by 40 · see citing works
Puebla, I., Polka, J., & Rieger, O. Y. (2022) · doi:10.3998/mpub.12412508 · Cited by 39 · see citing works
Kroon, C., Breuer, L., Jones, L., An, J., Akan, A., Mohamed Ali, E. A., Busch, F., Fislage, M., Ghosh, B., Hellrigel-Holderbaum, M., et al. (2022) · PLOS Biology · doi:10.1371/journal.pbio.3001783 · Cited by 31 · see citing works
Self-citationMansmann, U., Locher, C., Praßer, F., Weissgerber, T. L., Sax, U., Posch, M., Decullier, É., Cristea, I. A., Debray, T. P. A., Held, L., et al. (2023) · Nature Medicine · doi:10.1038/s41591-022-02080-y · Cited by 31 · see citing works
Self-citationPreprints: Their Evolving Role in Science CommunicationPreprint
Puebla, I., Polka, J., & Rieger, O. Y. (2021) · doi:10.31222/osf.io/ezfsk · Cited by 31 · see citing works
International multi‐stakeholder consensus statement on clinical trial integrity
Khan, K. S. (2023) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.17451 · Cited by 29 · see citing works
Technological advances in preclinical meta-research
Bannach‐Brown, A., Hair, K., Bahor, Z., Soliman, N., Macleod, M. R., & Liao, J. (2021) · BMJ Open Science · doi:10.1136/bmjos-2020-100131 · Cited by 25 · see citing works
Self-citationMenke, J., Eckmann, P., Ozyurt, I. B., Roelandse, M., Anderson, N., Grethe, J., Gamst, A., & Bandrowski, A. (2022) · Journal of Medical Internet Research · doi:10.2196/37324 · Cited by 25 · see citing works
Self-citationTransparency and reporting characteristics of COVID-19 randomized controlled trials
Kapp, P., Esmail, L., Ghosn, L., Ravaud, P., & Boutron, I. (2022) · BMC Medicine · doi:10.1186/s12916-022-02567-y · Cited by 25 · see citing works
Schulz, R., Langen, G., Prill, R., Cassel, M., & Weissgerber, T. L. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-059347 · Cited by 23 · see citing works
Self-citationThe use and acceptability of preprints in health and social care settings: A scoping review
Blatch-Jones, A., Recio‐Saucedo, A., & Giddins, B. (2023) · PLoS ONE · doi:10.1371/journal.pone.0291627 · Cited by 22 · see citing works
Mouse Models of Osteoarthritis: A Summary of Models and Outcomes Assessment
Drevet, S., Favier, B., Brun, E., Gavazzi, G., & Lardy, B. (2022) · Comparative Medicine · doi:10.30802/aalas-cm-21-000043 · Cited by 23 · see citing works
Self-citationJiang, L., Lan, M., Menke, J. D., Vorland, C. J., & Kilicoglu, H. (2024) · Scientific Reports · doi:10.1038/s41598-024-72130-7 · Cited by 18 · see citing works
Replacing bar graphs of continuous data with more informative graphics: are we making progress?
Riedel, N., Schulz, R., Kazezian, V., & Weissgerber, T. L. (2022) · Clinical Science · doi:10.1042/cs20220287 · Cited by 18 · see citing works
Self-citationReproducibility of COVID-19 pre-prints
Collins, A., & Alexander, R. (2022) · Scientometrics · doi:10.1007/s11192-022-04418-2 · Cited by 12 · see citing works
Institutional dashboards on clinical trial transparency for University Medical Centers: A case study
Franzen, D. L., Carlisle, B. G., Salholz‐Hillel, M., Riedel, N., & Strech, D. (2023) · PLoS Medicine · doi:10.1371/journal.pmed.1004175 · Cited by 12 · see citing works
Self-citationLimiting open science? Three approaches to bottom-up governance of dual-use research of concern
Yoshizawa, G., Shinomiya, N., Kawamoto, S., Kawahara, N., Kiga, D., Hanaki, K., & Minari, J. (2023) · Pathogens and Global Health · doi:10.1080/20477724.2023.2265626 · Cited by 11 · see citing works
Fernández, L. M., Hardwicke, T. E., & Vadillo, M. A. (2019) · doi:10.31234/osf.io/cszpy · Cited by 12 · see citing works
Retrieval augmented scientific claim verification
Liu, H., Soroush, A., Nestor, J. G., Park, E., Idnay, B., Fang, Y., Pan, J. Y., Liao, S., Bernard, M., Peng, Y., et al. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae021 · Cited by 9 · see citing works
Drude, N., Martinez‐Gamboa, L., Haven, T. L., Holman, C., Holst, M., Kniffert, S., McCann, S., Rackoll, T., Schulz, R., & Weschke, S. (2022) · BMC Research Notes · doi:10.1186/s13104-022-06108-x · Cited by 9 · see citing works
Self-citationLan, M., Cheng, M., Hoang, L., ter Riet, G., & Halil Kilicoglu, B. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104628 · Cited by 8 · see citing works
The Normalization of Preprints♦
Drury, L. O. (2022) · SRELS Journal of Information Management · doi:10.17821/srels/2022/v59i2/169462 · Cited by 8 · see citing works
Barnett, A. (2022) · F1000Research · doi:10.12688/f1000research.123002.1 · Cited by 7 · see citing works
Bannach‐Brown, A., Rackoll, T., Macleod, M. R., & McCann, S. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02524-2 · Cited by 6 · see citing works
Self-citationvan Aert, R. C. M., Nuijten, M. B., Olsson-Collentine, A., Stoevenbelt, A. H., van den Akker, O. R., Klein, R. A., & Wicherts, J. M. (2023) · Royal Society Open Science · doi:10.1098/rsos.202326 · Cited by 6 · see citing works
An attempt to explain the partial 'silent' withdrawal or retraction of a SAGE Advance preprint
da Silva, J. A. T. (2023) · Publishing Research · doi:10.48130/pr-2023-0004 · Cited by 6 · see citing works
The normalization of preprints
Drury, L. O. (2022) · doi:10.24948/2022.02 · Cited by 6 · see citing works
Jiang, L., Vorland, C. J., Ying, X., Brown, A. W., Menke, J. D., Hong, G., Lan, M., Mayo‐Wilson, E., & Halil Kilicoglu, B. (2025) · Scientific Data · doi:10.1038/s41597-025-04629-1 · Cited by 6 · see citing works
Lin, T.-L., Chen, W.-C., Hsiao, T.-F., Liu, H.-I., Yeh, Y.-H., Chan, Y.-K., Lien, W.-S., Kuo, P.-Y., Yu, P. S., & Shuai, H.-H. (2025) · Findings of the Association for Computational Linguistics: EMNLP 2025 · doi:10.18653/v1/2025.findings-emnlp.259 · Cited by 6
Drude, N., Baselly, C., Gazda, M. A., May, J., Tienken, L., Abbasi, P., Weissgerber, T. L., & Burgess, S. (2026) · Research Integrity and Peer Review · doi:10.1186/s41073-026-00188-0 · Cited by 5 · see citing works
Self-citationHamilton, D. G., Hong, K., Fraser, H. S., Rowhani-Farid, A., Fidler, F. M., & Page, M. J. (2023) · medRxiv · doi:10.1101/2023.03.22.23287607 · Cited by 5 · see citing works
Schulz, R., Langen, G., Prill, R., Cassel, M., & Weissgerber, T. (2021) · doi:10.1101/2021.07.20.21260565 · Cited by 23 · see citing works
Self-citationFranzen, D., Salholz-Hillel, M., Müller-Ohlraun, S., & Strech, D. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02482-9 · Cited by 4 · see citing works
Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials
Barnett, A. G. (2023) · F1000Research · doi:10.12688/f1000research.123002.2 · Cited by 3 · see citing works
The randomized clinical trial trustworthiness crisis
Butt, F., Fawzy, M. M., Wattar, B. H. A., Bueno‐Cavanillas, A., Khan, K. S., & Khalaf, Y. (2024) · Middle East Fertility Society Journal · doi:10.1186/s43043-023-00161-7 · Cited by 2 · see citing works
Turoman, N., Heyard, R., Schwab, S., Furrer, E., Vergauwe, E., & Held, L. (2023) · F1000Research · doi:10.12688/f1000research.129814.1 · Cited by 2 · see citing works
Epp, S., Hoffmann, M., Lell, N., Möhr, M., & Scherp, A. (2021) · arXiv (Cornell University) · Cited by 2 · see citing works
Automated detection of over- and under-dispersion in baseline tables in randomised controlled trialsPreprint
Barnett, A. G. (2022) · Research Square · doi:10.21203/rs.3.rs-1496802/v1 · Cited by 2 · see citing works
International multi-stakeholder consensus statement on clinical trial integrity
Khan, K. S. (2024) · Middle East Fertility Society Journal · doi:10.1186/s43043-024-00171-z · Cited by 1 · see citing works
Incentivize Peer Review Without Rewarding: Using OSS-Like Citation Pull Request
Miura, C., & Ito, K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78090-5_10 · Cited by 1 · see citing works
Eckmann, P., Barnett, A. G., Bannach‐Brown, A., Atria, E. B., Cabanac, G., Franzen, L. D. O., Gazda, M. A., Hair, K., Howison, J., Halil Kilicoglu, B., et al. (2026) · PLoS ONE · doi:10.1371/journal.pone.0342225 · Cited by 4 · see citing works
Self-citationSupporting reanalysis and reuse of clinical trial data: a case study
Burgwinkel, C., Chiam, H. C., Tai, K., Wang, J., Ali, M. H., Fallah, S. S., Matbouriahi, M., Obinwanne, T., Papapostolou, G., Riedha, M., et al. (2026) · Trials · doi:10.1186/s13063-026-09646-y · Cited by 1 · see citing works
Thornton, G. M., & Kroeker, E. (2023) · Canadian Journal of Information and Library Science · doi:10.5206/cjils-rcsib.v45i2.14730 · Cited by 1 · see citing works
Turoman, N., Heyard, R., Schwab, S., Furrer, E., Vergauwe, E., & Held, L. (2022) · doi:10.31222/osf.io/nb928
Preprint_vanAert_etal_Covid19Preprint
van Aert, R. C. M., Nuijten, M. B., Olsson-Collentine, A., Stoevenbelt, A. H., van den Akker, O. R., Klein, R. A., & Wicherts, J. M. (2023) · doi:10.31234/osf.io/asbfd
Turoman, N., Heyard, R., Schwab, S., Furrer, E., Vergauwe, E., & Held, L. (2024) · F1000Research · doi:10.12688/f1000research.129814.2
Turoman, N., Heyard, R., Schwab, S., Furrer, E., Vergauwe, E., & Held, L. (2024) · F1000Research · doi:10.12688/f1000research.129814.3
Documento de consenso internacional sobre la integridad de los ensayos clínicos
Núñez‐Núñez, M. (2024) · Medicina de Familia SEMERGEN · doi:10.1016/j.semerg.2024.102217
Partially or fully (“silently”) withdrawn or retracted Center for Open Science preprints
da Silva, J. A. T. (2024) · Central Asian Journal of Medical Hypotheses and Ethics · doi:10.47316/cajmhe.2024.5.4.05
Jiang, L., Vorland, C. J., Ying, X., Brown, A. W., Menke, J. D., Hong, G., Lan, M., Mayo‐Wilson, E., & Halil Kilicoglu, B. (2025) · medRxiv · doi:10.1101/2025.01.14.25320543
Hosseini, M., & Horbach, S. P. J. M. (2024) · Figshare · doi:10.6084/m9.figshare.26587827
Additional file 1 of The randomized clinical trial trustworthiness crisis
Butt, F. A., Fawzy, M. M., Wattar, B. H. A., Bueno‐Cavanillas, A., Khan, K. S., & Khalaf, Y. (2024) · Figshare · doi:10.6084/m9.figshare.26658337
Kim, D. H., Jeon, K., & You, S. C. (2026) · Journal of Evidence-Based Practice · doi:10.63528/jebp.2026.00007
- Authors
- Weissgerber, T., Riedel, N., Kilicoglu, H., Labbé, C., Eckmann, P., ter Riet, G., Byrne, J., Cabanac, G., Capes-Davis, A., Favier, B., Saladi, S., Grabitz, P., Bannach-Brown, A., Schulz, R., McCann, S., Bernard, R., & Bandrowski, A.
- Affiliations
- Berlin Institute of Health at Charité - Universitätsmedizin Berlin (DE)
- Institut polytechnique de Grenoble (FR)
- University of California San Diego (US)
- The University of Sydney (AU)
- Université Toulouse III - Paul Sabatier (FR)
- Translational Innovation in Medicine and Complexity (FR)
- California Institute of Technology (US)
- Humboldt-Universität zu Berlin (DE)
- Corresponding author
- Weissgerber, Tracey
- ORCID
- Venue
- Nature Medicine
- Published
- 2021-01
- Volume
- 27
- Issue
- 1
- Pages
- 6-7
- ISSN
- 1078-8956 (ISSN-L)
- 1546-170X
- PMC
- PMC8177099
- OpenAlex
- W3120711451
- Semantic Scholar
- 193b5c0edffbce59e4250a941050310ab09cc042
- Cited by
- 57
- References
- 8
- Citations by year
- 2026: 4 · 2025: 4 · 2024: 15 · 2023: 14 · 2022: 16 · 2021: 3 · 2019: 1
- FWCI
- 10.25
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Nature Medicine (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- Pure Amsterdam UMC (submittedVersion, cc-by)
- CaltechAUTHORS (California Institute of Technology) (submittedVersion)
- Data Archiving and Networked Services (DANS) (submittedVersion, other-oa)
- Data Archiving and Networked Services (DANS) (submittedVersion)
- HAL (submittedVersion, other-oa)
- The Sydney eScholarship Repository (The University of Sydney) (submittedVersion)
- Funders
- Bill and Melinda Gates Foundation
- International AIDS Vaccine Initiative
- Children's Medical Research
- California Institute of Technology
- NSW Health Pathology
- Georgia Clinical and Translational Science Alliance
- Amsterdam University Medical Centers
- Agence Nationale de la Recherche (ANR-11-LABX-0066)
- University of Sydney
- Universiteit van Amsterdam
- Université de Toulouse
- Centre National de la Recherche Scientifique
- Humboldt-Universität zu Berlin
- Freie Universität Berlin
- NSW Ministry of Health
- NeuroCure Exzellenzcluster
- Berlin Institute of Health
- National Institutes of Health (5u24da039832-02, 1r43od024432-01, UL1TR002550, DA039832, DK097771)
- University of Illinois at Urbana-Champaign
- University of California, San Diego
- Medical Research Council
- National Health and Medical Research Council (1184263)
- Faculty of Medicine and Health, University of Sydney
- National Institute of Mental Health (R44MH119094)
- National Institute on Drug Abuse (U24-DA039832)
- National Institute of Allergy and Infectious Diseases
- National Institute of Diabetes and Digestive and Kidney Diseases (U24DK097771)
- U.S. National Library of Medicine
- National Center for Advancing Translational Sciences
- Topics
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Physical Sciences › Computer Science › Information Systems › Research Data Management Practices
- Health Sciences › Medicine › Infectious Diseases › SARS-CoV-2 and COVID-19 Research
- Keywords
- automated screening, research transparency, reproducibility, author feedback
- MeSH terms
- Preprints as Topic, COVID-19, Automation, Bioethics, Humans, Publishing, Databases, Bibliographic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-04.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-04.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
