A checklist is associated with increased quality of reporting preclinical biomedical research: A systematic review
Open accessThe DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
Abstract
Irreproducibility of preclinical biomedical research has gained recent attention. It is suggested that requiring authors to complete a checklist at the time of manuscript submission would improve the quality and transparency of scientific reporting, and ultimately enhance reproducibility. Whether a checklist enhances quality and transparency in reporting preclinical animal studies, however, has not been empirically studied. Here we searched two highly cited life science journals, one that requires a checklist at submission (Nature) and one that does not (Cell), to identify in vivo animal studies. After screening 943 articles, a total of 80 articles were identified in 2013 (pre-checklist) and 2015 (post-checklist), and included for the detailed evaluation of reporting methodological and analytical information. We compared the quality of reporting preclinical animal studies between the two journals, accounting for differences between journals and changes over time in reporting. We find that reporting of randomization, blinding, and sample-size estimation significantly improved when comparing Nature to Cell from 2013 to 2015, likely due to implementation of a checklist. Specifically, improvement in reporting of the three methodological information was at least three times greater when a mandatory checklist was implemented than when it was not. Reporting the sex of animals and the number of independent experiments performed also improved from 2013 to 2015, likely from factors not related to a checklist. Our study demonstrates that completing a checklist at manuscript submission is associated with improved reporting of key methodological information in preclinical animal studies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Reporting guidelines and checklists provided to authors during the review or manuscript preparation processes have been reported to cause modest improvements limited to a few items in in vivo animal studies [19, 20, 28, 40].”
From Comparing quality of reporting between preprints and peer-reviewed articles in the biomedical literature (2020) · cites this paper as [20]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In contrast, other studies using shorter checklists (primarily focused on experimental design) with more editorial follow-up have shown a marked improvement in the nature and detail of the information included in publications [34–36].”
From The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research (2020) · cites this paper as [35]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Quality of reporting is used to evaluate study quality in meta-analyses [36], as well as the effect of interventions focused on improving transparency [19, 20, 40].”
From Comparing quality of reporting between preprints and peer-reviewed articles in the biomedical literature (2020) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Improving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · PLoS Biology · doi:10.1371/journal.pbio.1000412 · Cited by 7,300
Improving bioscience research reporting: The ARRIVE guidelines for reporting animal research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · Journal of Pharmacology and Pharmacotherapeutics · doi:10.4103/0976-500x.72351 · Cited by 6,277
No adjustments are needed for multiple comparisons.
Rothman, K. J. (1990) · Epidemiology · doi:10.1097/00001648-199001000-00010 · Cited by 5,965
A surgical safety checklist to reduce morbidity and mortality in a global population
Haynes, A. B., Weiser, T. G., Berry, W. R., Lipsitz, S. R., Breizat, A.-H. S., Dellinger, E. P., Herbosa, T., Joseph, S., Kibatala, P. L., Lapitan, M. C. M., et al. (2009) · New England Journal of Medicine · doi:10.1056/nejmsa0810119 · Cited by 5,688
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293
A call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196
Guidelines for the Design and Statistical Analysis of Experiments Using Laboratory Animals
Festing, M. F. W., & Altman, D. G. (2002) · ILAR Journal · doi:10.1093/ilar.43.4.244 · Cited by 1,110
Plint, A. C., Moher, D., Morrison, A., Schulz, K., Altman, D. G., Hill, C., & Gaboury, I. (2006) · Medical Journal of Australia · doi:10.5694/j.1326-5377.2006.tb00557.x · Cited by 872
Turner, L., Shamseer, L., Altman, D. G., Weeks, L., Peters, J., Kober, T., Dias, S., Schulz, K. F., Plint, A. C., & Moher, D. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000030.pub2 · Cited by 740
Translation of Research Evidence From Animals to Humans
Hackam, D. G., & Redelmeier, D. A. (2006) · JAMA · doi:10.1001/jama.296.14.1731 · Cited by 733
Bergs, J., Hellings, J., Cleemput, I., Zurel, Ö., De Troyer, V, Van Hiel, M., Demeere, J.-L., Claeys, D., & Vandijck, D. (2014) · British Journal of Surgery · doi:10.1002/bjs.9381 · Cited by 390
The Use and Interpretation of Quasi-Experimental Studies in Infectious Diseases
Eliopoulos, G. M., Harris, A. D., Bradham, D. D., Baumgarten, M., Zuckerman, I. H., Fink, J. C., & Perencevich, E. N. (2004) · Clinical Infectious Diseases · doi:10.1086/420936 · Cited by 330
Translation of highly promising basic science research into clinical applications
Contopoulos-Ioannidis, D. G., Ntzani, E. E., & Ioannidis, J. P. A. (2003) · The American Journal of Medicine · doi:10.1016/s0002-9343(03)00013-5 · Cited by 323
Baker, D., Lidster, K., Sottomayor, A., & Amor, S. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001756 · Cited by 305
Perspectives in quality: designing the WHO Surgical Safety Checklist
Weiser, T. G., Haynes, A. B., Lashoher, A., Dziekan, G., Boorman, D. J., Berry, W. R., & Gawande, A. A. (2010) · International Journal for Quality in Health Care · doi:10.1093/intqhc/mzq039 · Cited by 293
McNutt, M. (2014) · Science · doi:10.1126/science.1250475 · Cited by 291
Announcement: Reducing our irreproducibility
(2013) · Nature · doi:10.1038/496398a · Cited by 243
Reproducible research and Biostatistics
Peng, R. D. (2009) · Biostatistics · doi:10.1093/biostatistics/kxp014 · Cited by 237
Reporting in randomized clinical trials improved after adoption of the CONSORT statement
Kane, R. L., Wang, J., & Garrard, J. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2006.06.016 · Cited by 227
Devereaux, P. J., Manns, B. J., Ghali, W. A., Quan, H., & Guyatt, G. H. (2002) · Controlled Clinical Trials · doi:10.1016/s0197-2456(02)00214-3 · Cited by 166
Mobley, A., Linder, S. K., Braeuer, R., Ellis, L. M., & Zwelling, L. (2013) · PLoS ONE · doi:10.1371/journal.pone.0063221 · Cited by 166
Ramirez, F. D., Motazedian, P., Jung, R. G., Di Santo, P., MacDonald, Z. D., Moreland, R., Simard, T., Clancy, A. A., Russo, J. J., Welch, V. A., et al. (2017) · Circulation Research · doi:10.1161/circresaha.117.310628 · Cited by 138
Bias in the reporting of sex and age in biomedical research on mouse models
Flórez-Vargas, O., Brass, A., Karystianis, G., Bramhall, M., Stevens, R., Cruickshank, S., & Nenadic, G. (2016) · eLife · doi:10.7554/elife.13615 · Cited by 124
Simple intervention to improve detection of child abuse in emergency departments
Benger, J. R., & Pearce, A. V. (2002) · BMJ · doi:10.1136/bmj.324.7340.780 · Cited by 117
Casadevall, A., & Fang, F. C. (2010) · Infection and Immunity · doi:10.1128/iai.00908-10 · Cited by 111
Cousens, S., Hargreaves, J., Bonell, C., Armstrong, B., Thomas, J., Kirkwood, B. R., & Hayes, R. (2011) · Journal of Epidemiology and Community Health · doi:10.1136/jech.2008.082610 · Cited by 86
BÖHMER, A. B., WAPPLER, F., TINSCHMANN, T., KINDERMANN, P., RIXEN, D., BELLENDIR, M., SCHWANKE, U., BOUILLON, B., & GERBERSHAGEN, M. U. (2012) · Acta Anaesthesiologica Scandinavica · doi:10.1111/j.1399-6576.2011.02590.x · Cited by 85
The False Discovery Rate: A Key Concept in Large-Scale Genetic Studies
Chen, J. J., Robeson, P. K., & Schell, M. J. (2010) · Cancer Control · doi:10.1177/107327481001700108 · Cited by 55
Cramond, F., Irvine, C., Liao, J., Howells, D., Sena, E., Currie, G., & Macleod, M. (2016) · Scientometrics · doi:10.1007/s11192-016-1964-8 · Cited by 27
Rigid guidelines may restrict research
Morris, R. (2012) · Nature · doi:10.1038/491192c · Cited by 6
Research works printed but not linked: 1
- Drummond C, editor Replicability is not Reproducibility: Nor is it Good Science. Proc of the Evaluation Methods for Machine Learning Workshop at the 26th ICML; 2009; Montreal, Quebec, Canada.
Cited by
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000410 · Cited by 5,939
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · Experimental Physiology · doi:10.1113/ep088870 · Cited by 3,497
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research*
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1177/0271678x20943823 · Cited by 1,443
Leakage and the reproducibility crisis in machine-learning-based science
Kapoor, S., & Narayanan, A. (2023) · Patterns · doi:10.1016/j.patter.2023.100804 · Cited by 1,387
“The use of checklists has also been impactful in improving reporting practices in the few fields that have adopted them.(70) Although checklists and model cards provide concrete best practices for reporting standards, current efforts do not address pitfalls arising because of leakage.”
BackgroundThe ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · British Journal of Pharmacology · doi:10.1111/bph.15193 · Cited by 1,081
Metadata
- Authors
- Han, S., Olonisakin, T. F., Pribis, J. P., Zupetic, J., Yoon, J. H., Holleran, K. M., Jeong, K., Shaikh, N., Rubio, D. M., & Lee, J. S.
- Affiliations
- University of Pittsburgh (US)
- Baylor College of Medicine (US)
- Corresponding author
- Han, SeungHye
- ORCID
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2017-09-13
- Volume
- 12
- Issue
- 9
- Pages
- e0183591
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC5597130
- OpenAlex
- W2755142264
- Cited by
- 131
- References
- 33
- Citations by year
- 2026: 3 · 2025: 15 · 2024: 7 · 2023: 8 · 2022: 11 · 2021: 24 · 2020: 32 · 2019: 18 · 2018: 11 · 2017: 2
- FWCI
- 3.92
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- National Institutes of Health (UL1TR001857, HL086884, AI119042)
- American Heart Association (15POST25700096)
- breathe Pennsylvania (Lung Health Research Grant)
- Howard Hughes Medical Institute (Research Fellow Award)
- Topics
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Keywords
- reporting checklist, preclinical animal studies, research reproducibility, reporting quality, randomization, blinding, sample size determination, nature, cells
- MeSH terms
- Data Accuracy, Animals, Drug Evaluation, Preclinical, Humans, Publications, Research Design, Reproducibility of Results, Models, Animal, Biomedical Research, Checklist
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.