The devil is in the details: incomplete reporting in preclinical animal research
Open accessAbstract
Incomplete reporting of study methods and results has become a focal point for failures in the reproducibility and translation of findings from preclinical research. Here we demonstrate that incomplete reporting of preclinical research is not limited to a few elements of research design, but rather is a broader problem that extends to the reporting of the methods and results. We evaluated 47 preclinical research studies from a systematic review of acute lung injury that use mesenchymal stem cells (MSCs) as a treatment. We operationalized the ARRIVE (Animal Research: Reporting of In Vivo Experiments) reporting guidelines for pre-clinical studies into 109 discrete reporting sub-items and extracted 5,123 data elements. Overall, studies reported less than half (47%) of all sub-items (median 51 items; range 37-64). Across all studies, the Methods Section reported less than half (45%) and the Results Section reported less than a third (29%). There was no association between journal impact factor and completeness of reporting, which suggests that incomplete reporting of preclinical research occurs across all journals regardless of their perceived prestige. Incomplete reporting of methods and results will impede attempts to replicate research findings and maximize the value of preclinical studies.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Randomisation is reported in only 30%-40% of publications.
“randomisation (reported in only 30%–40% of publications)”
From The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research (2020) · cites this paper as [31]
The Results report randomization as a composite item in 23% of the 47 studies, not 30%–40%. They separately report that 47% used the term “randomization”; neither measure falls within the claimed range. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredStudies measuring the impact of ARRIVE on the quality of reporting have produced mixed results.
“Studies measuring the impact of ARRIVE on the quality of reporting have produced mixed results [6–11]”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [8]
The Results (page unavailable) report a statistically significant but small increase in reporting after ARRIVE was published; fewer than half of the assessed items were still reported. The authors also say extraneous factors may explain the increase. This study contributes a qualified finding, but it does not establish that studies of ARRIVE’s impact collectively produced mixed results. (LLM, conf=0.82)
Experimental — accuracy not yet measuredTo ensure scientific rigour, inclusion and exclusion criteria should be defined before the experiment starts and data are collected.
“To ensure scientific rigour, the criteria should be defined before the experiment starts and data are collected [8,33,48,49].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [8]
The Results section identifies inclusion and exclusion criteria as core reporting items, and the Discussion links rigorous implementation and reporting of such measures to reduced risk of bias. Our copy does not say that these criteria must be defined before the experiment starts or before data collection; that timing is not supported by this source. (LLM, conf=0.82)
Experimental — accuracy not yet measuredSample size justification is reported in less than 10% of publications.
“sample size justification (reported in less than 10% of publications)”
From The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research (2020) · cites this paper as [31]
The Results (page unavailable) report that elements of sample-size estimation were reported in 2% of the studies assessed, supporting the ‘less than 10%’ figure. The Materials and Methods, Selection of Studies (page unavailable) limits that assessment to 47 preclinical studies of mesenchymal stem cells for acute lung injury; it does not establish the rate across publications generally. (LLM, conf=0.86)
Experimental — accuracy not yet measuredAll basic animal characteristics are reported in less than 10% of publications.
“animal characteristics (all basic characteristics reported in less than 10% of publications)”
From The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research (2020) · cites this paper as [31]
The Results report poor reporting of animal-related details among 47 preclinical studies, and the Discussion says basic information such as sex and age was routinely omitted. But the Results’ 9% figure refers to allocating animals to experimental groups and to housing and husbandry—not to the proportion of publications reporting all basic animal characteristics. The claimed less-than-10% figure for all characteristics is therefore not established by our copy. (LLM, conf=0.83)
Experimental — accuracy not yet measured
Blinding is reported in only approximately 20% of publications.
“blinding (reported in only approximately 20% of publications)”
From The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research (2020) · cites this paper as [31]
The Results (page not available) identify blinding as a core reporting item and say that none of the core items were reported more than 40% of the time in the 47 studies assessed. Our copy does not give an approximately 20% figure for blinding, and the sample was limited to preclinical studies of acute lung injury treated with mesenchymal stem cells, not publications generally. (LLM, conf=0.78)
Experimental — accuracy not yet measuredImportant information as set out in the ARRIVE guidelines is still missing from most publications sampled.
“However, despite this level of support, recent studies have shown that important information as set out in the ARRIVE guidelines is still missing from most publications sampled.”
From The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research (2020) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
The citation lists haven’t been fetched yet
References 29 · Cited by 130
Research works printed but not linked: 19
- [1]Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, et al CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ. 2010; 340: c869 10.1136/bmj.c869 20332511 PMC2844943 · DOI · Paper page
- [2]Kane RL, Wang J, Garrard J. Reporting in randomized clinical trials improved after adoption of the CONSORT statement. J Clin Epidemiol. 2007; 60: 241– 9. 10.1016/j.jclinepi.2006.06.016 17292017 · DOI
- [4]Cressey D. Surge in support for animal-research guidelines. Nature. 2016; 10.1038/nature.2016.19274 · DOI
- [5]Kilkenny C, Browne WJ, Cuthill IC, Emerson M, Altman DG. Improving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research. PLoS Biol. 2010; 8: e1000412 10.1371/journal.pbio.1000412 20613859 PMC2893951 · DOI · Paper page
- [6]Kilkenny C, Parsons N, Kadyszewski E, Festing MFW, Cuthill IC, Fry D, et al Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PLoS One. 2009; 4: e7824 10.1371/journal.pone.0007824 19956596 PMC2779358 · DOI · Paper page
- [10]Collins FS, Tabak LA. Policy: NIH plans to enhance reproducibility. Nature. 2014; 505: 612– 613. 10.1038/505612a 24482835 PMC4058759 · DOI · Paper page
- [11]Henderson VC, Kimmelman J, Fergusson D, Grimshaw JM, Hackam DG. Threats to validity in the design and conduct of preclinical efficacy studies: a systematic review of guidelines for in vivo animal experiments. PLoS Med. 2013; 10: e1001489 10.1371/journal.pmed.1001489 23935460 PMC3720257 · DOI
- [14]Ioannidis JPA, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, et al Increasing value and reducing waste in research design, conduct, and analysis. Lancet. Elsevier Ltd; 2014; 383: 166– 175. 10.1016/S0140-6736(13)62227-8 24411645 PMC4697939 · DOI · Paper page
- [15]McIntyre LA, Moher D, Fergusson DA, Sullivan KJ, Mei SHJ, Lalu M, et al Efficacy of Mesenchymal Stromal Cell Therapy for Acute Lung Injury in Preclinical Animal Models: A Systematic Review. PLoS One. 2016; 11: e0147170 10.1371/journal.pone.0147170 26821255 PMC4731557 · DOI
- [16]Lalu MM, Moher D, Marshall J, Fergusson D, Mei SH, Macleod M, et al Efficacy and safety of mesenchymal stromal cells in preclinical models of acute lung injury: a systematic review protocol. Syst Rev. 2014; 3: 48 24887266 PMC4046388
- [19]Minnerup J, Wersching H, Diederich K, Schilling M, Ringelstein EB, Wellmann J, et al Methodological quality of preclinical stroke studies is not required for publication in high-impact journals. J Cereb Blood Flow Metab. 2010; 30: 1619– 1624. 10.1038/jcbfm.2010.74 20517323 PMC2949256 · DOI
- [20]Macleod MR, Lawson McLean A, Kyriakopoulou A, Serghiou S, de Wilde A, Sherratt N, et al Risk of Bias in Reports of In Vivo Research: A Focus for Improvement. PLoS Biol. 2015; 13: 1– 12. 10.1371/journal.pbio.1002273 26460723 PMC4603955 · DOI · Paper page
- [21]Thomason Reuters. 2015 Journal Citation Reports® Science Edition. 2013.
- [22]Baker D, Lidster K, Sottomayor A, Amor S. Two years later: journals are not yet enforcing the ARRIVE guidelines on reporting standards for pre-clinical animal studies. PLoS Biol. 2014; 12: e1001756 10.1371/journal.pbio.1001756 24409096 PMC3883646 · DOI · Paper page
- [23]Schwarz F, Iglhaut G, Becker J, Quality BJ, Periodontol C. Quality assessment of reporting of animal studies on pathogenesis and treatment of peri-implant mucositis and peri-implantitis. A systematic review using the ARRIVE guidelines. J Clin Periodontol. 2012; 39 Suppl 1: 63– 72. 10.1111/j.1600-051X.2011.01838.x 22533947 · DOI
- [25]Reardon S. A mouse’s house may ruin experiments. Nature. 2016; 530: 264– 264. 10.1038/nature.2016.19335 26887470 · DOI
- [26]Check Hayden E. Misleading mouse studies waste medical resources. Nature. 2014; 10.1038/nature.2014.14938 · DOI
- [28]Henderson VC, Demko N, Hakala A, MacKinnon N, Federico CA, Fergusson D, et al A meta-analysis of threats to valid clinical inference in preclinical research of sunitinib. Elife. 2015; 4: 1– 13. 10.7554/eLife.08351 26460544 PMC4600817 · DOI
- [30]Moberg-Mogren E, Nelson DL. Evaluating the quality of reporting occupational therapy randomized controlled trials by expanding the CONSORT criteria. Am J Occup Ther. 2006; 60: 226– 35. 16596926
- Authors
- Avey, M. T., Moher, D., Sullivan, K. J., Fergusson, D., Griffin, G., Grimshaw, J. M., Hutton, B., Lalu, M. M., Macleod, M., Marshall, J., Mei, S. H. J., Rudnicki, M., Stewart, D. J., Turgeon, A. F., McIntyre, L., & Canadian Critical Care Translational Biology Group
- Affiliations
- University of Ottawa (CA)
- Ottawa Hospital Research Institute (CA)
- University of Edinburgh (GB)
- University of Toronto (CA)
- Hôpital de l'Enfant-Jésus (CA)
- Corresponding author
- Avey, Marc T.
- Moher, David
- Sullivan, Katrina J.
- Fergusson, Dean
- Griffin, Gilly
- Grimshaw, Jeremy M.
- Hutton, Brian
- Lalu, Manoj M.
- Macleod, Malcolm
- Marshall, John
- Mei, Shirley H. J.
- Rudnicki, Michael
- Stewart, Duncan J.
- Turgeon, Alexis F.
- McIntyre, Lauralyn
- ORCID
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2016-11-17
- Volume
- 11
- Issue
- 11
- Pages
- e0166733
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC5113978
- OpenAlex
- W2553434692
- Cited by
- 130
- References
- 29
- Citations by year
- 2026: 5 · 2025: 10 · 2024: 10 · 2023: 15 · 2022: 17 · 2021: 15 · 2020: 26 · 2019: 15 · 2018: 6 · 2017: 11
- FWCI
- 16.40
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (publishedVersion, other-oa)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Canadian Institutes of Health Research (Post-Doctoral Fellowship - 201 21 OKPD-290743-HTA-ADYP-225697, New Investigator Award - 340151)
- Topics
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Keywords
- incomplete reporting, preclinical research, research reproducibility, acute lung injury, mesenchymal stem cells, research translation, journal impact factor, reporting completeness, animal studies
- MeSH terms
- Animals, Cell Line, National Institutes of Health (U.S.), United States, Guidelines as Topic, Animal Experimentation, Research Report
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
