CoChat
Sign Up
Cited by 72Animal testing and alternativesMeta-analysis and systematic reviewsAcademic Publishing and Open Access

The researchers’ view of scientific rigor—survey on the conduct and reporting of in vivo research

Open access
Reichlin, Thomas S., Vogt, Lucile, Würbel, Hanno·2016·PLOS ONE
DOI 10.1371/journal.pone.0165999PMC PMC5135049
Open this paper in CoChat — free accountOpen PDF

Abstract

Reproducibility in animal research is alarmingly low, and a lack of scientific rigor has been proposed as a major cause. Systematic reviews found low reporting rates of measures against risks of bias (e.g., randomization, blinding), and a correlation between low reporting rates and overstated treatment effects. Reporting rates of measures against bias are thus used as a proxy measure for scientific rigor, and reporting guidelines (e.g., ARRIVE) have become a major weapon in the fight against risks of bias in animal research. Surprisingly, animal scientists have never been asked about their use of measures against risks of bias and how they report these in publications. Whether poor reporting reflects poor use of such measures, and whether reporting guidelines may effectively reduce risks of bias has therefore remained elusive. To address these questions, we asked in vivo researchers about their use and reporting of measures against risks of bias and examined how self-reports relate to reporting rates obtained through systematic reviews. An online survey was sent out to all registered in vivo researchers in Switzerland (N = 1891) and was complemented by personal interviews with five representative in vivo researchers to facilitate interpretation of the survey results. Return rate was 28% (N = 530), of which 302 participants (16%) returned fully completed questionnaires that were used for further analysis. According to the researchers' self-report, they use measures against risks of bias to a much greater extent than suggested by reporting rates obtained through systematic reviews. However, the researchers' self-reports are likely biased to some extent. Thus, although they claimed to be reporting measures against risks of bias at much lower rates than they claimed to be using these measures, the self-reported reporting rates were considerably higher than reporting rates found by systematic reviews. Furthermore, participants performed rather poorly when asked to choose effective over ineffective measures against six different biases. Our results further indicate that knowledge of the ARRIVE guidelines had a positive effect on scientific rigor. However, the ARRIVE guidelines were known by less than half of the participants (43.7%); and among those whose latest paper was published in a journal that had endorsed the ARRIVE guidelines, more than half (51%) had never heard of these guidelines. Our results suggest that whereas reporting rates may underestimate the true use of measures against risks of bias, self-reports may overestimate it. To a large extent, this discrepancy can be explained by the researchers' ignorance and lack of knowledge of risks of bias and measures to prevent them. Our analysis thus adds significant new evidence to the assessment of research integrity in animal research. Our findings further question the confidence that the authorities have in scientific rigor, which is taken for granted in the harm-benefit analyses on which approval of animal experiments is based. Furthermore, they suggest that better education on scientific integrity and good research practice is needed. However, they also question reliance on reporting rates as indicators of scientific rigor and highlight a need for more reliable predictors.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pone.0165999

Cites

0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Studies measuring the impact of ARRIVE on the quality of reporting have produced mixed results.

    “Studies measuring the impact of ARRIVE on the quality of reporting have produced mixed results [6–11]”

    From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [9]

    In the Discussion (page unavailable), researchers who knew the ARRIVE guidelines scored higher on a self-reported reporting measure. The Introduction and Discussion also say reporting remained low after journal endorsement. These contrasting observations fit the idea of mixed results, but this survey did not directly measure ARRIVE’s impact on the quality of published reporting. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • This prioritisation enables research users to assess how much weight to ascribe to the findings and promotes the use of rigorous methodology in the planning and conduct of in vivo experiments, thus increasing the likelihood that the findings are reliable and reproducible.

    “This enables research users to assess how much weight to ascribe to the findings and, in parallel, promotes the use of rigorous methodology in the planning and conduct of in vivo experiments [37], thus increasing the likelihood that the findings are reliable and, ultimately, reproducible.”

    From The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research (2020) · cites this paper as [37]

    The Discussion section on reporting guidelines finds that researchers familiar with ARRIVE scored higher on measures of experimental rigor and says guidelines may improve the use of measures against bias. But the Conclusions hedge the link between rigor and reproducibility and say further research is needed; the paper does not establish that the cited prioritisation increases the likelihood of reliable, reproducible findings. The Conclusions also caution that reporting rates may not reliably tell readers how rigorous a study was. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

  • Awareness of ARRIVE is linked to the use of more rigorous experimental design standards.

    “There is some evidence that awareness of ARRIVE is linked to the use of more rigorous experimental design standards [37]”

    From The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research (2020) · cites this paper as [37]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 49

  1. [4]Kola I, Landis J. Can the pharmaceutical industry reduce attrition rates? Nat. Rev. Drug Discov. [Internet]. 2004; 3: 711– 5. Available: http://www.ncbi.nlm.nih.gov/pubmed/15286737 10.1038/nrd1470 15286737 · DOI · Paper page
  2. [5]van der Worp HB, Howells DW, Sena ES, Porritt MJ, Rewell S, O’Collins V, et al Can animal models of disease reliably inform human studies? PLoS Med. [Internet]. 2010; 7: e1000245 Available: http://www.plosmedicine.org/article/fetchObject.action?uri=info%253Adoi%252F10.1371%252Fjournal.pmed.1000245&representation=PDF 10.1371/journal.pmed.1000245 20361020 PMC2846855 · DOI · Paper page
  3. [6]Editor. Further confirmation needed. Nat. Biotechnol. 2012; 30: 806 10.1038/nbt.2335 22965029 · DOI
  4. [8]Richter SH, Garner JP, Würbel H. Environmental standardization: Cure or cause of poor reproducibility in animal experiments? Nat. Methods. 2009; 6: 257– 61. 10.1038/nmeth.1312 19333241 · DOI
  5. [9]Bailoo JD, Reichlin TS, Würbel H. Refinement of experimental design and conduct in laboratory animal research. ILAR J. [Internet]. 2014 [cited 2014 Dec 29]; 55: 383– 91. Available: http://ilarjournal.oxfordjournals.org/cgi/doi/10.1093/ilar/ilu037 10.1093/ilar/ilu037 25541540 · DOI
  6. [10]Collins FS, Tabak LA. Policy: NIH plans to enhance reproducibility. Nature [Internet]. 2014 [cited 2014 Jun 3]; 505: 612– 3. Available: http://europepmc.org/abstract/MED/24482835 24482835 PMC4058759
  7. [11]Festing MFW. We are not born knowing how to design and analyse scientific experiments. Altern. to Lab. Anim. ATLA. 2013; 41: 19– 21. 23781938
  8. [12]Fanelli D. Do pressures to publish increase scientists’ bias? An empirical support from US States data. PLoS One. 2010; 5: e10271 10.1371/journal.pone.0010271 20422014 PMC2858206 · DOI · Paper page
  9. [14]Ioannidis JPA. Why most published research findings are false. PLoS Med. [ Internet]. 2005 [cited 2013 May 21]; 2: e124 Available: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=1182327&tool=pmcentrez&rendertype=abstract PMC1182327 16060722
  10. [15]Macleod MR. Why animal research needs to improve. Nature. 2011; 477: 511 10.1038/477511a 21956292 · DOI
  11. [16]Ransohoff DF. Bias as a threat to the validity of cancer molecular-marker research. Nat. Rev. Cancer. 2005; 5: 142– 9. 10.1038/nrc1550 15685197 · DOI
  12. [17]Henderson VC, Kimmelman J, Fergusson D, Grimshaw JM, Hackam DG. Threats to validity in the design and conduct of preclinical efficacy studies: A systematic review of guidelines for in vivo animal experiments. PLoS Med. [Internet]. 2013 [cited 2014 Feb 20]; 10: e1001489 Available: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=3720257&tool=pmcentrez&rendertype=abstract 10.1371/journal.pmed.1001489 23935460 PMC3720257 · DOI
  13. [18]Steckler T. Editorial: preclinical data reproducibility for R&D—the challenge for neuroscience. Springerplus [Internet]. 2015; 4: 1 Available: http://www.ncbi.nlm.nih.gov/pubmed/25674489\n http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4320139/pdf/40064_2014_Article_1534.pdf 10.1186/2193-1801-4-1 25674489 PMC4320139 · DOI
  14. [20]Freedman LP, Gibson MC. The impact of pharmacogenomics research on drug development. Clin. Pharmacol. Ther. 2015; 97: 16– 8. 10.1002/cpt.9 25670378 · DOI
  15. [21]O’Collins VE, Macleod MR, Donnan GA, Horky LL, van der Worp HB, Howells DW. 1,026 experimental treatments in acute stroke. Ann. Neurol. [Internet]. 2006 [cited 2013 Jun 6]; 59: 467– 77. Available: http://www.ncbi.nlm.nih.gov/pubmed/16453316 10.1002/ana.20741 16453316 · DOI
  16. [22]Scannell JW, Blanckley A, Boldon H, Warrington B. Diagnosing the decline in pharmaceutical R&D efficiency. Nat. Rev. Drug Discov. [Internet]. Nature Publishing Group; 2012; 11: 191– 200. Available: http://www.ncbi.nlm.nih.gov/pubmed/22378269 10.1038/nrd3681 22378269 · DOI · Paper page
  17. [23]Henderson VC, Demko N, Hakala A, MacKinnon N, Federico CA, Fergusson D, et al A meta-analysis of threats to valid clinical inference in preclinical research of sunitinib. Elife [Internet]. 2015; 4: 1– 13. Available: http://elifesciences.org/lookup/doi/10.7554/eLife.08351 PMC4600817 26460544 · DOI
  18. [24]Sena ES, van der Worp HB, Howells DW, Macleod MR. How can we improve the pre-clinical development of drugs for stroke? Trends Neurosci. [Internet]. 2007 [cited 2013 May 15]; 30: 433– 9. Available: http://www.ncbi.nlm.nih.gov/pubmed/17765332 10.1016/j.tins.2007.06.009 17765332 · DOI
  19. [25]McCann SK, Irvine C, Mead GE, Sena ES, Currie GL, Egan KE, et al Efficacy of antidepressants in animal models of ischemic stroke: A systematic review and meta-analysis. Stroke [Internet]. 2014; 45: 3055– 63. Available: http://stroke.ahajournals.org/cgi/doi/10.1161/STROKEAHA.114.006304 10.1161/STROKEAHA.114.006304 25184357 · DOI
  20. [26]Macleod MR, Lawson McLean A, Kyriakopoulou A, Serghiou S, de Wilde A, Sherratt N, et al Risk of bias in reports of in vivo research: A focus for improvement. PLoS Biol. [Internet]. 2015; 13: e1002273 Available: http://dx.plos.org/10.1371/journal.pbio.1002273 10.1371/journal.pbio.1002273 26460723 PMC4603955 · DOI · Paper page
  21. [27]Frantzias J, Sena ES, Macleod MR, Al-Shahi Salman R. Treatment of intracerebral hemorrhage in animal models: meta-analysis. Ann. Neurol. [Internet]. 2011 [cited 2013 Jun 28]; 69: 389– 99. Available: http://www.ncbi.nlm.nih.gov/pubmed/21387381 10.1002/ana.22243 21387381 · DOI
  22. [28]Rooke EDM, Vesterinen HM, Sena ES, Egan KJ, Macleod MR. Dopamine agonists in animal models of Parkinson’s disease: A systematic review and meta-analysis. Parkinsonism Relat. Disord. [Internet]. Elsevier Ltd; 2011 [cited 2013 Jun 26]; 17: 313– 20. Available: http://www.ncbi.nlm.nih.gov/pubmed/21376651 10.1016/j.parkreldis.2011.02.010 21376651 · DOI
  23. [29]Vesterinen HM, Sena ES, Ffrench-Constant C, Williams A, Chandran S, Macleod MR. Improving the translational hit of experimental treatments in multiple sclerosis. Mult. Scler. 2010; 16: 1044– 55. 10.1177/1352458510379612 20685763 · DOI · Paper page
  24. [30]Currie GL, Delaney A, Bennett MI, Dickenson AH, Egan KJ, Vesterinen HM, et al Animal models of bone cancer pain: Systematic review and meta-analyses. Pain [Internet]. International Association for the Study of Pain; 2013 [cited 2013 Aug 7]; 154: 917– 26. Available: http://www.ncbi.nlm.nih.gov/pubmed/23582155 10.1016/j.pain.2013.02.033 23582155 · DOI
  25. [32]Macleod MR, O’Collins T, Howells DW, Donnan GA. Pooling of animal experimental data reveals influence of study design and publication bias. Stroke [Internet]. 2004 [cited 2013 Apr 29]; 35: 1203– 8. Available: http://www.ncbi.nlm.nih.gov/pubmed/15060322 10.1161/01.STR.0000125719.25853.20 15060322 · DOI
  26. [34]Kilkenny C, Browne WJ, Cuthill IC, Emerson M, Altman DG. Improving bioscience research reporting: The ARRIVE guidelines for reporting animal research. PLoS Biol. [Internet]. 2010 [cited 2013 May 24]; 8: e1000412 Available: http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=2893951&tool=pmcentrez&rendertype=abstract 10.1371/journal.pbio.1000412 20613859 PMC2893951 · DOI · Paper page
  27. [35]Baker D, Lidster K, Sottomayor A, Amor S. Two years later: Journals are not yet enforcing the ARRIVE guidelines on reporting standards for pre-clinical animal studies. Eisen JA, editor. PLoS Biol. [Internet]. Public Library of Science; 2014 [cited 2014 May 26]; 12: e1001756 Available: http://dx.plos.org/10.1371/journal.pbio.1001756 10.1371/journal.pbio.1001756 24409096 PMC3883646 · DOI · Paper page
  28. [37]Nuzzo R. Statistical errors. Nature. 2014; 506: 150– 2. 10.1038/506150a 24522584 · DOI · Paper page
  29. [38]Vaux DL. Know when your numbers are significant. Nature. 2012; 492: 180– 1. 10.1038/492180a 23235861 · DOI
  30. [39]Begley CG, Ioannidis JPA. Reproducibility in science: Improving the standard for basic and preclinical research. Circ. Res. [Internet]. 2015; 116: 116– 26. Available: http://circres.ahajournals.org/cgi/doi/10.1161/CIRCRESAHA.114.303819 10.1161/CIRCRESAHA.114.303819 25552691 · DOI · Paper page
  31. [41]Pound P, Bracken MB. Is animal research sufficiently evidence based to be a cornerstone of biomedical research? BMJ Br. Med. J. [Internet]. 2014 [cited 2014 Sep 4]; 348: g3387 Available: http://www.ncbi.nlm.nih.gov/pubmed/24879816 24879816
  32. [42]McNutt M. Journals unite for reproducibility. Nature. 2014; 515: 7. 25383411
  33. [44]Burnham KP, Anderson DR. Multimodel inference: Understanding AIC and BIC in model selection. Sociol. Methods Res. 2004; 33: 261– 304.
  34. [45]Aho K, Derryberry D, Peterson T. Model selection for ecologists: The worldviews of AIC and BIC. Ecology. 2014; 95: 631– 6. 24804445
  35. [48]Minnerup J, Zentsch V, Schmidt A, Fisher M, Schäbitz W- R. Methodological quality of experimental stroke studies published in the Stroke Journal. Stroke [Internet]. 2016; 47: 267– 72. Available: http://stroke.ahajournals.org/lookup/doi/10.1161/STROKEAHA.115.011695 10.1161/STROKEAHA.115.011695 26658439 · DOI
  36. [49]Ehrlinger J, Dunning D. How chronic self-views influence (and potentially mislead) estimates of performance. J. Pers. Soc. Psychol. 2003; 84: 5– 17. 12518967
  37. [50]Kruger J, Dunning D. Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J. Pers. Soc. Psychol. 1999; 7: 1121– 34. 10626367
  38. [51]Bello S, Krogsbøll LT, Gruber J, Zhao ZJ, Fischer D, Hróbjartsson A. Lack of blinding of outcome assessors in animal model experiments implies risk of observer bias. J. Clin. Epidemiol. [Internet]. 2014 [cited 2014 Aug 25]; 67: 973– 83. Available: http://www.ncbi.nlm.nih.gov/pubmed/24972762 10.1016/j.jclinepi.2014.04.008 24972762 · DOI
  39. [52]Howells DW, Sena ES, O’Collins V, Macleod MR. Improving the efficiency of the development of drugs for stroke. Int. J. Stroke [Internet]. 2012 [cited 2013 Jun 4]; 7: 371– 7. Available: http://www.ncbi.nlm.nih.gov/pubmed/22712738 10.1111/j.1747-4949.2012.00805.x 22712738 · DOI
  40. [54]Cressey D. Surge in support for animal-research guidelines. Nature. News [Internet]. 2016; Available: http://www.nature.com/doifinder/10.1038/nature.2016.19274 · DOI
  41. [55]Kontinen VK. From clear reporting to better research models. Scand. J. Pain [Internet]. Elsevier B.V.; 2013 [cited 2013 Oct 9]; 4: 57 Available: http://linkinghub.elsevier.com/retrieve/pii/S1877886013000074 29913898
  42. [56]Editor. Reducing our irreproducibility. Nature. 2013; 496: 198.
  43. [57]Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al Reducing waste from incomplete or unusable reports of biomedical research. Lancet [Internet]. Elsevier Ltd; 2014 [cited 2014 Jan 20]; 383: 267– 76. Available: http://www.ncbi.nlm.nih.gov/pubmed/24411647 10.1016/S0140-6736(13)62228-X 24411647 · DOI · Paper page
  44. [58]Kontinen V. Raising the standards of preclinical pain studies. Scand. J. Pain [Internet]. Elsevier B.V.; 2015; 7: 38– 9. Available: http://linkinghub.elsevier.com/retrieve/pii/S1877886015000063 29911596
  45. [59]Mullane K, Enna SJ, Piette J, Williams M. Guidelines for manuscript submission in the peer-reviewed pharmacological literature. Biochem. Pharmacol. [Internet]. 2015; 97: 225– 35. Available: http://www.sciencedirect.com/science/article/pii/S0006295215003585 10.1016/j.bcp.2015.06.023 26208784 · DOI
  46. [60]Würbel H. Behaviour and the standardization fallacy. Nat. Genet. [Internet]. 2000; 26: 263 Available: http://www.ncbi.nlm.nih.gov/pubmed/11062457 10.1038/81541 11062457 · DOI
  47. [61]Würbel H. Behavioral phenotyping enhanced–beyond (environmental) standardization. Genes, Brain Behav. [Internet]. 2002; 1: 3– 8. Available: http://www.ncbi.nlm.nih.gov/pubmed/12886944 12886944
  48. [62]Richter SH, Garner JP, Auer C, Kunert J, Würbel H. Systematic variation improves reproducibility of animal experiments. Nat. Methods [Internet]. Nature Publishing Group; 2010 [cited 2013 Apr 4]; 7: 167– 8. Available: http://www.ncbi.nlm.nih.gov/pubmed/20195246 10.1038/nmeth0310-167 20195246 · DOI
  49. [63]Voelkl B, Würbel H. Reproducibility crisis: Are we ignoring reaction norms? Trends Pharmacol. Sci. 2016; 37: 509– 10. 10.1016/j.tips.2016.05.003 27211784 · DOI

Cited by

Sorted by citations

Authors
Reichlin, T. S., Vogt, L., & Würbel, H.
Affiliations
University of Bern (CH)
Corresponding author
Würbel, Hanno
ORCID
Venue
PLOS ONE
Publisher
Public Library of Science (PLoS)
Published
2016-12-02
Volume
11
Issue
12
Pages
e0165999
ISSN
1932-6203 (ISSN-L)
DOI
10.1371/journal.pone.0165999
PMC
PMC5135049
OpenAlex
W2557934497
Cited by
72
References
64
Citations by year
2026: 2 · 2025: 1 · 2024: 11 · 2023: 3 · 2022: 6 · 2021: 7 · 2020: 19 · 2019: 8 · 2018: 7 · 2017: 6 · 2016: 2
FWCI
11.79
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS ONE (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • Bern Open Repository and Information System (University of Bern) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Figshare (submittedVersion, cc-by)
  • Open Access CRIS of the University of Bern
Funders
Bundesamt für Lebensmittelsicherheit und Veterinärwesen (2.13.01)
Topics
  • Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
Keywords
scientific rigor, animal studies, reporting guidelines, ARRIVE guidelines, reproducibility, reporting rates, research integrity, blinding, randomization
MeSH terms
Animals, Humans, Models, Biological, Periodicals as Topic, Switzerland, Guidelines as Topic, Biomedical Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review