CoChat
Sign Up
Cited by 262Animal testing and alternativesMeta-analysis and systematic reviewsStatistical Methods in Clinical Trials

Reproducibility of preclinical animal research improves with heterogeneity of study samples

Open access
Voelkl, Bernhard, Vogt, Lucile, Sena, Emily S., Würbel, Hanno·2018·PLOS Biology
DOI 10.1371/journal.pbio.2003693PMC PMC5823461
Open this paper in CoChat — free account

Abstract

Single-laboratory studies conducted under highly standardized conditions are the gold standard in preclinical animal research. Using simulations based on 440 preclinical studies across 13 different interventions in animal models of stroke, myocardial infarction, and breast cancer, we compared the accuracy of effect size estimates between single-laboratory and multi-laboratory study designs. Single-laboratory studies generally failed to predict effect size accurately, and larger sample sizes rendered effect size estimates even less accurate. By contrast, multi-laboratory designs including as few as 2 to 4 laboratories increased coverage probability by up to 42 percentage points without a need for larger sample sizes. These findings demonstrate that within-study standardization is a major cause of poor reproducibility. More representative study samples are required to improve the external validity and reproducibility of preclinical animal research and to prevent wasting animals and resources for inconclusive research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1371/journal.pbio.2003693

Cites

9 of 9 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • In toxicology, awareness of the reproducibility problem emerged gradually in association with insufficient in vivo reproducibility.

    “In toxicology, which may better reflect the background of most readers of this journal, awareness of this problem has emerged only gradually in association with insufficient in vivo reproducibility (Kilkenny et al., 2009; Voelkl et al., 2018).”

    From In Vitro research Reproducibility: Keeping up High Standards (2019) · cites this paper as Voelkl et al., 2018

    The Introduction (page unavailable) says reproducibility of preclinical animal research is alarmingly low, supporting the concern about in vivo reproducibility. It does not specifically discuss toxicology or say that awareness of the problem emerged gradually. (LLM, conf=0.83)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 40

  1. [1]Loken E, Gelman A. Measurement error and the replication crisis. Science. 2017; 355: 584– 585. doi: 10.1126/science.aal3618 28183939 · DOI · Paper page
  2. [4]Ioannidis JPA, Fanelli D, Dunne DD, Goodman SN. Meta-research: Evaluation and improvement of research methods and practices. PLoS Biol. 2015; 13: e1002264 doi: 10.1371/journal.pbio.1002264 26431313 PMC4592065 · DOI · Paper page
  3. [5]Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, du Sert NP, et al A manifesto for reproducible science. Nat Hum Behav. 2017; 1: 0021. PMC7610724 33954258
  4. [9]Beynen AC, Gärtner K, van Zutphen LFM. Standardization of animal experimentation In: Zutphen LFM, Baumans V, Beynen AC, editors. Principles of laboratory animal science. 2nd ed Amsterdam: Elsevier Ltd; 2003 pp. 103– 110.
  5. [11]Würbel H. Behaviour and the standardization fallacy. Nat Genet. 2000; 26: 263 doi: 10.1038/81541 11062457 · DOI
  6. [12]Voelkl B, Würbel H. Reproducibility crisis: Are we ignoring reaction norms? Trends Pharmacol Sci. 2016; 37: 509– 510. doi: 10.1016/j.tips.2016.05.003 27211784 · DOI
  7. [13]Crabbe JC, Wahlsten D, Dudek BC, Sibilia M, Wagner EF. Genetics of mouse behavior: Interactions with laboratory environment. Science. 1999; 284: 1670– 1672. 10356397
  8. [14]Richter SH, Garner JP, Auer C, Kunert J, Würbel H. Systematic variation improves reproducibility of animal experiments. Nat Methods. 2010; 7: 167– 168. doi: 10.1038/nmeth0310-167 20195246 · DOI
  9. [15]Richter SH, Garner JP, Würbel H. Environmental standardization: Cure or cause of poor reproducibility in animal experiments? Nat Methods. 2009; 6: 257– 261. doi: 10.1038/nmeth.1312 19333241 · DOI
  10. [16]Kafkafi N, Golani I, Jaljuli I, Morgan H, Sarig T, Würbel H, et al Addressing reproducibility in single-laboratory phenotyping experiments. Nat Methods. 2016; 14: 462– 464. 28448068
  11. [17]Karp NA, Speak AO, White JK, Adams DJ, Hrabe´ de Angelis M, Hérault Y, Mott RF. Impact of temporal variation on design and analysis of mouse knockout phenotyping studies. PLoS ONE. 2014; 9: e111239 doi: 10.1371/journal.pone.0111239 25343444 PMC4208881 · DOI
  12. [18]Schoener TW. The newest synthesis: Understanding ecological dynamics. Science. 2011; 331: 426– 429. 21273479
  13. [19]Würbel H. Behavioral phenotyping enhanced—beyond (environmental) standardization. Genes Brain Behav. 2002; 1: 3– 8. 12886944
  14. [20]Llovera G, Hofmann K, Roth S, Salas-Pérdomo A, Ferrer-Ferrer M, Perego M, et al Results of a preclinical randomized controlled multicenter trial (pRCT): Anti-CD49d treatment for acute brain ischemia. Sci Transl Med. 2015; 7: 299ra121– 299ra121. doi: 10.1126/scitranslmed.aaa9853 26246166 · DOI
  15. [22]Hagger MS, Chatzisarantis NL, Alberts H, Anggono CO, Batailler C, Birt AR, et al A multilab preregistered replication of the ego-depletion effect. Persp Psychol Sci. 2016; 11: 546– 73. 27474142
  16. [23]Kahan BC, Morris TP. Analysis of multicentre trials with continuous outcomes: When and how should we account for centre effects? Stat Med. 2013; 32: 1136– 1149. doi: 10.1002/sim.5667 23112128 · DOI
  17. [24]Cornfield J. Randomization by group: A formal analysis. Am J Epidemiol. 1978; 108: 100– 102. 707470
  18. [25]Localio AR, Berlin JA, Ten Have TR, Kimmel SE. Adjustments for center in multicenter studies: An overview. Ann Intern Med. 2001; 135: 112– 123. 11453711
  19. [26]Lefer DJ, Bolli R. Development of an NIH consortium for preclinicAl AssESsment of CARdioprotective therapies (CAESAR): A paradigm shift in studies of infarct size limitation. J Cardiovasc Pharmacol Ther. 2011; 16: 332– 339. doi: 10.1177/1074248411414155 21821536 · DOI
  20. [28]Collaborative Approach to Meta Analysis and Review of Animal Data from Experimental Studies (CAMARADES) website. www.camarades.info. Accessed on 30 September 2015.
  21. [29]Van Der Worp HB, Sena ES, Donnan GA, Howells DW, Macleod MR. Hypothermia in animal models of acute ischaemic stroke: A systematic review and meta-analysis. Brain. 2007; 130: 3063– 3074. doi: 10.1093/brain/awm083 17478443 · DOI
  22. [30]Wu T-C, Grotta JC. Hypothermia for acute ischaemic stroke. Lancet Neurol. 2013; 12: 275– 284. doi: 10.1016/S1474-4422(13)70013-9 23415567 · DOI
  23. [31]Reith J, Jørgensen HS, Pedersen PM, Nakamaya H, Jeppesen LL, Olsen TS, Raaschou HO. Body temperature in acute stroke: Relation to stroke severity, infarct size, mortality, and outcome. Lancet. 1996; 347: 422– 425. 8618482
  24. [33]Rooke ED, Vesterinen HM, Sena ES, Egan KJ, Macleod MR. Dopamine agonists in animal models of Parkinson’s disease: a systematic review and meta-analysis. Parkinsonism Relat D. 2011; 17: 313– 320. 21376651
  25. [35]InHout J, Ioannidis JPA, Borm GF. Obtaining evidence by a single well-powered trial or several modestly powered trials. Stat Meth Med Res. 2012; 25: 538– 552. 23070590
  26. [38]Sorge RE, Martin LJ, Isbester KA, Sotocinal SG, Rosen S, Tuttle AH, et al Olfactory exposure to males, including men, causes stress and related analgesia in rodents. Nat Methods. 2014; 11: 629– 632. doi: 10.1038/nmeth.2935 24776635 · DOI · Paper page
  27. [39]Chesler EJ, Wilson SG, Lariviere WR, Rodriguez-Zas SL, Mogil JS. Influences of laboratory environment on behavior. Nat Neurosci. 2012; 5: 1101– 1102. 12403996
  28. [40]Franklin CL, Ericsson AC. Microbiota and reproducibility of rodent models. LabAnimal. 2017; 46: 114– 122. PMC5762113 28328896
  29. [41]Johnson VE. Revised standards for statistical evidence. Proc Natl Acad Sci USA. 2013; 110: 19313– 19317. doi: 10.1073/pnas.1313476110 24218581 PMC3845140 · DOI · Paper page
  30. [42]Benjamin DJ, Berger JO, Johannesson M, Nosek BA, Wagenmakers E-J, Berk R, et al Redefine statistical significance. Nat Hum Behav. 2017; 1: 1. 30980045
  31. [43]Richter SH, Garner JP, Zipser B, Lewejohann L, Sachser N, Schindler B, et al Effect of population heterogenization on the reproducibility of mouse behavior: A multi-laboratory study. PLoS ONE. 2011; 6: e16461. 21305027 PMC3031565
  32. [45]Würbel H. More than 3Rs: The importance of scientific validity for harm-benefit analysis of animal research. Lab Anim. (NY). 2017; 46: 164– 166. 28328898
  33. [46]Sena E, van der Worp HB, Howells D, Macleod M. How can we improve the pre-clinical development of drugs for stroke? Trends Neurosci. 2007; 30: 433– 439. doi: 10.1016/j.tins.2007.06.009 17765332 · DOI
  34. [47]Viechtbauer W. Conducting meta-analyses in R with the metafor package. J Stat Softw. 2010; 36: 1– 48.
  35. [48]Cornfield J. Randomization by group: A formal analysis. Am J Epidemiol. 1978; 108: 100– 102. 707470
  36. [49]Vierron E, Giraudeau B. Design effect in multicenter studies: Gain or loss of power? BMC Med Res Methodol. 2009; 9: 39 doi: 10.1186/1471-2288-9-39 19538744 PMC2715424 · DOI
  37. [50]Reitsma A, Chu R, Thorpe J, McDonald S, Thabane L, Hutton E. Accounting for center in the early external cephalic version trials: An empirical comparison of statistical methods to adjust for center in a multicenter trial with binary outcomes. Trials. 2014; 15: 377 doi: 10.1186/1745-6215-15-377 25257928 PMC4192344 · DOI
  38. [51]Raudenbush SW. Hierarchical linear models as generalizations of certain common experimental design models In: Edwards L, editor. Applied analysis of variance in behavioral science. New York: Dekker; 1993 pp. 459– 496.
  39. [52]Moerbeek M, Van Breukelen GJP, Berger MPF. A comparison between traditional methods and multilevel regression for the analysis of multicenter intervention studies. J Clin Epidemiol. 2003; 56: 341– 350. 12767411
  40. [53]Kahan BC, Morris TP. Assessing potential sources of clustering in individually randomised trials. BMC Med Res Methodol. 2013; 13: 58 doi: 10.1186/1471-2288-13-58 23590245 PMC3643875 · DOI

Cited by

Sorted by citations

Authors
Voelkl, B., Vogt, L., Sena, E. S., & Würbel, H.
Affiliations
  • University of Bern (CH)
  • University of Edinburgh (GB)
Corresponding author
  • Voelkl, Bernhard
  • Vogt, Lucile
  • Sena, Emily S.
  • Würbel, Hanno
ORCID
Venue
PLOS Biology
Publisher
Public Library of Science (PLoS)
Published
2018-02-22
Volume
16
Issue
2
Pages
e2003693
ISSN
  • 1544-9173 (ISSN-L)
  • 1545-7885
DOI
10.1371/journal.pbio.2003693
PMC
PMC5823461
OpenAlex
W2787981664
Cited by
262
References
9
Citations by year
2026: 32 · 2025: 22 · 2024: 25 · 2023: 26 · 2022: 31 · 2021: 34 · 2020: 46 · 2019: 36 · 2018: 9 · 2017: 1
FWCI
44.41
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS Biology (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • RePEc: Research Papers in Economics (submittedVersion)
  • BORIS (University Library Bern) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Figshare (submittedVersion, cc-by)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
  • Open Access CRIS of the University of Bern
Funders
  • Bundesamt für Lebensmittelsicherheit und Veterinärwesen (2.13.01)
  • FP7 Ideas: European Research Council (322576)
Topics
  • Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords
reproducibility, effect size estimation, sample size, generalizability, animal models of stroke, myocardial infarction, breast cancer, simulation study
MeSH terms
Animals, Disease Models, Animal, Laboratories, Probability, Research Design, Reproducibility of Results, Sample Size, Animal Experimentation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review