Reproducibility and Rigor in Animal-Based Research
Open accessThe DOI matches its Crossref record; 49 of the 49 references it cites were found in a registry or an index.
Abstract
Increasing focus on issues of research reproducibility affords us the opportunity to review some of the key issues related in vivo research. First, we set out some key definitions, to guide the reader through the rest of the paper. Next we consider issues of epistemology, of how animal experiments lead to changes in our understanding of biomedicine and, potentially, to the development of new therapeutics. Here we consider the meaning of statistical significance; the importance of understanding whether findings have general truth; and the advances in knowledge which can result from ‘failed’ replication. Then, we consider weaknesses in the design, conduct and reporting of experiments, and review evidence for this from systematic reviews and from experimental studies addressing these issues. We consider the impact that these weaknesses have on the development of new treatments for human disease, and reflect on the response to these issues from the biomedical research community. Finally, we consider strategies for improvement including increased use of brief, pre-registered study protocols; pre-registration, open publication and open data; and the central importance of education in improving research performance.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Other factors have been identified that could improve replicability, such as blinding, randomization, and sample size planning (Landis et al., 2012; Macleod and Mohan, 2019).”
From Investigating the replicability of preclinical cancer biology (2021) · cites this paper as Macleod and Mohan, 2019
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Michie, S., van Stralen, M. M., & West, R. (2011) · Implementation Science · doi:10.1186/1748-5908-6-42 · Cited by 13,697
Improving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · PLoS Biology · doi:10.1371/journal.pbio.1000412 · Cited by 7,300
Improving bioscience research reporting: The ARRIVE guidelines for reporting animal research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · Journal of Pharmacology and Pharmacotherapeutics · doi:10.4103/0976-500x.72351 · Cited by 6,277
Can the pharmaceutical industry reduce attrition rates?
Kola, I., & Landis, J. (2004) · Nature Reviews Drug Discovery · doi:10.1038/nrd1470 · Cited by 4,167
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 1×
“This has been studied in retrospective observational [2,3] and prospective [4] studies.”
BackgroundMoving to a World Beyond “ p < 0.05”
Wasserstein, R. L., Schirm, A. L., & Lazar, N. A. (2019) · The American Statistician · doi:10.1080/00031305.2019.1583913 · Cited by 2,976
Avoidable waste in the production and reporting of research evidence
Chalmers, I., & Glasziou, P. (2009) · The Lancet · doi:10.1016/s0140-6736(09)60329-9 · Cited by 2,208
Fusobacterium nucleatum infection is prevalent in human colorectal carcinoma
Castellarin, M., Warren, R. L., Freeman, J. D., Dreolini, L., Krzywinski, M., Strauss, J., Barnes, R., Watson, P., Allen-Vercoe, E., Moore, R. A., et al. (2012) · Genome Research · doi:10.1101/gr.126516.111 · Cited by 2,043
Believe it or not: how much can we rely on published data on potential drug targets?
Prinz, F., Schlange, T., & Asadullah, K. (2011) · Nature Reviews Drug Discovery · doi:10.1038/nrd3439-c1 · Cited by 1,888 · cited 1×
“This has been studied in retrospective observational [2,3] and prospective [4] studies.”
BackgroundIncreasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597
Self-citationWhat does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · cited 3×
“Articulating this broader definition, and borrowing from Goodman [1], one might consider a hierarchy of characteristics that might give confidence in the ‘truth’ of a research finding: first, ‘reproducibility’ as originally described, based on reanalysis of an existing dataset (‘reproducibility of analysis’); secondly…”
BackgroundInfluentialA call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196
Self-citationMany Labs 2: Investigating variation in replicability across samples and settings
Klein, R. A., Vianello, M., Hasselman, F., Adams, B. G., Adams, R. B., Alper, S., Aveyard, M., Axt, J. R., Babalola, M. T., Bahník, Š., et al. (2018) · Advances in Methods and Practices in Psychological Science · doi:10.1177/2515245918810225 · Cited by 944 · cited 1×
“For instance, the recent Many Labs 2 replication project successfully replicated only 54% of findings [5].”
BackgroundNXY-059 for the Treatment of Acute Ischemic Stroke
Shuaib, A., Lees, K. R., Lyden, P., Grotta, J., Davalos, A., Davis, S. M., Diener, H.-C., Ashwood, T., Wasiewski, W. W., & Emeribe, U. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa070240 · Cited by 732
Publication bias in reports of animal stroke studies leads to major overstatement of efficacy
Sena, E. S., van der Worp, H. B., Bath, P. M. W., Howells, D. W., & Macleod, M. R. (2010) · PLoS Biology · doi:10.1371/journal.pbio.1000344 · Cited by 600 · cited 1×
“That is, the reports in the literature represent experiments where an effect was observed by chance but similar experiments where no effect was observed were never published (publication bias [8,9]).”
ResultSelf-citationAvoidable Waste in the Production and Reporting of Research Evidence
Chalmers, I., & Glasziou, P. (2009) · Obstetrics & Gynecology · doi:10.1097/aog.0b013e3181c3020d · Cited by 600
PREPARE: guidelines for planning animal research and testing
Smith, A. J., Clutton, R. E., Lilley, E., Hansen, K. E. A., & Brattelid, T. (2018) · Laboratory Animals · doi:10.1177/0023677217724823 · Cited by 587
The effect of experimenter bias on the performance of the albino rat
Rosenthal, R., & Fode, K. L. (2007) · Behavioral Science · doi:10.1002/bs.3830080302 · Cited by 384
Risk of bias in reports of in vivo research: a focus for improvement
Macleod, M. R., Lawson McLean, A., Kyriakopoulou, A., Serghiou, S., de Wilde, A., Sherratt, N., Hirst, T., Hemblade, R., Bahor, Z., Nunes-Fonseca, C., et al. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002273 · Cited by 358
Self-citationGood Laboratory Practice: Preventing Introduction of Bias at the Bench
Macleod, M. R., Fisher, M., O'Collins, V., Sena, E. S., Dirnagl, U., Bath, P. M. W., Buchan, A., van der Worp, H. B., Traystman, R., Minematsu, K., et al. (2009) · Stroke · doi:10.1161/strokeaha.108.525386 · Cited by 331
Self-citationUsing prediction markets to estimate the reproducibility of scientific research
Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., & Johannesson, M. (2015) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1516179112 · Cited by 331
Macleod, M. R., van der Worp, H. B., Sena, E. S., Howells, D. W., Dirnagl, U., & Donnan, G. A. (2008) · Stroke · doi:10.1161/strokeaha.108.515957 · Cited by 319
Self-citationBaker, D., Lidster, K., Sottomayor, A., & Amor, S. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001756 · Cited by 305
Evaluation of excess significance bias in animal studies of neurological diseases
Tsilidis, K. K., Panagiotou, O. A., Sena, E. S., Aretouli, E., Evangelou, E., Howells, D. W., Salman, R. A.-S., Macleod, M. R., & Ioannidis, J. P. A. (2013) · PLoS Biology · doi:10.1371/journal.pbio.1001609 · Cited by 295 · cited 1×
“That is, the reports in the literature represent experiments where an effect was observed by chance but similar experiments where no effect was observed were never published (publication bias [8,9]).”
ResultSelf-citationTGN1412: From Discovery to Disaster
Attarwala, H. (2010) · Journal of Young Pharmacists · doi:10.4103/0975-1483.66810 · Cited by 253 · cited 1×
“of results between species (ie, from nonhuman primates to humans), specifically in the difference in mechanism of action of the study drug at the cellular level [34].”
BackgroundAnnouncement: Reducing our irreproducibility
(2013) · Nature · doi:10.1038/496398a · Cited by 243
The IMPROVE Guidelines (Ischaemia Models: Procedural Refinements Of in Vivo Experiments)
Percie du Sert, N., Alfieri, A., Allan, S. M., Carswell, H. V., Deuchar, G. A., Farr, T. D., Flecknell, P., Gallagher, L., Gibson, C. L., Haley, M. J., et al. (2017) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1177/0271678x17709185 · Cited by 190
Self-citationon behalf of the IICARus Collaboration, Hair, K., Macleod, M. R., & Sena, E. S. (2019) · Research Integrity and Peer Review · doi:10.1186/s41073-019-0069-3 · Cited by 174
Self-citationA long journey to reproducible results
Lithgow, G. J., Driscoll, M., & Phillips, P. (2017) · Nature · doi:10.1038/548387a · Cited by 157
Cardiac Stem Cell Treatment in Myocardial Infarction
Zwetsloot, P.-P. M., Végh, A. M. D., Lorkeers, S. J. J. O., van Hout, G. P. J., Currie, G. L., Sena, E. S., Gremmels, H., Buikema, J. W., Goumans, M., Macleod, M. R., et al. (2016) · Circulation Research · doi:10.1161/circresaha.115.307676 · Cited by 155
Self-citationLucanic, M., Plummer, W. T., Chen, E., Harke, J., Foulger, A. C., Onken, B., Coleman-Hulbert, A. L., Dumas, K. J., Guo, S., Johnson, E., et al. (2017) · Nature Communications · doi:10.1038/ncomms14256 · Cited by 140
Ramirez, F. D., Motazedian, P., Jung, R. G., Di Santo, P., MacDonald, Z. D., Moreland, R., Simard, T., Clancy, A. A., Russo, J. J., Welch, V. A., et al. (2017) · Circulation Research · doi:10.1161/circresaha.117.310628 · Cited by 138 · cited 1×
“of animal research in 4 other journals from the same publisher even when adjusted for the disease under study and the species of animal used [44].”
BackgroundReproducibility Crisis: Are We Ignoring Reaction Norms?
Voelkl, B., & Würbel, H. (2016) · Trends in Pharmacological Sciences · doi:10.1016/j.tips.2016.05.003 · Cited by 103
Did a change in Nature journals’ editorial policy for life sciences research improve reporting?
The NPQIP Collaborative group, Study steering committee, Macleod, M., Sena, E., Howells, D., Macleod, M., Sena, E., Howells, D., Kiermer, V., Swaminathan, S., et al. (2019) · BMJ Open Science · doi:10.1136/bmjos-2017-000035 · Cited by 102
Self-citationLack of blinding of outcome assessors in animal model experiments implies risk of observer bias
Bello, S., Krogsbøll, L. T., Gruber, J., Zhao, Z. J., Fischer, D., & Hróbjartsson, A. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.04.008 · Cited by 102
Hirst, T. C., Vesterinen, H. M., Sena, E. S., Egan, K. J., Macleod, M. R., & Whittle, I. R. (2013) · British Journal of Cancer · doi:10.1038/bjc.2012.504 · Cited by 101
Self-citationNo publication without confirmation
Mogil, J. S., & Macleod, M. R. (2017) · Nature · doi:10.1038/542409a · Cited by 77
Self-citationDopamine agonists in animal models of Parkinson’s disease: A systematic review and meta-analysis
Rooke, E. D., Vesterinen, H. M., Sena, E. S., Egan, K. J., & Macleod, M. R. (2011) · Parkinsonism & Related Disorders · doi:10.1016/j.parkreldis.2011.02.010 · Cited by 75
Self-citationCriminalization of scientific misconduct
Bülow, W., & Helgesson, G. (2018) · Medicine Health Care and Philosophy · doi:10.1007/s11019-018-9865-7 · Cited by 72
Replication Study: Fusobacterium nucleatum infection is prevalent in human colorectal carcinoma
Repass, J., Reproducibility Project: Cancer Biology, Iorns, E., Denis, A., Williams, S. R., Perfito, N., & Errington, T. M. (2018) · eLife · doi:10.7554/elife.25801 · Cited by 63
Guidance for the Description of Animal Research in Scientific Publications
National Research Council (2011) · National Academies Press eBooks · doi:10.17226/13241 · Cited by 60
Minnerup, J., Zentsch, V., Schmidt, A., Fisher, M., & Schäbitz, W. (2015) · Stroke · doi:10.1161/strokeaha.115.011695 · Cited by 38
Research integrity is much more than misconduct
(2019) · Nature · doi:10.1038/d41586-019-01727-0 · Cited by 33
Hair, K., Macleod, M. R., Sena, E., et al. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/370874 · Cited by 24
Self-citationLearning lessons from MVA85A, a failed booster vaccine for BCG
Macleod, M. R. (2018) · BMJ · doi:10.1136/bmj.k66 · Cited by 21
Self-citationWang, Q., Liao, J., Hair, K., Bannach‐Brown, A., Bahor, Z., Currie, G. L., McCann, S., Howells, D. W., Sena, E. S., & Macleod, M. R. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/256776 · Cited by 14
Self-citationProtocols for laboratory research
Macleod, M. R., & Howells, D. J. (2016) · Evidence-based Preclinical Medicine · doi:10.1002/ebm2.21 · Cited by 7
Self-citationThe Cochrane Collaboration 20 years in
Grimshaw, J., Craig, J. C., Tovey, D. I., & Wilson, M. (2013) · Canadian Medical Association Journal · doi:10.1503/cmaj.131251 · Cited by 5
The reproducibility opportunity
Macleod, M. R. (2018) · Nature Human Behaviour · doi:10.1038/s41562-018-0398-0 · Cited by 4
Self-citation
Research works printed but not linked: 3
- Haywood, 2015, Letter re: Principles and Guidelines for Reporting Preclinical Research
- Macleod, 2015, Risk of bias in reports of in vivo research: A focus for improvement, PLoS Biol., 13, 10.1371/journal.pbio.1002273 · DOI · Paper page
- 2016, Enhancing Research Reproducibility: Recommendations from the Federation of American Societies for Experimental Biology
Cited by
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 462
What Research Institutions Can Do to Foster Research Integrity
Bouter, L. M. (2020) · Science and Engineering Ethics · doi:10.1007/s11948-020-00178-5 · Cited by 110
Kahnau, P., Mieske, P., Wilzopolski, J., Kalliokoski, O., Mandillo, S., Hölter, S. M., Võikar, V., Amfim, A., Badurek, S., Bartelik, A., et al. (2023) · BMC Biology · doi:10.1186/s12915-023-01751-7 · Cited by 91
Eckrich, J., Kugler, P., Buhr, C. R., Ernst, B. P., Mendler, S., Baumgart, J., Brieger, J., & Wiesmann, N. (2020) · Scientific Reports · doi:10.1038/s41598-020-75660-y · Cited by 79
Increasing the statistical power of animal experiments with historical control data
Bonapersona, V., Hoijtink, H., RELACS, Sarabdjitsingh, R. A., & Joëls, M. (2021) · Nature Neuroscience · doi:10.1038/s41593-020-00792-3 · Cited by 88
Metadata
- Authors
- Macleod, M., & Mohan, S.
- Affiliations
- University of Edinburgh (GB)
- National Institutes of Health (US)
- Corresponding author
- Macleod, Malcolm
- ORCID
- Macleod, Malcolm
- Venue
- ILAR Journal
- Publisher
- Oxford University Press (OUP)
- Published
- 2019
- Volume
- 60
- Issue
- 1
- Pages
- 17-23
- ISSN
- 1084-2020 (ISSN-L)
- 1930-6180
- PMC
- PMC7275809
- OpenAlex
- W2983895890
- Cited by
- 104
- References
- 48
- Citations by year
- 2026: 5 · 2025: 10 · 2024: 23 · 2023: 21 · 2022: 14 · 2021: 17 · 2020: 12 · 2019: 2
- FWCI
- 14.83
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ILAR Journal (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, other-oa)
- Funders
- National Institutes of Health
- Topics
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Social Sciences › Library and Information Sciences › Academic Publishing and Open Access
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- research reproducibility, animal experimentation, in vivo research, statistical significance, generalizability of findings, replication study, study design flaws, reporting quality, preregistration, open data, open science, research training, translational research
- MeSH terms
- Animals, Research Design, Animal Experimentation, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.