Threats to Validity in the Design and Conduct of Preclinical Efficacy Studies: A Systematic Review of Guidelines for In Vivo Animal Experiments
Open accessThe DOI matches its Crossref record; 65 of the 65 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The vast majority of medical interventions introduced into clinical development prove unsafe or ineffective. One prominent explanation for the dismal success rate is flawed preclinical research. We conducted a systematic review of preclinical research guidelines and organized recommendations according to the type of validity threat (internal, construct, or external) or programmatic research activity they primarily address. METHODS AND FINDINGS: We searched MEDLINE, Google Scholar, Google, and the EQUATOR Network website for all preclinical guideline documents published up to April 9, 2013 that addressed the design and conduct of in vivo animal experiments aimed at supporting clinical translation. To be eligible, documents had to provide guidance on the design or execution of preclinical animal experiments and represent the aggregated consensus of four or more investigators. Data from included guidelines were independently extracted by two individuals for discrete recommendations on the design and implementation of preclinical efficacy studies. These recommendations were then organized according to the type of validity threat they addressed. A total of 2,029 citations were identified through our search strategy. From these, we identified 26 guidelines that met our eligibility criteria--most of which were directed at neurological or cerebrovascular drug development. Together, these guidelines offered 55 different recommendations. Some of the most common recommendations included performance of a power calculation to determine sample size, randomized treatment allocation, and characterization of disease phenotype in the animal model prior to experimentation. CONCLUSIONS: By identifying the most recurrent recommendations among preclinical guidelines, we provide a starting point for developing preclinical guidelines in other disease domains. We also provide a basis for the study and evaluation of preclinical research practice. Please see later in the article for the Editors' Summary.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2002) · Cited by 13,211
Improving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · PLoS Biology · doi:10.1371/journal.pbio.1000412 · Cited by 7,300
Bellomo, R., Ronco, C., Kellum, J. A., Mehta, R. L., Palevsky, P., & workgroup, T. A. (2004) · Critical Care · doi:10.1186/cc2872 · Cited by 6,781
Improving bioscience research reporting: The ARRIVE guidelines for reporting animal research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · Journal of Pharmacology and Pharmacotherapeutics · doi:10.4103/0976-500x.72351 · Cited by 6,277 · cited 1×
“These include the establishment of preclinical data repositories [20], minimum reporting checklists for biomedical investigations [21], biomedical data ontologies [22], and reporting standards for animal studies [15].”
BackgroundCan the pharmaceutical industry reduce attrition rates?
Kola, I., & Landis, J. (2004) · Nature Reviews Drug Discovery · doi:10.1038/nrd1470 · Cited by 4,167
AGREE II: advancing guideline development, reporting and evaluation in health care
Brouwers, M. C., Kho, M. E., Browman, G. P., Burgers, J. S., Cluzeau, F., Feder, G., Fervers, B., Graham, I. D., Grimshaw, J., Hanna, S. E., et al. (2010) · Canadian Medical Association Journal · doi:10.1503/cmaj.090449 · Cited by 3,790 · cited 1×
“In the absence of wellcharacterized quality parameters for preclinical guideline documents (such as the AGREE II instrument for clinical guideline evaluation [16]), we did not include or exclude guidelines based on a quality score.”
BackgroundSelf-citationAnimal research: Reporting in vivo experiments: The ARRIVE guidelines
Kilkenny, C., Browne, W., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · British Journal of Pharmacology · doi:10.1111/j.1476-5381.2010.00872.x · Cited by 3,625
Clinical Practice Guidelines We Can Trust
Committee on Standards for Developing Trustworthy Clinical Practice Guidelines, Board on Health Care Services, & Institute of Medicine (2011) · National Academies Press eBooks · doi:10.17226/13058 · Cited by 3,081
Raise standards for preclinical cancer research
Begley, C. G., & Ellis, L. M. (2012) · Nature · doi:10.1038/483531a · Cited by 2,979 · cited 2×
“Various commentators have reported problems such as difficulty in replicating preclinical studies [5,6], publication bias [7], and the prevalence of methodological practices that result in threats to validity [8].”
BackgroundThe OBO Foundry: coordinated evolution of ontologies to support biomedical data integration
The OBI Consortium, Smith, B., Ashburner, M., Rosse, C., Bard, J., Bug, W., Ceusters, W., Goldberg, L. J., Eilbeck, K., Ireland, A., et al. (2007) · Nature Biotechnology · doi:10.1038/nbt1346 · Cited by 2,560
Cited by
Organs-on-chips at the frontiers of drug discovery
Esch, E. W., Bahinski, A., & Huh, D. (2015) · Nature Reviews Drug Discovery · doi:10.1038/nrd4539 · Cited by 1,185
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035
Animal models of human disease: challenges in enabling translation
McGonigle, P., & Ruggeri, B. (2014) · Biochemical Pharmacology · doi:10.1016/j.bcp.2013.08.006 · Cited by 610
Meta-analysis of data from animal studies: A practical guide
Vesterinen, H. M., Sena, E. S., Egan, K. J., Hirst, T. C., Churolov, L., Currie, G. L., Antonic, A., Howells, D. W., & Macleod, M. R. (2014) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2013.09.010 · Cited by 488
Pound, P., & Ritskes-Hoitinga, M. (2018) · Journal of Translational Medicine · doi:10.1186/s12967-018-1678-1 · Cited by 419
The need for randomization in animal trials: an overview of systematic reviews
Hirst, J. A., Howick, J., Aronson, J. K., Roberts, N., Perera, R., Koshiaris, C., & Heneghan, C. (2014) · PLoS ONE · doi:10.1371/journal.pone.0098856 · Cited by 294
“[61] The Animal Research: Reporting In Vivo Experiments (ARRIVE) guidelines, developed in 2010, [62] arguably constitute the leading candidate for becoming a requirement, although development work in this area continues [63].”
BackgroundAkyurekli, C., Le, Y., Richardson, R. B., Fergusson, D., Tay, J., & Allan, D. S. (2015) · Stem Cell Reviews and Reports · doi:10.1007/s12015-014-9545-9 · Cited by 285
Self-citationWhen Quality Beats Quantity: Decision Theory, Drug Discovery, and the Reproducibility Crisis
Scannell, J. W., & Bosley, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0147215 · Cited by 279
“We use the term PV because general terms such as “validity” and “validation” have a range of different meanings in the biomedical literature (see, for example: [66] [53] [67] [68] [69] [70] [71]).”
MethodSena, E. S., Currie, G. L., McCann, S. K., Macleod, M. R., & Howells, D. W. (2014) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.2014.28 · Cited by 264
Distinguishing between exploratory and confirmatory preclinical research will improve translation
Kimmelman, J., Mogil, J. S., & Dirnagl, U. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001863 · Cited by 236
“, calls for using larger sample sizes, ‘‘gold standard’’ animal models, or independent replication [9,10].”
MethodSelf-citationIs animal research sufficiently evidence based to be a cornerstone of biomedical research?
Pound, P., & Bracken, M. B. (2014) · BMJ · doi:10.1136/bmj.g3387 · Cited by 207
Ioannidis, J. P. A., Kim, B. Y. S., & Trounson, A. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0314-y · Cited by 175
Hooijmans, C. R., de Vries, R. B. M., Ritskes-Hoitinga, M., Rovers, M. M., Leeflang, M. M., IntHout, J., Wever, K. E., Hooft, L., de Beer, H., Kuijpers, T., et al. (2018) · PLOS ONE · doi:10.1371/journal.pone.0187271 · Cited by 176
“This list is largely based on the work by Henderson et al who identified threats to external/construct validity in preclinical research guidelines [47] and supplemented by our own findings.”
Methodde Vries, R. B. M., Wever, K. E., Avey, M. T., Stephens, M. L., Sena, E. S., & Leenaars, M. (2014) · ILAR Journal · doi:10.1093/ilar/ilu043 · Cited by 165
Ramirez, F. D., Motazedian, P., Jung, R. G., Di Santo, P., MacDonald, Z. D., Moreland, R., Simard, T., Clancy, A. A., Russo, J. J., Welch, V. A., et al. (2017) · Circulation Research · doi:10.1161/circresaha.117.310628 · Cited by 138
The devil is in the details: incomplete reporting in preclinical animal research
Avey, M. T., Moher, D., Sullivan, K. J., Fergusson, D., Griffin, G., Grimshaw, J. M., Hutton, B., Lalu, M. M., Macleod, M., Marshall, J., et al. (2016) · PLOS ONE · doi:10.1371/journal.pone.0166733 · Cited by 130
Self-citationHolman, C., Piper, S. K., Grittner, U., Diamantaras, A. A., Kimmelman, J., Siegerink, B., & Dirnagl, U. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002331 · Cited by 114
“Neither are routinely practiced in preclinical research, although interest in strategies for dealing with missing data is growing [18,26,27].”
BackgroundSelf-citationSamuel, G. O., Hoffmann, S., Wright, R. A., Lalu, M. M., Patlewicz, G., Becker, R. A., DeGeorge, G. L., Fergusson, D., Hartung, T., Lewis, R. J., et al. (2016) · Environment International · doi:10.1016/j.envint.2016.03.010 · Cited by 82
Self-citationThe researchers’ view of scientific rigor—survey on the conduct and reporting of in vivo research
Reichlin, T. S., Vogt, L., & Würbel, H. (2016) · PLOS ONE · doi:10.1371/journal.pone.0165999 · Cited by 72
Preclinical efficacy studies in investigator brochures: do they enable risk–benefit assessment?
Wieschowski, S., Chin, W. W. L., Federico, C., Sievers, S., Kimmelman, J., & Strech, D. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2004879 · Cited by 70
Self-citationOptimized design and analysis of preclinical intervention studies in vivo
Laajala, T. D., Jumppanen, M., Huhtaniemi, R., Fey, V., Kaur, A., Knuuttila, M., Aho, E., Oksala, R., Westermarck, J., Mäkelä, S., et al. (2016) · Scientific Reports · doi:10.1038/srep30723 · Cited by 48
Enhancing reproducibility: Failures from Reproducibility Initiatives underline core challenges
Mullane, K., & Williams, M. (2017) · Biochemical Pharmacology · doi:10.1016/j.bcp.2017.04.008 · Cited by 28
(2015) · PubMed Central · Cited by 20
Initial heritable genome editing: mapping a responsible pathway from basic research to the clinic
Ranisch, R., Trettenbach, K., & Arnason, G. (2023) · Medicine, Health Care and Philosophy · doi:10.1007/s11019-022-10115-x · Cited by 19
“In 2013, Henderson, Kimmelman, and colleagues proposed credibility alongside internal, construct, and external validity as components of a four-part framework to minimise uncertainty between different steps in translational research (Henderson et al. 2013).”
BackgroundMartos, D., Lőrinczi, B., Szatmári, I., Vécsei, L., & Tanaka, M. (2025) · doi:10.20944/preprints202504.1511.v2 · Cited by 13
Metadata
- Authors
- Henderson, V. C., Kimmelman, J., Fergusson, D., Grimshaw, J. M., & Hackam, D. G.
- Venue
- PLoS Medicine
- Publisher
- Public Library of Science (PLoS)
- Published
- 2013-07-23
- Volume
- 10
- Issue
- 7
- Pages
- e1001489
- ISSN
- 1549-1277 (ISSN-L)
- 1549-1676
- OpenAlex
- W1977680690
- Cited by
- 302
- References
- 65
- FWCI
- 6.59
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.