Survey of the Quality of Experimental Design, Statistical Analysis and Reporting of Research Using Animals
Open accessAbstract
For scientific, ethical and economic reasons, experiments involving animals should be appropriately designed, correctly analysed and transparently reported. This increases the scientific validity of the results, and maximises the knowledge gained from each experiment. A minimum amount of relevant information must be included in scientific publications to ensure that the methods and results of a study can be reviewed, analysed and repeated. Omitting essential information can raise scientific and ethical concerns. We report the findings of a systematic survey of reporting, experimental design and statistical analysis in published biomedical research using laboratory animals. Medline and EMBASE were searched for studies reporting research on live rats, mice and non-human primates carried out in UK and US publicly funded research establishments. Detailed information was collected from 271 publications, about the objective or hypothesis of the study, the number, sex, age and/or weight of animals used, and experimental and statistical methods. Only 59% of the studies stated the hypothesis or objective of the study and the number and characteristics of the animals used. Appropriate and efficient experimental design is a critical component of high-quality science. Most of the papers surveyed did not use randomisation (87%) or blinding (86%), to reduce bias in animal selection and outcome assessment. Only 70% of the publications that used statistical methods described their methods and presented the results with a measure of error or variability. This survey has identified a number of issues that need to be addressed in order to improve experimental design and reporting in publications describing research using animals. Scientific publication is a powerful and important source of information; the authors of scientific publications therefore have a responsibility to describe their methods and results comprehensively, accurately and transparently, and peer reviewers and journal editors share the responsibility to ensure that published studies fulfil these criteria.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In toxicology, awareness of the reproducibility problem emerged gradually in association with insufficient in vivo reproducibility.
“In toxicology, which may better reflect the background of most readers of this journal, awareness of this problem has emerged only gradually in association with insufficient in vivo reproducibility (Kilkenny et al., 2009; Voelkl et al., 2018).”
From In Vitro research Reproducibility: Keeping up High Standards (2019) · cites this paper as Kilkenny et al., 2009
The Discussion (Statistical Analysis) identifies reporting and analysis problems in animal studies that have implications for the reproducibility of results. The Conclusion also treats reproducibility as a concern. The paper does not establish that awareness emerged gradually in toxicology or document insufficient reproducibility of in vivo findings directly. (LLM, conf=0.83)
Experimental — accuracy not yet measuredEvidence shows that the majority of publications fail to include key information and there is significant scope to improve the reporting of studies involving animal research.
“However, evidence shows that the majority of publications fail to include key information and there is significant scope to improve the reporting of studies involving animal research [1–4].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [1]
Randomization and blinding of investigators who analyse the results of animal experiments are used in less than 20% of studies.
“Some pivotal aspects of study design, such as randomization and the blinding of investigators who analyse the results of animal experiments, are used in less than 20% of studies16–18”
From How to design preclinical studies in nanomedicine and cell therapy to maximize the prospects of clinical translation (2018) · cites this paper as [17]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · The Lancet · doi:10.1016/s0140-6736(07)61602-x · Cited by 21,320 · see citing works
Self-citationThe Principles of Humane Experimental Technique
(1960) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.1960.tb73127.x · Cited by 5,002 · see citing works
Effect size, confidence interval and statistical significance: a practical guide for biologists
Nakagawa, S., & Cuthill, I. C. (2007) · Biological Reviews · doi:10.1111/j.1469-185x.2007.00027.x · Cited by 3,835 · see citing works
Self-citationThe Revised CONSORT Statement for Reporting Randomized Trials: Explanation and Elaboration
Altman, D. G., Schulz, K. F., Moher, D., Egger, M., Davidoff, F., Elbourne, D., Gøtzsche, P. C., Lang, T., & for the CONSORT Group (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-8-200104170-00012 · Cited by 3,835 · see citing works
Self-citationGuidelines for the Design and Statistical Analysis of Experiments Using Laboratory Animals
Festing, M. F. W., & Altman, D. G. (2002) · ILAR Journal · doi:10.1093/ilar.43.4.244 · Cited by 1,110 · see citing works
Self-citationDell, R. B., Holleran, S., & Ramakrishnan, R. (2002) · ILAR Journal · doi:10.1093/ilar.43.4.207 · Cited by 746 · see citing works
Proposed minimum reporting standards for data analysis in metabolomics
Goodacre, R., Broadhurst, D., Smilde, A. K., Kristal, B. S., Baker, J. D., Beger, R. D., Bessant, C., Connor, S. C., Capuani, G., Craig, A., et al. (2007) · Metabolomics · doi:10.1007/s11306-007-0081-3 · Cited by 443 · see citing works
Quality of reporting of observational longitudinal research.
Tooth, L. (2005) · American Journal of Epidemiology · doi:10.1093/aje/kwi042 · Cited by 363 · see citing works
Good Laboratory Practice: Preventing Introduction of Bias at the Bench
Macleod, M. R., Fisher, M., O'Collins, V., Sena, E. S., Dirnagl, U., Bath, P. M. W., Buchan, A., van der Worp, H. B., Traystman, R., Minematsu, K., et al. (2009) · Stroke · doi:10.1161/strokeaha.108.525386 · Cited by 331 · see citing works
EQUATOR: reporting guidelines for health research
Altman, D. G., Simera, I., Hoey, J., Moher, D., & Schulz, K. (2008) · The Lancet · doi:10.1016/s0140-6736(08)60505-x · Cited by 325 · see citing works
Self-citationMacleod, M. R., van der Worp, H. B., Sena, E. S., Howells, D. W., Dirnagl, U., & Donnan, G. A. (2008) · Stroke · doi:10.1161/strokeaha.108.515957 · Cited by 319 · see citing works
Emergency medicine animal research: does use of randomization and blinding affect the results?
Bebarta, V., Luyten, D., & Heard, K. (2003) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.2003.tb00056.x · Cited by 244 · see citing works
Hainsworth, A. H., & Markus, H. S. (2008) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.2008.91 · Cited by 240 · see citing works
EQUATOR: reporting guidelines for health research.
Altman, D. G., Simera, I., Hoey, J., Moher, D., & Schulz, K. (2008) · PubMed Central · Cited by 224 · see citing works
Self-citationAltman, D. G. (2005) · BMJ · doi:10.1136/bmj.330.7499.1056 · Cited by 178 · see citing works
Self-citationPractical Aspects of Experimental Design in Animal Research
Johnson, P. D., & Besselsen, D. G. (2002) · ILAR Journal · doi:10.1093/ilar.43.4.202 · Cited by 161 · see citing works
Jafari, P., & Azuaje, F. J. (2006) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-6-27 · Cited by 156 · see citing works
Festing, M., Overend, P., Das, R., Cortina‐Borja, M., & Berdoy, M. (2002) · Cited by 140 · see citing works
Self-citationQuality of Reporting of Diagnostic Accuracy Studies
Smidt, N., Rutjes, A. W., van der Windt, D., Ostelo, R., Reitsma, J. B., Bossuyt, P. M., Bouter, L. M., & de Vet, H. C. W. (2005) · Radiology · doi:10.1148/radiol.2352040507 · Cited by 138 · see citing works
Stone, S. P., Cooper, B. S., Kibbler, C. C., Cookson, B., Roberts, J. A., Medley, G. F., Duckworth, G. J., Lai, R., Ebrahim, S. B. J., Brown, E. M., et al. (2007) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dkm055 · Cited by 130 · see citing works
Morris, C. E., Bardin, M., Berge, O., Frey‐Klett, P., Fromin, N., Girardin, H., Guinebretière, M., Lebaron, P., Thiery, J. M., & Troussellier, M. (2002) · Microbiology and Molecular Biology Reviews · doi:10.1128/mmbr.66.4.592-616.2002 · Cited by 122 · see citing works
Peters, J., Sutton, A. J., Jones, D. R. H., Rushton, L., & Abrams, K. R. (2006) · Journal of Environmental Science and Health Part B · doi:10.1080/03601230600857130 · Cited by 94 · see citing works
Use of factorial designs to optimize animal experiments and reduce animal use.
Shaw, R. J., Festing, M., Peers, I., & Furlong, L. (2002) · ILAR journal · doi:10.1093/ilar.43.4.223 · Cited by 93 · see citing works
Self-citationAssessment of ecological effects due to forest harvesting: approaches and statistical issues
Bennett, L. T., & Adams, M. A. (2004) · Journal of Applied Ecology · doi:10.1111/j.0021-8901.2004.00924.x · Cited by 86 · see citing works
Ellery, A. W. (1985) · Laboratory Animals · doi:10.1258/002367785780942714 · Cited by 64 · see citing works
Assessment of statistical procedures used in papers in the Australian Veterinary Journal
McCance, I. (1995) · Australian Veterinary Journal · doi:10.1111/j.1751-0813.1995.tb07534.x · Cited by 53 · see citing works
The Design of Animal Experiments
Festing, M. F. W. (2010) · doi:10.1002/9781444318777.ch3 · Cited by 49 · see citing works
Self-citationReporting animal use in scientific papers
Smith, J. A., Birke, L. I. A., & Sadler, D. (1997) · Laboratory Animals · doi:10.1258/002367797780596176 · Cited by 45 · see citing works
Animal definition: a necessity for the validity of animal experiments?
Örink, K. J., & Rehbinder, C. (2000) · Laboratory Animals · doi:10.1258/002367700780457608 · Cited by 42 · see citing works
Alfaro, V. (2005) · Methods and Findings in Experimental and Clinical Pharmacology · doi:10.1358/mf.2005.27.7.921309 · Cited by 28 · see citing works
Freeman, J. V., Walters, S. J., & Campbell, M. J. (2008) · Cited by 25 · see citing works
The History of the Measurement of III health
Cochrane, A. L. (1972) · International Journal of Epidemiology · doi:10.1093/ije/1.2.89 · Cited by 22 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Improving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · PLoS Biology · doi:10.1371/journal.pbio.1000412 · Cited by 7,300 · see citing works
Self-citationThe ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000410 · Cited by 5,939 · see citing works
Self-citationSYRCLE’s risk of bias tool for animal studies
Hooijmans, C. R., Rovers, M. M., de Vries, R. B., Leenaars, M., Ritskes-Hoitinga, M., & Langendam, M. W. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-43 · Cited by 4,415 · see citing works
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397 · see citing works
Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000411 · Cited by 2,677 · see citing works
Self-citationReducing waste from incomplete or unusable reports of biomedical research
Glasziou, P., Altman, D. G., Bossuyt, P., Boutron, I., Clarke, M., Julious, S., Michie, S., Moher, D., & Wager, E. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62228-x · Cited by 1,319 · see citing works
Self-citationA call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196 · see citing works
Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2012) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2012.02.010 · Cited by 728 · see citing works
Self-citationIncreasing value and reducing waste in biomedical research: who's listening?
Moher, D., Glasziou, P., Chalmers, I., Nasser, M., Bossuyt, P. M. M., Korevaar, D. A., Graham, I. D., Ravaud, P., & Boutron, I. (2016) · The Lancet · doi:10.1016/s0140-6736(15)00307-4 · Cited by 462 · see citing works
The problem of pseudoreplication in neuroscientific studies: is it affecting your analysis?
Lazic, S. E. (2010) · BMC Neuroscience · doi:10.1186/1471-2202-11-5 · Cited by 381 · see citing works
What exactly is ‘N’ in cell culture and animal experiments?
Lazic, S. E., Clarke-Williams, C. J., & Munafò, M. R. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2005282 · Cited by 291 · see citing works
Tractenberg, S. G., Levandowski, M. L., de Azeredo, L. A., Orso, R., Roithmann, L. G., Hoffmann, E. S., Brenhouse, H., & Grassi-Oliveira, R. (2016) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2016.06.021 · Cited by 267 · see citing works
Vasilevsky, N. A., Brush, M. H., Paddock, H., Ponting, L., Tripathy, S. J., LaRocca, G. M., & Haendel, M. A. (2013) · PeerJ · doi:10.7717/peerj.148 · Cited by 271 · see citing works
Evidence of experimental bias in the life sciences: why we need blind data recording
Holman, L., Head, M. L., Lanfear, R., & Jennions, M. D. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002190 · Cited by 263 · see citing works
Responsible Conduct of Research
Shamoo, A. E., & Resnik, D. B. (2022) · doi:10.1093/oso/9780197547090.001.0001 · Cited by 224 · see citing works
Is animal research sufficiently evidence based to be a cornerstone of biomedical research?
Pound, P., & Bracken, M. B. (2014) · BMJ · doi:10.1136/bmj.g3387 · Cited by 207 · see citing works
An analysis of the use of animal models in predicting human toxicology and drug safety
Bailey, J., Thew, M., & Balls, M. (2014) · Alternatives to Laboratory Animals · doi:10.1177/026119291404200306 · Cited by 182 · see citing works
on behalf of the IICARus Collaboration, Hair, K., Macleod, M. R., & Sena, E. S. (2019) · Research Integrity and Peer Review · doi:10.1186/s41073-019-0069-3 · Cited by 174 · see citing works
Ioannidis, J. P. A., Kim, B. Y. S., & Trounson, A. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0314-y · Cited by 175 · see citing works
Leung, V., Rousseau-Blass, F., Beauchamp, G., & Pang, D. S. J. (2018) · PLOS ONE · doi:10.1371/journal.pone.0197882 · Cited by 172 · see citing works
Hooijmans, C. R., de Vries, R. B. M., Ritskes-Hoitinga, M., Rovers, M. M., Leeflang, M. M., IntHout, J., Wever, K. E., Hooft, L., de Beer, H., Kuijpers, T., et al. (2018) · PLOS ONE · doi:10.1371/journal.pone.0187271 · Cited by 176 · see citing works
In Vitro research Reproducibility: Keeping up High Standards
Hirsch, C., & Schildknecht, S. (2019) · Frontiers in Pharmacology · doi:10.3389/fphar.2019.01484 · Cited by 161 · see citing works
Toward more predictive genetic mouse models of Alzheimer’s disease
Onos, K. D., Sukoff Rizzo, S. J., Howell, G. R., & Sasner, M. (2016) · Brain Research Bulletin · doi:10.1016/j.brainresbull.2015.12.003 · Cited by 156 · see citing works
Ramirez, F. D., Motazedian, P., Jung, R. G., Di Santo, P., MacDonald, Z. D., Moreland, R., Simard, T., Clancy, A. A., Russo, J. J., Welch, V. A., et al. (2017) · Circulation Research · doi:10.1161/circresaha.117.310628 · Cited by 138 · see citing works
The Experimental Design Assistant
Percie du Sert, N., Bamsey, I., Bate, S. T., Berdoy, M., Clark, R. A., Cuthill, I., Fry, D., Karp, N. A., Macleod, M., Moon, L., et al. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2003779 · Cited by 118 · see citing works
Self-citationWhole animal experiments should be more like human randomized controlled trials
Muhlhausler, B. S., Bloomfield, F. H., & Gillman, M. W. (2013) · PLoS Biology · doi:10.1371/journal.pbio.1001481 · Cited by 105 · see citing works
Banik, A., Brown, R. E., Bamburg, J., Lahiri, D. K., Khurana, D., Friedland, R. P., Chen, W., Ding, Y., Mudher, A., Padjen, A. L., et al. (2015) · Journal of Alzheimer’s Disease · doi:10.3233/jad-150136 · Cited by 102 · see citing works
Improving bioscience research reporting: ARRIVE-ing at a solution
Kilkenny, C., & Altman, D. G. (2010) · Laboratory Animals · doi:10.1258/la.2010.0010021 · Cited by 103 · see citing works
Self-citationThe Rigor and Transparency Index Quality Metric for Assessing Biological and Medical Science Methods
Menke, J., Roelandse, M., Ozyurt, B., Martone, M., & Bandrowski, A. (2020) · iScience · doi:10.1016/j.isci.2020.101698 · Cited by 90 · see citing works
Improving transparency and scientific rigor in academic publishing
Prager, E. M., Chambers, K. E., Plotkin, J. L., McArthur, D. L., Bandrowski, A. E., Bansal, N., Martone, M. E., Bergstrom, H. C., Bespalov, A., & Graf, C. (2019) · Journal of Neuroscience Research · doi:10.1002/jnr.24340 · Cited by 83 · see citing works
An analysis of the use of dogs in predicting human toxicology and drug safety
Bailey, J., Thew, M., & Balls, M. (2013) · Alternatives to Laboratory Animals · doi:10.1177/026119291304100504 · Cited by 71 · see citing works
Gulin, J. E. N., Rocco, D. M., & García-Bournissen, F. (2015) · PLOS Neglected Tropical Diseases · doi:10.1371/journal.pntd.0004194 · Cited by 66 · see citing works
Loss, C. M., Melleu, F. F., Domingues, K., Lino-de-Oliveira, C., & Viola, G. G. (2021) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2021.763428 · Cited by 44 · see citing works
A review of the literature on ethical issues related to scientific authorship
Hosseini, M., & Gordijn, B. (2020) · Accountability in Research · doi:10.1080/08989621.2020.1750957 · Cited by 38 · see citing works
Statistical Design Considerations in Animal Studies Published Recently in <i>Cancer Research</i>
Hess, K. R. (2011) · Cancer Research · doi:10.1158/0008-5472.can-10-3296 · Cited by 36 · see citing works
Brubaker, M., Kapadia, S., Maletic, V., Chambers, J., Watling, C., & Citrome, L. (2025) · Neuropsychiatric Disease and Treatment · doi:10.2147/ndt.s501207 · Cited by 7 · see citing works
- Authors
- Kilkenny, C., Parsons, N., Kadyszewski, E., Festing, M. F. W., Cuthill, I. C., Fry, D., Hutton, J., & Altman, D. G.
- Affiliations
- National Centre for the Replacement Refinement and Reduction of Animals in Research (GB)
- University of Warwick (GB)
- University of Bristol (GB)
- Home Office (GB)
- University of Oxford (GB)
- Corresponding author
- Kilkenny, Carol
- Venue
- PLoS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2009-11-30
- Volume
- 4
- Issue
- 11
- Pages
- e7824
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC2779358
- OpenAlex
- W2042190008
- Cited by
- 860
- References
- 32
- Citations by year
- 2026: 19 · 2025: 51 · 2024: 41 · 2023: 43 · 2022: 69 · 2021: 66 · 2020: 69 · 2019: 51 · 2018: 62 · 2017: 67 · 2016: 58 · 2015: 62 · 2014: 57 · 2013: 42 · 2012: 36
- FWCI
- 50.77
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS ONE (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Bristol Research (University of Bristol) (publishedVersion)
- Warwick Research Archive Portal (University of Warwick) (acceptedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- National Centre for the Replacement, Refinement and Reduction of Animals in Research
- National Institutes of Health
- Topics
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- animal studies, experimental design, statistical analysis, reporting quality, randomization, blinding, bias mitigation, sample size, laboratory animals, rats, mice, non-human primates, scientific validity, peer review, reproducibility
- MeSH terms
- Animals, Disease Models, Animal, Female, United Kingdom, Humans, Male, Periodicals as Topic, Research, Research Design, United States, MEDLINE, Peer Review, Research, Mice, Rats
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
