Risk of bias in reports of in vivo research: a focus for improvement
Open accessAbstract
The reliability of experimental findings depends on the rigour of experimental design. Here we show limited reporting of measures to reduce the risk of bias in a random sample of life sciences publications, significantly lower reporting of randomisation in work published in journals of high impact, and very limited reporting of measures to reduce the risk of bias in publications from leading United Kingdom institutions. Ascertainment of differences between institutions might serve both as a measure of research quality and as a tool for institutional efforts to improve research quality.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Randomisation is reported in only 30%-40% of publications.
“randomisation (reported in only 30%–40% of publications)”
From The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research (2020) · cites this paper as [11]
The paper reports randomisation in 35.7% of CAMARADES publications from 2009–2010, which falls within the claimed range. But the Abstract also reports lower overall rates in its datasets (20%, 24.8%, and 14.4%), so 30%–40% is not supported as an unqualified rate for publications generally. (LLM, conf=0.88)
Experimental — accuracy not yet measuredEvidence shows that the majority of publications fail to include key information and there is significant scope to improve the reporting of studies involving animal research.
“However, evidence shows that the majority of publications fail to include key information and there is significant scope to improve the reporting of studies involving animal research [1–4].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [4]
Important information as set out in the ARRIVE guidelines is still missing from most publications sampled.
“However, despite this level of support, recent studies have shown that important information as set out in the ARRIVE guidelines is still missing from most publications sampled.”
From The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research (2020) · cites this paper as [11]
Blinding is reported in only approximately 20% of publications.
“blinding (reported in only approximately 20% of publications)”
From The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research (2020) · cites this paper as [11]
Sample size justification is reported in less than 10% of publications.
“sample size justification (reported in less than 10% of publications)”
From The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research (2020) · cites this paper as [11]
All basic animal characteristics are reported in less than 10% of publications.
“animal characteristics (all basic characteristics reported in less than 10% of publications)”
From The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research (2020) · cites this paper as [11]
The paper’s Abstract (page unavailable) reports how often publications described randomisation, blinded outcome assessment, sample-size calculations and conflicts of interest. Searches for animal characteristics, including species, strain, sex, age and weight, found no passage giving a figure for publications reporting all basic characteristics. The less-than-10% claim therefore cannot be verified from our copy. (LLM, conf=0.72)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196 · see citing works
Self-citationEvaluation of excess significance bias in animal studies of neurological diseases
Tsilidis, K. K., Panagiotou, O. A., Sena, E. S., Aretouli, E., Evangelou, E., Howells, D. W., Salman, R. A.-S., Macleod, M. R., & Ioannidis, J. P. A. (2013) · PLoS Biology · doi:10.1371/journal.pbio.1001609 · Cited by 295 · see citing works
Self-citationThe need for randomization in animal trials: an overview of systematic reviews
Hirst, J. A., Howick, J., Aronson, J. K., Roberts, N., Perera, R., Koshiaris, C., & Heneghan, C. (2014) · PLoS ONE · doi:10.1371/journal.pone.0098856 · Cited by 294 · see citing works
Announcement: Reducing our irreproducibility
(2013) · Nature · doi:10.1038/496398a · Cited by 243 · see citing works
Abdel‐Sattar, M., Krauth, D., Anglemyer, A., & Bero, L. (2014) · Evidence-based Preclinical Medicine · doi:10.1002/ebm2.5 · Cited by 14 · see citing works
Flagged first, then by citations
Research works printed but not linked: 28
- [2]van der Worp HB, Howells DW, Sena ES, Porritt MJ, Rewell S, O'Collins V, et al Can animal models of disease reliably inform human studies? PLoS Med 2010 3 30; 7(3): e1000245 10.1371/journal.pmed.1000245 20361020 PMC2846855 · DOI · Paper page
- [3]Crossley NA, Sena E, Goehler J, Horn J, van der Worp B, Bath PMW, et al Empirical Evidence of Bias in the Design of Experimental Stroke Studies. A Metaepidemiologic Approach. Stroke 2008 1 31; 39(3): 929– 934. 10.1161/STROKEAHA.107.498725 18239164 · DOI
- [5]Macleod MR, O'Collins T, Horky LL, Howells DW, Donnan GA. Systematic review and metaanalysis of the efficacy of FK506 in experimental stroke. J Cereb Blood Flow Metab 2005 6; 25(6): 713– 21. 15703698
- [6]Rooke ED, Vesterinen HM, Sena ES, Egan KJ, Macleod MR. Dopamine agonists in animal models of Parkinson's disease: a systematic review and meta-analysis. Parkinsonism Relat Disord 2011 6; 17(5): 313– 20. 10.1016/j.parkreldis.2011.02.010 21376651 · DOI
- [7]Vesterinen HM, Sena ES, ffrench-Constant C, Williams A, Chandran S, Macleod MR. Improving the translational hit of experimental treatments in multiple sclerosis. Mult Scler 2010 9; 16(9): 1044– 55. 10.1177/1352458510379612 20685763 · DOI · Paper page
- [8]McNutt M. Journals unite for reproducibility. Science 2014 11 7; 346(6210): 679 25383411
- [10]Journals unite for reproducibility. Nature 2014 11 6; 515(7525): 7. 25373636
- [11]Currie GL, Delaney A, Bennett MI, Dickenson AH, Egan KJ, Vesterinen HM, et al Animal models of bone cancer pain: systematic review and meta-analyses. Pain 2013 6; 154(6): 917– 26. 10.1016/j.pain.2013.02.033 23582155 · DOI
- [12]Hirst TC, Vesterinen HM, Sena ES, Egan KJ, Macleod MR, Whittle IR. Systematic review and meta-analysis of temozolomide in animal models of glioma: was clinical efficacy predicted? Br J Cancer 2013 1 15; 108(1): 64– 71. 10.1038/bjc.2012.504 23321511 PMC3553514 · DOI
- [13]Frantzias J, Sena ES, Macleod MR, Al-Shahi SR. Treatment of intracerebral hemorrhage in animal models: meta-analysis. Ann Neurol 2011 2; 69(2): 389– 99. 10.1002/ana.22243 21387381 · DOI
- [14]Macleod MR, O'Collins T, Howells DW, Donnan GA. Pooling of animal experimental data reveals influence of study design and publication bias. Stroke 2004 5; 35(5): 1203– 8. 15060322
- [15]Kilkenny C, Parsons N, Kadyszewski E, Festing MF, Cuthill IC, Fry D, et al Survey of the quality of experimental design, statistical analysis and reporting of research using animals. PLoS One 2009 11 30; 4(11): e7824 10.1371/journal.pone.0007824 19956596 PMC2779358 · DOI · Paper page
- [16]Baker D, Lidster K, Sottomayor A, Amor S. Two years later: journals are not yet enforcing the ARRIVE guidelines on reporting standards for pre-clinical animal studies. PLoS Biol 2014 1; 12(1): e1001756 10.1371/journal.pbio.1001756 24409096 PMC3883646 · DOI · Paper page
- [17]Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, et al Increasing value and reducing waste in research design, conduct, and analysis. Lancet 2014 1 11; 383(9912): 166– 75. 10.1016/S0140-6736(13)62227-8 24411645 PMC4697939 · DOI · Paper page
- [18]Krauth D, Anglemyer A, Philipps R, Bero L. Nonindustry-Sponsored Preclinical Studies on Statins Yield Greater Efficacy Estimates Than Industry-Sponsored Studies: A Meta-Analysis. PLoS Biol 2014; 12(1): e1001770 10.1371/journal.pbio.1001770 24465178 PMC3897361 · DOI · Paper page
- [20]Lundh A, Sismondo S, Lexchin J, Busuioc OA, Bero L. Industry sponsorship and research outcome. Cochrane Database Syst Rev 2012 12 12; 12: MR000033 10.1002/14651858.MR000033.pub2 23235689 · DOI · Paper page
- [21]STAIR. Recommendations for standards regarding preclinical neuroprotective and restorative drug development. Stroke 1999 12; 30(12): 2752– 8. 10583007
- [22]Ludolph AC, Bendotti C, Blaugrund E, Chio A, Greensmith L, Loeffler JP, et al Guidelines for preclinical animal research in ALS/MND: A consensus meeting. Amyotroph Lateral Scler 2010; 11(1–2): 38– 45. 10.3109/17482960903545334 20184514 · DOI
- [23]Macleod MR, Fisher M, O'Collins V, Sena ES, Dirnagl U, Bath PM, et al Good laboratory practice: preventing introduction of bias at the bench. Stroke 2009 3; 40(3): e50– e52. 10.1161/STROKEAHA.108.525386 18703798 · DOI · Paper page
- [24]Shineman DW, Basi GS, Bizon JL, Colton CA, Greenberg BD, Hollister BA, et al Accelerating drug discovery for Alzheimer's disease: best practices for preclinical animal studies. Alzheimers Res Ther 2011 9 28; 3(5): 28 10.1186/alzrt90 21943025 PMC3218805 · DOI
- [25]Kilkenny C, Browne WJ, Cuthill IC, Emerson M, Altman DG. Improving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research. PLoS Biol 2010 6 29; 8(6): e1000412 10.1371/journal.pbio.1000412 20613859 PMC2893951 · DOI · Paper page
- [27]Munafo MR, Stothart G, Flint J. Bias in genetic association studies and impact factor. Mol Psychiatry 2009 2; 14(2): 119– 20. 10.1038/mp.2008.77 19156153 · DOI · Paper page
- [29]Sena ES, Currie GL, McCann SK, Macleod MR, Howells DW. Systematic reviews and meta-analysis of preclinical studies: why perform them and how to appraise them critically. J Cereb Blood Flow Metab 2014 5; 34(5): 737– 42. 10.1038/jcbfm.2014.28 24549183 PMC4013765 · DOI · Paper page
- [30]Vesterinen HM, Sena ES, Egan KJ, Hirst TC, Churolov L, Currie GL, et al Meta-analysis of data from animal studies: a practical guide. J Neurosci Methods 2014 1 15; 221: 92– 102. 10.1016/j.jneumeth.2013.09.010 Epub;%2013 Oct 4.:92–102. 24099992 · DOI
- [31]Nagel S, Papadakis M, Chen R, Hoyte LC, Brooks KJ, Gallichan D, et al Neuroprotection by dimethyloxalylglycine following permanent and transient focal cerebral ischemia in rats. J Cereb Blood Flow Metab 2011 1; 31(1): 132– 43. 10.1038/jcbfm.2010.60 20407463 PMC3049478 · DOI
- [32]Bara M, Joffe AR. The methodological quality of animal research in critical care: the public face of science. Ann Intensive Care 2014 7 29; 4: 26 10.1186/s13613-014-0026-8 25114829 PMC4126494 · DOI
- [33]Tressoldi PE, Giofre D, Sella F, Cumming G. High impact = high statistical standards? Not necessarily so. PLoS One 2013; 8(2): e56180 10.1371/journal.pone.0056180 23418533 PMC3571951 · DOI
- [34]Yordanov Y, Dechartres A, Porcher R, Boutron I, Altman DG, Ravaud P. Avoidable waste of research related to inadequate methods in clinical trials. BMJ 2015 3 24; 350: h809 10.1136/bmj.h809 25804210 PMC4372296 · DOI
Cited by
Sorted by citations
The ARRIVE guidelines 2.0: Updated guidelines for reporting animal research
Percie du Sert, N., Hurst, V., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000410 · Cited by 5,939 · see citing works
Self-citationReporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000411 · Cited by 2,677 · see citing works
Self-citationMindfulness mediates the physiological markers of stress: Systematic review and meta-analysis
Pascoe, M. C., Thompson, D. R., Jenkins, Z. M., & Ski, C. F. (2017) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2017.08.004 · Cited by 579 · see citing works
Yoga, mindfulness-based stress reduction and stress-related physiological measures: A meta-analysis.
Pascoe, M. C., Thompson, D. R., & Ski, C. F. (2017) · Psychoneuroendocrinology · doi:10.1016/j.psyneuen.2017.08.008 · Cited by 570 · see citing works
Investigating the replicability of preclinical cancer biology
Errington, T. M., Mathur, M., Soderberg, C. K., Denis, A., Perfito, N., Iorns, E., & Nosek, B. A. (2021) · eLife · doi:10.7554/elife.71601 · Cited by 393 · see citing works
“Indeed, there is accumulating evidence of the deleterious impacts of low power and small sample sizes, ignoring null results, failures of transparency of methodology, failing to publish all experimental data collected, and questionable research practices such as phacking on the inflation of false positives in the publ…”
BackgroundPrestigious Science Journals Struggle to Reach Even Average Reliability
Brembs, B. (2018) · Frontiers in Human Neuroscience · doi:10.3389/fnhum.2018.00037 · Cited by 213 · see citing works
“Analyzing the methods sections of publications reporting in vivo experiments of animal disease models where this design should have been applied, Macleod et al. (2015) found that the prevalence of reporting randomization before the treatment correlated negatively with journal rank, while FIGURE 4 |…”
MethodResultCurrent Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions
Higginson, A. D., & Munafò, M. R. (2016) · PLOS Biology · doi:10.1371/journal.pbio.2000995 · Cited by 192 · see citing works
Dechartres, A., Trinquart, L., Atal, I., Moher, D., Dickersin, K., Boutron, I., Perrodeau, E., Altman, D. G., & Ravaud, P. (2017) · BMJ · doi:10.1136/bmj.j2490 · Cited by 171 · see citing works
on behalf of the IICARus Collaboration, Hair, K., Macleod, M. R., & Sena, E. S. (2019) · Research Integrity and Peer Review · doi:10.1186/s41073-019-0069-3 · Cited by 174 · see citing works
Self-citationLeung, V., Rousseau-Blass, F., Beauchamp, G., & Pang, D. S. J. (2018) · PLOS ONE · doi:10.1371/journal.pone.0197882 · Cited by 172 · see citing works
Ramirez, F. D., Motazedian, P., Jung, R. G., Di Santo, P., MacDonald, Z. D., Moreland, R., Simard, T., Clancy, A. A., Russo, J. J., Welch, V. A., et al. (2017) · Circulation Research · doi:10.1161/circresaha.117.310628 · Cited by 138 · see citing works
“deemed crucial to improving the quality of preclinical research by addressing selection bias (randomization); minimizing performance, detection, and attrition bias (blinding); ensuring adequate statistical power and the ethical use of animals (sample size estimation); and promoting research relevance for both men and…”
Reis, D. J., Ilardi, S. S., & Punt, S. E. W. (2018) · PLOS ONE · doi:10.1371/journal.pone.0199041 · Cited by 123 · see citing works
The Experimental Design Assistant
Percie du Sert, N., Bamsey, I., Bate, S. T., Berdoy, M., Clark, R. A., Cuthill, I., Fry, D., Karp, N. A., Macleod, M., Moon, L., et al. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2003779 · Cited by 118 · see citing works
Self-citationReproducibility and Rigor in Animal-Based Research
Macleod, M., & Mohan, S. (2019) · ILAR Journal · doi:10.1093/ilar/ilz015 · Cited by 104 · see citing works
Self-citationDid a change in Nature journals’ editorial policy for life sciences research improve reporting?
The NPQIP Collaborative group, Study steering committee, Macleod, M., Sena, E., Howells, D., Macleod, M., Sena, E., Howells, D., Kiermer, V., Swaminathan, S., et al. (2019) · BMJ Open Science · doi:10.1136/bmjos-2017-000035 · Cited by 102 · see citing works
Self-citationImproving transparency and scientific rigor in academic publishing
Prager, E. M., Chambers, K. E., Plotkin, J. L., McArthur, D. L., Bandrowski, A. E., Bansal, N., Martone, M. E., Bergstrom, H. C., Bespalov, A., & Graf, C. (2019) · Journal of Neuroscience Research · doi:10.1002/jnr.24340 · Cited by 83 · see citing works
Macleod, M., Collings, A. M., Graf, C., Kiermer, V., Mellor, D., Swaminathan, S., Sweet, D., & Vinson, V. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2103238118 · Cited by 77 · see citing works
Self-citationLoss, C. M., Melleu, F. F., Domingues, K., Lino-de-Oliveira, C., & Viola, G. G. (2021) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2021.763428 · Cited by 44 · see citing works
Increasing value and reducing waste in stroke research
Berge, E., Salman, R. A.-S., van der Worp, H. B., Stapf, C., Sandercock, P., Sprigg, N., Macleod, M. R., Kelly, P. J., Nederkoorn, P. J., Ford, G. A., et al. (2017) · The Lancet Neurology · doi:10.1016/s1474-4422(17)30078-9 · Cited by 41 · see citing works
Self-citationWalker, V. R., Boyles, A. L., Pelch, K. E., Holmgren, S. D., Shapiro, A. J., Blystone, C. R., Devito, M. J., Newbold, R. R., Blain, R., Hartman, P., et al. (2018) · Environment International · doi:10.1016/j.envint.2017.12.032 · Cited by 28 · see citing works
How our approaches to assessing benefits and harms can be improved
Sena, E., & Currie, G. (2019) · Animal Welfare · doi:10.7120/09627286.28.1.107 · Cited by 23 · see citing works
Self-citationCait, J., Winder, C. B., & Mason, G. J. (2024) · Applied Animal Behaviour Science · doi:10.1016/j.applanim.2024.106361 · Cited by 21 · see citing works
Hunter, H., de Gracia Hahn, D., Duret, A., Im, Y. R., Cheah, Q., Dong, J., Fairey, M., Hjalmarsson, C., Li, A., Lim, H. K., et al. (2020) · eLife · doi:10.7554/elife.56573 · Cited by 12 · see citing works
- Authors
- Macleod, M. R., Lawson McLean, A., Kyriakopoulou, A., Serghiou, S., de Wilde, A., Sherratt, N., Hirst, T., Hemblade, R., Bahor, Z., Nunes-Fonseca, C., Potluru, A., Thomson, A., Baginskitae, J., Egan, K., Vesterinen, H., Currie, G. L., Churilov, L., Howells, D. W., & Sena, E. S.
- Affiliations
- University of Edinburgh (GB)
- Florey Institute of Neuroscience and Mental Health (AU)
- University of Tasmania (AU)
- Corresponding author
- Macleod, Malcolm R.
- ORCID
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2015-10-13
- Volume
- 13
- Issue
- 10
- Pages
- e1002273
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC4603955
- OpenAlex
- W1950977525
- Cited by
- 358
- References
- 5
- Citations by year
- 2026: 20 · 2025: 17 · 2024: 16 · 2023: 27 · 2022: 33 · 2021: 35 · 2020: 54 · 2019: 44 · 2018: 31 · 2017: 35 · 2016: 42 · 2015: 4
- FWCI
- 34.06
- Type
- journal article
- Language
- en
- Update notices
- correction (2015-11-10)
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS Biology (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Strathprints: The University of Strathclyde institutional repository (University of Strathclyde) (acceptedVersion, other-oa)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- eCite Digital Repository (University of Tasmania) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- Open Research (University of Surrey) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Surrey Research Insight Open Access (The University of Surrey) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- Funders
- National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC/L000970/1)
- National Institutes of Health
- Medical Research Council
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Keywords
- risk of bias, in vivo research, reporting quality, randomization, experimental design, research quality, journal impact
- MeSH terms
- Data Accuracy, Animals, Biological Science Disciplines, United Kingdom, Humans, Periodicals as Topic, Selection Bias, Guidelines as Topic, Publication Bias, Biomedical Research, Journal Impact Factor, Quality Improvement
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
