Publication bias in reports of animal stroke studies leads to major overstatement of efficacy
Open accessAbstract
The consolidation of scientific knowledge proceeds through the interpretation and then distillation of data presented in research reports, first in review articles and then in textbooks and undergraduate courses, until truths become accepted as such both amongst "experts" and in the public understanding. Where data are collected but remain unpublished, they cannot contribute to this distillation of knowledge. If these unpublished data differ substantially from published work, conclusions may not reflect adequately the underlying biological effects being described. The existence and any impact of such "publication bias" in the laboratory sciences have not been described. Using the CAMARADES (Collaborative Approach to Meta-analysis and Review of Animal Data in Experimental Studies) database we identified 16 systematic reviews of interventions tested in animal studies of acute ischaemic stroke involving 525 unique publications. Only ten publications (2%) reported no significant effects on infarct volume and only six (1.2%) did not report at least one significant finding. Egger regression and trim-and-fill analysis suggested that publication bias was highly prevalent (present in the literature for 16 and ten interventions, respectively) in animal studies modelling stroke. Trim-and-fill analysis suggested that publication bias might account for around one-third of the efficacy reported in systematic reviews, with reported efficacy falling from 31.3% to 23.8% after adjustment for publication bias. We estimate that a further 214 experiments (in addition to the 1,359 identified through rigorous systematic review; non publication rate 14%) have been conducted but not reported. It is probable that publication bias has an important impact in other animal disease models, and more broadly in the life sciences.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous analyses of studies using animal models have shown that small studies consistently give more favourable (positive) results than larger studies.
“Previous analyses of studies using animal models have shown that small studies consistently give more favourable (that is, 'positive') results than larger studies 78”
From Power failure: why small sample size undermines the reliability of neuroscience (2013) · cites this paper as [78]
The Results (Prevalence of Publication Bias; page not provided) report that all 16 datasets had an excess of imprecise studies with large effect sizes, supporting the favourable-results pattern in animal stroke studies. But the Discussion (page not provided) cautions that precision in these studies relates more to biological variability and measurement error than to study size. The paper therefore does not establish the claimed comparison of small studies with larger studies. (LLM, conf=0.86)
Experimental — accuracy not yet measuredFailure to disclose all the outcomes that were measured introduces bias in the literature, as positive outcomes are reported more often.
“Failure to disclose all the outcomes that were measured introduces bias in the literature, as positive outcomes (e.g., those statistically significant) are reported more often [88–91].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [91]
The Discussion reports that positive animal-stroke studies are more likely to be published than neutral ones, biasing the literature. But the paper examines missing experiments, not whether researchers selectively disclose positive outcomes among all outcomes measured in a study. (LLM, conf=0.88)
Experimental — accuracy not yet measuredThe greater refractoriness of HIV-related neuropathy and radiculopathy may be due to large placebo responses, a distinct clinical phenotype, or neglected psychosocial comorbidities.
“This may be due to large placebo responses in HIV-related neuropathy trials,45 a distinct clinical phenotype in subgroups of patients with radiculopathy,46 or psychological/psychosocial comorbidites, often neglected in large trials.”
From Pharmacotherapy for neuropathic pain in adults: a systematic review and meta-analysis (2015) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
The file drawer problem and tolerance for null results.
Rosenthal, R. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.3.638 · Cited by 9,068 · cited 1× · see citing works
“This is the ‘‘file drawer problem’’ [2,3]: at its most extreme, the 95% of studies that were truly neutral (that is, which reported no significant effects) remain in the files of the investigators, the 5% of experiments that were falsely positive are published, and reviewers conclude—falsely—that the literature repres…”
BackgroundFisher, R. A. (1936) · BMJ · doi:10.1136/bmj.1.3923.554-a · Cited by 4,980 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1× · see citing works
“However, in this context, the positive predictive value of apparently significant results is likely to be substantially lower than the 95% suggested by conventional statistical testing [31].”
BackgroundAssessing the quality of controlled clinical trials
Jüni, P., Altman, D. G., & Egger, M. (2001) · BMJ · doi:10.1136/bmj.323.7303.42 · Cited by 3,309 · cited 1× · see citing works
“assessing the impact of individual study quality items [38], across”
BackgroundA Nonparametric “Trim and Fill” Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. (2000) · Journal of the American Statistical Association · doi:10.1080/01621459.2000.10473905 · Cited by 2,955 · cited 2× · see citing works
“For each intervention, and for all interventions together (with individual experiments being pooled for a global analysis), the prevalence of publication bias was assessed using funnel plotting [36], Egger regression [37], and the Duval and Tweedie nonparametric trim-and-fill approach [27] (enabled in METATRIM, an add…”
MethodThe case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766 · cited 1× · see citing works
“However, there are a number of factors other than publication bias that can cause funnel plot asymmetry [29]:”
BackgroundAntman, E. M. (1992) · JAMA · doi:10.1001/jama.268.2.240 · Cited by 1,535 · see citing works
Summing up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,486 · see citing works
Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors
De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., et al. (2004) · New England Journal of Medicine · doi:10.1056/nejme048225 · Cited by 1,241 · see citing works
Comparison of Treatment Effects between Animal Experiments and Clinical Trials: Systematic Review
Perel, P., Roberts, I., Sena, E., Wheble, P., Briscoe, C., Sandercock, P., Macleod, M., Mignini, L. E., Jayaram, P., & Khan, K. S. (2007) · BMJ · doi:10.1136/bmj.39048.407928.be · Cited by 827 · cited 2× · see citing works
“tPA [15] 105 256 4,029 22.”
MethodSelf-citationA Nonparametric "Trim and Fill" Method of Accounting for Publication Bias in Meta-Analysis
Duval, S., & Tweedie, R. L. (2000) · Journal of the American Statistical Association · doi:10.2307/2669529 · Cited by 694 · see citing works
Pooling of Animal Experimental Data Reveals Influence of Study Design and Publication Bias
Macleod, M., O'Collins, T., Howells, D., & Donnan, G. (2004) · Stroke · doi:10.1161/01.str.0000125719.25853.20 · Cited by 692 · cited 1× · see citing works
“assessment of outcome, use of anaesthetic without significant intrinsic neuroprotective activity, appropriate animal model (aged, diabetic, or hypertensive), sample size calculation, compliance with animal welfare regulations, and statement of potential conflict of interests [11].”
BackgroundSelf-citationHypothermia in animal models of acute ischaemic stroke: a systematic review and meta-analysis.
van der Worp, H. B., Sena, E., Donnan, G., Howells, D., & Macleod, M. (2007) · Brain : a journal of neurology · doi:10.1093/brain/awm083 · Cited by 520 · see citing works
Self-citationWhere is the evidence that animal research benefits humans?
Pound, P., Ebrahim, S. B. J., Sandercock, P. A., Bracken, M. B., & Roberts, I. (2004) · BMJ · doi:10.1136/bmj.328.7438.514 · Cited by 507 · cited 1× · see citing works
“However, until recently there has been a paucity of systematic reviews of animal studies [8].”
BackgroundHow citation distortions create unfounded authority: analysis of a citation network
Greenberg, S. A. (2009) · BMJ · doi:10.1136/bmj.b2680 · Cited by 499 · cited 1× · see citing works
“A related group of biases, the citation biases [7], can be addressed through rigorous systematic review, in that an attempt is made to include all relevant publications describing data meeting predefined”
BackgroundThe Science of Reviewing Researcha
OXMAN, A. D., & GUYATT, G. H. (1993) · Annals of the New York Academy of Sciences · doi:10.1111/j.1749-6632.1993.tb26342.x · Cited by 472 · cited 1× · see citing works
“Attempts have been made to account for this bias using the technique of systematic review, in which there is prespecification of the biological question being addressed, the methods through which contributing data will be identified, and the criteria that will be used to select which data are included in the analysis…”
MethodHow can we improve the pre-clinical development of drugs for stroke?
Sena, E. S., van der Worp, H. B., Howells, D. W., & Macleod, M. R. (2007) · Trends in Neurosciences · doi:10.1016/j.tins.2007.06.009 · Cited by 365 · cited 1× · see citing works
“While the importance of these issues has been recognised for some years [32], they are rarely reported in contemporary reports of animal experiments [33].”
BackgroundSelf-citationGood Laboratory Practice: Preventing Introduction of Bias at the Bench
Macleod, M. R., Fisher, M., O'Collins, V., Sena, E. S., Dirnagl, U., Bath, P. M. W., Buchan, A., van der Worp, H. B., Traystman, R., Minematsu, K., et al. (2009) · Stroke · doi:10.1161/strokeaha.108.525386 · Cited by 331 · see citing works
Self-citationMacleod, M. R., van der Worp, H. B., Sena, E. S., Howells, D. W., Dirnagl, U., & Donnan, G. A. (2008) · Stroke · doi:10.1161/strokeaha.108.515957 · Cited by 319 · cited 1× · see citing works
“It is now clear that certain aspects of experimental design (particularly randomisation, allocation concealment, and the blinded assessment of outcome) can have a substantial impact on the reported outcome of experiments [14].”
BackgroundSelf-citationPublication bias in ecology and evolution: an empirical assessment using the ‘trim and fill’ method
Jennions, M. D., & Møller, A. P. (2002) · Biological reviews/Biological reviews of the Cambridge Philosophical Society · doi:10.1017/s1464793101005875 · Cited by 279 · cited 1× · see citing works
“Publication bias has also been observed in reports of genetic association studies [5] and in ecology and evolution, in which 40% of meta-analyses were confounded by publication bias, and adjusting for publication bias might have altered the conclusions in around one-third of cases [6].”
BackgroundAngelis, C. D., Drazen, J. M., Frizelle, F. A., Haug, C. J., Hoey, J., Horton, R., Kotzin, S., Laine, C. M., Marušić, A., Overbeke, A. J. P. M., et al. (2004) · PubMed · Cited by 275 · see citing works
Systematic Review and Metaanalysis of the Efficacy of FK506 in Experimental Stroke
Macleod, M., O'Collins, T., Horky, L. L., Howells, D., & Donnan, G. (2005) · Journal of Cerebral Blood Flow and Metabolism · doi:10.1038/sj.jcbfm.9600064 · Cited by 196 · cited 2× · see citing works
“FK506 [12] 27 96 1,596 32.”
MethodSelf-citationWillmot, M. R., Gray, L. J., Gibson, C. L., Murphy, S. P., & Bath, P. M. (2005) · Nitric Oxide · doi:10.1016/j.niox.2005.01.003 · Cited by 179 · cited 2× · see citing works
“Since 2004 the Collaborative Approach to Meta-Analysis and Review of Animal Data in Experimental Studies (CAMARADES) has curated data collected in the context of systematic reviews of reports of studies describing the efficacy in animals of candidate interventions for stroke [10–21].”
MethodSelf-citationEstrogens and Experimental Ischemic Stroke: A Systematic Review
Gibson, C. L., Gray, L. J., Murphy, S. P., & Bath, P. M. (2006) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/sj.jcbfm.9600270 · Cited by 142 · cited 2× · see citing works
“Estrogens [10] 27 99 1,452 26.”
MethodSelf-citationSystematic review and meta‐analysis of the efficacy of melatonin in experimental stroke
Macleod, M. R., O'Collins, T., Horky, L. L., Howells, D., & Donnan, G. A. (2005) · Journal of Pineal Research · doi:10.1111/j.1600-079x.2004.00172.x · Cited by 137 · cited 2× · see citing works
“Melatonin [13] 12 29 443 42.”
MethodSelf-citationWillmot, M. R., Gibson, C. L., Gray, L. J., Murphy, S. P., & Bath, P. M. (2005) · Free Radical Biology and Medicine · doi:10.1016/j.freeradbiomed.2005.03.028 · Cited by 129 · cited 1× · see citing works
“Since 2004 the Collaborative Approach to Meta-Analysis and Review of Animal Data in Experimental Studies (CAMARADES) has curated data collected in the context of systematic reviews of reports of studies describing the efficacy in animals of candidate interventions for stroke [10–21].”
MethodSelf-citationCraig, C. C., & Fisher, R. A. (1936) · American Mathematical Monthly · doi:10.2307/2300364 · Cited by 128 · see citing works
Banwell, V., Sena, E. S., & Macleod, M. R. (2009) · Journal of Stroke and Cerebrovascular Diseases · doi:10.1016/j.jstrokecerebrovasdis.2008.11.009 · Cited by 123 · see citing works
Self-citationEffects of NXY‐059 in experimental stroke: an individual animal meta‐analysis
Bath, P. M., Gray, L. J., Bath, A. J. G., Buchan, A. M., Miyata, T., & Green, A. R. (2009) · British Journal of Pharmacology · doi:10.1111/j.1476-5381.2009.00196.x · Cited by 122 · cited 1× · see citing works
“Many would consider these to be conservative estimates, and indeed a recent systematic review of individual animal data supporting the efficacy of NXY-059 showed that two of four unpublished experiments identified in the course of that review were neutral [28].”
ResultSelf-citationDawson, D. A., Kusumoto, K., Graham, D. I., McCulloch, J., & Macrae, I. M. (1992) · Neuroscience Letters · doi:10.1016/0304-3940(92)90361-a · Cited by 114 · cited 1× · see citing works
“Of these, only ten publications (2%; one publication [22] was represented in reviews of both tirilazad and tPA) described no significant effect on infarct volume, although four reported other statistically significant findings [23–26].”
ResultTimmer, A., Hilsden, R. J., Cole, J., Hailey, D. M., & Sutherland, L. R. (2002) · BMC Medical Research Methodology · doi:10.1186/1471-2288-2-7 · Cited by 103 · see citing works
Systematic Review and Meta-Analysis of the Efficacy of Tirilazad in Experimental Stroke
Sena, E. S., Wheble, P. C. R., Sandercock, P. A., & Macleod, M. R. (2007) · Stroke · doi:10.1161/01.str.0000254462.75851.22 · Cited by 95 · cited 2× · see citing works
“Tirilazad [16] 18 34 544 31.”
MethodSelf-citationMignini, L. E., & Khan, K. S. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-10 · Cited by 94 · see citing works
Wheble, P. C. R., Sena, E. S., & Macleod, M. R. (2007) · Cerebrovascular Diseases · doi:10.1159/000111493 · Cited by 41 · cited 1× · see citing works
“Piracetam and related compounds [18] 5 14 197 29.”
BackgroundSelf-citationPrado, R., Watson, B. D., Zhao, W., Yao, H., Busto, R., Dietrich, W. D., & Ginsberg, M. D. (1996) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1097/00004647-199607000-00011 · Cited by 31 · cited 1× · see citing works
“Of these, only ten publications (2%; one publication [22] was represented in reviews of both tirilazad and tPA) described no significant effect on infarct volume, although four reported other statistically significant findings [23–26].”
ResultJ. Light, R., & B. Pillemer, D. (1986) · Educational Researcher · doi:10.3102/0013189x015008016 · Cited by 17 · cited 1× · see citing works
“For each intervention, and for all interventions together (with individual experiments being pooled for a global analysis), the prevalence of publication bias was assessed using funnel plotting [36], Egger regression [37], and the Duval and Tweedie nonparametric trim-and-fill approach [27] (enabled in METATRIM, an add…”
MethodOprica, M., van Dam, A., Lundkvist, J., Iverfeldt, K., Winblad, B. G., Bartfai, T., & Schultzberg, M. (2004) · Acta Neuropathologica · doi:10.1007/s00401-004-0868-5 · Cited by 15 · cited 1× · see citing works
“Of these, only ten publications (2%; one publication [22] was represented in reviews of both tirilazad and tPA) described no significant effect on infarct volume, although four reported other statistically significant findings [23–26].”
ResultAntman, E. M. (1992) · JAMA · doi:10.1001/jama.1992.03490020088036 · Cited by 11 · cited 1× · see citing works
“This is the ‘‘file drawer problem’’ [2,3]: at its most extreme, the 95% of studies that were truly neutral (that is, which reported no significant effects) remain in the files of the investigators, the 5% of experiments that were falsely positive are published, and reviewers conclude—falsely—that the literature repres…”
BackgroundOrozco, J., Mendel, R. C., Hahn, M. R., Guthkelch, A. N., & Carter, L. P. (1995) · Neurological Research · doi:10.1080/01616412.1995.11740356 · Cited by 10 · cited 1× · see citing works
“Of these, only ten publications (2%; one publication [22] was represented in reviews of both tirilazad and tPA) described no significant effect on infarct volume, although four reported other statistically significant findings [23–26].”
ResultUp from ‘false positives’ in genetic—and other—epidemiology
Miettinen, O. S. (2008) · European Journal of Epidemiology · doi:10.1007/s10654-008-9295-6 · Cited by 9 · cited 1× · see citing works
“Publication bias has also been observed in reports of genetic association studies [5] and in ecology and evolution, in which 40% of meta-analyses were confounded by publication bias, and adjusting for publication bias might have altered the conclusions in around one-third of cases [6].”
BackgroundRoberts, T. P. L., Vexler, Z. S., Derugin, N., Koźniewska, E., Kucharczyk, J. R., & Emmett, C. J. (1995) · Brain Research · doi:10.1016/0006-8993(95)00857-m · Cited by 8 · cited 1× · see citing works
“Of these, only ten publications (2%; one publication [22] was represented in reviews of both tirilazad and tPA) described no significant effect on infarct volume, although four reported other statistically significant findings [23–26].”
ResultDealing with publication bias in translational stroke research
Liu, S. (2009) · Journal of Experimental Stroke and Translational Medicine · doi:10.6030/1939-067x-2.1.16 · Cited by 8 · cited 1× · see citing works
“Others have considered the issue of publication bias in animal stroke studies [34], and have made suggestions for how this might be addressed.”
Background
Flagged first, then by citations
Research works printed but not linked: 3
- [37]Egger M Davey S. G Schneider M Minder C 1997 Bias in meta-analysis detected by a simple, graphical test. BMJ 315 629 634 9310563 PMC2127453
- [40]van der Worp H. B Sena E. S Donnan G. A Howells D. W Macleod M. R 2007 Hypothermia in animal models of acute ischaemic stroke: a systematic review and meta-analysis. Brain 130 3063 3074 17478443
- [41]Willmot M Gibson C Gray L Murphy S Bath P 2005 Nitric oxide synthase inhibitors in experimental ischemic stroke and their effects on infarct size and cerebral blood flow: a systematic review. Free Radic Biol Med 39 412 425 15993340
Cited by
Sorted by citations
Power failure: why small sample size undermines the reliability of neuroscience
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013) · Nature Reviews Neuroscience · doi:10.1038/nrn3475 · Cited by 7,641 · see citing works
Improving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · PLoS Biology · doi:10.1371/journal.pbio.1000412 · Cited by 7,300 · see citing works
“The reviews have found that, in many cases, reporting omissions, in addition to the limitations of the animal models used in the individual studies assessed in the review, are a barrier to reaching any useful conclusion about the efficacy of the drugs and interventions being compared [2,3].”
Improving bioscience research reporting: The ARRIVE guidelines for reporting animal research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2010) · Journal of Pharmacology and Pharmacotherapeutics · doi:10.4103/0976-500x.72351 · Cited by 6,277 · see citing works
“The reviews have found that, in many cases, reporting omissions, in addition to the limitations of the animal models used in the individual studies assessed in the review, are a barrier to reaching any useful conclusion about the efficacy of the drugs and interventions being compared [2,3].”
Pharmacotherapy for neuropathic pain in adults: a systematic review and meta-analysis
Finnerup, N. B., Attal, N., Haroutounian, S., McNicol, E., Baron, R., Dworkin, R. H., Gilron, I., Haanpää, M., Hansson, P., Jensen, T. S., et al. (2015) · The Lancet Neurology · doi:10.1016/s1474-4422(14)70251-0 · Cited by 3,980 · see citing works
Self-citationReporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000411 · Cited by 2,677 · see citing works
Self-citationThe Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483 · see citing works
Can Animal Models of Disease Reliably Inform Human Studies?
van der Worp, H. B., Howells, D. W., Sena, E. S., Porritt, M. J., Rewell, S., O'Collins, V., & Macleod, M. R. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000245 · Cited by 1,354 · see citing works
“No study gave quantitative estimates of the impact on effect size of publication bias [47].”
BackgroundInfluentialSelf-citationA call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196 · see citing works
Self-citationChamorro, Á., Dirnagl, U., Urra, X., & Planas, A. M. (2016) · The Lancet Neurology · doi:10.1016/s1474-4422(16)00114-9 · Cited by 1,198 · see citing works
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research
Begley, C. G., & Ioannidis, J. P. A. (2015) · Circulation Research · doi:10.1161/circresaha.114.303819 · Cited by 1,035 · see citing works
“led the authors to conclude that serious publication bias may exist.(24) In another analysis of over 4445 data sets involving animal studies of neurological diseases, it was similarly con-”
The economics of reproducibility in preclinical research
Freedman, L. P., Cockburn, I. M., & Simcoe, T. S. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002165 · Cited by 1,055 · see citing works
“However, it is reasonable to state that cumulative errors in the following broad categories—as well as underlying biases that could contribute to each problem area [14] or even result in entire studies never being published or reported [15]—are the primary causes of irreproducibility [16]: (1) study design, (2) biolog…”
BackgroundLost in translation: animal models and clinical trials in cancer treatment.
(2014) · Europe PMC (PubMed Central) · Cited by 1,033 · see citing works
Increasing value and reducing waste: addressing inaccessible research
Chan, A.-W., Song, F., Vickers, A., Jefferson, T., Dickersin, K., Gøtzsche, P. C., Krumholz, H. M., Ghersi, D., & van der Worp, H. B. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62296-5 · Cited by 887 · see citing works
Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2012) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2012.02.010 · Cited by 728 · see citing works
Improving Bioscience Research Reporting: The ARRIVE Guidelines for Reporting Animal Research
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M., & Altman, D. G. (2014) · Animals · doi:10.3390/ani4010035 · Cited by 707 · see citing works
“The reviews have found that, in many cases, reporting omissions, in addition to the limitations of the animal models used in the individual studies assessed in the review, are a barrier to reaching any useful conclusion about the efficacy of the drugs and interventions being compared [2,3].”
Meta-evaluation of meta-analysis: ten appraisal questions for biologists
Nakagawa, S., Noble, D. W. A., Senior, A. M., & Lagisz, M. (2017) · BMC Biology · doi:10.1186/s12915-017-0357-7 · Cited by 579 · see citing works
Meta-analysis of data from animal studies: A practical guide
Vesterinen, H. M., Sena, E. S., Egan, K. J., Hirst, T. C., Churolov, L., Currie, G. L., Antonic‐Baker, A., Howells, D. W., & Macleod, M. R. (2013) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2013.09.010 · Cited by 477 · see citing works
Self-citationIncreasing value and reducing waste in biomedical research: who's listening?
Moher, D., Glasziou, P., Chalmers, I., Nasser, M., Bossuyt, P. M. M., Korevaar, D. A., Graham, I. D., Ravaud, P., & Boutron, I. (2016) · The Lancet · doi:10.1016/s0140-6736(15)00307-4 · Cited by 462 · see citing works
Kleinschnitz, C., Grund, H., Wingler, K., Armitage, M. E., Jones, E., Mittal, M., Barit, D., Schwarz, T., Geis, C., Kraft, P., et al. (2010) · PLoS Biology · doi:10.1371/journal.pbio.1000479 · Cited by 447 · see citing works
“Higher quality study designs are also required [6,7].”
BackgroundMethodPublication and other reporting biases in cognitive sciences: detection, prevalence, and prevention
Ioannidis, J. P. A., Munafò, M. R., Fusar-Poli, P., Nosek, B. A., & David, S. P. (2014) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2014.02.010 · Cited by 471 · see citing works
Henderson, V. C., Kimmelman, J., Fergusson, D. A., Grimshaw, J., & Hackam, D. G. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001489 · Cited by 300 · see citing works
“Various commentators have reported problems such as difficulty in replicating preclinical studies [5,6], publication bias [7], and the prevalence of methodological practices that result in threats to validity [8].”
BackgroundEvaluation of excess significance bias in animal studies of neurological diseases
Tsilidis, K. K., Panagiotou, O. A., Sena, E. S., Aretouli, E., Evangelou, E., Howells, D. W., Salman, R. A.-S., Macleod, M. R., & Ioannidis, J. P. A. (2013) · PLoS Biology · doi:10.1371/journal.pbio.1001609 · Cited by 295 · see citing works
“The estimate from this most precise study, other things being equal, should be closer to the true estimate than the results of less precise studies, especially if biases affect predominantly the literature of smaller studies (small-study effects) [44–46].”
ResultSelf-citationMarx, U. (2016) · ALTEX · doi:10.14573/altex.1603161 · Cited by 295 · see citing works
“Leist and Hartung, 2013; Matthews, 2008; Olson et al., 2000; Perel et al., 2007; Hartung and Leist, 2008; Schnerch et al., 2010; Sena et al., 2010; Seok et al., 2013; van der Worp et al., 2010; Hartung, 2013).”
The need for randomization in animal trials: an overview of systematic reviews
Hirst, J. A., Howick, J., Aronson, J. K., Roberts, N., Perera, R., Koshiaris, C., & Heneghan, C. (2014) · PLoS ONE · doi:10.1371/journal.pone.0098856 · Cited by 294 · see citing works
Restoration of sensorimotor functions after spinal cord injury
Dietz, V., & Fouad, K. (2013) · Brain · doi:10.1093/brain/awt262 · Cited by 293 · see citing works
“Such deficiencies include a low statistical power of animal experiments with relatively small numbers of animals, resulting in the probability of false positive results (Ioannidis, 2005) and a bias to publish such positive results (Sena et al., 2010).”
BackgroundInfluentialReproducibility of preclinical animal research improves with heterogeneity of study samples
Voelkl, B., Vogt, L., Sena, E. S., & Würbel, H. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2003693 · Cited by 262 · see citing works
“Both lack of scientific rigor [33] and publication bias [34] have been found to inflate summary effect sizes in meta-analyses.”
BackgroundSelf-citationSmith, A. M., & Dragunow, M. (2014) · Trends in Neurosciences · doi:10.1016/j.tins.2013.12.001 · Cited by 256 · see citing works
de Vries, R. B. M., Hooijmans, C. R., Langendam, M. W., van Luijk, J. A. K. R., Leenaars, M., Ritskes-Hoitinga, M. J., & Wever, K. E. (2015) · Evidence-based Preclinical Medicine · doi:10.1002/ebm2.7 · Cited by 264 · see citing works
Neuroprotection for Ischaemic Stroke: Translation from the Bench to the Bedside
Sutherland, B. A., Minnerup, J., Balami, J. S., Arba, F., Buchan, A. M., & Kleinschnitz, C. (2012) · International Journal of Stroke · doi:10.1111/j.1747-4949.2012.00770.x · Cited by 259 · see citing works
Baker, D., Lidster, K., Sottomayor, A., & Amor, S. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001756 · Cited by 305 · see citing works
“invariably positive [1,2,26].”
Lazic, S. E., & Essioux, L. (2013) · BMC Neuroscience · doi:10.1186/1471-2202-14-37 · Cited by 247 · see citing works
Neuroprotection for ischaemic stroke: Current status and challenges
Moretti, A., Ferrari, F., & Villa, R. F. (2014) · Pharmacology & Therapeutics · doi:10.1016/j.pharmthera.2014.09.003 · Cited by 242 · see citing works
“One reason for the failure is the heterogeneous or poor methodological quality, and the publication bias of many experimental studies leading to disputable or overstated preclinical evidence of efficacy (Philip et al., 2009; Sena et al., 2010).”
BackgroundDistinguishing between exploratory and confirmatory preclinical research will improve translation
Kimmelman, J., Mogil, J. S., & Dirnagl, U. (2014) · PLoS Biology · doi:10.1371/journal.pbio.1001863 · Cited by 236 · see citing works
“Because many preclinical studies poorly address internal validity threats [1], fail attempts at replication [2], are not published [3], or provide exaggerated estimates of clinical utility, numerous stakeholders are urging reforms in the way preclinical research is performed [4].”
BackgroundSena, E. S., Currie, G. L., McCann, S. K., Macleod, M. R., & Howells, D. W. (2014) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.2014.28 · Cited by 264 · see citing works
Self-citationGarner, J. P. (2014) · ILAR Journal · doi:10.1093/ilar/ilu047 · Cited by 225 · see citing works
Insights into the success and failure of systemic therapy for hepatocellular carcinoma
Bruix, J., da Fonseca, L. G., & Reig, M. (2019) · Nature Reviews Gastroenterology & Hepatology · doi:10.1038/s41575-019-0179-x · Cited by 220 · see citing works
Big data from small data: data-sharing in the 'long tail' of neuroscience
Ferguson, A. R., Nielson, J. L., Cragin, M. H., Bandrowski, A. E., & Martone, M. E. (2014) · Nature Neuroscience · doi:10.1038/nn.3838 · Cited by 211 · see citing works
Standardized mean differences cause funnel plot distortion in publication bias assessments
Zwetsloot, P.-P. M., van der Naald, M., Sena, E. S., Howells, D. W., IntHout, J., de Groot, J. A. H., Chamuleau, S. A. J., Macleod, M. R., & Wever, K. E. (2017) · eLife · doi:10.7554/elife.24260 · Cited by 214 · see citing works
“SE funnel plots, which formed the basis of a recent analysis 372 showing evidence for substantial publication bias in the animal stroke literature(20).”
BackgroundSelf-citationIs animal research sufficiently evidence based to be a cornerstone of biomedical research?
Pound, P., & Bracken, M. B. (2014) · BMJ · doi:10.1136/bmj.g3387 · Cited by 207 · see citing works
Neuroprotection for Stroke: Current Status and Future Perspectives
Minnerup, J., Sutherland, B. A., Buchan, A. M., & Kleinschnitz, C. (2012) · International Journal of Molecular Sciences · doi:10.3390/ijms130911753 · Cited by 209 · see citing works
“the overestimation of a treatment’s efficacy, was followed by some stroke journals by increasing the number of publications containing animal studies with neutral results [89–91].”
Look back in anger – what clinical studies tell us about preclinical work
Härtung, T. (2013) · ALTEX · doi:10.14573/altex.2013.3.275 · Cited by 203 · see citing works
“Publication bias was highly prevalent (Sena et al., 2010).”
Hooijmans, C. R., IntHout, J., Ritskes-Hoitinga, M. J., & Rovers, M. M. (2014) · ILAR Journal · doi:10.1093/ilar/ilu042 · Cited by 200 · see citing works
A step-by-step guide to systematically identify all relevant animal studies
Leenaars, M., Hooijmans, C. R., van Veggel, N., ter Riet, G., Leeflang, M., Hooft, L., van der Wilt, G. J., Tillema, A., & Ritskes-Hoitinga, M. J. (2011) · Laboratory Animals · doi:10.1258/la.2011.011087 · Cited by 198 · see citing works
The multifactorial role of the 3Rs in shifting the harm-benefit analysis in animal models of disease
Graham, M., & Prescott, M. (2015) · European Journal of Pharmacology · doi:10.1016/j.ejphar.2015.03.040 · Cited by 188 · see citing works
“…(2011), Baker et al. (2011) Pain Percie du Sert and Rice (2014) Sepsis Webb (2014) Stroke Crossley et al. (2008), Mergenthaler and Meisel (2012), Sena et al. (2010), van der Worp et al. (2010), Howells et al. (2014) Transplantation Graham and Schuurman (2013), Wijkstrom et al. (2013) scientists…”
BackgroundDirnagl, U., & Endres, M. (2014) · Stroke · doi:10.1161/strokeaha.113.004075 · Cited by 194 · see citing works
Neuroprotection in stroke: the importance of collaboration and reproducibility
Neuhaus, A. A., Couch, Y., Hadley, G., & Buchan, A. M. (2017) · Brain · doi:10.1093/brain/awx126 · Cited by 193 · see citing works
“It has been well documented in the stroke field that publication bias skews outcomes towards the positive, overstating the efficacy of novel compounds (Sena et al., 2010).”
BackgroundCarbone, L., & Austin, J. (2016) · PLOS ONE · doi:10.1371/journal.pone.0155001 · Cited by 191 · see citing works
“The ARRIVE and NAS guidelines, and many other systematic reviews of the quality of data of animal studies, emphasize the need for full and transparent reporting that allow for critical evaluation and reproduction of published data.[6, 15, 17, 64, 78, 79] Pain management has mostly stayed off their lists of criteria, o…”
BackgroundThe IMPROVE Guidelines (Ischaemia Models: Procedural Refinements Of in Vivo Experiments)
du Sert, N. P., Alfieri, A., Allan, S. M., Carswell, H. V. O., Deuchar, G. A., Farr, T. D., Flecknell, P. A., Gallagher, L., Gibson, C. L., Haley, M., et al. (2017) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1177/0271678x17709185 · Cited by 190 · see citing works
Self-citationImproving the translation of analgesic drugs to the clinic: animal models of neuropathic pain
du Sert, N. P., & Rice, A. S. C. (2014) · British Journal of Pharmacology · doi:10.1111/bph.12645 · Cited by 188 · see citing works
“In animal research, a study investigating publication bias in the field of stroke estimated that in addition to the 1359 experiments identified, a further 214 experiments (16%) yielding negative results had been carried out but not reported; publication bias accounted for around onethird of the efficacy reported in sy…”
BackgroundAn Estimate of the Number of Animals Used for Scientific Purposes Worldwide in 2015
Taylor, K., & Alvarez, L. R. (2019) · Alternatives to Laboratory Animals · doi:10.1177/0261192919899853 · Cited by 189 · see citing works
“A large proportion of research that is conducted will, naturally, produce negative results and will, therefore, not be published and this includes animal research.(73) As the current model has not been internally validated, we do not know whether its prediction will hold true, particularly for non-EU countries whose p…”
ResultAn analysis of the use of animal models in predicting human toxicology and drug safety
Bailey, J., Thew, M., & Balls, M. (2014) · Alternatives to Laboratory Animals · doi:10.1177/026119291404200306 · Cited by 182 · see citing works
Replication and reproducibility in spinal cord injury research
Steward, O., Popovich, P. G., Dietrich, W. D., & Kleitman, N. (2012) · Experimental Neurology · doi:10.1016/j.expneurol.2011.06.017 · Cited by 186 · see citing works
Therapeutic Hypothermia for Acute Ischemic Stroke: Ready to Start Large Randomized Trials?
van der Worp, H. B., Macleod, M. R., & Kollmar, R. (2010) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.2010.44 · Cited by 186 · see citing works
“Animal studies indicate a robust benefit of hypothermia after focal cerebral ischemia (van der Worp et al, 2007), without indicating critical publication bias (Sena et al, 2010).”
BackgroundResultSelf-citationNeuroprotective Strategies after Neonatal Hypoxic Ischemic Encephalopathy
Dixon, B., Reis, C., Ho, W., Tang, J., & Zhang, J. H. (2015) · International Journal of Molecular Sciences · doi:10.3390/ijms160922368 · Cited by 183 · see citing works
“Although much progress in experimental development and technology has been made in the past few decades, the issue of discovering disease models, that can mimic human pathophysiology sufficiently, remains an ongoing challenge [73,77,78].”
BackgroundLazic, S. E., & Essioux, L. (2016) · Cited by 182 · see citing works
Augustine, S., Avey, M. T., Harrison, B. J., Locke, T., Ghannad, M., Moher, D., & Thébaud, B. (2017) · Stem Cells Translational Medicine · doi:10.1002/sctm.17-0126 · Cited by 172 · see citing works
Improving the translational hit of experimental treatments in multiple sclerosis
Vesterinen, H. M., Sena, E. S., ffrench-Constant, C., Williams, A., Chandran, S., & Macleod, M. R. (2010) · Multiple Sclerosis Journal · doi:10.1177/1352458510379612 · Cited by 182 · see citing works
Self-citationTop Priorities for Cerebroprotective Studies—A Paradigm Shift: Report From STAIR XI
Lyden, P. D., Buchan, A. M., Boltze, J., Fisher, M., Ansari, S., Broderick, J. P., Campbell, B. C. V., Chaisinanunkul, N. N., Chen, C., Grotta, J. C., et al. (2021) · Stroke · doi:10.1161/strokeaha.121.034947 · Cited by 168 · see citing works
Hooijmans, C. R., de Vries, R. B. M., Ritskes-Hoitinga, M., Rovers, M. M., Leeflang, M. M., IntHout, J., Wever, K. E., Hooft, L., de Beer, H., Kuijpers, T., et al. (2018) · PLOS ONE · doi:10.1371/journal.pone.0187271 · Cited by 176 · see citing works
“Based on Egger regression and trim-and-fill analysis the authors concluded that publication bias was likely and treatment efficacy was over estimated [43].”
MethodSelf-citationProgress in using systematic reviews of animal studies to improve translational research
Hooijmans, C. R., & Ritskes-Hoitinga, M. (2013) · PLoS Medicine · doi:10.1371/journal.pmed.1001482 · Cited by 195 · see citing works
Inflammatory findings on species extrapolations: humans are definitely no 70-kg mice
Leist, M., & Härtung, T. (2013) · Archives of Toxicology · doi:10.1007/s00204-013-1038-0 · Cited by 164 · see citing works
“Findings on species differences tend to be neglected, and the skewing of the available data by a publication bias toward positive animal findings has only recently been unraveled (Sena et al. 2010).”
BackgroundSchrier, J. E., Harper, J. S., Hankinson, G. J., Hulsebos, L., Povar, M. L., Mathiesen, E., & Schrier, A. M. (2009) · Cited by 166 · see citing works
Cardiac Stem Cell Treatment in Myocardial Infarction
Zwetsloot, P.-P. M., Végh, A. M. D., Lorkeers, S. J. J. O., van Hout, G. P. J., Currie, G. L., Sena, E. S., Gremmels, H., Buikema, J. W., Goumans, M., Macleod, M. R., et al. (2016) · Circulation Research · doi:10.1161/circresaha.115.307676 · Cited by 155 · see citing works
Self-citationEstimation statistics should replace significance testing
Claridge‐Chang, A., & Assam, P. N. (2016) · Nature Methods · doi:10.1038/nmeth.3729 · Cited by 155 · see citing works
Ioannidis, J. P. A. (2013) · Journal of Mathematical Psychology · doi:10.1016/j.jmp.2013.03.002 · Cited by 150 · see citing works
Oliveri, R. S., Bello, S., & Biering‐Sørensen, F. (2013) · Neurobiology of Disease · doi:10.1016/j.nbd.2013.10.014 · Cited by 150 · see citing works
Neuroprotection After Stroke by Targeting NOX4 As a Source of Oxidative Stress
Radermacher, K. A., Wingler, K., Langhauser, F., Altenhöfer, S., Kleikers, P. W. M., Hermans, J. J. R., de Angelis, M. H., Kleinschnitz, C., & Schmidt, H. H. H. W. (2012) · Antioxidants and Redox Signaling · doi:10.1089/ars.2012.4797 · Cited by 147 · see citing works
“Finally, another aspect that should not be neglected is publication bias (55).”
Backgroundde Vries, R. B. M., Wever, K. E., Avey, M. T., Stephens, M. L., Sena, E. S., & Leenaars, M. (2014) · ILAR Journal · doi:10.1093/ilar/ilu043 · Cited by 165 · see citing works
“An SR of tPA in models of stroke (Sena et al. 2010a) showed that a total of 3,388 animals had been used.”
BackgroundMethodSelf-citationProvencher, S., Archer, S. L., Ramirez, F. D., Hibbert, B. M., Paulin, R., Boucherat, O., Lacasse, Y., & Bonnet, S. (2018) · Circulation Research · doi:10.1161/circresaha.117.312579 · Cited by 137 · see citing works
“There is a strong possibility that such studies are missing because of their uninteresting or unwelcome findings (ie, true publication bias), leading almost inevitably to major overstatements of efficacy.(87) More subtle publication biases arise when studies are published in obscure journals, rarely cited, inappropria…”
BackgroundAccelerating drug discovery for Alzheimer's disease: best practices for preclinical animal studies
Shineman, D. W., Basi, G. S., Bizon, J. L., Colton, C. A., Greenberg, B., Hollister, B. A., Lincecum, J. M., Leblanc, G. G., Lee, L. H., Luo, F. R., et al. (2011) · Alzheimer s Research & Therapy · doi:10.1186/alzrt90 · Cited by 136 · see citing works
“Publication bias fueled by a decreased ability or desire to publish negative results represents a huge problem for the fi eld [18].”
An Analysis of Mesenchymal Stem Cell-Derived Extracellular Vesicles for Preclinical Use
Tieu, A., Lalu, M. M., Slobodian, M., Gnyra, C., Fergusson, D. A., Montroy, J. R., Burger, D. E., Stewart, D. J., & Allan, D. (2020) · ACS Nano · doi:10.1021/acsnano.0c01363 · Cited by 136 · see citing works
Antonic‐Baker, A., Sena, E. S., Lees, J. S., Wills, T. E., Skeers, P., Batchelor, P., Macleod, M. R., & Howells, D. W. (2013) · PLoS Biology · doi:10.1371/journal.pbio.1001738 · Cited by 133 · see citing works
“Small study bias was suggested with asymmetry of the funnel plot (Figure 2A) and Egger regression (Figure 2B) but not by Trim and Fill.”
MethodSelf-citationStem Cell-Based Therapy for Experimental Stroke: A Systematic Review and Meta-Analysis
Lees, J. S., Sena, E. S., Egan, K. J., Antonic‐Baker, A., Koblar, S. A., Howells, D. W., & Macleod, M. R. (2012) · International Journal of Stroke · doi:10.1111/j.1747-4949.2012.00797.x · Cited by 132 · see citing works
Self-citationter Riet, G., Korevaar, D. A., Leenaars, M., Sterk, P. J., Noorden, C. J. F. V., Bouter, L. M., Lutter, R., Elferink, R. P. J. O., & Hooft, L. (2012) · PLoS ONE · doi:10.1371/journal.pone.0043404 · Cited by 131 · see citing works
“[6] Evidence from clinical research on humans suggests that between 46 and 67 percent of studies are not published [14–17], and that in those published, positive findings are over-emphasized.”
Bringing rigour to translational medicine
Howells, D. W., Sena, E. S., & Macleod, M. R. (2014) · Nature Reviews Neurology · doi:10.1038/nrneurol.2013.232 · Cited by 130 · see citing works
Self-citationLai, M.-C., Chandrasekera, P. C., & Barnard, N. D. (2014) · Nutrition and Diabetes · doi:10.1038/nutd.2014.30 · Cited by 129 · see citing works
Extrapolating from animals to humans
Ioannidis, J. P. A. (2012) · Science Translational Medicine · doi:10.1126/scitranslmed.3004631 · Cited by 124 · see citing works
“For example, one such empirical evaluation assessed the accumulated evidence on 16 interventions tested in animal models of acute ischemic stroke, a total of 525 unique scientif c publications (8).”
BackgroundCerebroprotection for Acute Ischemic Stroke: Looking Ahead
Lyden, P. D. (2021) · Stroke · doi:10.1161/strokeaha.121.032241 · Cited by 125 · see citing works
More than 3Rs: The importance of scientific validity for harm-benefit analysis of animal research
Würbel, H. (2017) · Lab Animal · doi:10.1038/laban.1220 · Cited by 152 · see citing works
Evidence-Based Research Series-Paper 1: What Evidence-Based Research is and why is it important?
Robinson, K. A., Brunnhuber, K., Ciliska, D., Juhl, C. B., Christensen, R., & Lund, H. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.07.020 · Cited by 146 · see citing works
Reproducibility and Rigor in Animal-Based Research
Macleod, M., & Mohan, S. (2019) · ILAR Journal · doi:10.1093/ilar/ilz015 · Cited by 104 · see citing works
“That is, the reports in the literature represent experiments where an effect was observed by chance but similar experiments where no effect was observed were never published (publication bias [8,9]).”
ResultSelf-citationLalu, M. M., Sullivan, K., Mei, S. H. J., Moher, D., Straus, A., Fergusson, D. A., Stewart, D. J., Jazi, M., Macleod, M. R., Winston, B. W., et al. (2016) · eLife · doi:10.7554/elife.17850 · Cited by 116 · see citing works
“This is of particular concern given that studies replicating high profile experiments fail in up to 50–90% of attempts (Begley and Ellis, 2012; Scott et al., 2008; Steward et al., 2012) and significant publication bias results in a skewed representation of effects (Sena et al., 2010).”
BackgroundSelf-citationSchmidt‐Pogoda, A., Bonberg, N., Koecke, M. H. M., Strecker, J., Wellmann, J., Bruckmann, N. M., Beuker, C., Schäbitz, W., Meuth, S. G., Wiendl, H., et al. (2019) · Annals of Neurology · doi:10.1002/ana.25643 · Cited by 115 · see citing works
“The publication bias in experimental studies, early clinical trials, and phase 3 clinical trials was estimated by funnel plot asymmetry and a trim-and-fill approach, as previously described.(8) To prepare funnel plots, we plotted the effect estimates from individual studies against each study’s precision, that is, the…”
BackgroundMethodInfluentialA search filter for increasing the retrieval of animal studies in Embase
de Vries, R. B. M., Hooijmans, C. R., Tillema, A., Leenaars, M., & Ritskes-Hoitinga, M. J. (2011) · Laboratory Animals · doi:10.1258/la.2011.011056 · Cited by 114 · see citing works
Holman, C., Piper, S. K., Grittner, U., Diamantaras, A. A., Kimmelman, J., Siegerink, B., & Dirnagl, U. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002331 · Cited by 114 · see citing works
“In our simulation, our starting point was a sample size of 8 + 8, which is representative of many published experiments [21,22].”
BackgroundMethodInfluentialOpportunities for improving animal welfare in rodent models of epilepsy and seizures
Lidster, K., Jefferys, J. G. R., Blümcke, I., Crunelli, V., Flecknell, P. A., Frenguelli, B. G., Gray, W., Kaminski, R. M., Pitkänen, A., Ragan, I., et al. (2015) · Journal of Neuroscience Methods · doi:10.1016/j.jneumeth.2015.09.007 · Cited by 114 · see citing works
Fighting Publication Bias: Introducing the Negative Results Section
Dirnagl, U., & Lauritzen, M. (2010) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.2010.51 · Cited by 112 · see citing works
“Macleod and colleagues have now, for the first time, quantified publication bias in animal stroke studies and demonstrated that it leads to major overstatements of efficacy (Sena et al, 2010).”
BackgroundMacDougall, N., & Muir, K. W. (2010) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.2010.210 · Cited by 111 · see citing works
Predicting Harms and Benefits in Translational Trials: Ethics, Evidence, and Uncertainty
Kimmelman, J., & London, A. J. (2011) · PLoS Medicine · doi:10.1371/journal.pmed.1001010 · Cited by 108 · see citing works
“In one recent analysis, publication bias in preclinical stroke studies led to a 30% overestimation of treatment effect size [20].”
BackgroundKorevaar, D. A., Hooft, L., & ter Riet, G. (2011) · Laboratory Animals · doi:10.1258/la.2011.010121 · Cited by 110 · see citing works
Can animal data predict human outcome? Problems and pitfalls of translational animal research
Martić-Kehl, M. I., Schibli, R., & Schubiger, P. A. (2012) · European Journal of Nuclear Medicine and Molecular Imaging · doi:10.1007/s00259-012-2175-z · Cited by 110 · see citing works
“In a recent publication, Sena and colleagues came to the conclusion that publication bias accounts for one third of the effect in animal stroke studies that was assessed in a systematic review [26].”
BackgroundSimilar Effect of Autologous and Allogeneic Cell Therapy for Ischemic Heart Disease
Lorkeers, S. J. J. O., Eding, J., Vesterinen, H. M., van der Spoel, T. I. G., Sena, E. S., Duckers, H. J., Doevendans, P. A., Macleod, M. R., & Chamuleau, S. A. J. (2014) · Circulation Research · doi:10.1161/circresaha.116.304872 · Cited by 108 · see citing works
“1%.(40) Largest contributors to publication bias are authors or researchers not willing to put effort in publishing negative results and editors who tend to select papers that are most exciting.”
InfluentialSelf-citationvan Assen, M. A. L. M., van Aert, R. C. M., Nuijten, M. B., & Wicherts, J. M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0084896 · Cited by 110 · see citing works
“[4–13].”
Sena, E., Briscoe, C., Howells, D., Donnan, G., Sandercock, P., & Macleod, M. (2010) · Journal of Cerebral Blood Flow and Metabolism · doi:10.1038/jcbfm.2010.116 · Cited by 118 · see citing works
Self-citationDo stroke models model stroke?
Mergenthaler, P., & Meisel, A. (2012) · Disease Models & Mechanisms · doi:10.1242/dmm.010033 · Cited by 106 · see citing works
“Second, most investigators disregard the fact that most stroke patients are not young or middle-aged males without any comorbidities (Howells et al., 2010; Sena et al., 2010).”
BackgroundInfluentialTracking Replicability as a Method of Post-Publication Open Evaluation
Hartshorne, J. K., & Schachner, A. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00008 · Cited by 102 · see citing works
“…(Dickersin et al., 1987, 1992; Easterbrook et al., 1991; Callaham et al., 1998; Misakian and Bero, 1998; Olson et al., 2002; Dwan et al., 2008; Sena et al., 2010), with additional reports in political science (Gerberg et al., 2001), ecology and evolution (Jennions and Møller, 2002a), and…”
BackgroundIntroducing Therioepistemology: the study of how knowledge is gained from animal research
Garner, J. P., Gaskill, B. N., Weber, E. M., Dallaire, J. A., & Pritchett‐Corning, K. R. (2017) · Lab Animal · doi:10.1038/laban.1224 · Cited by 106 · see citing works
Banik, A., Brown, R. E., Bamburg, J., Lahiri, D. K., Khurana, D., Friedland, R. P., Chen, W., Ding, Y., Mudher, A., Padjen, A. L., et al. (2015) · Journal of Alzheimer’s Disease · doi:10.3233/jad-150136 · Cited by 102 · see citing works
“Thus, the validity of the outcomes may be questionable, widening the chasm between preclinical and clinical investigations [218].”
Software Tools for Literature Screening in Systematic Reviews in Biomedical Research
van der Mierden, S. (2019) · ALTEX · doi:10.14573/altex.1902131 · Cited by 99 · see citing works
Alaverdashvili, M., & Whishaw, I. Q. (2013) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2013.03.026 · Cited by 95 · see citing works
- Authors
- Sena, E. S., van der Worp, H. B., Bath, P. M. W., Howells, D. W., & Macleod, M. R.
- Affiliations
- The University of Melbourne (AU)
- University of Nottingham (GB)
- NHS Forth Valley (GB)
- Corresponding author
- Macleod, Malcolm R.
- Venue
- PLoS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2010-03-30
- Volume
- 8
- Issue
- 3
- Pages
- e1000344
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- PMC
- PMC2846857
- OpenAlex
- W2164759539
- Cited by
- 600
- References
- 42
- Citations by year
- 2026: 34 · 2025: 33 · 2024: 19 · 2023: 26 · 2022: 28 · 2021: 32 · 2020: 48 · 2019: 41 · 2018: 26 · 2017: 39 · 2016: 44 · 2015: 41 · 2014: 61 · 2013: 36 · 2012: 42
- FWCI
- 42.33
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- PLoS Biology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Minerva Access (University of Melbourne) (submittedVersion, cc-by)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- RePEc: Research Papers in Economics (submittedVersion)
- WBI Studies Repository (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Figshare (submittedVersion, cc-by)
- Funders
- Medical Research Council
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Health Sciences › Veterinary › Small Animals › Animal testing and alternatives
- Keywords
- publication bias, animal models of stroke, acute ischemic stroke, systematic reviews, infarct size, animal disease models
- MeSH terms
- Animals, Disease Models, Animal, Humans, Review Literature as Topic, Meta-Analysis as Topic, Models, Statistical, Treatment Outcome, Publication Bias, Stroke
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
