Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
The replicability of research results has been a cause of increasing concern to the scientific community. The long-held belief that experimental standardization begets replicability has also been recently challenged, with the observation that the reduction of variability within studies can lead to idiosyncratic, lab-specific results that cannot be replicated. An alternative approach is to, instead, deliberately introduce heterogeneity, known as “heterogenization” of experimental design. Here, we explore a novel perspective in the heterogenization program in a meta-analysis of variability in observed phenotypic outcomes in both control and experimental animal models of ischemic stroke. First, by quantifying interindividual variability across control groups, we illustrate that the amount of heterogeneity in disease state (infarct volume) differs according to methodological approach, for example, in disease induction methods and disease models. We argue that such methods may improve replicability by creating diverse and representative distribution of baseline disease state in the reference group, against which treatment efficacy is assessed. Second, we illustrate how meta-analysis can be used to simultaneously assess efficacy and stability (i.e., mean effect and among-individual variability). We identify treatments that have efficacy and are generalizable to the population level (i.e., low interindividual variability), as well as those where there is high interindividual variability in response; for these, latter treatments translation to a clinical setting may require nuance. We argue that by embracing rather than seeking to minimize variability in phenotypic outcomes, we can motivate the shift toward heterogenization and improve both the replicability and generalizability of preclinical research.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
mice: Multivariate Imputation by Chained Equations in R
Buuren, S. V., & Groothuis-Oudshoorn, K. (2011) · Journal of Statistical Software · doi:10.18637/jss.v045.i03 · Cited by 13,870
Cumpston, M., Li, T., Page, M. J., Chandler, J., Welch, V. A., Higgins, J. P., & Thomas, J. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.ed000142 · Cited by 7,218
Feature Selection with the Boruta Package
Kursa, M. B., & Rudnicki, W. R. (2010) · Journal of Statistical Software · doi:10.18637/jss.v036.i11 · Cited by 5,659
1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209
Schafer, J. L. (1999) · Statistical Methods in Medical Research · doi:10.1177/096228029900800102 · Cited by 3,949 · cited 1×
“For all analyses, we dealt with missing data via multiple imputation [65,66] using the package mice [67] as follows: We first generated multiple, simulated datasets (m = 20) by replacing missing values with possible values under the assumption that data are missing at random (MAR) [68,69].”
MethodAggregation, Variance and the Mean
TAYLOR, L. R. (1961) · Nature · doi:10.1038/189732a0 · Cited by 2,496 · cited 2×
“, Taylor’s law, where the variance increases with an increase in the mean [42]; S1 Fig).”
BackgroundPolicy: NIH to balance sex in cell and animal studies
Clayton, J. A., & Collins, F. S. (2014) · Nature · doi:10.1038/509282a · Cited by 1,816
Genetics of Mouse Behavior: Interactions with Laboratory Environment
Crabbe, J. C., Wahlsten, D., & Dudek, B. C. (1999) · Science · doi:10.1126/science.284.5420.1670 · Cited by 1,666
What does research reproducibility mean?
Goodman, S. N., Fanelli, D., & Ioannidis, J. P. A. (2016) · Science Translational Medicine · doi:10.1126/scitranslmed.aaf5027 · Cited by 1,293 · cited 2×
“We further discuss these results below in the context of their implications for improving the replicability (also defined as “Results reproducibility” [1]) and generalizability of preclinical studies.”
BackgroundStanley, T. D., & Doucouliagos, H. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1095 · Cited by 916
Cited by
The Stroke Preclinical Assessment Network: Rationale, Design, Feasibility, and Stage 1 Results
Lyden, P. D., Bosetti, F., Diniz, M. A., Rogatko, A., Koenig, J. I., Lamb, J., Nagarkatti, K. A., Cabeen, R. P., Hess, D. C., Kamat, P. K., et al. (2022) · Stroke · doi:10.1161/strokeaha.121.038047 · Cited by 73
Jaric, I., Voelkl, B., Clerc, M., Schmid, M. W., Novak, J., Rosso, M., Rufener, R., von Kortzfleisch, V. T., Richter, S. H., Buettner, M., et al. (2022) · PLoS Biology · doi:10.1371/journal.pbio.3001837 · Cited by 32
Cosden, M., Jinn, S., Yao, L., Gretzula, C. A., Kandebo, M., Toolan, D., Hatcher, N. G., Ma, L., Lemaire, W., Adam, G. C., et al. (2021) · Neurobiology of Disease · doi:10.1016/j.nbd.2021.105507 · Cited by 23
Metadata
- Authors
- Usui, T., Macleod, M. R., McCann, S. K., Senior, A. M., & Nakagawa, S.
- Venue
- PLOS Biology
- Publisher
- Public Library of Science (PLoS)
- Published
- 2021-05-19
- Volume
- 19
- Issue
- 5
- Pages
- e3001009
- ISSN
- 1544-9173 (ISSN-L)
- 1545-7885
- OpenAlex
- W3162160784
- Cited by
- 72
- References
- 15
- FWCI
- 2.95
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research. https://app.cochat.ai/papers/10.1371/journal.pbio.3001009
- The paper
- Usui, T., Macleod, M. R., McCann, S. K., Senior, A. M., & Nakagawa, S. (2021). Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research. PLOS Biology. https://doi.org/10.1371/journal.pbio.3001009
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.