The DOI matches its Crossref record; 92 of the 92 references it cites were found in a registry or an index.
The problem of pseudoreplication in neuroscientific studies: is it affecting your analysis?
Open accessAbstract
BACKGROUND: Pseudoreplication occurs when observations are not statistically independent, but treated as if they are. This can occur when there are multiple observations on the same subjects, when samples are nested or hierarchically organised, or when measurements are correlated in time or space. Analysis of such data without taking these dependencies into account can lead to meaningless results, and examples can easily be found in the neuroscience literature. RESULTS: A single issue of Nature Neuroscience provided a number of examples and is used as a case study to highlight how pseudoreplication arises in neuroscientific studies, why the analyses in these papers are incorrect, and appropriate analytical methods are provided. 12% of papers had pseudoreplication and a further 36% were suspected of having pseudoreplication, but it was not possible to determine for certain because insufficient information was provided. CONCLUSIONS: Pseudoreplication can undermine the conclusions of a statistical analysis, and it would be easier to detect if the sample size, degrees of freedom, the test statistic, and precise p-values are reported. This information should be a requirement for all publications.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Pseudoreplication has been addressed repeatedly in ecology, agriculture, psychology, and neuroscience and acknowledged as one of the most common statistical mistakes in scientific literature4–9.”
From A practical solution to pseudoreplication bias in single-cell studies (2021) · cites this paper as [8]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Reporting n = 50 in this case is an example of pseudoreplication [26].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“It underestimates the true variability in a study, which can lead to false positives and invalidate the analysis and resulting conclusions [26,27].”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [26]
The cited source says what the claim attributes to it.
In the citing paper
“Conflating experimental units with subsamples or repeated measurements can lead to artificial inflation of the sample size.”
From Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0 (2020) · cites this paper as [26]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
R: A Language and Environment for Statistical Computing
R Core Team (2000) · doi:10.32614/r.manuals · Cited by 355,893
Mixed effects models and extensions in ecology with R
Zuur, A. F., Ieno, E. N., Walker, N., Saveliev, A. A., & Smith, G. M. (2009) · Statistics for Biology and Health · doi:10.1007/978-0-387-87458-6 · Cited by 17,963
Data Analysis Using Regression and Multilevel/Hierarchical Models
Gelman, A., & Hill, J. (2006) · doi:10.1017/cbo9780511790942 · Cited by 13,597
Gentleman R: R: A language for data analysis and graphics
Ihaka, R., & Gentleman, R. (1996) · Journal of Computational and Graphical Statistics · doi:10.1080/10618600.1996.10474713 · Cited by 11,747
Mixed-Effects Models in Sand S-PLUS
Pinheiro, J. C., & Bates, D. M. (2000) · Statistics and Computing · doi:10.1007/978-1-4419-0318-1 · Cited by 11,187
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Mixed-Effects Models in S and S-PLUS
(2001) · Technometrics · doi:10.1198/tech.2001.s574 · Cited by 9,278
Pseudoreplication and the Design of Ecological Field Experiments
Hurlbert, S. H. (1984) · Ecological Monographs · doi:10.2307/1942661 · Cited by 8,377
Crawley, M. J. (2007) · Wiley eBooks · doi:10.1002/9780470515075 · Cited by 6,980
Crawley, M. J. (2007) · Medical Entomology and Zoology · Cited by 5,431
Cited by
Sorted by citations
Reporting animal research: Explanation and elaboration for the ARRIVE guidelines 2.0
Percie du Sert, N., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2020) · PLOS Biology · doi:10.1371/journal.pbio.3000411 · Cited by 2,677
“Reporting n = 50 in this case is an example of pseudoreplication [26].”
BackgroundSelf-citationA call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196
Self-citationExperimental design and analysis and their reporting: new guidance for publication in BJP
Curtis, M. J., Bond, R. A., Spina, D., Ahluwalia, A., Alexander, S. P. H., Giembycz, M. A., Gilchrist, A., Hoyer, D., Insel, P. A., Izzo, A. A., et al. (2015) · British Journal of Pharmacology · doi:10.1111/bph.12856 · Cited by 1,057
“This mistake is known as pseudo-replication (Lazic, 2010) and increases the chance of false positive results.”
BackgroundSuperPlots: Communicating reproducibility and variability in cell biology
Lord, S. J., Velle, K. B., Mullins, R. D., & Fritz-Laylin, L. K. (2020) · Journal of Cell Biology · doi:10.1083/jcb.202001064 · Cited by 718
“…standard error of the mean should be calculated using biological replicates—independent measurements of a population of interest, typically from independent samples or separate experiments (Hurlbert, 1984; Lazic, 2010; Vaux et al., 2012; Aarts et al., 2015; Naegle et al., 2015; Lazic et al., 2018).”
BackgroundMethodReporting animal research: Explanation and Elaboration for the ARRIVE guidelines 2019Preprint
du Sert, N. P., Ahluwalia, A., Alam, S., Avey, M. T., Baker, M., Browne, W. J., Clark, A., Cuthill, I. C., Dirnagl, U., Emerson, M., et al. (2019) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/703355 · Cited by 557
“This may invalidate the analysis and resulting conclusions [21, 22] (see also item 7 – Statistical methods).”
BackgroundMethodSynaptic degeneration in Alzheimer disease
Tzioras, M., McGeachan, R. I., Durrant, C. S., & Spires-Jones, T. L. (2023) · Nature Reviews Neurology · doi:10.1038/s41582-022-00749-z · Cited by 504
Linear models and linear mixed effects models in R with linguistic applicationsPreprint
Winter, B. (2013) · arXiv.org · doi:10.48550/arxiv.1308.5499 · Cited by 658
“… – so much in fact, that there’s a whole literature associated with this violation, starting from Hurlbert (1984) for ecology, Freeberg and Lucas (2009) for psychology, Lazic (2010) for neuroscience and my own small paper for…”
McLaren, S. J., Macpherson, T. W., Coutts, A. J., Hurst, C., Spears, I. R., & Weston, M. (2017) · Sports Medicine · doi:10.1007/s40279-017-0830-z · Cited by 421
“The inevitable repeated measures nature of this work should be met with the appropriate analyses to avoid inference error arising from pseudoreplication [107].”
BackgroundComputation of measures of effect size for neuroscience data sets
Hentschke, H., & Stüttgen, M. C. (2011) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2011.07902.x · Cited by 421
“…the documented existence of these misconceptions, the application of NHST suffers from frequent analysis mistakes (Garcı´a-Berthou & Alcaraz, 2004; Lazic, 2010; Nieuwenhuis et al. , 2011), and it has been suggested that replacing NHST with an analysis couched in the framework of MES and CIs is…”
MethodA practical solution to pseudoreplication bias in single-cell studies
Zimmerman, K. D., Espeland, M. A., & Langefeld, C. D. (2021) · Nature Communications · doi:10.1038/s41467-021-21038-1 · Cited by 356
Fair Statistical Communication in HCI
Dragicevic, P. (2016) · Human-computer interaction series · doi:10.1007/978-3-319-26633-6_13 · Cited by 365
What exactly is ‘N’ in cell culture and animal experiments?
Lazic, S. E., Clarke-Williams, C. J., & Munafò, M. R. (2018) · PLOS Biology · doi:10.1371/journal.pbio.2005282 · Cited by 291
Self-citationMagezi, D. A. (2015) · Frontiers in Psychology · doi:10.3389/fpsyg.2015.00002 · Cited by 314
Acoustic indices as proxies for biodiversity: a meta‐analysis
Alcocer, I., Lima, H., Sugai, L. S. M., & Llusia, D. (2022) · Biological Reviews · doi:10.1111/brv.12890 · Cited by 273
“The percentage of pseudoreplicated studies was similar to those recently identified in other scientific areas (e.g. Lazic, 2010; Waller et al., 2013).”
ResultLazic, S. E., & Essioux, L. (2013) · BMC Neuroscience · doi:10.1186/1471-2202-14-37 · Cited by 260
“animals are nested within litters) and avoids artificially inflating the sample size (also known as pseudoreplication [12, 15]).”
BackgroundMethodSelf-citationLazic, S. E., & Essioux, L. (2016) · Cited by 182
Self-citationGoedhart, J. (2021) · Molecular Biology of the Cell · doi:10.1091/mbc.e20-09-0583 · Cited by 181
“This approach is statistically valid (Galbraith et al. , 2010; Aarts et al. , 2014), but it is recommended to keep the number of measurements for technical each replicate similar (Lazic, 2010).”
Fundus-controlled perimetry (microperimetry): Application as outcome measure in clinical trials
Pfau, M., Jolly, J. K., Wu, Z., Denniss, J., Lad, E. M., Guymer, R. H., Fleckenstein, M., Holz, F. G., & Schmitz-Valckenberg, S. (2020) · Progress in Retinal and Eye Research · doi:10.1016/j.preteyeres.2020.100907 · Cited by 158
“Accordingly, simple linear regression analyses or correlation analyses across individual test-points (as unfortunately commonly reported) will lead to incorrectly low P-values (Lazic, 2010).”
BackgroundAssessment of cerebral autoregulation: the quandary of quantification
Tzeng, Y., Ainslie, P. N., Cooke, W. H., Peebles, K. C., Willie, C. K., MacRae, B. A., Smirl, J. D., Horsman, H. M., & Rickards, C. A. (2012) · American Journal of Physiology-Heart and Circulatory Physiology · doi:10.1152/ajpheart.00328.2012 · Cited by 155
Sikkel, M. B., Francis, D. P., Howard, J., Gordon, F., Rowlands, C. T., Peters, N. S., Lyon, A. R., Harding, S. E., & MacLeod, K. T. (2017) · Cardiovascular Research · doi:10.1093/cvr/cvx151 · Cited by 150
“This problem, the ability of large sample numbers to deceive us when we do not recognize clustering, has been previously identified in the field of neuroscience.(12,14)”
BackgroundResultMixed Models Offer No Freedom from Degrees of Freedom
Arnqvist, G. (2020) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2019.12.004 · Cited by 135
Payne, B. R., Lee, C., & Federmeier, K. D. (2015) · Psychophysiology · doi:10.1111/psyp.12515 · Cited by 131
“…statistical dependency that arises from the kinds of nested and hierarchically structured data that are common in these fields (Aarts, Verhage, Veenvliet, Dolan, & van der Sluis, 2014; Baayen, Davidson, & Bates, 2008; Jaeger, 2008; Lazic, 2010; Locker, Hoffman, & Bovaird, 2007; Payne et al., 2014).”
Application of the Hierarchical Bootstrap to Multi-Level Data in Neuroscience
Saravanan, V., Berman, G. J., & Sober, S. J. (2019) · doi:10.1101/819334 · Cited by 170
“While pseudoreplication was first extensively reported on in ecological studies [5, 48], it has since been identified as a common problem in other fields, including neuroscience [6].”
BackgroundMurphy, A. E., & Skene, N. (2022) · Nature Communications · doi:10.1038/s41467-022-35519-4 · Cited by 121
The Design and Statistical Analysis of Animal Experiments
Bate, S. T., & Clark, R. A. (2014) · doi:10.1017/cbo9781139344319 · Cited by 150
Cait, J., Cait, A., Scott, R. W., Winder, C. B., & Mason, G. J. (2022) · BMC Biology · doi:10.1186/s12915-021-01184-0 · Cited by 114
Using Biological Insight and Pragmatism When Thinking about Pseudoreplication
Colegrave, N., & Ruxton, G. D. (2017) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2017.10.007 · Cited by 91
rmcorrShiny: A web and standalone application for repeated measures correlation
Marusich, L. R., & Bakdash, J. Z. (2021) · F1000Research · doi:10.12688/f1000research.55027.1 · Cited by 87
Donnelly, N. A., Holtzman, T., Rich, P. D., Nevado‐Holgado, A. J., Fernando, A. B. P., Dijck, G. V., Holzhammer, T., Paul, O., Ruther, P., Paulsen, O., et al. (2014) · PLoS ONE · doi:10.1371/journal.pone.0111300 · Cited by 90
“Simply grouping all available trials across all electrodes and rats without accounting for this structure violates the assumption of independence between observations required for standard statistical tests, and increases the type 1 error rate through pseudoreplication [69,70].”
BackgroundImproving transparency and scientific rigor in academic publishing
Prager, E. M., Chambers, K. E., Plotkin, J. L., McArthur, D. L., Bandrowski, A. E., Bansal, N., Martone, M. E., Bergstrom, H. C., Bespalov, A., & Graf, C. (2019) · Journal of Neuroscience Research · doi:10.1002/jnr.24340 · Cited by 83
Data visualization, bar naked: A free tool for creating interactive graphics
Weissgerber, T. L., Savic, M. D., Winham, S. J., Stanisavljevic, D. M., Garovic, V. D., & Milic, N. M. (2017) · Journal of Biological Chemistry · doi:10.1074/jbc.ra117.000147 · Cited by 76
“Clusters of non-independent data, such as technical replicates or mice from the same litter, are common in basic science research, yet these data are often inappropriately analyzed (32).”
BackgroundMoen, E. L., Fricano-Kugler, C. J., Luikart, B. W., & O’Malley, A. J. (2016) · PLoS ONE · doi:10.1371/journal.pone.0146721 · Cited by 78
“However, most of those papers (82%) did not have sufficient information for the reader to determine whether each observation was considered independent in the analyses [2].”
BackgroundPseudoreplication in physiology: More means less
Eisner, D. A. (2021) · The Journal of General Physiology · doi:10.1085/jgp.202012826 · Cited by 79
“The purpose of this tutorial is to concentrate on one statistical issue that, although widely discussed (Lazic, 2010; Sikkel et al., 2017; Lazic et al., 2018), is still a major problem: this is the subject of pseudoreplication (Hurlbert, 1984) in which data points are treated as independent…”
BackgroundAarts, E., Dolan, C. V., Verhage, M., & van der Sluis, S. (2015) · BMC Neuroscience · doi:10.1186/s12868-015-0228-5 · Cited by 75
“This design has received some attention in the neuroscience literature, focusing specifically on ways to correctly analyze such data [1–4].”
Hensel, N., Schön, A., Konen, T., Lübben, V., Förthmann, B., Baron, O., Grothe, C., Leifheit‐Nestler, M., Claus, P., & Haffner, D. (2016) · Journal of Neurochemistry · doi:10.1111/jnc.13585 · Cited by 78
Reproducibility of science: Fraud, impact factors and carelessness
Eisner, D. A. (2017) · Journal of Molecular and Cellular Cardiology · doi:10.1016/j.yjmcc.2017.10.009 · Cited by 80
Systematic heterogenization for better reproducibility in animal experimentation
Richter, S. H. (2017) · Lab Animal · doi:10.1038/laban.1330 · Cited by 73
Repeated measurements, bilateral observations and pseudoreplicates, why does it matter?
Ranstam, J. (2012) · Osteoarthritis and Cartilage · doi:10.1016/j.joca.2012.02.011 · Cited by 73
Pseudoreplication: a widespread problem in primate communication research
Waller, B. M., Warmelink, L., Liebal, K., Micheletta, J., & Slocombe, K. E. (2013) · Animal Behaviour · doi:10.1016/j.anbehav.2013.05.038 · Cited by 68
Reinventing Biostatistics Education for Basic Scientists
Weissgerber, T. L., Garovic, V. D., Milin-Lazovic, J. S., Winham, S. J., Zoran Obradovic, A., Trzeciakowski, J. P., & Milic, N. M. (2016) · PLoS Biology · doi:10.1371/journal.pbio.1002430 · Cited by 73
“This includes using incorrect or suboptimal tests [1,2], summarizing data that were analyzed by nonparametric techniques as mean and standard deviation or standard error [4], reporting p-values that are inconsistent with the test statistic [5,6], p-hacking [7], and analyzing nonindependent data as though they are inde…”
BackgroundMethodInfluentialIndependence and generalizability in linguistics
Winter, B., & Grice, M. (2021) · Linguistics · doi:10.1515/ling-2019-0049 · Cited by 74
Mickleburgh, H. L., Schwalbe, E. C., Bonicelli, A., Mizukami, H., Sellitto, F., Starace, S., Wescott, D. J., Carter, D. O., & Procopio, N. (2021) · Journal of Proteome Research · doi:10.1021/acs.jproteome.0c00992 · Cited by 65
Fundamental relationships between blood pressure and cerebral blood flow in humans
Tzeng, Y., MacRae, B. A., Ainslie, P. N., & Chan, G. S. H. (2014) · Journal of Applied Physiology · doi:10.1152/japplphysiol.00366.2014 · Cited by 65
Betts, J. A., Gonzalez, J. T., Burke, L. M., Close, G. L., Garthe, I., James, L. J., Jeukendrup, A. E., Morton, J. P., Nieman, D. C., Peeling, P., et al. (2020) · International Journal of Sport Nutrition and Exercise Metabolism · doi:10.1123/ijsnem.2019-0326 · Cited by 63
“14b Researchers should avoid pooling data for participants over multiple experimental conditions or groups and analyzing the data without modeling these conditions and groups, a problem known as “pseudoreplication” (Lazic, 2010).”
BackgroundToward more robust plant‐soil feedback research
Rinella, M. J., & Reinhart, K. O. (2018) · Ecology · doi:10.1002/ecy.2146 · Cited by 62
Improving transparency and scientific rigor in academic publishing
Prager, E. M., Chambers, K. E., Plotkin, J. L., McArthur, D. L., Bandrowski, A. E., Bansal, N., Martone, M. E., Bergstrom, H. C., Bespalov, A., & Graf, C. (2018) · Brain and Behavior · doi:10.1002/brb3.1141 · Cited by 70
Vinken, K., Vogels, R., & de Beeck, H. O. (2017) · Current Biology · doi:10.1016/j.cub.2017.02.024 · Cited by 58
Pelucchi, S., Vandermeulen, L., Pizzamiglio, L., Aksan, B., Yan, J., Konietzny, A., Bonomi, E., Borroni, B., Padovani, A., Rust, M. B., et al. (2020) · Brain Communications · doi:10.1093/braincomms/fcaa086 · Cited by 55
“…the use of mixedeffects models to fully account for the complex design of multi-level hierarchically structured imaging data (Paternoster et al., 2018); this approach improved our ability to control variability at culture and neuron levels and provided us more precise mean estimates (Lazic, 2010).”
BackgroundSympathetic regulation of the human cerebrovascular response to carbon dioxide
Peebles, K. C., Ball, O. G., MacRae, B. A., Horsman, H. M., & Tzeng, Y. (2012) · Journal of Applied Physiology · doi:10.1152/japplphysiol.00614.2012 · Cited by 58
“In contrast to conventional least squares regression, mixed effects models explicitly account for the fact that repeated PETCO2 and MCAV mean measurements made within subjects are correlated in nature and therefore violate the case independence assumption required for least squares regression (24).”
Development of the Biological Experimental Design Concept Inventory (BEDCI)
Deane, T., Nomme, K., Jeffery, E., Pollock, C. A., & Bírol, G. (2014) · CBE—Life Sciences Education · doi:10.1187/cbe.13-11-0218 · Cited by 56
“For example, the appropriate application of independent sampling continues to be a very difficult concept for researchers (Hurlbert, 1984; Lazic, 2010).”
BackgroundHill, J. M., Egan, J. F., Stauffer, G. E., & Diefenbach, D. R. (2014) · PLoS ONE · doi:10.1371/journal.pone.0098064 · Cited by 58
“Failure to account for this lack of independence would not necessarily favor insecticide acute toxicity in the modeling process, but the violation of basic linear model assumptions can result in spurious results [41].”
BackgroundMacartney, E. L., Lagisz, M., & Nakagawa, S. (2022) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2022.104554 · Cited by 61
PSEUDOREPLICATION IN PHONETIC RESEARCH
Winter, B. (2011) · ICPhS · Cited by 54
Estes, M. L., Prendergast, K., MacMahon, J. A., Cameron, S., Aboubechara, J. P., Farrelly, K., Sell, G. L., Haapanen, L. A. D., Schauer, J. D., Horta, A., et al. (2020) · Brain Behavior and Immunity · doi:10.1016/j.bbi.2020.04.061 · Cited by 59
“However, even using the optimized lot and most effective 30 mg/kg dose of pure poly(I:C), the behavioral outcomes for grooming for young adult male MIA offspring were so variable that there was only a non-statistically significant trend towards increased repetitive grooming (Figure 1I), which was also evident when we…”
BackgroundHopkins, W. D. (2013) · American Journal of Physical Anthropology · doi:10.1002/ajpa.22248 · Cited by 53
“…and others (Machlis et al., 1985) discussed independence of data points (or what Machlis et al. termed the “pooling fallacy” or “pseudoreplication” (Hurlbert, 1984; Lazic, 2010), they were referring to the necessity that each data point in any statistical analysis represent a single subject’s data.”
BackgroundSchewe, J., Seidel, E., Forslund, S. K., Markó, L., Peters, J., Müller, D. N., Fahlke, C. M., Stölting, G., & Scholl, U. I. (2019) · Nature Communications · doi:10.1038/s41467-019-13033-4 · Cited by 55
The effects of 17 alpha-estradiol to inhibit inflammation in vitro
Santos, R. D. S., de Fátima, L. A., Frank, A. P., Carneiro, E. M., & Clegg, D. J. (2017) · Biology of Sex Differences · doi:10.1186/s13293-017-0151-9 · Cited by 51
“For primary cells, two mice were pooled and processed for each experiment, which yielded three biological replicates [36, 37].”
MethodA Bayesian predictive approach for dealing with pseudoreplication
Lazic, S., Mellor, J., Ashby, M. C., & Munafo, M. (2019) · Scientific Reports · doi:10.1038/s41598-020-59384-7 · Cited by 44
Self-citationFarrar, B. G., Voudouris, K., & Clayton, N. S. (2021) · Animal Behavior and Cognition · doi:10.26451/abc.08.02.14.2021 · Cited by 53
“This leads to pseudoreplication (Hurlbert, 1984; Lazic, 2010) if this non-random sampling is not accounted for in the statistical models, and the consequence is that uncertainty intervals will be overly narrow, and the results will struggle to replicate in new samples – or generalize to them.”
BackgroundReynolds, P. S., Middleton, P., McCarthy, H. L., & Spiess, B. D. (2016) · Anesthesia & Analgesia · doi:10.1213/ane.0000000000001362 · Cited by 49
Using meta-analysis for evidence synthesis: The case of incomplete neutralization in German
Nicenboim, B., Roettger, T. B., & Vasishth, S. (2018) · Journal of Phonetics · doi:10.1016/j.wocn.2018.06.001 · Cited by 45
Baba, R., Matsuda, S., Arakawa, Y., Yamada, R. X., Suzuki, N., Ando, T., Oki, H., Igaki, S., Daini, M., Hattori, Y., et al. (2021) · Science Advances · doi:10.1126/sciadv.aba1187 · Cited by 46
Kelly, D. M., Strudwick, A. J., Atkinson, G., Drust, B., & Gregson, W. A. (2016) · Journal of Sports Sciences · doi:10.1080/02640414.2016.1142669 · Cited by 45
“…withinparticipant correlations according to the most appropriate statistical approach, which models the longitudinal dataset as a whole using the correct degrees of freedom, rather than by calculating correlations for individual players (Atkinson et al., 2011; Bland & Altman, 1995; Lazic, 2010).”
MethodLoss, C. M., Melleu, F. F., Domingues, K., Lino-de-Oliveira, C., & Viola, G. G. (2021) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2021.763428 · Cited by 44
“Some frequent sources of biases are pseudoreplication (Freeberg and Lucas, 2009; Lazic, 2010; Lazic et al., 2020; Eisner, 2021; Zimmerman et al., 2021) and violations of rules for experimental design, such as a priori calculating the sample size, unbiased allocation of samples to groups (randomization), blinded assess…”
BackgroundMethodBehavioural nudges for water conservation in unequal settings: Experimental evidence from Cape Town
Brick, K., Martino, S. D., & Visser, M. (2023) · Journal of Environmental Economics and Management · doi:10.1016/j.jeem.2023.102852 · Cited by 45
Visual Categorization of Natural Movies by Rats
Vinken, K., Vermaercke, B., & de Beeck, H. O. (2014) · Journal of Neuroscience · doi:10.1523/jneurosci.3663-13.2014 · Cited by 49
Young-Bernier, M., Kamil, Y., Tremblay, F., & Davidson, P. S. R. (2012) · Behavioral and Brain Functions · doi:10.1186/1744-9081-8-17 · Cited by 47
“When we performed an analysis across all individuals [52,53]; but see [54,55], SAI significantly predicted the memory factor score (r=−0.”
MethodJeffery, N. D., Hamilton, L., & Granger, N. P. (2011) · Veterinary Record · doi:10.1136/vr.d475 · Cited by 43
“…experiments cannot always be readily replicated (eg, compare lu and others 2002 with Steward and others 2006) and, furthermore, it is not unknown for statistical analysis of laboratory experiments to be erroneous, leading to incorrect interpretation of data (Kilkenny and others 2009, lazic 2010).”
BackgroundTrabecular bone patterning across the human hand
Stephens, N. B., Kivell, T. L., Pahr, D. H., Hublin, J., & Skinner, M. M. (2018) · Journal of Human Evolution · doi:10.1016/j.jhevol.2018.05.004 · Cited by 42
Bilius, L.-B., & Vatavu, R.-D. (2020) · Journal of Intelligent Transportation Systems · doi:10.1080/15472450.2020.1846127 · Cited by 42
The EGF receptor ligand amphiregulin controls cell division via FoxM1
Stoll, S. W., Stuart, P., Swindell, W. R., Tsoi, L. C., Li, B., Gandarillas, A., Lambert, S., Johnston, A., Nair, R. P., & Elder, J. (2015) · Oncogene · doi:10.1038/onc.2015.269 · Cited by 42
Kung, G. L., Vaseghi, M., Gahm, J. K., Shevtsov, J., Garfinkel, A., Shivkumar, K., & Ennis, D. B. (2018) · Frontiers in Physiology · doi:10.3389/fphys.2018.00826 · Cited by 41
“Last, to test whether DT invariants significantly remodeled between remote, BZ, and infarct regions for pooled data from all infarcted hearts, we performed a bootstrap analog to repeated measures ANOVA of the de-correlated data (Lazic, 2010).”
MethodKis, A., Hernádi, A., Miklósi, B., Kanizsàr, O., & Topál, J. (2017) · Frontiers in Behavioral Neuroscience · doi:10.3389/fnbeh.2017.00210 · Cited by 39
“…et al., 2012) and in preferential looking (image projection) paradigms (Faragó et al., 2010; Racca et al., 2012; Péter et al., 2013), which raises concerns of pseudoreplication (Lazic, 2010), e.g., the effect found might be specific to those images only and might not generalize to other stimuli.”
BackgroundMarche, K., Martel, A., & Apicella, P. (2017) · Frontiers in Systems Neuroscience · doi:10.3389/fnsys.2017.00052 · Cited by 39
“It should be underlined that neuronal data collected in two animals face us with potential problems related to repeated measures taken from the same subjects, thus leading to lack of statistical independence in the data (the so-called ‘‘pseudoreplication’’, for a review see Lazic, 2010).”
BackgroundCosta, J. A., Brito, J., Nakamura, F. Y., Figueiredo, P., & Rebelo, A. (2019) · The Journal of Strength and Conditioning Research · doi:10.1519/jsc.0000000000003407 · Cited by 39
“For instance, rather than pooling all the data, or calculating correlations separately for individual subjects, this approach quantifies the correlation, and associated 95% confidence interval (95% CI), between a covariate and outcome while taking into account the withinsubject nature of the study design (3,7,26).”
BackgroundDesigning, conducting, and reporting reproducible animal experiments
Wilson, E., Ramage, F. J., Wever, K. E., Sena, E. S., Macleod, M. R., & Currie, G. L. (2023) · Journal of Endocrinology · doi:10.1530/joe-22-0330 · Cited by 43
“Pseudoreplication, where the reported sample size is artificially inflated, is a common issue that may produce false-positive results and violates the assumptions of statistical tests used (Lazic 2010, Eisner 2021, Bannach-Brown et al.”
ResultWalcott, E. C., Higgins, E. A., & Desai, N. S. (2011) · Journal of Neuroscience · doi:10.1523/jneurosci.1341-11.2011 · Cited by 36
“This method of analysis (“by animal” rather than “by neuron”) obviates the potential problem of pseudoreplication, which can arise when recordings from a single animal are not statistically independent but are treated as such (Lazic, 2010).”
MethodDevin, S., Giambérini, L., & Pain-Devin, S. (2014) · Environment International · doi:10.1016/j.envint.2014.08.002 · Cited by 36
“Experimental design (for recent review and analysis of pseudoreplication, see Bastos et al., 2013; Lazic, 2010; Drummond and Vowler, 2012; Nikinmaa et al., 2012).”
MethodDevelopment of an oxidative stress in vitro assay in zebrafish (Danio rerio) cell lines
Lungu-Mitea, S., Oskarsson, A., & Lundqvist, J. (2018) · Scientific Reports · doi:10.1038/s41598-018-30880-1 · Cited by 36
WALKER, M. D., Fureix, C., Palme, R., Newman, J. A., Dallaire, J. A., & Mason, G. J. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0113-7 · Cited by 36
“Cage was included as a factor in the model in order to avoid pseudoreplication because mice housed in the same cage are non-independent [39, 40], and was set as a random effect (the only one in the model) so that inferences can be made that go beyond just the cages used in this experiment [41].”
Soybean resilience to drought is supported by partial recovery of photosynthetic traits
Elsalahy, H. H., & Reckling, M. (2022) · Frontiers in Plant Science · doi:10.3389/fpls.2022.971893 · Cited by 35
“As a remedy for the pseudoreplicates in our experiment, while performing the statistical analysis, we used Mixed-effects models that allowed using the concept of random effects to emulate the randomness inherent in the data (Lazic, 2010; Lazic et al., 2020).”
MethodArtificial intelligence for neurodegenerative experimental models
Marzi, S. J., Schilder, B. M., Nott, A., Frigerio, C. S., Willaime‐Morawek, S., Bucholc, M., Hanger, D. P., James, C. R., Lewis, P. A., Lourida, I., et al. (2023) · Alzheimer s & Dementia · doi:10.1002/alz.13479 · Cited by 35
“170–172 Here too, ML can be of great utility.”
BackgroundDiurnal Variation in the Mechanical and Neural Components of the Baroreflex
Taylor, C. E., Atkinson, G., Willie, C. K., Jones, H., Ainslie, P. N., & Tzeng, Y. (2011) · Hypertension · doi:10.1161/hypertensionaha.111.171512 · Cited by 34
Williams, D. R., Carlsson, R., & Bürkner, P. (2017) · Frontiers in Neuroendocrinology · doi:10.1016/j.yfrne.2017.08.003 · Cited by 33
“Therefore, the statistical assumption that the observations are independent will routinely be violated (Lazic, 2010).”
BackgroundTransparent reporting for reproducible science
Weissgerber, T. L., Garovic, V. D., Winham, S. J., Milic, N. M., & Prager, E. M. (2016) · Journal of Neuroscience Research · doi:10.1002/jnr.23785 · Cited by 33
“…et al. 2007), providing adequate detail about the test (i.e. paired vs. unpaired t-test) (Strasak et al. 2007), not stating whether the assumptions of the statistical tests were examined (Strasak et al. 2007; Weissgerber et al. 2015), or not stating how replicates were analyzed (Lazic 2010).”
BackgroundMethodInfluentialLiu, Q., Chen, W., Deng, J. D., Gu, T., Zhang, H., Yu, Z., & Zhang, J. (2017) · IEEE Transactions on Cybernetics · doi:10.1109/tcyb.2017.2659659 · Cited by 32
Trehalose Improves Human Fibroblast Deficits in a New CHIP-Mutation Related Ataxia
Casarejos, M. J., Perucho, J., López-Sendón, J. L., de Yébenes, J. G., Bettencourt, C., Gómez, A., Ruíz, C., Heutink, P., Rizzu, P., & Mena, M. A. (2014) · PLoS ONE · doi:10.1371/journal.pone.0106931 · Cited by 31
“CHIP-mutant group we have multiple observations on the same subject [28].”
BackgroundPessah, I. N., Rogawski, M. A., Tancredi, D. J., Wulff, H., Żółkowska, D., Bruun, D. A., Hammock, B. D., & Lein, P. J. (2016) · Annals of the New York Academy of Sciences · doi:10.1111/nyas.13137 · Cited by 33
Atkinson, G. (2014) · Physiological Measurement · doi:10.1088/0967-3334/35/9/1825 · Cited by 30
Lazic, S. E. (2015) · BMC Research Notes · doi:10.1186/s13104-015-1062-7 · Cited by 30
“Note that all models assume that observations are independent [41].”
BackgroundSelf-citationKenakin, T., Bylund, D. B., Toews, M. L., Mullane, K., Winquist, R. J., & Williams, M. J. (2013) · Biochemical Pharmacology · doi:10.1016/j.bcp.2013.10.024 · Cited by 28
WALKER, M. D., Fureix, C., Palme, R., & Mason, G. J. (2013) · PLoS ONE · doi:10.1371/journal.pone.0077541 · Cited by 29
“Cage is a blocking factor in order to avoid pseudoreplication because mice housed in the same cage are non-independent (see [51,52]), and was set as a random effect [53].”
BackgroundOnline mobile map effect: how smartphone map use impairs spatial memory
Sugimoto, M., Kusumi, T., Nagata, N., & Ishikawa, T. (2021) · Spatial Cognition and Computation · doi:10.1080/13875868.2021.1969401 · Cited by 28
“To avoid the problem of overfitting owing to the treatment of repeated measures as data from independent participants (known as pseudoreplication in ecology and biostatistics; Lazic, 2010), we adopted a multilevel modeling approach.”
MethodSuedel, B. C., Clarke, J. U., Wilkens, J. L., Lutz, C. H., & Clarke, D. G. (2014) · Estuaries and Coasts · doi:10.1007/s12237-014-9835-0 · Cited by 28
A community-led initiative for training in reproducible research
Auer, S., Haelterman, N. A. T., Weissgerber, T. L., Erlich, J. C., Susilaradeya, D., Julkowska, M., Gazda, M. A., Schwessinger, B., Jadavji, N. M., Reproducibility for Everyone Team, et al. (2021) · eLife · doi:10.7554/elife.64719 · Cited by 28
Application of the hierarchical bootstrap to multi-level data in neuroscience
Saravanan, V., Berman, G. J., & Sober, S. J. (2020) · arXiv · doi:10.48550/arxiv.2007.07797 · Cited by 26
“While pseudoreplication was first extensively reported on in ecological studies [5, 48], it has since been identified as a common problem in other fields, including neuroscience [6].”
BackgroundSample size estimation for power and accuracy in the experimental comparison of algorithms
Campelo, F., & Takahashi, F. C. (2018) · Journal of Heuristics · doi:10.1007/s10732-018-9396-7 · Cited by 25
Cavarsan, C. F., Steele, P. R., Genry, L. T., Reedich, E. J., McCane, L. M., LaPre, K. J., Puritz, A., Manuel, M., Katenka, N., & Quinlan, K. A. (2022) · The Journal of Physiology · doi:10.1113/jp284192 · Cited by 24
“The linear mixed model was used to avoid errors of pseudoreplication (Lazic, 2010; Yu et al., 2022) and dependency of P-value on sample size (Gómez-de-Mariscal et al.”
MethodAppropriate within-subjects statistical models for the analysis of baroreflex sensitivity
Atkinson, G., Batterham, A. M., Jones, H., Taylor, C. E., Willie, C. K., & Tzeng, Y. (2010) · Clinical Physiology and Functional Imaging · doi:10.1111/j.1475-097x.2010.00974.x · Cited by 24
“…each individual leads to loss of information (e.g. the precision of each individuals regression slope), makes inferences from small samples quite susceptible to outliers and can be suboptimal in terms of statistical power, as the correlated nature of the data is completely overlooked (Lazic, 2010).”
BackgroundMethodResultInfluentialIdiosyncratic perception: a link between acuity, perceived position and apparent size
Wang, Z., Murai, Y., & Whitney, D. C. (2020) · Proceedings of the Royal Society B Biological Sciences · doi:10.1098/rspb.2020.0825 · Cited by 23
“We are aware that any analysis based on the ‘super subject’ data may be subject to the problem of pseudo-replication [17], which is why we constructed a linear mixed effect regression (see Data analysis section) and calculated individual correlations for each subject (electronic supplementary material, figure S3).”
Metadata
- Authors
- Lazic, S. E.
- Affiliations
- F. Hoffmann-La Roche AG (CH)
- Corresponding author
- Lazic, Stanley E
- ORCID
- Lazic, Stanley E
- Venue
- BMC Neuroscience
- Published
- 2010-01-14
- Volume
- 11
- Issue
- 1
- Pages
- 5-5
- ISSN
- 1471-2202 (ISSN-L)
- PMC
- PMC2817684
- OpenAlex
- W2109626712
- Cited by
- 381
- References
- 90
- Citations by year
- 2026: 68 · 2025: 30 · 2024: 19 · 2023: 32 · 2022: 20 · 2021: 21 · 2020: 31 · 2019: 16 · 2018: 23 · 2017: 22 · 2016: 15 · 2015: 19 · 2014: 20 · 2013: 17 · 2012: 19
- FWCI
- 12.28
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Neuroscience (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- KAIST Institutional Repository (KAIST) (submittedVersion)
- King Abdullah University of Science and Technology Repository (King Abdullah University of Science and Technology) (submittedVersion, other-oa)
- Funders
- Cancer Research UK (10218)
- University of Oxford
- Cambridge Commonwealth Trust
- King Abdullah University of Science and Technology (KUK‐C1‐013‐04)
- Korea Science and Engineering Foundation
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
- Keywords
- pseudoreplication, statistical dependence, nested data, hierarchical data, repeated measures, time series correlation, spatial correlation, statistical analysis errors, sample size reporting, degrees of freedom, p-values
- MeSH terms
- Analysis of Variance, Animals, Computer Simulation, Neurosciences, Periodicals as Topic, Time Factors, Models, Statistical, Linear Models, Biomedical Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
