The DOI matches its Crossref record; 60 of the 60 references it cites were found in a registry or an index.
Computation of measures of effect size for neuroscience data sets
Abstract
The overwhelming majority of research in the neurosciences employs P‐values stemming from tests of statistical significance to decide on the presence or absence of an effect of some treatment variable. Although a continuous variable, the P‐value is commonly used to reach a dichotomous decision about the presence of an effect around an arbitrary criterion of 0.05. This analysis strategy is widely used, but has been heavily criticized in the past decades. To counter frequent misinterpretations of P‐values, it has been advocated to complement or replace P‐values with measures of effect size (MES). Many psychological, biological and medical journals now recommend reporting appropriate MES. One hindrance to the more frequent use of MES may be their scarcity in standard statistical software packages. Also, the arguably most widespread data analysis software in neuroscience, matlab, does not provide MES beyond correlation and receiver‐operating characteristic analysis. Here we review the most common criticisms of significance testing and provide several examples from neuroscience where use of MES conveys insights not amenable through the use of P‐values alone. We introduce an open‐access matlab toolbox providing a wide range of MES to complement the frequently used types of hypothesis tests, such as t‐tests and analysis of variance. The accompanying documentation provides calculation formulae, intuitive explanations and example calculations for each measure. The toolbox described is usable without sophisticated statistical knowledge and should be useful to neuroscientists wishing to enhance their repertoire of statistical reporting.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007) · Behavior Research Methods · doi:10.3758/bf03193146 · Cited by 65,971
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Distribution Theory for Glass's Estimator of Effect Size and Related Estimators
Hedges, L. V. (1981) · Journal of Educational Statistics · doi:10.2307/1164588 · Cited by 5,695
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Effect size, confidence interval and statistical significance: a practical guide for biologists
Nakagawa, S., & Cuthill, I. C. (2007) · Biological Reviews · doi:10.1111/j.1469-185x.2007.00027.x · Cited by 3,835
Belief in the law of small numbers.
Tversky, A., & Kahneman, D. (1971) · Psychological Bulletin · doi:10.1037/h0031322 · Cited by 3,246
Statistical methods in psychology journals: Guidelines and explanations.
Wilkinson, L., & Task Force on Statistical Inference (1999) · American Psychologist · doi:10.1037/0003-066x.54.8.594 · Cited by 2,791
Meta-analysis of psychotherapy outcome studies.
Smith, M. L., & Glass, G. V. (1977) · American Psychologist · doi:10.1037/0003-066x.32.9.752 · Cited by 2,154 · cited 1×
“Distilling statistical results through meta-analysis has proved immensely useful in several fields of science, such as psychology and especially medicine (e.g. Smith & Glass, 1977; Mann, 1990; Baigent et al. , 2005), and MES are central to that endeavor (Thompson, 2002).”
MethodA farewell to Bonferroni: the problems of low statistical power and publication bias
Nakagawa, S. (2004) · Behavioral Ecology · doi:10.1093/beheco/arh107 · Cited by 2,097
Motulsky, H. J., & Christopoulos, A. (2004) · Cited by 1,972
Tukey, J. W. (1962) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177704711 · Cited by 1,860
Practical significance: a concept whose time has come
Kirk, R. E. (1996) · Educational and Psychological Measurement · doi:10.1177/0013164496056005002 · Cited by 1,778 · cited 1×
“…have been many articles highlighting the shortcomings of NHST in a variety of research areas, among them medicine (Fleiss, 1986; Goodman, 1999a,b), biology (Nakagawa, 2004), ergonomics (Vicente & Torenvliet, 2000), consumer research (Iacobucci, 2005) and education (Kirk, 1996, 2001; Morgan, 2003).”
BackgroundInference by Eye: Confidence Intervals and How to Read Pictures of Data.
Cumming, G., & Finch, S. (2005) · American Psychologist · doi:10.1037/0003-066x.60.2.170 · Cited by 1,449 · cited 1×
“The misuse of NHST has perhaps been most hotly debated in psychology (Loftus, 1996; Cumming & Finch, 2005).”
BackgroundToward Evidence-Based Medical Statistics. 1: The P Value Fallacy
Goodman, S. N. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-12-199906150-00008 · Cited by 1,185
Beyond significance testing: Reforming data analysis methods in behavioral research.
Kline, R. B. (2004) · doi:10.1037/10693-000 · Cited by 1,034
The case against statistical significance testing
Carver, R. (1978) · Harvard Educational Review · doi:10.17763/haer.48.3.t490261645281841 · Cited by 931
Toward Evidence-Based Medical Statistics. 2: The Bayes Factor
Goodman, S. N. (1999) · Annals of Internal Medicine · doi:10.7326/0003-4819-130-12-199906150-00019 · Cited by 912
Erroneous analyses of interactions in neuroscience: A problem of significance
Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011) · Nature Neuroscience · doi:10.1038/nn.2886 · Cited by 864
Contrasts and Effect Sizes in Behavioral Research: A Correlational Approach
Rosenthal, R., Rosnow, R. L., & Rubin, D. B. (1999) · Cited by 854
Schmidt, F. L. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.115 · Cited by 830
Schmidt, F. L. (1996) · Psychological Methods · doi:10.1037//1082-989x.1.2.115 · Cited by 783
The Empire of chance: how probability changed science and everyday life
(1989) · Choice Reviews Online · doi:10.5860/choice.27-2084 · Cited by 597
What If There Were No Significance Tests?
(2016) · Routledge eBooks · doi:10.4324/9781315629049 · Cited by 542
Thompson, B. (2002) · Educational Researcher · doi:10.3102/0013189x031003025 · Cited by 538
The problem of pseudoreplication in neuroscientific studies: is it affecting your analysis?
Lazic, S. E. (2010) · BMC Neuroscience · doi:10.1186/1471-2202-11-5 · Cited by 381 · cited 2×
“…the documented existence of these misconceptions, the application of NHST suffers from frequent analysis mistakes (Garcı´a-Berthou & Alcaraz, 2004; Lazic, 2010; Nieuwenhuis et al. , 2011), and it has been suggested that replacing NHST with an analysis couched in the framework of MES and CIs is…”
MethodPsychology Will Be a Much Better Science When We Change the Way We Analyze Data
Loftus, G. R. (1996) · Current Directions in Psychological Science · doi:10.1111/1467-8721.ep11512376 · Cited by 361
Coding of somatic sensory input by vibrissae neurons in the rat's trigeminal ganglion
Zucker, E., & Welker, W. I. (1969) · Brain Research · doi:10.1016/0006-8993(69)90061-4 · Cited by 359
Needed: A Ban on the Significance Test
Hunter, J. E. (1997) · Psychological Science · doi:10.1111/j.1467-9280.1997.tb00534.x · Cited by 314
Null hypothesis significance testing: On the survival of a flawed method.
Krueger, J. (2001) · American Psychologist · doi:10.1037//0003-066x.56.1.16 · Cited by 282
Effect sizes and p values: What should be reported and what should be replicated?
Greenwald, A., Gonzalez, R., Harris, R. J., & Guthrie, D. (1996) · Psychophysiology · doi:10.1111/j.1469-8986.1996.tb02121.x · Cited by 274
Contrasts and Effect Sizes in Behavioral Research: A Correlational Approach
Boik, R. J. (2001) · Journal of the American Statistical Association · doi:10.1198/jasa.2001.s432 · Cited by 257
Fidler, F., Thomason, N., Cumming, G., Finch, S., & Leeman, J. (2004) · Psychological Science · doi:10.1111/j.0963-7214.2004.01502008.x · Cited by 244
Confidence Intervals for Standardized Effect Sizes: Theory, Application, and Implementation
Kelley, K. (2007) · Journal of Statistical Software · doi:10.18637/jss.v020.i08 · Cited by 236
The Interpretation of Levels of Significance by Psychological Researchers
Rosenthal, R., & Gaito, J. (1963) · The Journal of Psychology · doi:10.1080/00223980.1963.9916596 · Cited by 185
Promoting good statistical practices: Some suggestions
Kirk, R. E. (2001) · Educational and Psychological Measurement · doi:10.1177/00131640121971185 · Cited by 184
Incongruence between test statistics and P values in medical papers
García-Berthou, E., & Alcaraz, C. (2004) · BMC Medical Research Methodology · doi:10.1186/1471-2288-4-13 · Cited by 181
Should Significance Tests be Banned? Introduction to a Special Section Exploring the Pros and Cons
Shrout, P. E. (1997) · Psychological Science · doi:10.1111/j.1467-9280.1997.tb00533.x · Cited by 180
Smithson, M. (2003) · Sage eBooks · doi:10.4135/9781412983761 · Cited by 179
Kline, R. B. (2004) · Cited by 157
The Empire of Chance: How Probability Changed Science and Everyday Life.
Guttorp, P. M., Gigerenzer, G., Swijtink, Z. G., Porter, T. M., Daston, L., Beatty, J., & Krüger, L. (1990) · Journal of the American Statistical Association · doi:10.2307/2289806 · Cited by 151
On the Tyranny of Hypothesis Testing in the Social Sciences
Loftus, G. R. (1991) · Contemporary Psychology · doi:10.1037/029395 · Cited by 136
Gibson, J. M., & Welker, W. (1983) · Somatosensory Research · doi:10.3109/07367228309144540 · Cited by 126
Statistical Reform in Psychology: Is Anything Changing?
Cumming, G., Fidler, F., Leonard, M., Kalinowski, P., Christiansen, A., Kleinig, A., Lo, J., McMenamin, N., & Wilson, S. (2007) · Psychological Science · doi:10.1111/j.1467-9280.2007.01881.x · Cited by 122
Computing Correct Confidence Intervals for ANOVA Fixed- and Random-Effects Effect Sizes
Fidler, F. M., & Thompson, B. (2001) · Educational and Psychological Measurement · doi:10.1177/0013164401614003 · Cited by 115
Stüttgen, M. C., Rüter, J., & Schwarz, C. (2006) · Journal of Neuroscience · doi:10.1523/jneurosci.1864-06.2006 · Cited by 112
Self-citationCoulson, M. (2010) · Frontiers in Psychology · doi:10.3389/fpsyg.2010.00026 · Cited by 77
Muscarinic blockade weakens interaction of gamma with theta rhythms in mouse hippocampus
Hentschke, H., Perkins, M. G., Pearce, R. A., & Banks, M. I. (2007) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2007.05779.x · Cited by 77
Self-citationGreat challenges with few subjects: Statistical strategies for neuroscientists
Nakagawa, S., & Hauber, M. E. (2010) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2010.06.003 · Cited by 75
Confidence intervals vs significance tests: quantitative interpretation.
Fleiss, J. L. (1986) · American Journal of Public Health · doi:10.2105/ajph.76.5.587 · Cited by 70
Jordan, D., Steiner, M., Kochs, E. F., & Schneider, G. H. (2010) · Anesthesia & Analgesia · doi:10.1213/ane.0b013e3181fb919e · Cited by 56
Reporting statistical methods and statistical results in EJN
Sarter, M. F., & Fritschy, J. (2008) · European Journal of Neuroscience · doi:10.1111/j.1460-9568.2008.06581.x · Cited by 49
Altered GABAA,slowInhibition and Network Oscillations in Mice Lacking the GABAAReceptor β3Subunit
Hentschke, H., Benkwitz, C., Banks, M. I., Perkins, M. G., Homanics, G. E., & Pearce, R. A. (2009) · Journal of Neurophysiology · doi:10.1152/jn.00651.2009 · Cited by 46
Self-citationResponses of Rat Trigeminal Ganglion Neurons to Longitudinal Whisker Stimulation
Stüttgen, M. C., Kullmann, S., & Schwarz, C. (2008) · Journal of Neurophysiology · doi:10.1152/jn.90511.2008 · Cited by 42
Self-citationGilbert, M., Healey, M., Fidler, F., & Cumming, G. (2010) · Figshare · doi:10.26181/16926727 · Cited by 37
The Earth is spherical (p < 0.05): alternative methods of statistical inference
Vicente, K. J., & Torenvliet, G. (2000) · Theoretical Issues in Ergonomics Science · doi:10.1080/14639220110037065 · Cited by 28
Morgan, P. L. (2003) · Exceptionality · doi:10.1207/s15327035ex1104_2 · Cited by 21
Iacobucci, D. (2005) · Journal of Consumer Research · doi:10.1086/430648 · Cited by 10
Cited by
Gorilla in our midst: An online behavioral experiment builder
Anwyl-Irvine, A. L., Massonnié, J., Flitton, A., Kirkham, N., & Evershed, J. K. (2020) · Behavior Research Methods · doi:10.3758/s13428-019-01237-x · Cited by 1,862
The idiosyncratic brain: distortion of spontaneous connectivity patterns in autism spectrum disorder
Hahamy, A., Behrmann, M., & Malach, R. (2015) · Nature Neuroscience · doi:10.1038/nn.3919 · Cited by 416
Serotonergic neurons signal reward and punishment on multiple timescales
Cohen, J. Y., Amoroso, M. W., & Uchida, N. (2015) · eLife · doi:10.7554/elife.06346 · Cited by 388
Alekseichuk, I., Turi, Z., Amador de Lara, G., Antal, A., & Paulus, W. (2016) · Current Biology · doi:10.1016/j.cub.2016.04.035 · Cited by 376
Performance of the Emotiv Epoc headset for P300-based applications
Duvinage, M., Castermans, T., Petieau, M., Hoellinger, T., Cheron, G., & Dutoit, T. (2013) · BioMedical Engineering OnLine · doi:10.1186/1475-925x-12-56 · Cited by 322
Metadata
- Authors
- Hentschke, H., & Stüttgen, M. C.
- Venue
- European Journal of Neuroscience
- Publisher
- Wiley
- Published
- 2011-11-14
- Volume
- 34
- Issue
- 12
- Pages
- 1887-1894
- ISSN
- 0953-816X (ISSN-L)
- 1460-9568
- OpenAlex
- W1948588766
- Cited by
- 410
- References
- 60
- FWCI
- 10.47
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.