The DOI matches its Crossref record; 62 of the 62 references it cites were found in a registry or an index.
The Insignificance of Statistical Significance Testing
Abstract
Despite their wide use in scientific journals such as The Journal of Wildlife Management, statistical hypothesis tests add very little value to the products of research. Indeed, they frequently confuse the interpretation of data. This paper describes how statistical hypothesis tests are often viewed, and then contrasts that interpretation with the correct one. I discuss the arbitrariness of P-values, conclusions that the null hypothesis is true, power analysis, and distinctions between statistical and biological significance. Statistical hypothesis testing, in which the null hypothesis about the properties of a population is almost always known a priori to be false, is contrasted with scientific hypothesis testing, which examines a credible null hypothesis about phenomena in nature. More meaningful alternatives are briefly outlined, including estimation and confidence intervals for determining the importance of factors, decision theory for guiding actions in the face of uncertainty, and Bayesian approaches to hypothesis testing and other statistical practices.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Muller, K. (1989) · Technometrics · doi:10.1080/00401706.1989.10488618 · Cited by 99,167
Statistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
The Logic of Scientific Discovery
(2013) · A Realist Theory of Science · doi:10.4324/9780203090732-9 · Cited by 7,731
Statistical Decision Theory and Bayesian Analysis
Berger, J. O. (1985) · Springer Series in Statistics · doi:10.1007/978-1-4757-4286-2 · Cited by 7,722
The Logic of Scientific Discovery.
Hutchison, T. W., & Popper, K. R. (1959) · Economica · doi:10.2307/2550489 · Cited by 7,499
Applied Nonparametric Statistics
Wozniak, P. J., & Daniel, W. W. (1991) · Technometrics · doi:10.2307/1268795 · Cited by 4,987
Model Selection and Inference: A Practical Information-Theoretic Approach
Strawderman, R. L., Burnham, K. P., & Anderson, D. R. (2000) · Journal of the American Statistical Association · doi:10.2307/2669574 · Cited by 4,194
Cohen, J. (1994) · American Psychologist · doi:10.1037/0003-066x.49.12.997 · Cited by 4,003
Bayesian Inference in Statistical Analysis.
Kadane, J. B., Box, G. E. P., & Tiao, G. C. (1975) · Journal of the American Statistical Association · doi:10.2307/2285412 · Cited by 3,824
Experiments in ecology their logical design and interpretation using analysis of variance
Underwood, A. J. (1997) · Cited by 3,799
Cited by
CLASSIFICATION AND REGRESSION TREES: A POWERFUL YET SIMPLE TECHNIQUE FOR ECOLOGICAL DATA ANALYSIS
De'ath, G., & Fabricius, K. E. (2000) · Ecology · doi:10.1890/0012-9658(2000)081[3178:cartap]2.0.co;2 · Cited by 3,167
Evaluating resource selection functions
Boyce, M. S., Vernier, P. R., Nielsen, S. E., & Schmiegelow, F. K. A. (2002) · Ecological Modelling · doi:10.1016/s0304-3800(02)00200-4 · Cited by 2,566
Ellis, P. D. (2010) · doi:10.1017/cbo9780511761676 · Cited by 2,461
STRENGTH OF EVIDENCE FOR DENSITY DEPENDENCE IN ABUNDANCE TIME SERIES OF 1198 SPECIES
Brook, B. W., & Bradshaw, C. J. A. (2006) · Ecology · doi:10.1890/0012-9658(2006)87[1445:soefdd]2.0.co;2 · Cited by 2,082
Null Hypothesis Testing: Problems, Prevalence, and an Alternative
Anderson, D. R., Burnham, K. P., & Thompson, W. L. (2000) · The Journal of Wildlife Management · doi:10.2307/3803199 · Cited by 1,930
Why do we still use stepwise modelling in ecology and behaviour?
WHITTINGHAM, M. J., STEPHENS, P. A., BRADBURY, R. B., & FRECKLETON, R. P. (2006) · Journal of Animal Ecology · doi:10.1111/j.1365-2656.2006.01141.x · Cited by 1,520
Avoiding Pitfalls When Using Information-Theoretic Methods
Anderson, D. R., & Burnham, K. P. (2002) · The Journal of Wildlife Management · doi:10.2307/3803155 · Cited by 1,324
Ecological models and data in R
Bolker, B. M. (2008) · doi:10.2307/j.ctvcm4g37 · Cited by 1,297
“…values of biologically meaningful parameters and finding their confidence limits rather than worrying too much about whether p is greater or less than 0.05 (Yoccoz, 1991; Johnson, 1999; Osenberg et al., 2002) (although Stephens et al. (2005) remind us that hypothesis testing can still be useful).”
BackgroundInfluentialBOOSTED TREES FOR ECOLOGICAL MODELING AND PREDICTION
De'ath, G. (2007) · Ecology · doi:10.1890/0012-9658(2007)88[243:btfema]2.0.co;2 · Cited by 1,293
Kruschke, J. K., & Liddell, T. M. (2018) · Psychonomic Bulletin & Review · doi:10.3758/s13423-016-1221-4 · Cited by 1,288
“…include “the occurrence of sheep remains in coyote (Canis latrans) scats differed among seasons (p = 0.03, n = 467), (2) duckling body mass differed among years (p 0.0001), and (3) the density of large trees was greater in unlogged forest stands than in logged stands (p = 0.02)” (Johnson, 1999).”
InfluentialMULTIVARIATE REGRESSION TREES: A NEW TECHNIQUE FOR MODELING SPECIES–ENVIRONMENT RELATIONSHIPS
De'ath, G. (2002) · Ecology · doi:10.1890/0012-9658(2002)083[1105:mrtant]2.0.co;2 · Cited by 1,202
Nutrient co‐limitation of primary producer communities
Harpole, W. S., Ngai, J. T., Cleland, E. E., Seabloom, E. W., Borer, E. T., Bracken, M. E. S., Elser, J. J., Gruner, D. S., Hillebrand, H., Shurin, J. B., et al. (2011) · Ecology Letters · doi:10.1111/j.1461-0248.2011.01651.x · Cited by 1,088
Kullback-Leibler information as a basis for strong inference in ecological studies
Burnham, K. P., & Anderson, D. R. (2001) · Wildlife Research · doi:10.1071/wr99107 · Cited by 926
A UNIFIED APPROACH TO ANALYZING NEST SUCCESS
Shaffer, T. L. (2004) · The Auk · doi:10.1642/0004-8038(2004)121[0526:auatan]2.0.co;2 · Cited by 803
The fickle P value generates irreproducible results
Halsey, L. G., Curran-Everett, D., Vowler, S. L., & Drummond, G. B. (2015) · Nature Methods · doi:10.1038/nmeth.3288 · Cited by 765
McCarthy, M. A. (2007) · Cambridge University Press eBooks · doi:10.1017/cbo9780511802454 · Cited by 756
The Other Half of the Story: Effect Size Analysis in Quantitative Research
Maher, J. M., Markey, J. C., & Ebert-May, D. (2013) · CBE—Life Sciences Education · doi:10.1187/cbe.13-04-0082 · Cited by 625
Model selection and model averaging in behavioural ecology: the utility of the IT-AIC framework
Richards, S. A., Whittingham, M. J., & Stephens, P. A. (2011) · Behavioral Ecology and Sociobiology · doi:10.1007/s00265-010-1035-8 · Cited by 566
Risks and decisions for conservation and environmental management
(2005) · Choice Reviews Online · doi:10.5860/choice.43-2184 · Cited by 523
Traditional Ecological Knowledge in Environmental Assessment and Management
Usher, P. J. (2000) · ARCTIC · doi:10.14430/arctic849 · Cited by 522
Estimating and comparing thermal performance curves
Angilletta, M. J. (2006) · Journal of Thermal Biology · doi:10.1016/j.jtherbio.2006.06.002 · Cited by 510
Risks and Decisions for Conservation and Environmental Management
Burgman, M. (2005) · Cambridge University Press eBooks · doi:10.1017/cbo9780511614279 · Cited by 498
Scale for resource selection functions
Boyce, M. S. (2006) · Diversity and Distributions · doi:10.1111/j.1366-9516.2006.00243.x · Cited by 496
Why most conservation monitoring is, but need not be, a waste of time
Legg, C. J., & Nagy, L. (2006) · Journal of Environmental Management · doi:10.1016/j.jenvman.2005.04.016 · Cited by 483
Suggestions for Presenting the Results of Data Analyses
Anderson, D. R., Link, W. A., Johnson, D. H., & Burnham, K. P. (2001) · The Journal of Wildlife Management · doi:10.2307/3803088 · Cited by 465
Self-citationGronum, S., Verreynne, M.-L., & Kastelle, T. (2012) · Journal of Small Business Management · doi:10.1111/j.1540-627x.2012.00353.x · Cited by 459
Cross validation for model selection: A review with examples from ecology
Yates, L. A., Aandahl, Z., Richards, S. A., & Brook, B. W. (2023) · Ecological Monographs · doi:10.1002/ecm.1557 · Cited by 459
Beyond description: the active and effective way to infer processes from spatial patterns
McIntire, E. J. B., & Fajardo, A. (2009) · Ecology · doi:10.1890/07-2096.1 · Cited by 442
Mazerolle, M. (2006) · Amphibia-Reptilia · doi:10.1163/156853806777239922 · Cited by 429
Swift, T. L., & Hannon, S. J. (2010) · Biological Reviews · doi:10.1111/j.1469-185x.2009.00093.x · Cited by 394
Scale and heterogeneity in habitat selection by elk in Yellowstone National Park
Boyce, M. S., Mao, J. S., Merrill, E. H., Fortin, D., Turner, M. G., Fryxell, J., & Turchin, P. (2003) · Écoscience · doi:10.1080/11956860.2003.11682790 · Cited by 391
Ecologists should not use statistical significance tests to interpret simulation model results
White, J. W., Rassweiler, A., Samhouri, J. F., Stier, A. C., & White, C. (2014) · Oikos · doi:10.1111/j.1600-0706.2013.01073.x · Cited by 387
Effect sizes, confidence intervals, and confidence intervals for effect sizes
Thompson, B. (2007) · Psychology in the Schools · doi:10.1002/pits.20234 · Cited by 386
Alternatives To Statistical Hypothesis Testing In Ecology: A Guide To Self Teaching
Hobbs, N. T., & Hilborn, R. (2006) · Ecological Applications · doi:10.1890/04-0645 · Cited by 384
Beck, H. E., van Dijk, A. I. J. M., Miralles, D. G., de Jeu, R. A. M., Sampurno Bruijnzeel, L. A., McVicar, T. R., & Schellekens, J. (2013) · Water Resources Research · doi:10.1002/2013wr013918 · Cited by 361
Information theory and hypothesis testing: a call for pluralism
STEPHENS, P. A., BUSKIRK, S. W., HAYWARD, G. D., & MARTÍNEZ DEL RIO, C. (2005) · Journal of Applied Ecology · doi:10.1111/j.1365-2664.2005.01002.x · Cited by 360
RELATING RESOURCES TO A PROBABILISTIC MEASURE OF SPACE USE: FOREST FRAGMENTS AND STELLER'S JAYS
Marzluff, J. M., Millspaugh, J. J., Hurvitz, P., & Handcock, M. S. (2004) · Ecology · doi:10.1890/03-0114 · Cited by 359
Halsey, L. G. (2019) · Biology Letters · doi:10.1098/rsbl.2019.0174 · Cited by 357
Block, W. M., Franklin, A. B., Ward, J. P., Ganey, J. L., & White, G. C. (2001) · Restoration Ecology · doi:10.1046/j.1526-100x.2001.009003293.x · Cited by 336
The role of administrative data in the big data revolution in social science research
Connelly, R., Playford, C. J., Gayle, V., & Dibben, C. (2016) · Social Science Research · doi:10.1016/j.ssresearch.2016.04.015 · Cited by 330
Impact of traffic oscillations on freeway crash occurrences
Zheng, Z., Ahn, S., & Monsere, C. M. (2010) · Accident Analysis & Prevention · doi:10.1016/j.aap.2009.10.009 · Cited by 325
Kader, M. A. (2005) · Journal and proceedings of the Royal Society of New South Wales · doi:10.5962/p.361564 · Cited by 320
Murtaugh, P. A. (2014) · Ecology · doi:10.1890/13-0590.1 · Cited by 317
The earth is flat ( p > 0.05): significance thresholds and the crisis of unreplicable research
Amrhein, V., Korner-Nievergelt, F., & Roth, T. (2017) · PeerJ · doi:10.7717/peerj.3544 · Cited by 308
Wang, P., Xu, B., Xu, J., Tian, G., Liu, C.-L., & Hao, H. (2016) · Neurocomputing · doi:10.1016/j.neucom.2015.09.096 · Cited by 304
Correlation and studies of habitat selection: problem, red herring or opportunity?
Fieberg, J., Matthiopoulos, J., Hebblewhite, M., Boyce, M. S., & Frair, J. L. (2010) · Philosophical Transactions of the Royal Society B: Biological Sciences · doi:10.1098/rstb.2010.0079 · Cited by 302
Final Collapse of the Neyman-Pearson Decision Theoretic Framework and Rise of the neoFisherian
Hurlbert, S. H., & Lombardi, C. M. (2009) · Annales Zoologici Fennici · doi:10.5735/086.046.0501 · Cited by 301
Evidence of an Edge Effect on Avian Nest Success
BATÁRY, P., & BÁLDI, A. (2004) · Conservation Biology · doi:10.1111/j.1523-1739.2004.00184.x · Cited by 296
LESSONS FROM MONITORING TRENDS IN ABUNDANCE OF MARINE MAMMALS
Taylor, B. L., Martinez, M., Gerrodette, T., Barlow, J., & Hrovat, Y. N. (2007) · Marine Mammal Science · doi:10.1111/j.1748-7692.2006.00092.x · Cited by 293
Impacts of Traffic Noise and Traffic Volume on Birds of Roadside Habitats
Parris, K. M., & Schneider, A. (2009) · Ecology and Society · doi:10.5751/es-02761-140129 · Cited by 286
HOW SHOULD DETECTION PROBABILITY BE INCORPORATED INTO ESTIMATES OF RELATIVE ABUNDANCE?
MacKenzie, D. I., & Kendall, W. L. (2002) · Ecology · doi:10.1890/0012-9658(2002)083[2387:hsdpbi]2.0.co;2 · Cited by 284
Inference in ecology and evolution
Stephens, P. A., Buskirk, S. W., & del Rio, C. M. (2007) · Trends in Ecology & Evolution · doi:10.1016/j.tree.2006.12.003 · Cited by 268
Stable isotopes from multiple tissues reveal diet switching in sharks
MacNeil, M., Skomal, G., & Fisk, A. (2005) · Marine Ecology Progress Series · doi:10.3354/meps302199 · Cited by 259
Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error
Correll, M., & Gleicher, M. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346298 · Cited by 258
“By only showing information about one kind of statistical inference, viewers are unable to draw their own conclusions for their own standards of proof, exacerbating existing problems with null-hypothesis significance testing [6, 18, 28].”
The Importance of Replication in Wildlife Research
Johnson, D. H. (2002) · The Journal of Wildlife Management · doi:10.2307/3802926 · Cited by 256
Self-citationConsidering aspects of the 3Rs principles within experimental animal biology
Sneddon, L. U., Halsey, L. G., & Bury, N. R. (2017) · Journal of Experimental Biology · doi:10.1242/jeb.147058 · Cited by 254
Fontaine, J. B., & Kennedy, P. L. (2012) · Ecological Applications · doi:10.1890/12-0009.1 · Cited by 205
Survival and Cause‐Specific Mortality of Red Foxes in Agricultural and Urban Areas of Illinois
GOSSELINK, T. E., VAN DEELEN, T. R., WARNER, R. E., & MANKIN, P. C. (2007) · The Journal of Wildlife Management · doi:10.2193/2006-020 · Cited by 123
Null hypothesis significance testing: effect size matters
Gliner, J. A., Vaske, J. J., & Morgan, G. A. (2001) · Human Dimensions of Wildlife · doi:10.1080/108712001753473966 · Cited by 45
“Recognition of need to report effect sizes is apparent in a variety of disciplines (e.g., education, psychology, wildlife biology) and several recent articles (see, for example, Anderson, Burnham, & Thompson, 2000; Johnson, 1999; Kirk, 1996; Snyder & Lawson, 1993; Thompson, 1996, 1997).”
Background(2009) · PubMed · Cited by 13
Metadata
- Authors
- Johnson, D. H.
- Venue
- The Journal of Wildlife Management
- Publisher
- JSTOR
- Published
- 1999-07
- Volume
- 63
- Issue
- 3
- Pages
- 763
- ISSN
- 0022-541X (ISSN-L)
- OpenAlex
- W2157002521
- Cited by
- 1,104
- References
- 62
- FWCI
- 50.57
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.