The DOI matches its Crossref record; 112 of the 112 references it cites were found in a registry or an index.
Severe Testing as a Basic Concept in a Neyman–Pearson Philosophy of Induction
Abstract
Despite the widespread use of key concepts of the Neyman–Pearson (N–P) statistical paradigm—type I and II errors, significance levels, power, confidence levels—they have been the subject of philosophical controversy and debate for over 60 years. Both current and long-standing problems of N–P tests stem from unclarity and confusion, even among N–P adherents, as to how a test's (pre-data) error probabilities are to be used for (post-data) inductive inference as opposed to inductive behavior. We argue that the relevance of error probabilities is to ensure that only statistical hypotheses that have passed severe or probative tests are inferred from the data. The severity criterion supplies a meta-statistical principle for evaluating proposed statistical inferences, avoiding classic fallacies from tests that are overly sensitive, as well as those not sensitive enough to particular errors and discrepancies. 1. Introduction and overview1.1Behavioristic and inferential rationales for Neyman–Pearson (N–P) tests1.2Severity rationale: induction as severe testing1.3Severity as a meta-statistical concept: three required restrictions on the N–P paradigm2. Error statistical tests from the severity perspective2.1N–P test T(α): type I, II error probabilities and power2.2Specifying test T(α) using p-values3. Neyman's post-data use of power3.1Neyman: does failure to reject H warrant confirming H?4. Severe testing as a basic concept for an adequate post-data inference4.1The severity interpretation of acceptance (SIA) for test T(α)4.2The fallacy of acceptance (i.e., an insignificant difference): Ms Rosy4.3Severity and power5. Fallacy of rejection: statistical vs. substantive significance5.1Taking a rejection of H0 as evidence for a substantive claim or theory5.2A statistically significant difference from H0 may fail to indicate a substantively important magnitude5.3Principle for the severity interpretation of a rejection (SIR)5.4Comparing significant results with different sample sizes in T(α): large n problem5.5General testing rules for T(α), using the severe testing concept6. The severe testing concept and confidence intervals6.1Dualities between one and two-sided intervals and tests6.2Avoiding shortcomings of confidence intervals7. Beyond the N–P paradigm: pure significance, and misspecification tests8. Concluding comments: have we shown severity to be a basic concept in a N–P philosophy of induction?
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Power Analysis for the Behavioral Sciences
Cohen, J. (2013) · doi:10.4324/9780203771587 · Cited by 25,373
The Logic of Scientific Discovery
(2013) · A Realist Theory of Science · doi:10.4324/9780203090732-9 · Cited by 7,731
Journal of The American Statistical Association
Fienberg, S. E. (2005) · Encyclopedia of Statistical Sciences · doi:10.1002/0471667196.ess1317 · Cited by 6,125
(1947) · The American Statistician · doi:10.1080/00031305.1947.10501534 · Cited by 3,660
Journal of the Royal Statistical Society (B)
Fildes, R. (1993) · International Journal of Forecasting · doi:10.1016/0169-2070(93)90088-5 · Cited by 3,437
(2010) · Encyclopedia of Curriculum Studies · doi:10.4135/9781412958806.n175 · Cited by 2,556
Logical Foundations of Probability.
Burks, A. W., & Carnap, R. (1951) · The Journal of Philosophy · doi:10.2307/2021419 · Cited by 2,370
Logical Foundations of Probability
Savage, L. J., & Carnap, R. (1952) · Econometrica · doi:10.2307/1907654 · Cited by 2,033
Parametric measures of effect size.
Rosenthal, R. (1994) · Cited by 1,847
Methodology of Scientific Research Programmes
(1998) · The Handbook of Economic Methodology · doi:10.4337/9781781954249.00075 · Cited by 1,815
Bayesian statistical inference for psychological research.
Edwards, W., Lindman, H., & Savage, L. J. (1963) · Psychological Review · doi:10.1037/h0044139 · Cited by 1,734
ON THE PROBLEM OF THE MOST EFFICIENT TESTS OF STATISTICAL HYPOTHESES
(1967) · Joint Statistical Papers · doi:10.1525/9780520339897-008 · Cited by 1,504
Error and the Growth of Experimental Knowledge
Mayo, D. G. (1996) · doi:10.7208/chicago/9780226511993.001.0001 · Cited by 1,325
Self-citationThe Selection of Prior Distributions by Formal Rules
Kass, R. E., & Wasserman, L. (1996) · Journal of the American Statistical Association · doi:10.1080/01621459.1996.10477003 · Cited by 1,308
Statistical Methods and Scientific Inference.
Bartlett, M. S., & Fisher, R. A. (1957) · Biometrika · doi:10.2307/2333276 · Cited by 1,278
The Logic of Inductive Inference
Fisher, R. A. (1935) · Journal of the Royal Statistical Society · doi:10.2307/2342435 · Cited by 1,265
Scientific reasoning: the Bayesian approach
(1990) · Choice Reviews Online · doi:10.5860/choice.27-2696 · Cited by 1,245
The Logic of Scientific Discovery
Popper, K. (2005) · Routledge eBooks · doi:10.4324/9780203994627 · Cited by 1,142
Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability
Neyman, J. (1937) · Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences · doi:10.1098/rsta.1937.0005 · Cited by 1,087
Realism and the Aim of Science
Popper, K. (2013) · Routledge eBooks · doi:10.4324/9780203713969 · Cited by 1,063
LINDLEY, D. V. (1957) · Biometrika · doi:10.1093/biomet/44.1-2.187 · Cited by 977
Scandinavian Journal of Statistics
(2010) · Scandinavian Journal of Statistics · doi:10.1111/(issn)1467-9469 · Cited by 976
Testing Statistical Hypotheses.
Biggins, J. D., & Lehmann, E. L. (1988) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2982206 · Cited by 966
Theory-Testing in Psychology and Physics: A Methodological Paradox
Meehl, P. E. (1967) · Philosophy of Science · doi:10.1086/288135 · Cited by 962
Testing Statistical Hypotheses
Neath, A. A. (2006) · Journal of the American Statistical Association · doi:10.1198/jasa.2006.s100 · Cited by 948
124: The Logic of Inductive Inference.
Fisher, R. A. (1935) · Adelaide Research & Scholarship (AR&S) (University of Adelaide) · Cited by 881
Statistical Evidence: A Likelihood Paradigm
Reeves, J. H., & Royall, R. M. (1998) · Journal of the American Statistical Association · doi:10.2307/2669869 · Cited by 814
Statistical Methods and Scientific Inference.
Bailey, N. T. J., & Fisher, R. (1957) · Journal of the Royal Statistical Society. Series A (General) · doi:10.2307/2342557 · Cited by 760
The Foundations Of Scientific Inference
Salmon, W. C. (1967) · University of Pittsburgh Press eBooks · doi:10.2307/j.ctt5hjqm2 · Cited by 678
Principles of Statistical Inference
Cox, D. R. (2006) · doi:10.1017/cbo9780511813559 · Cited by 670
What If There Were No Significance Tests?
(2013) · Psychology Press eBooks · doi:10.4324/9781315827353 · Cited by 667
Logic of Statistical Inference
Hacking, I., & Romeijn, J.-W. (2016) · Cambridge University Press eBooks · doi:10.1017/cbo9781316534960 · Cited by 611
Logic of Statistical Inference.
Dempster, A. P., & Hacking, I. (1966) · Journal of the American Statistical Association · doi:10.2307/2283215 · Cited by 599
Statistical Methods and Scientific Induction
Fisher, R. (1955) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1955.tb00180.x · Cited by 596
Communications in statistics: theory and methods
Monari, P., Piccolo, D., Rampichini, C., et al. (2014) · Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) · Cited by 573
Consistency in Statistical Inference and Decision
Smith, C. A. B. (1961) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1961.tb00388.x · Cited by 557
Studies in Subjective Probability.
Hill, B. M., Kyburg, H. E., & Smokler, H. E. (1965) · Journal of the American Statistical Association · doi:10.2307/2283162 · Cited by 544
A Handbook for data analysis in the behavioral sciences: methodological issues
(1993) · Choice Reviews Online · doi:10.5860/choice.31-0602 · Cited by 543
Some Problems Connected with Statistical Inference
Cox, D. R. (1958) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177706618 · Cited by 536
Bayes or Bust? A Critical Examination of Bayesian Confirmation Theory.
Christensen, D., & Earman, J. (1994) · The Philosophical Review · doi:10.2307/2185741 · Cited by 495
Cox, D. R., & Hinkley, D. V. (1979) · Chapman & Hall/CRC eBooks · doi:10.1201/b14832 · Cited by 473
Fisher, R. A. (1930) · Mathematical Proceedings of the Cambridge Philosophical Society · doi:10.1017/s0305004100016297 · Cited by 455
Philosophy, science, and method : essays in honor of Ernest Nagel
Nagel, E., Morgenbesser, S., Suppes, P., & White, M. (1969) · St Martin's Press eBooks · Cited by 443
The Significance Test Controversy.
Carter, L. F., Morrison, D. E., & Henkel, R. E. (1971) · Social Forces · doi:10.2307/2576949 · Cited by 437
Probability, Induction and Statistics.
Godambe, V. P., & de Finetti, B. (1974) · Journal of the American Statistical Association · doi:10.2307/2285706 · Cited by 436
Horwich, P., & Ayer, A. J. (1973) · The Philosophical Review · doi:10.2307/2183724 · Cited by 414
Understanding Scientific Reasoning
Gardner, M. R., Loewer, B., & Philosophy Documentation Center (1983) · Teaching Philosophy · doi:10.5840/teachphil19836239 · Cited by 397
Popper, K. (1976) · Boston Studies in the Philosophy of Science · doi:10.1007/978-94-009-3779-6_2 · Cited by 382
The Fisher, Neyman-Pearson Theories of Testing Hypotheses: One Theory or Two?
Lehmann, E. L. (1993) · Journal of the American Statistical Association · doi:10.1080/01621459.1993.10476404 · Cited by 382
Why Isn't Everyone a Bayesian?
Efron, B. (1986) · The American Statistician · doi:10.1080/00031305.1986.10475342 · Cited by 377
Could Fisher, Jeffreys and Neyman Have Agreed on Testing?
Berger, J. O. (2003) · Statistical Science · doi:10.1214/ss/1056397485 · Cited by 346
AERA Editorial Policies regarding Statistical Significance Testing: Three Suggested Reforms
Thompson, B. (1996) · Educational Researcher · doi:10.2307/1176337 · Cited by 339
Statistical Methods and Scientific Inference
Snell, J. L., & Fisher, S. R. (1959) · Econometrica · doi:10.2307/1907790 · Cited by 336
Probability Theory and Statistical Inference: Econometric Modeling with Observational Data
Spanos, A. (1984) · Cambridge University Press eBooks · doi:10.1017/cbo9780511754081 · Cited by 328
Self-citationAchinstein, P. (2001) · doi:10.1093/0195143892.001.0001 · Cited by 322
Inference, Method and Decision
Rosenkrantz, R. D. (1977) · Springer eBooks · doi:10.1007/978-94-010-1237-9 · Cited by 317
The Logical Foundations of Statistical Inference.
Seidenfeld, T., & Kyburg, H. E. (1977) · The Journal of Philosophy · doi:10.2307/2025734 · Cited by 316
Scientific Reasoning: The Bayesian Approach.
AA, Howson, C., & Urbach, P. (1994) · Journal of the American Statistical Association · doi:10.2307/2291033 · Cited by 293
Henry, N. W., Glymour, C., Scheines, R., Spirtes, P., & Kelly, K. (1989) · Journal of Educational Statistics · doi:10.2307/1164612 · Cited by 291
Howson, C. (1995) · The British Journal for the Philosophy of Science · doi:10.1093/bjps/46.1.1 · Cited by 287
Foundations of Probability Theory, Statistical Inference, and Statistical Theories of Science
(1976) · Springer eBooks · doi:10.1007/978-94-010-1436-6 · Cited by 238
Frequentist probability and frequentist statistics
Neyman, J. (1977) · Synthese · doi:10.1007/bf00485695 · Cited by 230
P-Values: Interpretation and Methodology
Gibbons, J. D., & Pratt, J. W. (1975) · The American Statistician · doi:10.1080/00031305.1975.10479106 · Cited by 228
The Fisher, Neyman-Peerson Theories of Testing Hypotheses: One Theory or Two?
Lehmann, E. L. (2012) · Selected Works of E. L. Lehmann · doi:10.1007/978-1-4614-1412-4_19 · Cited by 220
The Superego, the Ego, and the Id in Statistical Reasoning
Gigerenzer, G. (2002) · Adaptive Thinking · doi:10.1093/acprof:oso/9780195153729.003.0013 · Cited by 203
THE CHOICE OF STATISTICAL TESTS ILLUSTRATED ON THE INTERPRETATION OF DATA CLASSED IN A 2×2 TABLE
PEARSON, E. S. (1947) · Biometrika · doi:10.1093/biomet/34.1-2.139 · Cited by 203
Stuart, A., & Edwards, A. W. F. (1973) · Applied Statistics · doi:10.2307/2346929 · Cited by 199
"Inductive Behavior" as a Basic Concept of Philosophy of Science
Neyman, J. (1957) · Revue de l'Institut International de Statistique / Review of the International Statistical Institute · doi:10.2307/1401671 · Cited by 195
Pearce, S. C., Glymour, C., Scheines, R., Spirtes, P., & Kelly, K. (1989) · Biometrics · doi:10.2307/2532061 · Cited by 192
The Interpretation of Levels of Significance by Psychological Researchers
Rosenthal, R., & Gaito, J. (1963) · The Journal of Psychology · doi:10.1080/00223980.1963.9916596 · Cited by 185
Neyman, J., & Pearson, E. S. (1967) · University of California Press eBooks · doi:10.1525/9780520339897 · Cited by 179
Probability, Induction and Statistics. The Art of Guessing
Rosenblatt-Roth, M., & de Finetti, B. (1979) · International Statistical Review / Revue Internationale de Statistique · doi:10.2307/1403216 · Cited by 177
Probability, Statistics, and Data Analysis.
Welsch, R. E., Kempthorne, O., & Folks, L. (1973) · Journal of the American Statistical Association · doi:10.2307/2284189 · Cited by 176
Conditional Confidence Statements and Confidence Estimators
Kiefer, J. (1977) · Journal of the American Statistical Association · doi:10.1080/01621459.1977.10479956 · Cited by 168
Fetzer, J. H. (1981) · Boston Studies in the Philosophy of Science · doi:10.1007/978-94-009-8558-2 · Cited by 162
Novel Evidence and Severe Tests
Mayo, D. G. (1991) · Philosophy of Science · doi:10.1086/289639 · Cited by 154
Self-citationThe handbook of research synthesis
Richards, J. D. (1995) · Evaluation Practice · doi:10.1016/0886-1633(95)90027-6 · Cited by 150
Bayesian Statistical Inference
(2019) · Princeton University Press eBooks · doi:10.2307/j.ctvrxk1hs.8 · Cited by 139
The Foundations of Statistics Reconsidered
Savage, L. J. (1961) · Cited by 133
Lectures and Conferences on Mathematical Statistics and Probability
Whittle, P. J. L., & Neyman, J. (1954) · Econometrica · doi:10.2307/1907447 · Cited by 128
Frequentist statistics as a theory of inductive inference
Mayo, D. G., & Cox, D. (2006) · Institute of Mathematical Statistics eBooks · doi:10.1214/074921706000000400 · Cited by 123
Self-citationMethodology in Practice: Statistical Misspecification Testing
Mayo, D. G., & Spanos, A. (2004) · Philosophy of Science · doi:10.1086/425064 · Cited by 118
Self-citationA theory of probable inference.
Peirce, C. S. (2011) · Little, Brown eBooks · doi:10.1037/12811-007 · Cited by 113
The Philosophy of Science : An Encyclopedia
Sarkar, S., & Pfeifer, J. (2002) · Medical Entomology and Zoology · Cited by 109
Birnbaum, A. (1977) · Synthese · doi:10.1007/bf00485690 · Cited by 104
Some Thoughts on Statistical Inference
Pearson, E. S. (1962) · The Annals of Mathematical Statistics · doi:10.1214/aoms/1177704566 · Cited by 103
Probability, Statistics and Data Analysis.
Nelder, J. A., Kempthorne, O., & Folks, L. (1972) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2344621 · Cited by 100
Statistical Concepts in Their Relation to Reality
Pearson, E. S. (1955) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1955.tb00194.x · Cited by 92
Watkins, J. P. (1996) · Philosophy · doi:10.1017/s0031819100041553 · Cited by 88
Note on an Article by Sir Ronald Fisher
Neyman, J. (1956) · Journal of the Royal Statistical Society Series B (Statistical Methodology) · doi:10.1111/j.2517-6161.1956.tb00236.x · Cited by 88
Howson, C. (1997) · Philosophy of Science · doi:10.1086/392551 · Cited by 85
Inference, Method, and Decision: Towards a Bayesian Philosophy of Science.
Jaynes, E. T., & Rosenkrantz, R. D. (1979) · Journal of the American Statistical Association · doi:10.2307/2287026 · Cited by 80
ON QUESTIONS RAISED BY THE COMBINATION OF TESTS BASED ON DISCONTINUOUS DISTRIBUTIONS
PEARSON, E. S. (1950) · Biometrika · doi:10.1093/biomet/37.3-4.383 · Cited by 80
Behavioristic, Evidentialist, and Learning Models of Statistical Testing
Mayo, D. G. (1985) · Philosophy of Science · doi:10.1086/289272 · Cited by 72
Self-citationThe problem of inductive inference
Neyman, J. (1955) · Communications on Pure and Applied Mathematics · doi:10.1002/cpa.3160080103 · Cited by 71
Inference, Method and Decision: Towards a Bayesian Philosophy of Science.
Spielman, S., & Rosenkrantz, R. D. (1981) · The Journal of Philosophy · doi:10.2307/2026129 · Cited by 68
Some issues in the foundation of statistics
Freedman, D. A. (1995) · Foundations of Science · doi:10.1007/bf00208723 · Cited by 68
Principles of Inference and Their Consequences
Mayo, D. G., & Kruse, M. (2001) · Applied logic series · doi:10.1007/978-94-017-1586-7_16 · Cited by 65
Self-citationPhilosophical Problems of Statistical Inference. Learning from R. A. Fisher.
Marriott, F. H. C., & Seidenfeld, T. (1980) · Journal of the Royal Statistical Society Series A (General) · doi:10.2307/2981990 · Cited by 61
Tests of statistical hypotheses and their use in studies of natural phenomena
Neyman, J. (1976) · Communication in Statistics- Theory and Methods · doi:10.1080/03610927608827392 · Cited by 56
An objective theory of statistical testing
Mayo, D. G. (1983) · Synthese · doi:10.1007/bf01064701 · Cited by 54
Self-citationHorwich, P. (2016) · Cambridge University Press eBooks · doi:10.1017/cbo9781316494219 · Cited by 54
Mayo, D. G. (1979) · Cited by 43
Self-citationRevisiting the omitted variables argument: Substantive vs. statistical adequacy
Spanos, A. (2006) · Journal of Economic Methodology · doi:10.1080/13501780600730687 · Cited by 41
Self-citationA note on metastatistics or ?an essay toward stating a problem in the doctrine of chances?
LeCam, L. M. (1977) · Synthese · doi:10.1007/bf00485696 · Cited by 37
Empirical Probability, Objective Statistical Methods, and Scientific Inquiry
Giere, R. N. (1976) · Springer eBooks · doi:10.1007/978-94-010-1436-6_3 · Cited by 34
Chen, D., Geng, Z., & Ma, Z.-M. (2011) · Frontiers of Mathematics · doi:10.1007/s11464-011-0159-0 · Cited by 34
Statistics and the Philosophers
Kempthorne, O. (1976) · Springer eBooks · doi:10.1007/978-94-010-1436-6_8 · Cited by 33
Science at century's end: philosophical questions on the progress and limits of science
(2000) · Choice Reviews Online · doi:10.5860/choice.38-2130 · Cited by 29
Downton, F., & Bailey, D. E. (1972) · International Statistical Review · doi:10.2307/1402768 · Cited by 21
An Error-Statistical Philosophy of Evidence
Mayo, D. G. (2004) · University of Chicago Press eBooks · doi:10.7208/chicago/9780226789583.003.0004 · Cited by 20
Self-citationA discussion of the question: for what use are tests of hypotheses and tests of significance
Pratt, J. W. (1976) · Communication in Statistics- Theory and Methods · doi:10.1080/03610927608827395 · Cited by 11
Cited by
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016) · European Journal of Epidemiology · doi:10.1007/s10654-016-0149-3 · Cited by 3,200
Fay, M. P., & Proschan, M. A. (2010) · Statistics Surveys · doi:10.1214/09-ss051 · Cited by 1,118
Philosophy and the practice of Bayesian statistics
Gelman, A., & Shalizi, C. R. (2013) · British Journal of Mathematical and Statistical Psychology · doi:10.1111/j.2044-8317.2011.02037.x · Cited by 825
The fallacy of placing confidence in confidence intervals
Morey, R. D., Hoekstra, R., Rouder, J. N., Lee, M. D., & Wagenmakers, E.-J. (2016) · Psychonomic Bulletin & Review · doi:10.3758/s13423-015-0947-8 · Cited by 577
“…procedure, warn readers that the confidence interval cannot be 8Some recent writers have suggested the replacement of Neyman’s behavioral view on confidence intervals with a frequentist view focused on tests at various levels of “stringency” (see, e.g., Mayo & Cox, 2006; Mayo & Spanos, 2006).”
BackgroundRisks and decisions for conservation and environmental management
(2005) · Choice Reviews Online · doi:10.5860/choice.43-2184 · Cited by 523
Metadata
- Authors
- Mayo, D. G., & Spanos, A.
- Venue
- The British Journal for the Philosophy of Science
- Publisher
- University of Chicago Press
- Published
- 2006-06-01
- Volume
- 57
- Issue
- 2
- Pages
- 323-357
- ISSN
- 0007-0882 (ISSN-L)
- 1464-3537
- OpenAlex
- W2141673338
- Cited by
- 270
- References
- 112
- FWCI
- 12.51
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.