CoChat
Sign Up
Cited by 387Philosophy of Biology and Scientific ExplanationLogic and Philosophy of ScienceLiterary, Cultural, Historical Analysis

The DOI matches its Crossref record; 313 of the 325 references it cites were found in a registry or an index.

Statistical Inference as Severe Testing: How to Get Beyond the Statistics Wars

Open access
Mayo, Deborah G.·2018·Cambridge University Press eBooks
DOI 10.1017/9781107286184
Open this paper in CoChat — free accountOpen PDF

Abstract

Mounting failures of replication in social and biological sciences give a new urgency to critically appraising proposed reforms. This book pulls back the cover on disagreements between experts charged with restoring integrity to science. It denies two pervasive views of the role of probability in inference: to assign degrees of belief, and to control error rates in a long run. If statistical consumers are unaware of assumptions behind rival evidence reforms, they can't scrutinize the consequences that affect them (in personalized medicine, psychology, etc.). The book sets sail with a simple tool: if little has been done to rule out flaws in inferring a claim, then it has not passed a severe test. Many methods advocated by data experts do not stand up to severe scrutiny and are in tension with successful strategies for blocking or accounting for cherry picking and selective reporting. Through a series of excursions and exhibits, the philosophy and history of inductive inference come alive. Philosophical tools are put to work to solve problems about science and pseudoscience, induction and falsification.

Overview

Integrity signals

Registry

Crossref · 10.1017/9781107286184

Cites

313 of 313 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

The text we found for this paper looks like another document, so its claims and self-check weren’t run. The registry and citation checks still apply.

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The text we found for this paper looks like another document, so the self-check wasn’t run.

Citations

Cites

1 retracted

Show all 325 references

Research works printed but not linked: 167

  1. Ziliak, 2008, The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives
  2. Worrall, 1978, Progress and Rationality in Science
  3. Wilson, 1971, Feminine Forever
  4. Will, 1986, Was Einstein Right?: Putting General Relativity to the Test
  5. Wilks, 1962, Mathematical Statistics
  6. Wasserman, 2007, The Science of Bradley Efron
  7. von Mises, 1957, Probability, Statistics and Truth
  8. 10.4159/9780674970199 · DOI
  9. Sprott, 2000, Comments on the Paper by Lindley, Journal of the Royal Statistical Society, Series D, 49
  10. Spanos, 2019, Probability Theory and Statistical Inference: Empirical Modeling with Observational Data, 10.1017/9781316882825 · DOI
  11. Spanos, 2011, Revisiting the Welch Uniform Model: A Case for Conditional Inference?, Advances and Applications in Statistical Science, 5, 33
  12. Spanos, 2008, The New Palgrave Dictionary of Economics
  13. 10.1017/CBO9780511806285 · DOI
  14. Skyrms, 1986, Choice and Chance: An Introduction to Inductive Logic
  15. Simmons, 2012, A 21 word solution, Dialogue: The Official Newsletter of the Society for Personality and Social Psychology, 26, 4
  16. Silberstein, 1919, Contribution to “Joint Eclipse Meeting of the Royal Society and the Royal Astronomical Society”, The Observatory, 42
  17. Savage, 1964, Studies in Subjective Probability, 173
  18. Savage, 1954, The Foundations of Statistics
  19. Salmon, 1966, The Foundations of Scientific Inference
  20. 455 Good, I. J. (1971b). ‘Commentary on D. J. Bartholomew’ in Godambe, V. and Sprott, D. (eds.), p. 431.
  21. Armitage, P. (1962). ‘Contribution to Discussion’, in Savage, L. J. (ed.), pp. 62–103.
  22. Laudan, 1978, Progress and Its Problems
  23. Gelman, 2012, Ethics and the Statistical Use of Prior Information, Chance Magazine, 25
  24. Cox, 1977, The Role of Significance Tests, Scandinavian Journal of Statistics, 4, 49
  25. ATLAS Collaboration (2012a). ‘Latest Results from ATLAS Higgs Search’, Press statement, ATLAS Updates, July 4, 2012. http://atlas.cern/updates/press-statement/latest-results-atlas-higgs-search.
  26. Cox, 1995, Comment on “Model Uncertainty, Data Mining and Statistical Inference by C. Chatfield”, Journal of the Royal Statistical Society: Series A, 158
  27. 448 Berger, J. (2008). ‘A Comparison of Testing Methodologies’, in Proceedings of the PHYSTAT-LHC Workshop on Statistical Issues for LHC Physics, June 2008, CERN 2008-001, pp. 8–19.
  28. Fisher, 1938, Presidential Address, Sankhyā: The Indian Journal of Statistics, 4, 14
  29. Fisher, 1926, The Arrangement of Field Experiments, Journal of Ministry of Agriculture, 33
  30. Laudan, 1996, Beyond Positivism and Relativism: Theory, Method, and Evidence
  31. Box, 1978, R. A. Fisher: The Life of a Scientist
  32. Popper, 1983, Realism and the Aim of Science
  33. Lindemann, 1919, Contribution to “Discussion on the Theory of Relativity”, Monthly Notices of the Royal Astronomical Society, 80, 114
  34. Godambe, 1971, Foundations of Statistical Inference
  35. Dupré, 1993, The Disorder of Things
  36. Dienes, 2008, Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference
  37. Lodge, 1919, Contribution to “Discussion on the Theory of Relativity”, Monthly Notices of the Royal Astronomical Society, 80
  38. de Finetti, 1974, Theory of Probability: A Critical Introductory Treatment, 2
  39. Box, 1976, Time Series Analysis: Forecasting and Control
  40. Cumming, 2012, Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-analysis
  41. Logan, B. (2012). ‘Jackie Mason Review’, The Guardian (2/21/2012).
  42. Cox, 2011, A Statistical Scientist Meets a Philosopher of Science: A Conversation between Sir David Cox and Deborah Mayo, Rationality, Markets and Morals (RMM), 2
  43. Pearson, 1966, The Selected Papers of E. S. Pearson
  44. Bernardo, 1997, Non-informative Priors Do Not Exist: A Discussion, Journal of Statistical Planning and Inference, 65
  45. Casella, 2002, Statistical Inference
  46. Spanos, 2008, Review of S. T. Ziliak and D. N. McCloskey’s The Cult of Statistical Significance, Erasmus Journal for Philosophy and Economics, 1
  47. Morrison, 1970, The Significance Test Controversy: A Reader
  48. Good, I. J. (1971a). ‘The Probabilistic Explication of Information, Evidence, Surprise, Causality, Explanation, and Utility’ and ‘Reply’, in Godambe, V. and Sprott, D. (eds.), pp. 108–22, 131–41.
  49. Robert, 2007, The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation
  50. Mayo, 2016, Don’t Throw Out the Error Control Baby with the Bad Statistics Bathwater: A Commentary on Wasserstein, R. L. and Lazar, N. A. 2016, “The ASA’s Statement on p-Values: Context, Process, and Purpose”’, The American Statistician, 70
  51. Worrall, 1989, The Uses of Experiment: Studies in the Natural Sciences
  52. Benjamin, 2016, Comment: A Simple Alternative to P-values on Wasserstein, R. and Lazar, N. 2016, “The ASA’s Statement on p-Values: Context, Process, and Purpose, The American Statistician, 70
  53. Senn, 2019, Statistical Issues in Drug Development
  54. 450CMS Experiment (2012). ‘Observation of a New Particle with a Mass of 125 GeV’, Press statement, Compact Muon Solenoid Experiment at CERN [blog], July 4, 2012. http://cms.web.cern.ch/news/observation-new-particle-mass-125-gev.
  55. Cartwright, 2012, The Oxford Handbook of Philosophy of Social Science
  56. Barnard, G. (1962). ‘Contribution to Discussion’, in Savage, L. J. (ed.), pp. 62–103.
  57. Nelder, 2000, Commentary on Lindley’s “The Philosophy of Statistics”, Journal of the Royal Statistical Society: Series D, 49
  58. Hume, 1739, A Treatise of Human Nature
  59. Armitage, 2000, Comments on the Paper by Lindley, Journal of the Royal Statistical Society: Series D, 49
  60. Banerjee, 2011, Poor Economics: A Radical Rethinking of the Way to Fight Global Poverty
  61. Glymour, C. (2010). ‘Explanation and Truth’, in Mayo, D. and Spanos, A. (eds.), pp. 331–50.
  62. Dominus, 2017, The New York Times Magazine
  63. Fisher, 1935, The Design of Experiments
  64. Gillies, 2000, Philosophical Theories of Probability.
  65. Reid, 1998, Neyman
  66. Fisher, 1950, Contributions to Mathematical Statistics
  67. Fraser, 1998, R. A. Fisher in the 21st Century]: Comment, Statistical Science, 13, 118
  68. Royall, 1997, Statistical Evidence: A Likelihood Paradigm
  69. Eddington, 1920, Space, Time and Gravitation: An Outline of the General Relativity Theory
  70. Goldacre, 2012, Bad Pharma: How Drug Companies Mislead Doctors and Harm Patients
  71. Jeffreys, 1939, Theory of Probability
  72. Vigen, 2015, Spurious Correlations
  73. Earman, 1992, Bayes or Bust?: A Critical Examination of Bayesian Confirmation Theory
  74. Bertrand, 1889, Calcul des Probabilités
  75. Byrd, 2018, Recovery, Analysis, and Identification of Commingled Human Remains, 199
  76. Box, 1979, Robustness in Statistics
  77. Wasserman, L. (2012c). ‘Nate Silver is a Frequentist: Review of “The Signal and the Noise”’, Blogpost on normaldeviate.wordpress.com (12/4/2012).
  78. Haidt, J. and Iyer, R. (2016). ‘How to Get Beyond Our Tribal Politics’, Wall Street Journal (11/10/2016).
  79. 2011, Reference Manual on Scientific Evidence
  80. Pearson, 1970, Studies in History of Statistics and Probability, I
  81. Barnard, 1972, The Logic of Statistical Inference (Review of “The Logic of Statistical Inference” by Ian Hacking), British Journal for the Philosophy of Science, 23
  82. Taleb, 2018, Skin in the Game: The Thrills and Logic of Risk Taking
  83. Good, 1983, Good Thinking: The Foundations of Probability and Its Applications
  84. Birnbaum, 1969, Philosophy, Science, and Method: Essays in Honor of Ernest Nagel
  85. Gelman, 2004, Bayesian Data Analysis
  86. Carnap, 1962, Logical Foundations of Probability
  87. Armitage, 1975, Sequential Medical Trials
  88. Efron, 1998, R. A. Fisher in the 21st Century, Statistical Science, 13, 95
  89. Borel, 1914, Le Hasard
  90. Spanos, A. (2010d). ‘Graphical Causal Modeling and Error Statistics: Exchanges with Clark Glymour’, in Mayo, D. and Spanos, A. (eds.), 364–75.
  91. Mayo, 2012, Statistical Science Meets Philosophy of Science Part 2: Shallow Versus Deep Explorations, Rationality, Markets and Morals (RMM), 3, 71
  92. Bowley, A. (1934). ‘Discussion and Commentary’ pp. 131–3 in Neyman 1934.
  93. Chalmers, A. (2010). ‘Can Scientific Theories Be Warranted?’, in Mayo, D. and Spanos, A. (eds.), pp. 58–72.
  94. Goodman, 1999, Toward Evidence-Based Medical Statistics. 2: The Bayes Factor, Annals of Internal Medicine, 130
  95. Burnham, 2002, Model Selection and Multimodal Inference: A Practical Information-Theoretic Approach
  96. Cox, 2006, Statistical Problems in Particle Physics, Astrophysics and Cosmology
  97. Dawid, 2000, Comment on a Paper by Lindley, Journal of the Royal Statistical Society: Series D, 49
  98. Goldacre, 2008, Bad Science
  99. Efron, 2000, Comments on the Paper by Lindley, Journal of the Royal Statistical Society: Series D, 49
  100. Savage, 1961, Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability
  101. de Finetti, 1972, Probability, Induction and Statistics: The Art of Guessing
  102. Prusiner, 2014, Madness and Memory: The Discovery of Prions: A New Biological Principle of Disease
  103. Cohen, 1988, Statistical Power Analysis for the Behavioral Sciences
  104. Fisher, 1956, Statistical Methods and Scientific Inference
  105. Mayo, D. (2010d). ‘An Ad Hoc Save of a Theory of Adhocness?: Exchanges with John Worrall’, in Mayo, D. and Spanos, A. (eds.), pp. 155–169.
  106. Cox, 2000, Comments on the Paper by Lindley, Journal of the Royal Statistical Society: Series D, 49
  107. Hacking, 1980, Science, Belief and Behavior: Essays in Honour of R. B. Braithwaite
  108. 458 Kyburg, H. (2003). ‘Probability as a Guide to Life’, in Kyburg, H. E. and Thalos, M. (eds.), pp. 135–52.
  109. Hand, 2014, The Improbability Principle: Why Coincidences, Miracles, and Rare Events Happen Every Day
  110. Barnard, G. (1971). ‘Scientific Inferences and Day to Day Decisions’, in Godambe, V. and Sprott, D. (eds.), pp. 289–300.
  111. Harper, 1976, Foundations of Probability Theory, Statistical Inference and Statistical Theories of Science, II
  112. Glymour, 1980, Theory and Evidence
  113. Howson, 1993, Scientific Reasoning: The Bayesian Approach
  114. Neyman, 1950, First Course in Probability and Statistics
  115. Pearson, 1947, The Choice of Statistical Tests Illustrated on the Interpretation of Data Classed in a 2 × 2 Table, Biometrika, 34, 139
  116. Huff, 1954, How to Lie with Statistics
  117. Keynes, 1921, A Treatise on Probability
  118. Mayo, 2013, JSM Proceedings
  119. Kyburg, 2003, Probability Is the Very Guide of Life: The Philosophical Uses of Chance
  120. Edwards, 1972, Likelihood
  121. Goldman, 1986, Epistemology and Cognition
  122. Kaku, 2005, Einstein’s Cosmos: How Albert Einstein’s Vision Transformed Our Understanding of Space and Time (Great Discoveries)
  123. Prusiner Labs , University of California, press release, San Francisco (2004, July 30). Prion Finding Offers Insight into Spontaneous Protein Diseases. ScienceDaily.
  124. Savage, 1962, The Foundations of Statistical Inference: A Discussion
  125. Fisher, 1934, Statistical Methods for Research Workers
  126. O’Neil, 2016, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy
  127. Laudan, 1983, Physics, Philosophy and Psychoanalysis
  128. Mayo, 2003, Could Fisher, Jeffreys and Neyman Have Agreed on Testing? Commentary on J. Berger’s Fisher Address, Statistical Science, 18, 19
  129. Lehmann, 1993, Statistics and Probability: A Raghu Raj Bahadur Festschrift
  130. Mayo, 2011, Statistical Science and Philosophy of Science: Where Do/Should They Meet in 2011 (and Beyond)?, Rationality, Markets and Morals (RMM), 2, 79
  131. Lehmann, 2012, Selected Works of E. L. Lehmann
  132. Lehmann, 2005, Testing Statistical Hypotheses
  133. Levi, 1980, The Enterprise of Knowledge: An Essay on Knowledge, Credal Probability, and Change
  134. Efron, 2013, A 250-Year Argument: Belief, Behavior, and the Bootstrap, Bulletin of the American Mathematical Society, 50
  135. Lindley, D. (1976). ‘Bayesian Statistics’, in Harper, W. and Hooker, C. (eds.), pp. 353–62.
  136. Ghosh, 2010, An Introduction to Bayesian Analysis: Theory and Methods
  137. Mayo, D. (2003a). ‘Severe Testing as a Guide for Inductive Learning’, in Kyburg, H. E. and Thalos, M. (eds.), pp. 89–117.
  138. Mayo, D. (2005b). ‘Evidence as Passing Severe Tests: Highly Probable versus Highly Probed Hypotheses’, in Achinstein, P. (ed.), Scientific Evidence, Johns Hopkins, pp. 95–127.
  139. Feynman, 1974, Surely You’re Joking Mr. Feynman!: Adventures of a Curious Character
  140. Berger, 1987, Testing a Point Null Hypothesis: The Irreconcilability of P Values and Evidence (with Discussion and Rejoinder), Journal of the American Statistical Association, 82
  141. Mayo, D. (2010b). ‘An Error in the Argument from Conditionality and Sufficiency to the Likelihood Principle’, in Mayo, D. and Spanos, A. (eds.), pp. 305–14.
  142. Mayo, D. (2010c). ‘Sins of the Epistemic Probabilist: Exchanges with Peter Achinstein’, in Mayo, D. and Spanos, A. (eds.), pp. 189–201.
  143. Fisher. R. A. (1934c). ‘Discussion and Commentary’ pp. 137–8 in Neyman 1934.
  144. Mayo, 2014, Error and Uncertainty in Scientific Practice, 57
  145. Kass, 1998, [R. A. Fisher in the 21st Century]: Comment, Statistical Science, 13
  146. Mayo, 2010, Error and Inference: Recent Exchanges on Experimental Reasoning, Reliability, and the Objectivity and Rationality of Science
  147. Mayo, 2011, Philosophy of Statistics, 7, Handbook of the Philosophy of Science
  148. Mill, 1888, A System of Logic
  149. Miller, J. (2008). Naturalism & Objectivity: Methods and Meta-methods (dissertation), Virginia Tech.
  150. Westfall, 1993, Resampling-Based Multiple Testing: Examples and Methods for P-Value Adjustment
  151. Musgrave, A. (2010). ‘Critical Rationalism, Explanation and Severe Tests’, in Mayo, D. and Spanos, A. (eds.), pp. 88–112.
  152. 2002, The Truth About Hormone Replacement Therapy: How to Break Free from the Medical Myths of Menopause
  153. Achinstein, P. (2010). ‘Mill’s Sins or Mayo’s Errors?’ In Mayo, D. and Spanos, A. (eds.), pp. 170–88.
  154. Neyman, 1952, Lectures and Conferences on Mathematical Statistics and Probability
  155. Neyman, 1957, The Use of the Concept of Power in Agricultural Experimentation, Journal of the Indian Society of Agricultural Statistics, IX, 9
  156. Neyman, 1967, Early Statistical Papers of J. Neyman
  157. Neyman, 1936, Contributions to the Theory of Testing Statistical Hypotheses, Statistical Research Memoirs, 1, 1
  158. Birnbaum, 1962, Breakthroughs in Statistics, 1, 478
  159. Akaike, 1973, 2nd International Symposium on Information Theory
  160. Peirce, 1931, Collected Papers, 1
  161. Popper, 1959, The Logic of Scientific Discovery
  162. Popper, 1962, Conjectures and Refutations: The Growth of Scientific Knowledge
  163. Potti, 2007, Potti et al. Reply, Nature Medicine, 13
  164. Pratt, 1995, Introduction to Statistical Decision Theory
  165. Reid, 2003, Could Fisher, Jeffreys and Neyman Have Agreed on Testing? Commentary on J. Berger’s Fisher Address, Statistical Science, 18, 27
  166. 465Robbins, 1956, Proceedings Third Berkeley Symposium on Mathematical Statistics and Probability, Vol. 1
  167. Rothman, 2013, Brief to United States Supreme Court of Amici Curiae in Support of Petitioner

Cited by

Sorted by citations

Related

See all

Metadata

Authors
Mayo, D. G.
Affiliations
Virginia Tech (US)
Corresponding author
Mayo, Deborah G.
ORCID
Mayo, Deborah G.
Venue
Cambridge University Press eBooks
Publisher
Cambridge University Press
Published
2018-09-14
DOI
10.1017/9781107286184
OpenAlex
W3171064310
Cited by
376
References
325
Citations by year
2026: 32 · 2025: 57 · 2024: 35 · 2023: 42 · 2022: 60 · 2021: 72 · 2020: 43 · 2019: 28 · 2018: 5
FWCI
54.96
Type
book
Language
en
Open access
Licence
www.cambridge.org/core/terms (unspecified)
Locations
Cambridge University Press eBooks (publishedVersion)
Topics
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Philosophy of Biology and Scientific Explanation
  • Social Sciences › Arts and Humanities › History and Philosophy of Science › Logic and Philosophy of Science
  • Social Sciences › Arts and Humanities › Philosophy › Literary, Cultural, Historical Analysis
Keywords
severe testing, statistical inference, replication crisis, degrees of belief, cherry-picking, selective reporting, inductive reasoning, falsification, pseudoscience, science reform

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review