CoChat
Sign Up
Cited by 2,490Statistical Methods and Bayesian InferenceMeta-analysis and systematic reviewsProbability and Statistical Research

A practical solution to the pervasive problems ofp values

Open access
Wagenmakers, Eric-Jan·2007·Psychonomic Bulletin & Review
DOI 10.3758/bf03194105
Open this paper in CoChat — free accountOpen PDF

Abstract

In the field of psychology, the practice of p value null-hypothesis testing is as widespread as ever. Despite this popularity, or perhaps because of it, most psychologists are not aware of the statistical peculiarities of the p value procedure. In particular, p values are based on data that were never observed, and these hypothetical data are themselves influenced by subjective intentions. Moreover, p values do not quantify statistical evidence. This article reviews these p value problems and illustrates each problem with concrete examples. The three problems are familiar to statisticians but may be new to psychologists. A practical solution to these p value problems is to adopt a model selection perspective and use the Bayesian information criterion (BIC) for statistical inference (Raftery, 1995). The BIC provides an approximation to a Bayesian hypothesis test, does not require the specification of priors, and can be easily calculated from SPSS output.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.3758/bf03194105

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Bayesian analysis provided strong evidence for the null hypothesis across all pairwise comparisons (BF01 range: 11.9-34.6).

    “Bayesian analysis provided strong evidence for the null hypothesis across all pairwise comparisons (BF 01 range: 11 . 9-34 . 6; see Supplementary Information, [45]).”

    From Humanlike AI Design Increases Anthropomorphism but Yields Divergent Outcomes on Engagement and Trust Globally (2025) · cites this paper as [45]

    Table 3 (section “That Wretched Prior . . .”; page unavailable) classifies BF01 values of 20–150 as strong evidence for the null, but values of 3–20 as positive evidence. Thus 34.6 meets that scheme’s strong threshold while 11.9 does not. The paper does not report the cited pairwise comparisons or their BF01 range. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

  • The shortcomings of null hypothesis significance testing are well documented.

    “The limitations of NHST methods are well documented and need not be rehashed here in their entirety”

    From Reevaluating the effectiveness of n-back training on transfer through the Bayesian lens: Support for the null (2016) · cites this paper as Wagenmakers (2007)

  • Assuming a potentially infinite number of tests, a data collection stopping rule of p ≤ α guarantees a significant result.

    “assuming a potentially infinite number of tests, a data collection stopping rule of p ≤ α guarantees a significant result.”

    From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Wagenmakers, 2007, pp. 784-785

  • If researchers merely aim to decide whether or not a test result is significantly different from a point null hypothesis (in either direction), then a p ≤ α stopping rule will result in 'sampling to a forgone conclusion'.

    “if researchers merely aim to decide whether or not a test result is significantly different from a point null hypothesis (in either direction), then a p ≤ α stopping rule will result in 'sampling to a forgone conclusion'”

    From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Wagenmakers, 2007, pp. 784-785

  • The Bayesian framework represents parameter uncertainty directly through posterior distributions, offering interpretable estimates without relying on sampling distributions from null hypotheses.

    “The Bayesian framework was chosen specifically for its ability to represent parameter uncertainty directly through posterior distributions, offering interpretable estimates without relying on sampling distributions from null hypotheses, as required in frequentist approaches [79, 80, 81].”

    From Interpersonal Synchronization of Autonomic Physiology via Mere Visual Contact (2026) · cites this paper as [79]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 99

  1. Anscombe, F. J. (1963). Sequential medical trials. Journal of the Ameri- can Statistical Association, 58, 365-383.
  2. Armitage, P., McPherson, C. K., & Rowe, B. C. (1969). Repeated significance tests on accumulating data. Journal of the Royal Statisti- cal Society: Series A, 132, 235-244.
  3. Bakan, D. (1966). The test of significance in psychological research. Psychological Bulletin, 66, 423-437.
  4. Berger, J. O. (1985). Statistical decision theory and Bayesian analysis (2nd ed.). New York: Springer.
  5. Berger, J. O., & Berry, D. A. (1988a). The relevance of stopping rules in statistical inference. In S. S. Gupta & J. O. Berger (Eds.), Statistical decision theory and related topics IV (Vol. 1, pp. 29-72). New York: Springer.
  6. Berger, J. O., Brown, L., & Wolpert, R. (1994). A unified conditional frequentist and Bayesian test for fixed and sequential hypothesis test- ing. Annals of Statistics, 22, 1787-1807.
  7. Berger, J. O., & Mortera, J. (1999). Default Bayes factors for non- nested hypothesis testing. Journal of the American Statistical Associa- tion, 94, 542-554.
  8. Berger, J. O., & Pericchi, L. R. (1996). The intrinsic Bayes factor for model selection and prediction. Journal of the American Statistical Association, 91, 109-122.
  9. Berger, J. O., & Sellke, T. (1987). Testing a point null hypothesis: The irreconcilability of p values and evidence. Journal of the American Statistical Association, 82, 112-139.
  10. Berger, J. O., & Wolpert, R. L. (1988). The likelihood principle (2nd ed.). Hayward, CA: Institute of Mathematical Statistics.
  11. Birnbaum, A. (1962). On the foundations of statistical inference (with discussion). Journal of the American Statistical Association, 53, 259- 326.
  12. Birnbaum, A. (1977). The Neyman-Pearson theory as decision theory, and as inference theory; with a criticism of the Lindley-Savage argu- ment for Bayesian theory. Synthese, 36, 19-49.
  13. Box, G. E. P., & Tiao, G. C. (1973). Bayesian inference in statistical analysis. Reading, MA: Addison-Wesley.
  14. Browne, M. (2000). Cross-validation methods. Journal of Mathemati- cal Psychology, 44, 108-132.
  15. Christensen, R. (2005). Testing Fisher, Neyman, Pearson, and Bayes. American Statistician, 59, 121-126.
  16. Cox, D. R. (1971). The choice between alternative ancillary statistics. Journal of the Royal Statistical Society: Series B, 33, 251-255.
  17. Cox, R. T. (1946). Probability, frequency and reasonable expectation. American Journal of Physics, 14, 1-13.
  18. De Finetti, B. (1974). Theory of probability: A critical introductory treatment (Vols. 1 & 2; A. Machí & A. Smith, Trans.). London: Wiley.
  19. Diamond, G. A., & Forrester, J. S. (1983). Clinical trials and statisti- cal verdicts: Probable grounds for appeal. Annals of Internal Medi- cine, 98, 385-394.
  20. Dickey, J. M. (1973). Scientific reporting and personal probabilities: Student's hypothesis. Journal of the Royal Statistical Society: Se- ries B, 35, 285-305.
  21. Dickey, J. M. (1977). Is the tail area useful as an approximate Bayes fac- tor? Journal of the American Statistical Association, 72, 138-142.
  22. Dixon, P. (2003). The p value fallacy and how to avoid it. Canadian Journal of Experimental Psychology, 57, 189-202.
  23. Djuric ´, P. M. (1998). Asymptotic MAP criteria for model selection. IEEE Transactions on Signal Processing, 46, 2726-2735.
  24. Edwards, A. W. F. (1992). Likelihood. Baltimore: Johns Hopkins Uni- versity Press.
  25. Edwards, W., Lindman, H., & Savage, L. J. (1963). Bayesian statisti- cal inference for psychological research. Psychological Review, 70, 193-242.
  26. Efron, B. (2005). Bayesians, frequentists, and scientists. Journal of the American Statistical Association, 100, 1-5.
  27. Efron, B., & Tibshirani, R. (1997). Improvements on cross-validation: The .632 bootstrap method. Journal of the American Statistical As- sociation, 92, 548-560.
  28. Feller, W. (1940). Statistical aspects of ESP. Journal of Parapsychol- ogy, 4, 271-298.
  29. Feller, W. (1970). An introduction to probability theory and its appli- cations: Vol. 1 (2nd ed.). New York: Wiley.
  30. Fine, T. L. (1973). Theories of probability: An examination of founda- tions. New York: Academic Press.
  31. Fisher, R. A. (1934). Statistical methods for research workers (5th ed.). London: Oliver & Boyd.
  32. Fisher, R. A. (1935b). The logic of inductive inference (with discus- sion). Journal of the Royal Statistical Society, 98, 39-82.
  33. Fisher, R. A. (1958). Statistical methods for research workers (13th ed.). New York: Hafner.
  34. Friedman, L. M., Furberg, C. D., & DeMets, D. L. (1998). Funda- mentals of clinical trials (3rd ed.). New York: Springer.
  35. Gigerenzer, G. (1993). The superego, the ego, and the id in statistical reasoning. In G. Keren & C. Lewis (Eds.), A handbook for data analy- sis in the behavioral sciences: Methodological issues (pp. 311-339).
  36. Gilks, W. R., Richardson, S., & Spiegelhalter, D. J. (Eds.) (1996). Markov chain Monte Carlo in practice. Boca Raton, FL: Chapman & Hall/CRC.
  37. Gill, J. (2002). Bayesian methods: A social and behavioral sciences approach. Boca Raton, FL: CRC Press.
  38. Glover, S., & Dixon, P. (2004). Likelihood ratios: A simple and flex- ible statistic for empirical psychologists. Psychonomic Bulletin & Re- view, 11, 791-806.
  39. Good, I. J. (1983). Good thinking: The foundations of probability and its applications. Minneapolis: University of Minnesota Press.
  40. Good, I. J. (1985). Weight of evidence: A brief survey. In J. M. Bernardo, M. H. DeGroot, D. V. Lindley, & A. F. M. Smith (Eds.), Bayesian sta- tistics 2: Proceedings of the Second Valencia International Meeting, September 6/10, 1983 (pp. 249-269). Amsterdam: North-Holland.
  41. Goodman, S. N. (1993). p values, hypothesis tests, and likelihood: Im- plications for epidemiology of a neglected historical debate. American Journal of Epidemiology, 137, 485-496.
  42. Grünwald, P. [D.] (2000). Model selection based on minimum descrip- tion length. Journal of Mathematical Psychology, 44, 133-152.
  43. Grünwald, P. D., Myung, I. J., & Pitt, M. A. (Eds.) (2005). Advances in minimum description length: Theory and applications. Cambridge, MA: MIT Press.
  44. Hacking, I. (1965). Logic of statistical inference. Cambridge: Cam- bridge University Press.
  45. Hannan, E. J. (1980). The estimation of the order of an ARMA process. Annals of Statistics, 8, 1071-1081.
  46. Howson, C., & Urbach, P. (2005). Scientific reasoning: The Bayesian approach (3rd. ed.). Chicago: Open Court.
  47. Jaynes, E. T. (1968). Prior probabilities. IEEE Transactions on Systems Science & Cybernetics, 4, 227-241.
  48. Jaynes, E. T. (2003). Probability theory: The logic of science. Cam- bridge: Cambridge University Press.
  49. Jeffreys, H. (1961). Theory of probability. Oxford: Oxford University Press.
  50. Jennison, C., & Turnbull, B. W. (1990). Statistical approaches to in- terim monitoring of medical trials: A review and commentary. Statisti- cal Science, 5, 299-317.
  51. Kadane, J. B., Schervish, M. J., & Seidenfeld, T. (1996). Reasoning to a foregone conclusion. Journal of the American Statistical Associa- tion, 91, 1228-1235.
  52. Karabatsos, G. (2006). Bayesian nonparametric model selection and model testing. Journal of Mathematical Psychology, 50, 123-148.
  53. Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 377-395.
  54. Kass, R. E., & Wasserman, L. (1996). The selection of prior distribu- tions by formal rules. Journal of the American Statistical Association, 91, 1343-1370.
  55. Killeen, P. R. (2006). Beyond statistical inference: A decision theory for science. Psychonomic Bulletin & Review, 13, 549-562.
  56. Lee, M. D. (2002). Generating additive clustering models with limited stochastic complexity. Journal of Classification, 19, 69-85.
  57. Lee, M. D., & Pope, K. J. (2006). Model selection for the rate prob- lem: A comparison of significance testing, Bayesian, and minimum description length statistical inference. Journal of Mathematical Psy- chology, 50, 193-202.
  58. Lee, M. D., & Wagenmakers, E.-J. (2005). Bayesian statistical infer- ence in psychology: Comment on Trafimow (2003). Psychological Review, 112, 662-668.
  59. Lee, P. M. (1989). Bayesian statistics: An introduction. New York: Ox- ford University Press.
  60. Lindley, D. V. (1972). Bayesian statistics: A review. Philadelphia: So- ciety for Industrial & Applied Mathematics.
  61. Lindley, D. V. (1982). Scoring rules and the inevitability of probability. International Statistical Review, 50, 1-26.
  62. Lindley, D. V., & Scott, W. F. (1984). New Cambridge elementary statistical tables. Cambridge: Cambridge University Press.
  63. Loftus, G. R. (2002). Analysis, interpretation, and visual presentation of experimental data. In H. Pashler (Ed. in Chief ) & J. Wixted (Vol. Ed.), Stevens' Handbook of experimental psychology: Vol. 4. Method- ology in experimental psychology (3rd ed., pp. 339-390). New York: Wiley.
  64. Myung, I. J. (2000). The importance of complexity in model selection. Journal of Mathematical Psychology, 44, 190-204.
  65. Myung, I. J., Forster, M. R., & Browne, M. W. (Eds.) (2000). Model selection [Special issue]. Journal of Mathematical Psychology, 44(1).
  66. Myung, I. J., Navarro, D. J., & Pitt, M. A. (2006). Model selection by normalized maximum likelihood. Journal of Mathematical Psychol- ogy, 50, 167-179.
  67. Myung, I. J., & Pitt, M. A. (1997). Applying Occam's razor in model- ing cognition: A Bayesian approach. Psychonomic Bulletin & Review, 4, 79-95.
  68. Neyman, J., & Pearson, E. S. (1933). On the problem of the most ef- ficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society: Series A, 231, 289-337.
  69. Nickerson, R. S. (2000). Null hypothesis statistical testing: A review of an old and continuing controversy. Psychological Methods, 5, 241- 301.
  70. O'Hagan, A. (1997). Fractional Bayes factors for model comparison. Journal of the Royal Statistical Society: Series B, 57, 99-138.
  71. O'Hagan, A., & Forster, J. (2004). Kendall's advanced theory of sta- tistics: Vol. 2B. Bayesian inference (2nd ed.). London: Arnold.
  72. Peto, R., Pike, M. C., Armitage, P., Breslow, N. E., Cox, D. R., How- ard, S. V., et al. (1976). Design and analysis of randomized clinical trials requiring prolonged observation of each patient: I. Introduction and design. British Journal of Cancer, 34, 585-612.
  73. Pitt, M. A., Myung, I. J., & Zhang, S. (2002). Toward a method of selecting among computational models of cognition. Psychological Review, 109, 472-491.
  74. Pocock, S. J. (1983). Clinical trials: A practical approach. New York: Wiley.
  75. Review of Lehmann, E. L., Testing statistical hy- potheses]. Journal of the American Statistical Association, 56, 163- 167.
  76. Pratt, J. W. (1962). On the foundations of statistical inference: Discus- sion. Journal of the American Statistical Association, 57, 314-315.
  77. Raftery, A. E. (1993). Bayesian model selection in structural equation models. In K. A. Bollen & J. S. Long (Eds.), Testing structural equa- tion models (pp. 163-180). Newbury Park, CA: Sage.
  78. Raftery, A. E. (1995). Bayesian model selection in social research. In P. V. Marsden (Ed.), Sociological methodology 1995 (pp. 111-196). Cambridge, MA: Blackwell.
  79. Raftery, A. E. (1996). Hypothesis testing and model selection. In W. R. Gilks, S. Richardson, & D. J. Spiegelhalter (Eds.), Markov chain Monte Carlo in practice (pp. 163-187). Boca Raton, FL: Chapman & Hall/CRC.
  80. Rissanen, J. (2001). Strong optimality of the normalized ML models as universal codes and information in data. IEEE Transactions on Infor- mation Theory, 47, 1712-1717.
  81. Robert, C. P., & Casella, G. (1999). Monte Carlo statistical meth- ods. New York: Springer.
  82. Rosenthal, R., & Gaito, J. (1963). The interpretation of levels of significance by psychological researchers. Journal of Psychology, 55, 33-38.
  83. Rouder, J. N., & Lu, J. (2005). An introduction to Bayesian hierar- chical models with an application in the theory of signal detection. Psychonomic Bulletin & Review, 12, 573-604.
  84. Rouder, J. N., Lu, J., Speckman, P., Sun, D., & Jiang, Y. (2005). A hierarchical model for estimating response time distributions. Psy- chonomic Bulletin & Review, 12, 195-223.
  85. Savage, L. J. (1954). The foundations of statistics. New York: Wiley.
  86. Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461-464.
  87. Shafer, G. (1982). Lindley's paradox. Journal of the American Statis- tical Association, 77, 325-351.
  88. Siegmund, D. (1985). Sequential analysis: Tests and confidence inter- vals. New York: Springer.
  89. Smith, A. F. M., & Spiegelhalter, D. J. (1980). Bayes factors and choice criteria for linear models. Journal of the Royal Statistical Society: Series B, 42, 213-220.
  90. Stone, M. (1974). Cross-validatory choice and assessment of statisti- cal predictions (with discussion). Journal of the Royal Statistical Society: Series B, 36, 111-147.
  91. Stuart, A., Ord, J. K., & Arnold, S. (1999). Kendall's advanced theory of statistics: Vol. 2A. Classical inference and the linear model (6th ed.). London: Arnold.
  92. Trafimow, D. (2003). Hypothesis testing and theory evaluation at the boundaries: Surprising insights from Bayes's theorem. Psychologi- cal Review, 110, 526-535.
  93. Wagenmakers, E.-J. (2003). How many parameters does it take to fit an elephant? [Book review]. Journal of Mathematical Psychology, 47, 580-586.
  94. Wagenmakers, E.-J., Grünwald, P., & Steyvers, M. (2006). Ac- cumulative prediction error and the selection of time series models. Journal of Mathematical Psychology, 50, 149-166.
  95. Wagenmakers, E.-J., Ratcliff, R., Gomez, P., & Iverson, G. J. (2004). Assessing model mimicry using the parametric bootstrap. Journal of Mathematical Psychology, 48, 28-50.
  96. Wagenmakers, E.-J., & Waldorp, L. (Eds.) (2006). Model selection: Theoretical developments and applications [Special issue]. Journal of Mathematical Psychology, 50(2).
  97. Wallace, C. S., & Dowe, D. L. (1999). Refinements of MDL and MML coding. Computer Journal, 42, 330-337.
  98. Wasserman, L. (2000). Bayesian model selection and model averag- ing. Journal of Mathematical Psychology, 44, 92-107.
  99. Wasserman, L. (2004). All of statistics: A concise course in statistical inference. New York: Springer.

Cited by

Sorted by citations

Authors
Wagenmakers, E.-J.
Affiliations
University of Amsterdam (NL)
Corresponding author
Wagenmakers, Eric-Jan
ORCID
Wagenmakers, Eric-Jan
Venue
Psychonomic Bulletin & Review
Publisher
Springer Science and Business Media LLC
Published
2007-10
Volume
14
Issue
5
Pages
779-804
ISSN
  • 1069-9384 (ISSN-L)
  • 1531-5320
DOI
10.3758/bf03194105
OpenAlex
W2017146523
Cited by
2,490
References
12
Citations by year
2026: 116 · 2025: 151 · 2024: 162 · 2023: 170 · 2022: 184 · 2021: 214 · 2020: 203 · 2019: 290 · 2018: 231 · 2017: 186 · 2016: 150 · 2015: 103 · 2014: 89 · 2013: 71 · 2012: 49
FWCI
18.31
Type
journal article
Language
en
Open access
Licence
www.springer.com/tdm (tdm)
Locations
  • Psychonomic Bulletin & Review (publishedVersion)
  • PubMed (publishedVersion)
  • UvA-DARE (University of Amsterdam) (submittedVersion)
  • Data Archiving and Networked Services (DANS) (submittedVersion)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
Keywords
p-values, null hypothesis significance testing, statistical evidence, model selection, Bayesian information criterion, Bayesian hypothesis testing, prior specification, statistical inference
MeSH terms
Bayes Theorem, Humans, Models, Psychological, Psychology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review