CoChat
Sign Up
Cited by 740Statistical Methods and Bayesian InferenceBayesian Modeling and Causal InferenceStatistical Distribution Estimation and Applications

Bayesian hypothesis testing for psychologists: A tutorial on the Savage–Dickey method

Open access
Wagenmakers, Eric-Jan, Lodewyckx, Tom, Kuriyal, Himanshu, Grasman, Raoul·2010·Cognitive Psychology
DOI 10.1016/j.cogpsych.2009.12.001
Open this paper in CoChat — free accountOpen PDF

Abstract

In the field of cognitive psychology, the p-value hypothesis test has established a stranglehold on statistical reporting. This is unfortunate, as the p-value provides at best a rough estimate of the evidence that the data provide for the presence of an experimental effect. An alternative and arguably more appropriate measure of evidence is conveyed by a Bayesian hypothesis test, which prefers the model with the highest average likelihood. One of the main problems with this Bayesian hypothesis test, however, is that it often requires relatively sophisticated numerical methods for its computation. Here we draw attention to the Savage-Dickey density ratio method, a method that can be used to compute the result of a Bayesian hypothesis test for nested models and under certain plausible restrictions on the parameter priors. Practical examples demonstrate the method's validity, generality, and flexibility.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.cogpsych.2009.12.001

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Researchers may demonstrate the robustness of their results by using Bayesian hypothesis tests, because optional stopping is not problematic for Bayesian hypothesis testing.

    “They may also demonstrate the robustness of their results by using Bayesian hypothesis tests, because optional stopping is not problematic for Bayesian hypothesis testing”

    From Does preregistration improve the credibility of research findings? (2020) · cites this paper as Wagenmakers, Lodewyckx, Kuriyal, & Grasman, 2010, p. 167

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 73

  1. Aitchison, J., & Dunsmore, I. R. (1975). Statistical prediction analysis. Cambridge: Cambridge University Press.
  2. Anscombe, F. J. (1963). Sequential medical trials. Journal of the American Statistical Association, 58, 365-383.
  3. Batchelder, W. H., & Riefer, D. M. (1999). Theoretical and empirical review of multinomial process tree modeling. Psychonomic Bulletin & Review, 6, 57-86.
  4. Berger, J. O., & Berry, D. A. (1988a). The relevance of stopping rules in statistical inference. In S. S. Gupta & J. O. Berger (Eds.). Statistical decision theory and related topics (Vol. 1, pp. 29-72). New York: Springer Verlag.
  5. Berger, J. O., & Mortera, J. (1999). Default Bayes factors for nonnested hypothesis testing. Journal of the American Statistical Association, 94, 542-554.
  6. Berger, J. O., & Pericchi, L. R. (1996). The intrinsic Bayes factor for model selection and prediction. Journal of the American Statistical Association, 91, 109-122.
  7. Berger, J. O., & Pericchi, L. R. (2001). Objective Bayesian methods for model selection: Introduction and comparison (with discussion). In P. Lahiri (Ed.). Model selection (Vol. 38, pp. 135-207). Beachwood, OH: Institute of Mathematical Statistics Lecture Notes-Monograph Series.
  8. Bernardo, J. M., & Smith, A. F. M. (1994). Bayesian theory. New York: Wiley.
  9. Bowers, J. S. (1999). Priming is not all bias: Commentary on Ratcliff and McKoon (1997). Psychological Review, 106, 582-596.
  10. Brückner, H., & Bearman, P. (2005). After the promise: The STD consequences of adolescent virginity pledges. Journal of Adolescent Health, 36, 271-278.
  11. Carlin, B. P., & Chib, S. (1995). Bayesian model choice via Markov chain Monte Carlo methods. Journal of the Royal Statistical Society, Series B, 57, 473-484.
  12. Chib, S. (1995). Marginal likelihood from the Gibbs output. Journal of the American Statistical Association, 90, 1313-1321.
  13. Chib, S., & Jeliazkov, I. (2001). Marginal likelihood from the Metropolis-Hastings output. Journal of the American Statistical Association, 96, 270-281.
  14. Christensen, R. (2005). Testing fisher, Neyman, Pearson, and Bayes. The American Statistician, 59, 121-126.
  15. Congdon, P. (2003). Applied Bayesian modelling. Chichester, UK: Wiley.
  16. Dennis, S., & Humphreys, M. S. (2001). A context noise model of episodic word recognition. Psychological Review, 108, 452-477.
  17. Dennis, S. J., Lee, M. D., & Kinnell, A. (2008). Bayesian analysis of recognition memory: The case of the list -length effect. Journal of Memory and Language, 59, 361-376.
  18. Dey, D. K., Ghosh, S. K., & Mallick, B. K. (Eds.). (2000). Generalized linear models: A Bayesian perspective. Boca Raton, FL: Taylor & Francis.
  19. Dickey, J. M. (1971). The weighted likelihood ratio, linear hypotheses on normal location parameters. The Annals of Mathematical Statistics, 42, 204-223.
  20. Dickey, J. M., & Lientz, B. P. (1970). The weighted likelihood ratio, sharp hypotheses about chances, the order of a Markov chain. The Annals of Mathematical Statistics, 41, 214-226.
  21. Draper, D. (1995). Assessment and propagation of model uncertainty. Journal of the Royal Statistical Society, Series B, 57, 45-97.
  22. Edwards, W., Lindman, H., & Savage, L. J. (1963). Bayesian statistical inference for psychological research. Psychological Review, 70, 193-242.
  23. Forster, K. I., Mohan, K., & Hector, J. (2003). The mechanics of masked priming. In S. Kinoshita & S. J. Lupker (Eds.), Masked priming: The state of the art (pp. 3-38). New York, NY: Psychology Press.
  24. Gallistel, C. R. (2009). The importance of proving the null. Psychological Review, 116, 439-453.
  25. Gelfand, A. E., Smith, A. F. M., & Lee, T.-M. (1992). Bayesian analysis of constrained parameter and truncated data problems using Gibbs sampling. Journal of the American Statistical Association, 87, 523-532.
  26. Gelman, A. (1996). Inference and monitoring convergence. In W. R. Gilks, S. Richardson, & D. J. Spiegelhalter (Eds.), Markov chain Monte Carlo in practice (pp. 131-143). Boca Raton, FL: Chapman & Hall/CRC.
  27. Gelman, A., & Hill, J. (2007). Data analysis using regression and multilevel/hierarchical models. Cambridge: Cambridge University Press.
  28. Gigerenzer, G. (1993). The Superego, the Ego, and the Id in statistical reasoning. In C. Lewis & G. Keren (Eds.), A handbook for data analysis in the behavioral sciences: Methodological issues (pp. 311-339). Hillsdale, NJ: Erlbaum.
  29. Gigerenzer, G. (2004). Mindless statistics. The Journal of Socio-Economics, 33, 587-606.
  30. Gilks, W. R., Richardson, S., & Spiegelhalter, D. J. (Eds.). (1996). Markov chain Monte Carlo in practice. Boca Raton, FL: Chapman & Hall/CRC.
  31. Gill, J. (2002). Bayesian methods: A social and behavioral sciences approach. Boca Raton, FL: CRC Press.
  32. Good, I. J. (1985). Weight of evidence: A brief survey. In J. M. Bernardo, M. H. DeGroot, D. V. Lindley, & A. F. M. Smith (Eds.), Bayesian statistics 2 (pp. 249-269). New York: Elsevier.
  33. Grant, D. A., & Berg, E. A. (1948). A behavioral analysis of degree of reinforcement and ease of shifting to new responses in a Weigl-type card-sorting problem. Journal of Experimental Psychology, 38, 404-411.
  34. Hoijtink, H., Klugkist, I., & Boelen, P. (2008). Bayesian evaluation of informative hypotheses that are of practical value for social scientists. New York: Springer.
  35. Hsiao, C. K. (1997). Approximate Bayes factors when a mode occurs on the boundary. Journal of the American Statistical Association, 92, 656-663.
  36. Jaynes, E. T. (2003). Probability theory: The logic of science. Cambridge, UK: Cambridge University Press.
  37. Jeffreys, H. (1961). Theory of probability. Oxford, UK: Oxford University Press.
  38. Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773-795.
  39. Kass, R. E., & Vaidyanathan, S. K. (1992). Approximate Bayes factors and orthogonal parameters, with application to testing equality of two binomial proportions. Journal of the Royal Statistical Society, Series B, 54, 129-144.
  40. Kass, R. E., & Wasserman, L. (1995). A reference Bayesian test for nested hypotheses and its relationship to the Schwarz criterion. Journal of the American Statistical Association, 90, 928-934.
  41. Kruschke, J. K. (1992). ALCOVE: An exemplar-based connectionist model of category learning. Psychological Review, 99, 22-44.
  42. Lauritzen, S. L. (1996). Graphical models. Oxford: Clarendon.
  43. Lee, M. D. (2008). Three case studies in the Bayesian analysis of cognitive models. Psychonomic Bulletin & Review, 15, 1-15.
  44. Lewis, S. M., & Raftery, A. E. (1997). Estimating Bayes factors via posterior simulation with the Laplace-Metropolis estimator. Journal of the American Statistical Association, 92, 648-655.
  45. Liang, F., Paulo, R., Molina, G., Clyde, M. A., & Berger, J. O. (2008). Mixtures of g priors for Bayesian variable selection. Journal of the American Statistical Association, 103, 410-423.
  46. Lindley, D. V. (1972). Bayesian statistics, a review. Philadelphia, PA: SIAM.
  47. Liu, C. C., & Aitkin, M. (2008). Bayes factors: Prior sensitivity and model generalizability. Journal of Mathematical Psychology, 52, 362-375.
  48. MacKay, D. J. C. (2003). Information theory, inference, and learning algorithms. Cambridge: Cambridge University Press.
  49. Mulder, J., Klugkist, I., van de Schoot, R., Meeus, W.H.J., Selfhout, M., & Hoijtink, H. (in press). Bayesian model selection of informative hypotheses for repeated measurements. Journal of Mathematical Psychology.
  50. Myung, I. J. (2003). Tutorial on maximum likelihood estimation. Journal of Mathematical Psychology, 47, 90-100.
  51. Myung, I. J., Forster, M. R., & Browne, M. W. (2000). Model selection [Special issue]. Journal of Mathematical Psychology, 44(1-2).
  52. Myung, I. J., & Pitt, M. A. (1997). Applying Occam's razor in modeling cognition: A Bayesian approach. Psychonomic Bulletin & Review, 4, 79-95.
  53. Nickerson, R. S. (2000). Null hypothesis statistical testing: A review of an old and continuing controversy. Psychological Methods, 5, 241-301.
  54. Ntzoufras, I. (2009). Bayesian modeling using WinBUGS. Hoboken, NJ: Wiley.
  55. O'Hagan, A. (1995). Fractional Bayes factors for model comparison. Journal of the Royal Statistical Society B, 57, 99-138.
  56. O'Hagan, A., & Forster, J. (2004). Kendall's advanced theory of statistics (2nd ed.). Bayesian inference (Vol. 2B). London: Arnold.
  57. Ratcliff, R., & McKoon, G. (1997). A counter model for implicit priming in perceptual word identification. Psychological Review, 104, 319-343.
  58. Rouder, J. N., & Lu, J. (2005). An introduction to Bayesian hierarchical models with an application in the theory of signal detection. Psychonomic Bulletin & Review, 12, 573-604.
  59. Rouder, J. N., Lu, J., Morey, R. D., Sun, D., & Speckman, P. L. (2008). A hierarchical process dissociation model. Journal of Experimental Psychology: General, 137, 370-389.
  60. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t-tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225-237.
  61. Sisson, S. A. (2005). Transdimensional Markov chains: A decade of progress and future perspectives. Journal of the American Statistical Association, 100, 1077-1089.
  62. Smith, A. F. M., & Spiegelhalter, D. J. (1980). Bayes factors and choice criteria for linear models. Journal of the Royal Statistical Society B, 42, 213-220.
  63. Stone, C. J., Hansen, M. H., Kooperberg, C., & Truong, Y. K. (1997). Polynomial splines and their tensor products in extended linear modeling (with discussion). The Annals of Statistics, 25, 1371-1470.
  64. Vandekerckhove, J., Tuerlinckx, F., & Lee, M. D. (2008). A Bayesian approach to diffusion process models of decision-making. In V. Sloutsky, B. Love, & K. McRae (Eds.), Proceedings of the 30th annual conference of the cognitive science society (pp. 1429-1434). Cognitive Science Society.
  65. Verdinelli, I., & Wasserman, L. (1995). Computing Bayes factors using a generalization of the Savage-Dickey density ratio. Journal of the American Statistical Association, 90, 614-618.
  66. Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review, 14, 779-804.
  67. Wagenmakers, E.-J. (2009). Methodological and empirical developments for the Ratcliff diffusion model of response times and accuracy. European Journal of Cognitive Psychology, 21, 641-671.
  68. Wagenmakers, E.-J., Lee, M. D., Lodewyckx, T., & Iverson, G. (2008). Bayesian versus frequentist inference. In H. Hoijtink, I. Klugkist, & P. A. Boelen (Eds.), Bayesian evaluation of informative hypotheses (pp. 181-207). New York: Springer Verlag.
  69. Wagenmakers, E.-J., Ratcliff, R., Gomez, P., & Iverson, G. J. (2004). Assessing model mimicry using the parametric bootstrap. Journal of Mathematical Psychology, 48, 28-50.
  70. Wagenmakers, E.-J., & Waldorp, L. (2006). Model selection: Theoretical developments and applications [Special issue]. Journal of Mathematical Psychology, 50(2).
  71. Wagenmakers, E.-J., Zeelenberg, R., & Raaijmakers, J. G. W. (2003). Testing the counter model for perceptual identification: Effects of repetition priming and word frequency. Psychonomic Bulletin & Review, 7, 662-667.
  72. Wetzels, R., Vandekerckhove, J., Tuerlinckx, F., & Wagenmakers, E.-J. (in press). Bayesian parameter estimation in the Expectancy Valence model of the Iowa gambling task. Journal of Mathematical Psychology.
  73. Zeelenberg, R., Wagenmakers, E.-J., & Raaijmakers, J. G. W. (2002). Priming in implicit memory tasks: Prior study causes enhanced discriminability, not only bias. Journal of Experimental Psychology: General, 131, 38-47.

Cited by

Sorted by citations

Authors
Wagenmakers, E.-J., Lodewyckx, T., Kuriyal, H., & Grasman, R.
Affiliations
  • University of Amsterdam (NL)
  • KU Leuven (BE)
  • Indian Institute of Technology Kharagpur (IN)
Corresponding author
Wagenmakers, Eric-Jan
ORCID
Venue
Cognitive Psychology
Publisher
Elsevier BV
Published
2010-05
Volume
60
Issue
3
Pages
158-189
ISSN
0010-0285 (ISSN-L)
DOI
10.1016/j.cogpsych.2009.12.001
OpenAlex
W2058189433
Cited by
740
References
8
Citations by year
2026: 69 · 2025: 70 · 2024: 66 · 2023: 58 · 2022: 58 · 2021: 72 · 2020: 59 · 2019: 51 · 2018: 59 · 2017: 55 · 2016: 39 · 2015: 25 · 2014: 21 · 2013: 9 · 2012: 12
FWCI
9.61
Type
journal article
Language
en
Open access
Licence
www.elsevier.com/tdm/userlicense/1.0/ (tdm)
Locations
  • Cognitive Psychology (publishedVersion)
  • PubMed (publishedVersion)
  • UvA-DARE (University of Amsterdam) (publishedVersion)
  • UvA-DARE (University of Amsterdam) (submittedVersion)
  • Lirias (KU Leuven) (submittedVersion, other-oa)
  • Data Archiving and Networked Services (DANS) (submittedVersion)
Funders
  • Universiteit van Amsterdam
  • Nederlandse Organisatie voor Wetenschappelijk Onderzoek
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
  • Physical Sciences › Computer Science › Artificial Intelligence › Bayesian Modeling and Causal Inference
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Distribution Estimation and Applications
Keywords
Savage-Dickey density ratio, Bayesian hypothesis testing, nested models, parameter priors, statistical reporting
MeSH terms
Bayes Theorem, Data Interpretation, Statistical, Humans, Models, Statistical, Likelihood Functions, Cognitive Science, Behavioral Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review