CoChat
Sign Up
Cited by 126Statistical Methods and Bayesian InferenceProbability and Statistical ResearchStatistical Methods in Clinical Trials

The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.

Rejection odds and rejection ratios: A proposal for statistical practice in testing hypotheses

Open access
Bayarri, M.J., Benjamin, Daniel J., Berger, James O., Sellke, Thomas M.·2016·Journal of Mathematical Psychology
DOI 10.1016/j.jmp.2015.12.007PMC PMC6358203
Open this paper in CoChat — free account

Abstract

Much of science is (rightly or wrongly) driven by hypothesis testing. Even in situations where the hypothesis testing paradigm is correct, the common practice of basing inferences solely on p-values has been under intense criticism for over 50 years. We propose, as an alternative, the use of the odds of a correct rejection of the null hypothesis to incorrect rejection. Both pre-experimental versions (involving the power and Type I error) and post-experimental versions (depending on the actual data) are considered. Implementations are provided that range from depending only on the p-value to consideration of full Bayesian analysis. A surprise is that all implementations–even the full Bayesian analysis–have complete frequentist justification. Versions of our proposal can be implemented that require only minor modifications to existing practices yet overcome some of their most severe shortcomings.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jmp.2015.12.007

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • For Clinton's supporters, the least-squares estimate of β is 0.15 (SE 0.16), so β=0 but not β=0.5 is within the usual margin of error; for Trump's supporters, the estimated slope is 0.44 (SE 0.09), close to 0.5, with p≈10^{-6} for testing β=0 and Bayes factor ≈26,628, giving strong odds to reject the null of no selective non-response bias.

    “Lo and behold, for Clinton's supporters, the least-squares estimator for β is 0.15 with (estimated) standard error 0.16, so β = 0-but not β = 0 . 5-is well within the usual margin of error. In contrast, the estimated slope for Trump's supporters is 0.44, quite close to the theoretical value 0 . 5 in the presence of a biased R -mechanism. Its estimated standard error is 0.09, and hence the usual z”

    From Statistical paradises and paradoxes in big data (I): Law of large populations, big data paradox, and the 2016 US presidential election (2018) · cites this paper as Bayarri et al. (2016)

    Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Flagged first, then by citations

Research works printed but not linked: 34

  1. Anscombe, 1954, Fixed-sample-size analysis of sequential observations, Biometrics, 10, 89, 10.2307/3001665 · DOI
  2. Benjamin, 2010, Social identity and preferences, American Economic Review, 100, 1913, 10.1257/aer.100.4.1913 · DOI
  3. Berger, 2003, Could Fisher, Jeffreys and Neyman have agreed on testing (with Discussion)?, Statistical Science, 18, 1, 10.1214/ss/1056397485 · DOI
  4. Berger, 1997, Unified frequentist and Bayesian testing of a precise hypothesis (with discussion), Statistical Science, 12, 133, 10.1214/ss/1030037904 · DOI
  5. Berger, 1994, A unified conditional frequentist and Bayesian test for fixed and sequential hypothesis testing, The Annals of Statistics, 22, 1787, 10.1214/aos/1176325757 · DOI
  6. Berger, 1999, Default Bayes factors for non-nested hypothesis testing, Journal of the American Statistical Association, 94, 542, 10.1080/01621459.1999.10474149 · DOI
  7. Berger, 2001, Objective Bayesian methods for model selection: introduction and comparison (with Discussion), vol. 38, 135
  8. Coffman, 2015, Pre-analysis plans have limited upside especially where replications are feasible, Journal of Economic Perspectives, 29, 81, 10.1257/jep.29.3.81 · DOI · Paper page
  9. Dass, 2003, Unified Bayesian and conditional frequentist testing of composite hypotheses, Scandinavian Journal of Statistics, 30, 193, 10.1111/1467-9469.00326 · DOI
  10. Edwards, 1963, Bayesian statistical inference for psychological research, Psychological Review, 70, 193, 10.1037/h0044139 · DOI · Paper page
  11. Fischhoff, 1975, Hindsight ≠ foresight. The effect of outcome knowledge on judgment under uncertainty, Journal of Experimental Psychology: Human Perception and Performance, 1, 288
  12. Fischhoff, 1983, Hypothesis evaluation from a Bayesian perspective, Psychological Review, 90, 239, 10.1037/0033-295X.90.3.239 · DOI
  13. Garner, 2007, Upward bias in odds ratio estimates from genome-wide association studies, Genetic Epidemiology, 31, 288, 10.1002/gepi.20209 · DOI · Paper page
  14. Gelman, 2014, Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors, Perspectives on Psychological Science, 9, 641, 10.1177/1745691614551642 · DOI · Paper page
  15. Gilbert, 2011, Statistical interpretation of the RV144 HIV vaccine efficacy trial in Thailand: A case study for statistical issues in efficacy trials, The Journal of Infectious Diseases, 203, 969, 10.1093/infdis/jiq152 · DOI
  16. Ioannidis, 2008, Effect of formal statistical significance on the credibility of observational associations, American Journal of Epidemiology, 168, 374, 10.1093/aje/kwn156 · DOI
  17. John, 2012, Measuring the prevalence of questionable research practices with incentives for truth-telling, Psychological Science, 23, 524, 10.1177/0956797611430953 · DOI · Paper page
  18. Johnson, 2013, Revised standards for statistical evidence, Proceedings of the National Academy of Sciences, 110, 19313, 10.1073/pnas.1313476110 · DOI · Paper page
  19. Kruschke, 2011, Bayesian assessment of null values via parameter estimation and model comparison, Perspectives on Psychological Science, 6, 299, 10.1177/1745691611406925 · DOI
  20. Lucas, 2000, Semantic priming without association: A meta-analytic review, Psychonomic Bulletin and Review, 6, 618, 10.3758/BF03212999 · DOI
  21. Lucke, 2009, A critique of the false-positive report probability, Genetic Epidemiology, 33, 145, 10.1002/gepi.20363 · DOI
  22. Neyman, 1977, Frequentist probability and frequentist statistics, Synthese, 36, 97, 10.1007/BF00485695 · DOI
  23. Olken, 2015, Promises and perils of pre-analysis plans, The Journal of Economic Perspectives, 29, 61, 10.1257/jep.29.3.61 · DOI
  24. Rosenthal, 1979, The “file drawer problem” and tolerance for null results, Psychological Bulletin, 86, 641, 10.1037/0033-2909.86.3.638 · DOI · Paper page
  25. Schkade, 1998, Does living in california make people happy? A focusing illusion in judgments of life satisfaction, Psychological Science, 9, 340, 10.1111/1467-9280.00066 · DOI
  26. Sellke, 2001, Calibration of p values for testing precise null hypotheses, The American Statistician, 55, 62, 10.1198/000313001300339950 · DOI
  27. Vankov, 2014, On the persistence of low power in psychological science, The Quarterly Journal of Experimental Psychology, 67, 1037, 10.1080/17470218.2014.885986 · DOI · Paper page
  28. Vovk, 1993, A logic of probability, with application to the foundations of statistics, Journal of the Royal Statistical Society: Series B, 55, 317, 10.1111/j.2517-6161.1993.tb01904.x · DOI
  29. Wacholder, 2004, Assessing the probability that a positive report is false: An approach for molecular epidemiology studies, Journal of the National Cancer Institute, 96, 434, 10.1093/jnci/djh075 · DOI
  30. Wagenmakers, E.-J., Verhagen, A.J., Ly, A., Matzke, D., Steingroever, H., Rouder, J.N., Lilienfeld, S.O., & Waldman, I. et al. (in press). The need for Bayesian hypothesis testing in psychological science. In Psychological science under scrutiny: Recent challenges and proposed solutions. John Wiley and Sons.
  31. Wagenmakers, 2011, Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011), Journal of Personality and Social Psychology, 100, 426, 10.1037/a0022790 · DOI · Paper page
  32. 2007, Genome-wide association study of 14,000 cases of seven common diseases and 3000 shared controls, Nature, 447, 661, 10.1038/nature05911 · DOI · Paper page
  33. Wood, 2014, Defining the role of common variation in the genomic and biological architecture of adult human height, Nature Genetics, 46, 1173, 10.1038/ng.3097 · DOI · Paper page
  34. Zhang, L., & Ortmann, A. (2013). Exploring the Meaning of Significance in Experimental Economics. Australian School of Business Research Paper 2013–32. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2356018.

Cited by

Sorted by citations

Metadata

Authors
Bayarri, M. J., Benjamin, D. J., Berger, J. O., & Sellke, T. M.
Affiliations
  • Universitat de València (ES)
  • University of Southern California (US)
  • Duke University (US)
  • Purdue University West Lafayette (US)
Corresponding author
Berger, James O.
ORCID
Venue
Journal of Mathematical Psychology
Publisher
Elsevier BV
Published
2016-06
Volume
72
Pages
90-103
ISSN
0022-2496 (ISSN-L)
DOI
10.1016/j.jmp.2015.12.007
PMC
PMC6358203
OpenAlex
W3124008874
Cited by
126
References
14
Citations by year
2026: 2 · 2025: 4 · 2024: 1 · 2023: 5 · 2022: 11 · 2021: 10 · 2020: 17 · 2019: 13 · 2018: 37 · 2017: 16 · 2016: 10
FWCI
26.34
Type
journal article
Language
en
Update notices
erratum (2019-04-01)
Open access
Licence
Locations
  • Journal of Mathematical Psychology (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (acceptedVersion, cc-by)
  • PubMed Central (acceptedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
  • National Science Foundation (DMS-1007773, DMS-1407775, BCS-1521855)
  • National Institutes of Health / National Institute on Aging (R01-AG042568)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
  • Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Keywords
rejection ratio, hypothesis testing, p-values, statistical evidence, power, type I error, Bayesian analysis, null hypothesis testing, statistical inference

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review