CoChat

A Gentle Introduction to the Comparison Between Null Hypothesis Testing and Bayesian Analysis: Reanalysis of Two Randomized Controlled Trials

Open access
Bendtsen, Marcus · 2018 · Journal of Medical Internet Research

Published version of a preprint

The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index; none of the 17 sources its claims cite is checked for support yet.

Abstract

The debate on the use and misuse of P values has risen and fallen throughout their almost century-long existence in scientific discovery. Over the past few years, the debate has again received front-page attention, particularly through the public reminder by the American Statistical Association on how P values should be used and interpreted. At the core of the issue lies a fault in the way that scientific evidence is dichotomized and research is subsequently reported, and this fault is exacerbated by researchers giving license to statistical models to do scientific inference. This paper highlights a different approach to handling the evidence collected during a randomized controlled trial, one that does not dichotomize, but rather reports the evidence collected. Through the use of a coin flipping experiment and reanalysis of real-world data, the traditional approach of testing null hypothesis significance is contrasted with a Bayesian approach. This paper is meant to be understood by those who rely on statistical models to draw conclusions from data, but are not statisticians and may therefore not be able to grasp the debate that is primarily led by statisticians.

Overview

CoChat Summary

## TL;DR

This paper is a gentle, nontechnical introduction contrasting null hypothesis significance testing (NHST) with Bayesian analysis, using a coin-flipping example and reanalysis of two randomized controlled trials. The Bayesian approach is presented as a way to report posterior probabilities and avoid dichotomizing results into “significant” and “not significant.”

## Key findings

  • In the coin-flip example, 540 heads out of 1000 flips gave a maximum likelihood estimate of q = 0.54; the NHST approach produced a P value of .0114 and 95% confidence limits of 0.509 and 0.571.
  • For the NEXit trial, 1590 participants were allocated and 1502 responded to follow-up on the primary outcomes; the NHST analysis rejected the null hypothesis (P values < .05), while the Bayesian analysis found the intervention had a positive effect with posterior probabilities of more than 99% for both prolonged abstinence and point prevalence.
  • For the NEXit Bayesian results, the paper reports that the odds ratio was greater than 1.5 with probability more than 99%, greater than 2.0 for prolonged abstinence, and greater than 2.5 with probability 7.05%.
  • For the AMADEUS trial, 896 participants were randomized and 816 primary outcome responses were collected; the NHST analysis did not reject the null hypothesis, while the Bayesian analysis found 3.3% probability of a major preference for the novel intervention (IRR < 0.9), 55.4% probability of a minor preference for the novel intervention, 39.6% probability of a minor preference for referral to the questionnaire, and 1.8% probability that referral should be majorly preferred.

## Methods

  • Conceptual/tutorial article contrasting NHST and Bayesian reasoning, first with a coin-flipping experiment and then with reanalyses of 2 randomized controlled trials: NEXit (smoking cessation) and AMADEUS (alcohol reduction).
  • For NEXit, the paper treats binary outcomes with Bernoulli/logistic regression and reports reanalysis using flat priors and posterior distributions over the odds ratio.
  • For AMADEUS, the paper treats the outcome as a count variable and uses negative binomial regression; it reanalyzes the trial with flat priors and posterior distributions over the incidence rate ratio (IRR).

## Limitations

  • The paper states that it uses flat priors for the trial reanalyses “so that we can defer any discussion about nonflat priors,” which it explicitly notes goes against its general recommendation.
  • The paper says it shall not delve into the details of the interventions for NEXit and AMADEUS and focuses on the statistical comparison instead.
  • Evident from the text: the paper is primarily a didactic reanalysis, so its goal is explanation and illustration rather than a new substantive evaluation of the interventions.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.2196/10873

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Figure 1 (a) A normal distribution with a mean value of 165 cm and standard deviation of 4 cm. (b) A beta distribution with shape parameters 8 and 3. (c) A Bernoulli distribution with q=0.6. (d) A negative binomial distribution with parameters 8 and 0.5 (failures and success probability).
Figure 1 (a) A normal distribution with a mean value of 165 cm and standard deviation of 4 cm. (b) A beta distribution with shape parameters 8 and 3. (c) A Bernoulli distribution with q=0.6. (d) A negative binomial distribution with parameters 8 and 0.5 (failures and success probability).CC BY · PubMed Central
Figure 2 Sampling distribution of q under the null hypothesis that the population value is 0.5. The X marks the maximum likelihood estimate of the coin flip experiment (540 heads over 1000 flips).
Figure 2 Sampling distribution of q under the null hypothesis that the population value is 0.5. The X marks the maximum likelihood estimate of the coin flip experiment (540 heads over 1000 flips).CC BY · PubMed Central
Figure 3 (a) Uniform prior distribution (flat prior). (b) A prior distribution that encodes that fair coins are more likely. (c) A prior distribution that encodes that biased coins are more likely.
Figure 3 (a) Uniform prior distribution (flat prior). (b) A prior distribution that encodes that fair coins are more likely. (c) A prior distribution that encodes that biased coins are more likely.CC BY · PubMed Central
Figure 4 Data likelihood for different values of q in the coin flipping experiment.
Figure 4 Data likelihood for different values of q in the coin flipping experiment.CC BY · PubMed Central
Figure 5 Three examples of the use of Bayes’ theorem for the coin flipping experiment. Top row: posterior distribution when using a uniform prior distribution (flat prior). Middle row: posterior distribution when using a weak prior distribution that makes fair coins more likely. Bottom row: posterior distribution when using a strong prior distribution that makes biased coins more likely.
Figure 5 Three examples of the use of Bayes’ theorem for the coin flipping experiment. Top row: posterior distribution when using a uniform prior distribution (flat prior). Middle row: posterior distribution when using a weak prior distribution that makes fair coins more likely. Bottom row: posterior distribution when using a strong prior distribution that makes biased coins more likely.CC BY · PubMed Central
Figure 6 (a) Posterior distribution of q after observing 540 heads given 1000 flips (0≤q ≤1). (b) Posterior distribution of q after observing 540 heads given 1000 flips (0.45≤q ≤0.6).
Figure 6 (a) Posterior distribution of q after observing 540 heads given 1000 flips (0≤q ≤1). (b) Posterior distribution of q after observing 540 heads given 1000 flips (0.45≤q ≤0.6).CC BY · PubMed Central
Figure 7 (a) NEXit trial prolonged abstinence: an approximation of the density of the posterior distribution of the odds ratio comparing intervention versus control. (b) NEXit trial point prevalence: an approximation of the density of the posterior distribution of the odds ratio comparing intervention versus control.
Figure 7 (a) NEXit trial prolonged abstinence: an approximation of the density of the posterior distribution of the odds ratio comparing intervention versus control. (b) NEXit trial point prevalence: an approximation of the density of the posterior distribution of the odds ratio comparing intervention versus control.CC BY · PubMed Central
Figure 8 Approximation of the density of the posterior distribution of the incident rate ratio, comparing intervention versus control in the AMADEUS trial.
Figure 8 Approximation of the density of the posterior distribution of the incident rate ratio, comparing intervention versus control in the AMADEUS trial.CC BY · PubMed Central

Verification

Claims the paper makes

0 Valid · 2 Not valid · 14 Not checked · 40 Own results and interpretations

16 more claims

Create a free account to see all 56 claims and to verify the 17 references still open.

See all claims — free account
  • Probably not a substantial claim. 27
  • It may state two separate claims as one. 19

Create a free account to see these claims.

Citations

Cites

Show all 22 references

Research works printed but not linked: 2

  1. McShaneBBGalDGelmanARobertCTackettJLAbandon statistical significance. arXiv:1709.0758820170921https://arxiv.org/pdf/1709.07588v1.pdf
  2. CarvalhoCMPolsonNGScottJGHandling sparsity via the horseshoe200912th International Conference on Artificial Intelligence and Statistics (AISTATS)Apr 16-18, 2009Clearwater Beach, FL, USA

Cited by

See all 105

Metadata

Authors
Bendtsen, M.
Affiliations
Linköping University (SE)
Corresponding author
Bendtsen, Marcus
Venue
Journal of Medical Internet Research
Published
2018-10-24
Volume
20
Issue
10
Pages
e10873
ISSN
1438-8871 (ISSN-L)
OpenAlex
W2889111765
Cited by
105
References
22
Citations by year
2026: 8 · 2025: 8 · 2024: 23 · 2023: 15 · 2022: 22 · 2021: 15 · 2020: 9 · 2019: 5
FWCI
6.02
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • Journal of Medical Internet Research (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • KTH Publication Database DiVA (KTH Royal Institute of Technology) (submittedVersion, other-oa)
Funders
  • National Science Foundation
  • Office of Naval Research
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Keywords
null hypothesis significance testing, Bayesian analysis, p-values, statistical inference
MeSH terms
Bayes Theorem, Humans, Research Design, Randomized Controlled Trials as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.