CoChat

The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index; 1 of the 8 sources its claims cite were checked for support.

The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant

Gelman, Andrew, Stern, Hal · 2006 · The American Statistician

Abstract

It is common to summarize statistical comparisons by declarations of statistical significance or nonsignificance. Here we discuss one problem with such declarations, namely that changes in statistical significance are often not themselves statistically significant. By this, we are not merely making the commonplace observation that any particular threshold is arbitrary—for example, only a small change is required to move an estimate from a 5.1% significance level to 4.9%, thus moving it into statistical significance. Rather, we are pointing out that even large changes in significance levels can correspond to small, nonsignificant changes in the underlying quantities.The error we describe is conceptually different from other oft-cited problems—that statistical significance is not the same as practical importance, that dichotomization into significant and nonsignificant results encourages the dismissal of observed differences in favor of the usually less interesting null hypothesis of no difference, and that any particular threshold for declaring significance is arbitrary. We are troubled by all of these concerns and do not intend to minimize their importance. Rather, our goal is to bring attention to this additional error of interpretation. We illustrate with a theoretical example and two applied examples. The ubiquity of this statistical error leads us to suggest that students and practitioners be made more aware that the difference between “significant” and “not significant” is not itself statistically significant.

Overview

CoChat Summary

TL;DR

The paper argues that comparing results by whether they are statistically significant versus not significant is misleading, because the difference between a significant and a non-significant estimate is often not itself statistically significant. It illustrates this with a theoretical example and two applied examples, including a multilevel alternative to separate significance judgments.

Key findings

  • In a theoretical example, two independent studies estimate effects of 25 ± 10 and 10 ± 10; the first is significant at the 1% level and the second is not, but their estimated difference is 15 with standard error √(10² + 10²) = 14, so the difference is not statistically significant.
  • In Blanchard and Bogaert’s (1996) data on 302 homosexual men and 302 matched heterosexual men, number of older brothers has β = 0.29, SE = 0.11, p = 0.007, while number of older sisters has β = 0.08, SE = 0.10, p = 0.43; the authors argue the difference between these coefficients likely is not statistically significant, so the conclusion that “only” older brothers matter is better described as suggestive.
  • In Blackman et al.’s (1988) chick-brain electromagnetic-field experiments across frequencies (1 Hz, 15 Hz, 30 Hz, …, 510 Hz), the original analysis separated effects by statistical significance; a multilevel analysis partially pools estimates toward a common average near 0.1, and features such as the negative estimate at 480 Hz and the significant pair at 405 Hz may be sampling variability rather than real underlying effects.

Methods

  • Conceptual statistical commentary using one theoretical example and two applied examples; no new primary data are collected.
  • The theoretical example compares two independent studies and a third larger study, focusing on the statistical significance of the difference between estimates rather than the difference between their significance levels.
  • The applied examples reanalyze or redisplay published results: Blanchard and Bogaert’s logistic regression on sibling characteristics and sexual orientation, and Blackman et al.’s calcium-efflux experiments; for the latter, the authors fit a simple multilevel model, y_j ∼ N(θ_j, σ_j²), θ_j ∼ N(μ, τ²), with partial pooling toward the average.

Limitations

  • Stated: the article “does not attempt to provide a comprehensive discussion of significance testing”; the simple hierarchical model is “not intended to be definitive”; and the authors lack scientific expertise to justify a smooth-function model for the frequency data.
  • Stated: for the Blanchard and Bogaert example, the authors “cannot be certain” because the regression coefficients are not independent, and they only “strongly suspect” that the difference between older-brothers and older-sisters coefficients is not significant.
  • Evident: the applied examples rely on published summaries and figures rather than formal tests of all compared differences, and the multilevel model assumes exchangeability across frequencies, which the authors note some might object to.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1198/000313006x152649

Cites

11 of 11 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

1 Valid · 0 Not valid · 7 Not checked · 10 Own results and interpretations

4 more claims

Create a free account to see all 18 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 5
  • Probably not a substantial claim. 4

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 11 references

Research works printed but not linked: 1

  1. Bower B., 2006, Science News, 170, 1

Cited by

See all 1,407

Metadata

Authors
Gelman, A., & Stern, H.
Affiliations
University of California, Irvine (US)
Venue
The American Statistician
Published
2006-11
Volume
60
Issue
4
Pages
328-331
ISSN
  • 0003-1305 (ISSN-L)
  • 1537-2731
OpenAlex
W2130663341
Cited by
1,407
References
9
Citations by year
2026: 264 · 2025: 129 · 2024: 82 · 2023: 101 · 2022: 63 · 2021: 98 · 2020: 105 · 2019: 118 · 2018: 99 · 2017: 75 · 2016: 62 · 2015: 50 · 2014: 41 · 2013: 27 · 2012: 25
FWCI
9.21
Type
journal article
Language
en
Open access
closed
Locations
The American Statistician (publishedVersion)
Topics
  • Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
  • Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
Keywords
statistical significance, significance threshold, statistical interpretation, dichotomization, null hypothesis

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-18, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-18 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant. https://app.cochat.ai/papers/10.1198/000313006x152649
The paper
Gelman, A., & Stern, H. (2006). The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant. The American Statistician. https://doi.org/10.1198/000313006x152649

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.