The DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index; 1 of the 8 sources its claims cite were checked for support.
The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant
Abstract
It is common to summarize statistical comparisons by declarations of statistical significance or nonsignificance. Here we discuss one problem with such declarations, namely that changes in statistical significance are often not themselves statistically significant. By this, we are not merely making the commonplace observation that any particular threshold is arbitrary—for example, only a small change is required to move an estimate from a 5.1% significance level to 4.9%, thus moving it into statistical significance. Rather, we are pointing out that even large changes in significance levels can correspond to small, nonsignificant changes in the underlying quantities.The error we describe is conceptually different from other oft-cited problems—that statistical significance is not the same as practical importance, that dichotomization into significant and nonsignificant results encourages the dismissal of observed differences in favor of the usually less interesting null hypothesis of no difference, and that any particular threshold for declaring significance is arbitrary. We are troubled by all of these concerns and do not intend to minimize their importance. Rather, our goal is to bring attention to this additional error of interpretation. We illustrate with a theoretical example and two applied examples. The ubiquity of this statistical error leads us to suggest that students and practitioners be made more aware that the difference between “significant” and “not significant” is not itself statistically significant.
Overview
CoChat Summary
TL;DR
The paper argues that comparing results by whether they are statistically significant versus not significant is misleading, because the difference between a significant and a non-significant estimate is often not itself statistically significant. It illustrates this with a theoretical example and two applied examples, including a multilevel alternative to separate significance judgments.
Key findings
- In a theoretical example, two independent studies estimate effects of 25 ± 10 and 10 ± 10; the first is significant at the 1% level and the second is not, but their estimated difference is 15 with standard error √(10² + 10²) = 14, so the difference is not statistically significant.
- In Blanchard and Bogaert’s (1996) data on 302 homosexual men and 302 matched heterosexual men, number of older brothers has β = 0.29, SE = 0.11, p = 0.007, while number of older sisters has β = 0.08, SE = 0.10, p = 0.43; the authors argue the difference between these coefficients likely is not statistically significant, so the conclusion that “only” older brothers matter is better described as suggestive.
- In Blackman et al.’s (1988) chick-brain electromagnetic-field experiments across frequencies (1 Hz, 15 Hz, 30 Hz, …, 510 Hz), the original analysis separated effects by statistical significance; a multilevel analysis partially pools estimates toward a common average near 0.1, and features such as the negative estimate at 480 Hz and the significant pair at 405 Hz may be sampling variability rather than real underlying effects.
Methods
- Conceptual statistical commentary using one theoretical example and two applied examples; no new primary data are collected.
- The theoretical example compares two independent studies and a third larger study, focusing on the statistical significance of the difference between estimates rather than the difference between their significance levels.
- The applied examples reanalyze or redisplay published results: Blanchard and Bogaert’s logistic regression on sibling characteristics and sexual orientation, and Blackman et al.’s calcium-efflux experiments; for the latter, the authors fit a simple multilevel model, y_j ∼ N(θ_j, σ_j²), θ_j ∼ N(μ, τ²), with partial pooling toward the average.
Limitations
- Stated: the article “does not attempt to provide a comprehensive discussion of significance testing”; the simple hierarchical model is “not intended to be definitive”; and the authors lack scientific expertise to justify a smooth-function model for the frequency data.
- Stated: for the Blanchard and Bogaert example, the authors “cannot be certain” because the regression coefficients are not independent, and they only “strongly suspect” that the difference between older-brothers and older-sisters coefficients is not significant.
- Evident: the applied examples rely on published summaries and figures rather than formal tests of all compared differences, and the multilevel model assumes exchangeability across frequencies, which the authors note some might object to.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 0 Not valid · 7 Not checked · 10 Own results and interpretations
In the paper
“However, no evidence is presented that would indicate that this difference is statistically significant.”
What the sources say
- Not matchedcites Blanchard and Bogaert (1996)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
hedgedn = 302- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“But the data are also consistent with the hypothesis that only the birth order (the total number of older siblings) matters.”
What the sources say
- Not matchedcites Blanchard and Bogaert (1996)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“The graph and table illustrate that, in these data, homosexuality is more strongly associated with number of older brothers than with number of older sisters.”
What the sources say
- Not matchedcites Blanchard and Bogaert (1996)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plainstates an associationn = 302- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
In the paper
“After all, the data are consistent with the hypothesis that only the number of older brothers matters.”
What the sources say
- Not matchedcites Blanchard and Bogaert (1996)
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Rather, we are pointing out that even large changes in significance levels can correspond to small, non-significant changes in the underlying variables.”
- Also stated in: § 1 Introduction · p. 1 · plain
“Rather, we are pointing out that even large changes in significance levels can correspond to small, non-significant changes in the underlying variables.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § 1 Introduction · p. 1 · plain
4 more claims
Create a free account to see all 18 claims and every source verdict.
- It may state two separate claims as one. 5
- Probably not a substantial claim. 4
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Third, it is inappropriate to compare the nominal significance of two independent statistical tests as a means of drawing a conclusion about differential effects (30).”
From Issues with data and analyses: Errors, underlying themes, and potential solutions (2018) · cites this paper as [30]
The cited source says what the claim attributes to it.
In the citing paper
“That said, although the significance levels of these contrasts differed for these two studies, the no log condition always fell between the two logged streak conditions for all measures (as predicted), and there were no differences in the estimated effect sizes of these contrasts (cf. Gelman and Stern 2006).”
From On or Off Track: How (Broken) Streaks Affect Consumer Decisions (2023) · cites this paper as Gelman and Stern 2006
Citations
Cites
Gelman, A., Carlin, J. B., Stern, H. S., & Rubin, D. B. (2003) · doi:10.1201/9780429258480 · Cited by 27,276
Self-citationBayesian Data Analysis, 2nd edn
Brix, A. (2005) · Journal of the Royal Statistical Society Series A: Statistics in Society · doi:10.1111/j.1467-985x.2004.00347_4.x · Cited by 1,767
Nedelman, J. R. (2005) · Computational Statistics · doi:10.1007/bf02741321 · Cited by 1,078
Homosexuality in men and number of older brothers
(1996) · American Journal of Psychiatry · doi:10.1176/ajp.153.1.27 · Cited by 284
Biological versus nonbiological older brothers and men’s sexual orientation
Bogaert, A. F. (2006) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.0511152103 · Cited by 238
The Null Hypothesis Testing Controversy in Psychology
Krantz, D. H. (1999) · Journal of the American Statistical Association · doi:10.1080/01621459.1999.10473888 · Cited by 158
Blackman, C. F., Benane, S. G., Elliott, D. J., House, D. E., & Pollock, M. M. (1988) · Bioelectromagnetics · doi:10.1002/bem.2250090303 · Cited by 152
Replication and Meta-Analysis in Parapsychology
Utts, J. M. (1991) · Statistical Science · doi:10.1214/ss/1177011577 · Cited by 136
The Null Hypothesis Testing Controversy in Psychology
Krantz, D. H. (1999) · Journal of the American Statistical Association · doi:10.2307/2669949 · Cited by 34
[Replication and Meta-Analysis in Parapsychology]: Rejoinder
Utts, J. M. (1991) · Statistical Science · doi:10.1214/ss/1177011585 · Cited by 3
Research works printed but not linked: 1
- Bower B., 2006, Science News, 170, 1
Cited by
The ASA Statement on p -Values: Context, Process, and Purpose
Wasserstein, R. L., & Lazar, N. A. (2016) · The American Statistician · doi:10.1080/00031305.2016.1154108 · Cited by 6,306
An index and test of linear moderated mediation
Hayes, A. F. (2015) · Multivariate Behavioral Research · doi:10.1080/00273171.2014.962683 · Cited by 3,979
Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations
Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016) · European Journal of Epidemiology · doi:10.1007/s10654-016-0149-3 · Cited by 3,200
“…and Boen 1972; Berkson 1942; Bland and Altman 2015; Chia 1997; Cohen 1994; Evans et al. 1988; Fidler and Loftus 2009; Gardner and Altman 1986; Gelman 2013; Gelman and Loken 2014; Gelman and Stern 2006; Gigerenzer 2004; Gigerenzer and Marewski 2015; Goodman 1992, 1993, 1999, 2008; Greenland 2011,”
BackgroundTesting measurement invariance of composites using partial least squares
Henseler, J., Ringle, C. M., & Sarstedt, M. (2016) · International Marketing Review · doi:10.1108/imr-09-2014-0304 · Cited by 2,964
“At this stage, the significance of the indicator weights is irrelevant, because differences in their significance (an indicator may be significant in one group, but not in another) do not imply that the coefficients differ significantly (Gelman and Stern, 2006).”
BackgroundMoving to a World Beyond “ p < 0.05”
Wasserstein, R. L., Schirm, A. L., & Lazar, N. A. (2019) · The American Statistician · doi:10.1080/00031305.2019.1583913 · Cited by 2,976
“As Gelman and Stern (2006) famously observed, the difference between “significant” and “not significant” is not itself statistically significant.”
Metadata
- Authors
- Gelman, A., & Stern, H.
- Affiliations
- University of California, Irvine (US)
- Venue
- The American Statistician
- Publisher
- Informa UK Limited
- Published
- 2006-11
- Volume
- 60
- Issue
- 4
- Pages
- 328-331
- ISSN
- 0003-1305 (ISSN-L)
- 1537-2731
- OpenAlex
- W2130663341
- Cited by
- 1,407
- References
- 9
- Citations by year
- 2026: 264 · 2025: 129 · 2024: 82 · 2023: 101 · 2022: 63 · 2021: 98 · 2020: 105 · 2019: 118 · 2018: 99 · 2017: 75 · 2016: 62 · 2015: 50 · 2014: 41 · 2013: 27 · 2012: 25
- FWCI
- 9.21
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- The American Statistician (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Keywords
- statistical significance, significance threshold, statistical interpretation, dichotomization, null hypothesis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-18, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-18 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant. https://app.cochat.ai/papers/10.1198/000313006x152649
- The paper
- Gelman, A., & Stern, H. (2006). The Difference Between “Significant” and “Not Significant” is not Itself Statistically Significant. The American Statistician. https://doi.org/10.1198/000313006x152649
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.