The DOI matches its Crossref record; 33 of the 33 references it cites were found in a registry or an index.
On judging the significance of differences by examining the overlap between confidence intervals.
Abstract
To judge whether the difference between two point estimates is statistically significant, data analysts often examine the overlap between the two associated confidence intervals. We compare this technique to the standard method of testing significance under the common assumptions of consistency, asymptotic normality, and asymptotic independence of the estimates. Rejection of the null hypothesis by the method of examining overlap implies rejection by the standard method, whereas failure to reject by the method of examining overlap does not imply failure to reject by the standard method. As a consequence, the method of examining overlap is more conservative (i.e., rejects the null hypothesis less often) than the standard method when the null hypothesis is true, and it mistakenly fails to reject the null hypothesis more frequently than does the standard method when the null hypothesis is false. Although the method of examining overlap is simple and especially convenient when lists or graphs of confidence intervals have been presented, we conclude that it should not be used for formal significance testing unless the data analyst is aware of its deficiencies and unless the information needed to carry out a more appropriate procedure is unavailable.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Smithers, D. W. (1956) · The Lancet · doi:10.1016/s0140-6736(56)91184-9 · Cited by 1,791
Cancer epidemiology biomarkers & prevention
Press, H. (1991) · Cited by 1,398
Statistical Theory and Methodology in Science and Engineering
Lloyd, E. (1961) · Technometrics · doi:10.1080/00401706.1961.10489977 · Cited by 1,063
Tersmette, A. C., Petersen, G. M., Offerhaus, G. J. A., Falatko, F. C., Brune, K. A., Goggins, M., Rozenblum, E., Wilentz, R. E., Yeo, C. J., Cameron, J. L., et al. (2001) · PubMed · Cited by 309
Doses of Nicotine and Lung Carcinogens Delivered to Cigarette Smokers
Djordjevic, M. V., Stellman, S. D., & Zang, E. (2000) · JNCI: Journal of the National Cancer Institute · doi:10.1093/jnci/92.2.106 · Cited by 282
Sorsby, A. (1963) · The Lancet · doi:10.1016/s0140-6736(63)90662-7 · Cited by 262
Jaccard, J., & Guilamo-Ramos, V. (2002) · Journal of Clinical Child & Adolescent Psychology · doi:10.1207/s15374424jccp3101_15 · Cited by 248
Evaluation of the Faces Pain Scale for Use with the Elderly
Herr, K. A., Mobily, P. R., Kohout, F. J., & Wagenaar, D. (1998) · The Clinical Journal of Pain · doi:10.1097/00002508-199803000-00005 · Cited by 240
Sont, W. N., Zielinski, J. M., Ashmore, J. P., Jiang, H., Krewski, D., Fair, M. E., Band, P. R., & Létourneau, E. G. (2001) · American Journal of Epidemiology · doi:10.1093/aje/153.4.309 · Cited by 231
Slavotinek, A. M., Rosenberg, M. J., Knight, S. J. L., Gaunt, L., Fergusson, W., Killoran, C., Clayton‐Smith, J., Kingston, H. M., Campbell, R. H. A., Flint, J., et al. (1999) · Journal of Medical Genetics · doi:10.1136/jmg.36.5.405 · Cited by 134
Research works printed but not linked: 2
- Eising E. G., 1998, Journal of Nuclear Medicine, 39, 1062
- Martinez M. E., 1999, Cancer Research, 59, 5181
Cited by
Chao, A., Gotelli, N. J., Hsieh, T. C., Sander, E. L., Ma, K. H., Colwell, R. K., & Ellison, A. M. (2014) · Ecological Monographs · doi:10.1890/13-0133.1 · Cited by 4,256
Coverage‐based rarefaction and extrapolation: standardizing samples by completeness rather than size
Chao, A., & Jost, L. (2012) · Ecology · doi:10.1890/11-1952.1 · Cited by 2,213
Zhao, H., Seibert, S. E., & Lumpkin, G. T. (2010) · Journal of Management · doi:10.1177/0149206309335187 · Cited by 1,521
Increasing CO2 threatens human nutrition
Myers, S. S., Zanobetti, A., Kloog, I., Huybers, P., Leakey, A. D. B., Bloom, A. J., Carlisle, E., Dietterich, L. H., Fitzgerald, G., Hasegawa, T., et al. (2014) · Nature · doi:10.1038/nature13179 · Cited by 1,470
Inference by Eye: Confidence Intervals and How to Read Pictures of Data.
Cumming, G., & Finch, S. (2005) · American Psychologist · doi:10.1037/0003-066x.60.2.170 · Cited by 1,449
Metadata
- Authors
- Schenker, N., & Gentleman, J. F.
- Affiliations
- National Center for Health Statistics (US)
- Venue
- The American Statistician
- Publisher
- Informa UK Limited
- Published
- 2001-08
- Volume
- 55
- Issue
- 3
- Pages
- 182-186
- ISSN
- 0003-1305 (ISSN-L)
- 1537-2731
- OpenAlex
- W2036002592
- Cited by
- 1,218
- References
- 33
- Citations by year
- 2026: 57 · 2025: 63 · 2024: 85 · 2023: 77 · 2022: 105 · 2021: 75 · 2020: 63 · 2019: 61 · 2018: 64 · 2017: 64 · 2016: 53 · 2015: 50 · 2014: 63 · 2013: 51 · 2012: 39
- FWCI
- 19.30
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- The American Statistician (publishedVersion)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Probability and Statistical Research
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Bayesian Inference
- Physical Sciences › Mathematics › Statistics and Probability › Statistics Education and Methodologies
- Keywords
- confidence intervals, significance testing, null hypothesis rejection, conservative tests, asymptotic normality, asymptotic independence, consistency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: On judging the significance of differences by examining the overlap between confidence intervals. https://app.cochat.ai/papers/10.1198/000313001317097960
- The paper
- Schenker, N., & Gentleman, J. F. (2001). On judging the significance of differences by examining the overlap between confidence intervals. The American Statistician. https://doi.org/10.1198/000313001317097960
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.