The DOI matches its Crossref record; 5 of the 5 references it cites were found in a registry or an index.
The (mis)use of overlap of confidence intervals to assess effect modification
Open accessAbstract
In randomized controlled trials as well as in observational studies, researchers are often interested in effects of treatment or exposure in different subgroups, i.e. effect modification [1, 2]. There are several methods to assess effect modification and the debate on which method is best is still ongoing [2–5]. In this article we focus on an invalid method to assess effect modification, which is often used in articles in health sciences journals [6], namely concluding that there is no effect modification if the confidence intervals of the subgroups are overlapping [7–9]. When assessing effect modification by looking at overlap of the 95% confidence intervals in subgroups, a type 1 error probability of 0.05 is often mistakenly assumed. In other words, if the confidence intervals are overlapping, the difference in effect estimates between the two subgroups is judged to be statistically insignificant. By using mathematical derivation, we calculated that the chance of finding non-overlapping 95% confidence intervals under the null hypothesis is 0.0056 if the variance of both effect estimates is equal and the effect estimates are independent (see Supplemental material for derivation of this probability). If the variance of the effect estimates is not equal, the chance of finding non-overlapping 95% confidence intervals can be calculated by taking into account ρ, i.e. the ratio between the standard deviations in the subgroups, σ2/σ1 (Supplementary material, formula (3)). Figure. 1 shows the relation between ρ and the type 1 error probability if the effect estimates are independent. If the effect estimates are not independent, the correlation coefficient between the effect estimates can also be accounted for (Supplementary material, formula (3)). Fig. 1 Relation between ρ, which is the ratio of σ2 and σ1, and the probability of non-overlapping confidence intervals under the null hypothesis (type 1 error) To arrive at a type 1 error probability of 0.05, 83.4% confidence intervals should be calculated around the effect estimates in subgroups if the variance is equal and the effect estimates are independent (see Supplementary material for derivation of this percentage). If the variance is not equal, ρ should be taken into account (Supplementary material, formula (11)). Figure. 2 shows the relation between ρ and the level of the confidence interval. If the effect estimates are not independent, the correlation coefficient should be taken into account (Supplementary material, formula (11)). Adapting the level of the confidence interval can be especially useful for graphical presentations, for example in meta-analyses [10]. However, it is necessary to explicitly and clearly state which percentage confidence interval is calculated and its meaning should be thoroughly explained to the reader. Many readers will still interpret this ‘new’ confidence interval as if it were a 95% confidence interval, because this percentage is so commonly used. To prevent such confusion, other methods to assess effect modification could be used, such as calculating a 95% confidence interval around the difference in effect estimates [8]. Fig. 2 Relation between ρ, which is the ratio of σ2 and σ1, and the percentage confidence intervals to be calculated to arrive at a type 1 error probability of 0.05 The assumption used in the formulas presented in the appendices is that the effect estimators in the subgroups are normally distributed. Assuming that epidemiologic effect measures, such as the odds ratio, risk ratio, hazard ratio and risk difference, follow a normal distribution, the methods presented can also be used for these epidemiologic measures. Note that the assumption for normality is generally unreasonable in small samples, but a satisfactory approximation in large samples.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials
Wang, R., Lagakos, S. W., Ware, J. H., Hunter, D. J., & Drazen, J. M. (2007) · New England Journal of Medicine · doi:10.1056/nejmsr077003 · Cited by 1,445
On judging the significance of differences by examining the overlap between confidence intervals.
Schenker, N., & Gentleman, J. F. (2001) · The American Statistician · doi:10.1198/000313001317097960 · Cited by 1,218
The graphical presentation of a collection of means
Goldstein, H., & Healy, M. J. R. (1995) · Journal of the Royal Statistical Society. Series A (Statistics in Society) · doi:10.2307/2983411 · Cited by 639
Interactions in epidemiology: relevance, identification, and estimation
Greenland, S. (2009) · Epidemiology · doi:10.1097/ede.0b013e318193e7b5 · Cited by 204
When One Depends on the Other: Reporting of Interaction in Case-Control and Cohort Studies
Knol, M. J., Egger, M., Scott, P., Geerlings, M. I., & Vandenbroucke, J. P. (2009) · Epidemiology · doi:10.1097/ede.0b013e31818f6651 · Cited by 113
Self-citation
Cited by
Risk of End-Stage Renal Disease Following Live Kidney Donation
Muzaale, A. D., Massie, A. B., Wang, M.-C., Montgomery, R. A., McBride, M. A., Wainright, J. L., & Segev, D. L. (2014) · JAMA · doi:10.1001/jama.2013.285141 · Cited by 982
Rueger, S. Y., Malecki, C. K., Pyun, Y., Aycock, C., & Coyle, S. (2016) · Psychological Bulletin · doi:10.1037/bul0000058 · Cited by 946
Do people who feel connected to nature do more to protect it? A meta-analysis
Mackay, C. M. L., & Schmitt, M. T. (2019) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2019.101323 · Cited by 538
Meta-analysis of the Age-Dependent Efficacy of Multiple Sclerosis Treatments
Weideman, A. M., Tapia-Maltos, M. A., Johnson, K., Greenwood, M., & Bielekova, B. (2017) · Frontiers in Neurology · doi:10.3389/fneur.2017.00577 · Cited by 310
Mansournia, M. A., Collins, G. S., Nielsen, R. O., Nazemipour, M., Jewell, N. P., Altman, D. G., & Campbell, M. J. (2021) · British Journal of Sports Medicine · doi:10.1136/bjsports-2020-103652 · Cited by 263
Metadata
- Authors
- Knol, M. J., Pestman, W. R., & Grobbee, D. E.
- Venue
- European Journal of Epidemiology
- Published
- 2011-03-19
- Volume
- 26
- Issue
- 4
- Pages
- 253-254
- ISSN
- 0393-2990 (ISSN-L)
- 1573-7284
- OpenAlex
- W1995695717
- Cited by
- 249
- References
- 5
- FWCI
- 2.50
- Type
- journal article
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: The (mis)use of overlap of confidence intervals to assess effect modification. https://app.cochat.ai/papers/10.1007/s10654-011-9563-8
- The paper
- Knol, M. J., Pestman, W. R., & Grobbee, D. E. (2011). The (mis)use of overlap of confidence intervals to assess effect modification. European Journal of Epidemiology. https://doi.org/10.1007/s10654-011-9563-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.