Subgroup analyses in randomized trials: risks of subgroup-specific analyses;

Brookes, Sara T.; Whitely, Elise; Egger, Matthias; Smith, George Davey; Mulheran, Paul A.; Peters, Tim J.

doi:10.1016/j.jclinepi.2003.08.009

Cited by 775Statistical Methods in Clinical TrialsMeta-analysis and systematic reviewsAdvanced Causal Inference Techniques

Subgroup analyses in randomized trials: risks of subgroup-specific analyses;

Brookes, Sara T., Whitely, Elise, Egger, Matthias, Smith, George Davey, Mulheran, Paul A., Peters, Tim J.·2004·Journal of Clinical Epidemiology

DOI 10.1016/j.jclinepi.2003.08.009

Open this paper in CoChat — free account

Abstract

Objective Despite guidelines recommending the use of formal tests of interaction in subgroup analyses in clinical trials, inappropriate subgroup-specific analyses continue. Moreover, trials designed to detect overall treatment effects have limited power to detect treatment-subgroup interactions. This article quantifies the error rates associated with subgroup analyses. Study design and setting Simulations quantified the risks of misinterpreting subgroup analyses as evidence of differential subgroup effects and the limited power of the interaction test in trials designed to detect overall treatment effects. Results Although formal interaction tests performed as expected with respect to false positives, subgroup-specific tests were considerably less reliable: A significant effect in one subgroup only was observed in 7% to 64% of simulations depending on trial characteristics. Regarding power of the interaction test, a trial with 80% power for the overall effect had only 29% power to detect an interaction effect of the same magnitude. For interactions of this size to be detected with the same power as the overall effect, sample sizes should be inflated fourfold, increasing dramatically for interactions smaller than 20% of the overall effect. Conclusion Although it is generally recognized that subgroup analyses can produce spurious results, the extent of the problem may be underestimated.

Integrity signals

Registry

Crossref · 10.1016/j.jclinepi.2003.08.009

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

This paper’s full text isn’t openly available, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Testing the interaction effect requires a much larger sample size than detecting the main effect with the same magnitude.
“comparing methylation to disease associations between the sexes by testing the interaction effect would require a much larger sample size than detecting the main effect with the same magnitude [24]”
From Sex-specific DNA methylation differences in Alzheimer’s disease pathology (2021) · cites this paper as [24]

The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · The Lancet · doi:10.1016/s0140-6736(00)04337-3 · Cited by 5,373 · see citing works
The CONSORT statement: revised recommendations for improving the quality of reports of parallel group randomized trials
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-2 · Cited by 2,755 · see citing works
The Importance of Beta, the Type II Error and Sample Size in the Design and Interpretation of the Randomized Control Trial: Survey of 71 Negative Trials
Freiman, J. A., Chalmers, T. C., Smith, H., & Kuebler, R. R. (1978) · New England Journal of Medicine · doi:10.1056/nejm197809282991304 · Cited by 1,628 · see citing works
Sifting the evidence---what's wrong with significance tests? Another comment on the role of statistical methods
Sterne, J. A. C. (2001) · BMJ · doi:10.1136/bmj.322.7280.226 · Cited by 1,466 · see citing works
<i>meta-analysis</i> bias in location and selection of studies
Egger, M., & Smith, G. D. (1998) · BMJ · doi:10.1136/bmj.316.7124.61 · Cited by 1,374 · see citing works
Self-citation
Analysis and interpretation of treatment effects in subgroups of patients in randomized clinical trials
Yusuf, S. (1991) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1991.03470010097038 · Cited by 1,014 · see citing works
Tests for interaction in epidemiologic studies: A review and a study of power
Greenland, S. (1983) · Statistics in Medicine · doi:10.1002/sim.4780020219 · Cited by 280 · see citing works

Flagged first, then by citations

Sorted by citations

Statistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials
Wang, R., Lagakos, S. W., Ware, J. H., Hunter, D. J., & Drazen, J. M. (2007) · New England Journal of Medicine · doi:10.1056/nejmsr077003 · Cited by 1,445 · see citing works
Impacts of adverse childhood experiences on health, mental health, and substance use in early adulthood: A cohort study of an urban, minority sample in the U.S.
Mersky, J. P., Topitzes, J., & Reynolds, A. J. (2013) · Child Abuse & Neglect · doi:10.1016/j.chiabu.2013.07.011 · Cited by 729 · see citing works
Effect of Low-Fat vs Low-Carbohydrate Diet on 12-Month Weight Loss in Overweight Adults and the Association With Genotype Pattern or Insulin Secretion: The DIETFITS Randomized Clinical Trial
Gardner, C. D., Trepanowski, J. F., Del Gobbo, L. C., Hauser, M. E., Rigdon, J., Ioannidis, J. P. A., Desai, M., & King, A. C. (2018) · JAMA · doi:10.1001/jama.2018.0245 · Cited by 709 · see citing works
Sample Size Planning for Statistical Power and Accuracy in Parameter Estimation
Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008) · Annual Review of Psychology · doi:10.1146/annurev.psych.59.103006.093735 · Cited by 542 · see citing works
Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal
Kent, D. M., Rothwell, P. M., Ioannidis, J. P., Altman, D. G., & Hayward, R. A. (2010) · Trials · doi:10.1186/1745-6215-11-85 · Cited by 504 · see citing works
The Predictive Approaches to Treatment effect Heterogeneity (PATH) statement
Kent, D. M., Paulus, J. K., van Klaveren, D., D'Agostino, R., Goodman, S., Hayward, R., Ioannidis, J. P. A., Patrick-Lake, B., Morton, S., Pencina, M., et al. (2020) · Annals of Internal Medicine · doi:10.7326/m18-3667 · Cited by 469 · see citing works
Digital interventions for the treatment of depression: a meta-analytic review
Moshe, I., Terhorst, Y., Philippi, P., Domhardt, M., Cuijpers, P., Cristea, I., Pulkki-Råback, L., Baumeister, H., & Sander, L. B. (2021) · Psychological Bulletin · doi:10.1037/bul0000334 · Cited by 390 · see citing works
Beyond a catalogue of differences: A theoretical frame and good practice guidelines for researching sex/gender in human health
Springer, K. W., Mager Stellman, J., & Jordan-Young, R. M. (2012) · Social Science & Medicine · doi:10.1016/j.socscimed.2011.05.033 · Cited by 411 · see citing works
High-Intensity Interval Training in Patients With Heart Failure With Reduced Ejection Fraction
Ellingsen, Ø., Halle, M., Conraads, V., Støylen, A., Dalen, H., Delagardelle, C., Larsen, A.-I., Hole, T., Mezzani, A., Van Craenenbroeck, E. M., et al. (2017) · Circulation · doi:10.1161/circulationaha.116.022924 · Cited by 414 · see citing works
An introduction to multiplicity issues in clinical trials: the what, why, when and how
Li, G., Taljaard, M., Van den Heuvel, E. R., Levine, M. A., Cook, D. J., Wells, G. A., Devereaux, P. J., & Thabane, L. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyw320 · Cited by 381 · see citing works
Influence of initial severity of depression on effectiveness of low intensity interventions: meta-analysis of individual patient data
Bower, P., Kontopantelis, E., Sutton, A., Kendrick, T., Richards, D. A., Gilbody, S., Knowles, S., Cuijpers, P., Andersson, G., Christensen, H., et al. (2013) · BMJ · doi:10.1136/bmj.f540 · Cited by 341 · see citing works
Do guided internet-based interventions result in clinically relevant changes for patients with depression? An individual participant data meta-analysis
Karyotaki, E., Ebert, D. D., Donkin, L., Riper, H., Twisk, J., Burger, S., Rozental, A., Lange, A., Williams, A. D., Zarski, A. C., et al. (2018) · Clinical Psychology Review · doi:10.1016/j.cpr.2018.06.007 · Cited by 303 · see citing works
Randomized, Placebo-Controlled Trial of Olanzapine as Maintenance Therapy in Patients With Bipolar I Disorder Responding to Acute Treatment With Olanzapine
Tohen, M., Calabrese, J. R., Sachs, G. S., Banov, M. D., Detke, H. C., Risser, R., Baker, R. W., Chou, J. C.-Y., & Bowden, C. L. (2006) · American Journal of Psychiatry · doi:10.1176/appi.ajp.163.2.247 · Cited by 317 · see citing works
Three simple rules to ensure reasonably credible subgroup analyses
Burke, J. F., Sussman, J. B., Kent, D. M., & Hayward, R. A. (2015) · BMJ · doi:10.1136/bmj.h5651 · Cited by 312 · see citing works
Sex and gender differences research design for basic, clinical, and population studies: essentials for investigators
Rich-Edwards, J. W., Kaiser, U. B., Chen, G. L., Manson, J. E., & Goldstein, J. M. (2018) · Endocrine Reviews · doi:10.1210/er.2017-00246 · Cited by 273 · see citing works
Reverse Causation and Illness-related Weight Loss in Observational Studies of Body Weight and Mortality
Flegal, K. M., Graubard, B. I., Williamson, D. F., & Cooper, R. S. (2011) · American Journal of Epidemiology · doi:10.1093/aje/kwq341 · Cited by 223 · see citing works
Effect of cognitive-behavioral therapy for anxiety disorders on quality of life: A meta-analysis.
Hofmann, S. G., Wu, J. Q., & Boettcher, H. (2014) · Journal of Consulting and Clinical Psychology · doi:10.1037/a0035491 · Cited by 191 · see citing works
A systematic review of predictors and moderators of response to psychological therapies in OCD: Do we have enough empirical evidence to target treatment?
Knopp, J., Knowles, S., Bee, P., Lovell, K., & Bower, P. (2013) · Clinical Psychology Review · doi:10.1016/j.cpr.2013.08.008 · Cited by 164 · see citing works
GENDER AS PREDICTOR AND MODERATOR OF OUTCOME IN COGNITIVE BEHAVIOR THERAPY AND PHARMACOTHERAPY FOR ADULT DEPRESSION: AN “INDIVIDUAL PATIENT DATA” META-ANALYSIS: Research Article: Gender as Moderator of Treatment Outcome
Cuijpers, P., Weitz, E., Twisk, J., Kuehner, C., Cristea, I., David, D., DeRubeis, R. J., Dimidjian, S., Dunlop, B. W., Faramarzi, M., et al. (2014) · Depression and Anxiety · doi:10.1002/da.22328 · Cited by 158 · see citing works
Ketamine and other glutamate receptor modulators for depression in adults
Caddy, C., Amit, B. H., McCloud, T. L., Rendell, J. M., Furukawa, T. A., McShane, R., Hawton, K., & Cipriani, A. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011612.pub2 · Cited by 174 · see citing works
Does Internet-based guided-self-help for depression cause harm? An individual participant data meta-analysis on deterioration rates and its moderators in randomized controlled trials
Ebert, D. D., Donkin, L., Andersson, G., Andrews, G., Berger, T., Carlbring, P., Rozenthal, A., Choi, I., Laferton, J. A. C., Johansson, R., et al. (2016) · Psychological Medicine · doi:10.1017/s0033291716001562 · Cited by 154 · see citing works
Subgroup analyses in therapeutic cardiovascular clinical trials: Are most of them misleading?
Hernández, A. V., Boersma, E., Murray, G. D., Habbema, J. D. F., & Steyerberg, E. W. (2006) · American Heart Journal · doi:10.1016/j.ahj.2005.04.020 · Cited by 131 · see citing works
Personalized Psychotherapy for Adult Depression: A Meta-Analytic Review
Cuijpers, P., Ebert, D. D., Acarturk, C., Andersson, G., & Cristea, I. A. (2016) · Behavior Therapy · doi:10.1016/j.beth.2016.04.007 · Cited by 128 · see citing works
Clinical effectiveness and cost-effectiveness of cognitive behavioural therapy as an adjunct to pharmacotherapy for treatment-resistant depression in primary care: the CoBalT randomised controlled trial
Wiles, N., Thomas, L., Abel, A., Barnes, M., Carroll, F., Ridgway, N., Sherlock, S., Turner, N., Button, K., Odondi, L., et al. (2014) · Health Technology Assessment · doi:10.3310/hta18310 · Cited by 127 · see citing works
Self-citation
Substance use outcomes 5½ years past baseline for partnership-based, family-school preventive interventions
Spoth, R. L., Randall, G. K., Trudeau, L., Shin, C., & Redmond, C. (2008) · Drug and Alcohol Dependence · doi:10.1016/j.drugalcdep.2008.01.023 · Cited by 117 · see citing works
COGNITIVE BEHAVIORAL THERAPY AGE EFFECTS IN CHILD AND ADOLESCENT ANXIETY: AN INDIVIDUAL PATIENT DATA METAANALYSIS: Research Article: CBT Age Effects in Child and Adolescent Anxiety
Bennett, K., Manassis, K., Walter, S. D., Cheung, A., Wilansky-Traynor, P., Diaz-Granados, N., Duda, S., Rice, M., Baer, S., Barrett, P., et al. (2013) · Depression and Anxiety · doi:10.1002/da.22099 · Cited by 104 · see citing works
Ketamine and other glutamate receptor modulators for depression in bipolar disorder in adults
McCloud, T. L., Caddy, C., Jochim, J., Rendell, J. M., Diamond, P. R., Shuttleworth, C., Brett, D., Amit, B. H., McShane, R., Hamadi, L., et al. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd011611.pub2 · Cited by 102 · see citing works
The REFLECT Statement: Reporting guidelines for randomized controlled trials in livestock and food safety: Explanation and Elaboration
Sargeant, J. M., O’connor, A. M., Gardner, I. A., Dickson, J. S., Torrence, M. E., Dohoo, C. M. P. I. R., Lefebvre, S. L., Morley, P. S., Ramirez, A., & Snedeker, K. (2010) · Journal of Food Protection · doi:10.4315/0362-028x-73.3.579 · Cited by 97 · see citing works
The effectiveness and cost-effectiveness of a mindfulness training programme in schools compared with normal school provision (MYRIAD): study protocol for a randomised controlled trial
the MYRIAD team, Kuyken, W., Nuthall, E., Byford, S., Crane, C., Dalgleish, T., Ford, T., Greenberg, M. T., Ukoumunne, O. C., Viner, R. M., et al. (2017) · Trials · doi:10.1186/s13063-017-1917-4 · Cited by 77 · see citing works
A Randomized Controlled Trial on the Effectiveness of a Classification-Based System for Subacute and Chronic Low Back Pain
Apeldoorn, A. T., Ostelo, R. W., van Helvoirt, H., Fritz, J. M., Knol, D. L., van Tulder, M. W., & de Vet, H. C. (2012) · Spine · doi:10.1097/brs.0b013e31824d9f2b · Cited by 74 · see citing works
Sex-specific DNA methylation differences in Alzheimer’s disease pathology
Zhang, L., Young, J. I., Gomez, L., Silva, T. C., Schmidt, M. A., Cai, J., Chen, X., Martin, E. R., & Wang, L. (2021) · Acta Neuropathologica Communications · doi:10.1186/s40478-021-01177-8 · Cited by 58 · see citing works
“Also, comparing methylation to disease associations between the sexes by testing the interaction effect would require a much larger sample size than detecting the main effect with the same magnitude [24].”
Background
Predicting successful treatment outcome of web-based self-help for problem drinkers: secondary analysis from a randomized controlled trial
Riper, H., Kramer, J., Keuken, M., Smit, F., Schippers, G., & Cuijpers, P. (2008) · Journal of Medical Internet Research · doi:10.2196/jmir.1102 · Cited by 52 · see citing works
Moderators of Outcome for Youth Anxiety Treatments: Current Findings and Future Directions
Norris, L. A., & Kendall, P. C. (2021) · Journal of Clinical Child & Adolescent Psychology · doi:10.1080/15374416.2020.1833337 · Cited by 46 · see citing works
Variation in results from randomized, controlled trials: stochastic or systematic?
Jane-wit, D., Horwitz, R. I., & Concato, J. (2010) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.02.010 · Cited by 33 · see citing works
Usual care and a self-management support programme versus usual care and a relaxation programme for people living with chronic headache disorders: A randomised controlled trial protocol (CHESS)
Patel, S., Achana, F., Carnes, D., Eldridge, S., Ellard, D. R., Griffiths, F., Haywood, K., Hee, S. W., Mistry, D., Mistry, H., et al. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-033520 · Cited by 24 · see citing works
“46 We will assess the level of missingness in the primary outcome, and if required, we will use appropriate multiple imputation techniques to impute data and estimate the treatment effect as a form of sensitivity analysis.”
Method
Predicting response to a cognitive-behavioral approach to treating low back pain: Secondary analysis of the BeST data set
Underwood, M., Mistry, D., Lall, R., & Lamb, S. (2011) · Arthritis Care & Research · doi:10.1002/acr.20518 · Cited by 23 · see citing works
The effect of patient's choice of cognitive behavioural or psychodynamic therapy on outcomes for panic disorder: a doubly randomised controlled preference trial
Svensson, M., Nilsson, T., Perrin, S., Johansson, H., Viborg, G., Falkenström, F., & Sandell, R. (2021) · Psychotherapy and Psychosomatics · doi:10.1159/000511469 · Cited by 17 · see citing works

Authors: Brookes, S. T., Whitely, E., Egger, M., Smith, G. D., Mulheran, P. A., & Peters, T. J.
Affiliations: University of Bristol (GB)
University of Bern (CH)
University of Reading (GB)
Corresponding author: Brookes, Sara T.
ORCID: Brookes, Sara T.
Egger, Matthias
Smith, George Davey
Mulheran, Paul A.
Venue: Journal of Clinical Epidemiology
Publisher: Elsevier BV
Published: 2004-03
Volume: 57
Issue: 3
Pages: 229-236
ISSN: 0895-4356 (ISSN-L)
DOI: 10.1016/j.jclinepi.2003.08.009
OpenAlex: W1592228396
Cited by: 775
References: 7
Citations by year: 2026: 83 · 2025: 46 · 2024: 49 · 2023: 52 · 2022: 45 · 2021: 38 · 2020: 53 · 2019: 45 · 2018: 31 · 2017: 25 · 2016: 43 · 2015: 33 · 2014: 27 · 2013: 37 · 2012: 16
FWCI: 15.20
Type: journal article
Language: en
Open access: closed
Licence: www.elsevier.com/tdm/userlicense/1.0/ (tdm)
Locations: Journal of Clinical Epidemiology (publishedVersion)
PubMed (publishedVersion)
Bristol Research (University of Bristol) (submittedVersion)
Funders: Health Technology Assessment Programme
Topics: Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Keywords: subgroup analysis, interaction testing, statistical power, false positives, randomized controlled trials, differential treatment effects
MeSH terms: Data Interpretation, Statistical, Humans, Research Design, Randomized Controlled Trials as Topic, Treatment Outcome, Sample Size

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Subgroup analyses in randomized trials: risks of subgroup-specific analyses;

Abstract

Integrity signals

Claims the paper makes

Claims other papers make about this one

Cites

Cited by

Record

About these checks