Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses
Abstract
How can we tell the difference between spurious and real subgroup effects? This article identifies new criteria and proposes a checklist for judging the credibility of subgroup analyses
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The method for adjusting for confounders when different hypothesized life course associations have different potential confounders does not allow for the potential combination of more than one hypothesized association with different potential confounders.
“However, this method does not allow for the potential combination of more than one hypothesized association with different potential confounders, as may be encountered if effect modification is considered.37”
From A structured approach to hypotheses involving continuous exposures over the life course (2016) · cites this paper as [37]
Cites
CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials
the CONSORT Group, Schulz, K. F., Altman, D. G., & Moher, D. (2010) · BMC Medicine · doi:10.1186/1741-7015-8-18 · Cited by 13,371 · see citing works
Intensive insulin therapy in the medical ICU
Van den Berghe, G., Wilmer, A., Hermans, G., Meersseman, W., Wouters, P. J., Milants, I., Van Wijngaerden, E., Bobbaers, H., & Bouillon, R. (2006) · New England Journal of Medicine · doi:10.1056/nejmoa052521 · Cited by 3,639 · see citing works
Thatcher, N., Chang, A., Parikh, P., Rodrigues Pereira, J., Ciuleanu, T., von Pawel, J., Thongprasert, S., Tan, E. H., Pemberton, K., Archer, V., et al. (2005) · The Lancet · doi:10.1016/s0140-6736(05)67625-8 · Cited by 2,079 · see citing works
Babyak, M. A. (2004) · Psychosomatic Medicine · doi:10.1097/01.psy.0000127692.23278.a9 · Cited by 2,061 · see citing works
Vasopressin versus Norepinephrine Infusion in Patients with Septic Shock
Russell, J. A., Walley, K. R., Singer, J., Gordon, A. C., Hébert, P. C., Cooper, D. J., Holmes, C. L., Mehta, S., Granton, J. T., Storms, M. M., et al. (2008) · New England Journal of Medicine · doi:10.1056/nejmoa067373 · Cited by 1,924 · see citing works
Statistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials
Wang, R., Lagakos, S. W., Ware, J. H., Hunter, D. J., & Drazen, J. M. (2007) · New England Journal of Medicine · doi:10.1056/nejmsr077003 · Cited by 1,445 · see citing works
Users' guides to the medical literature : a manual for evidence-based clinical practice
Guyatt, G. H., Rennie, D., Meade, M. O., & Cook, D. J. (2002) · Cited by 1,196 · see citing works
Self-citationSubgroup analysis and other (mis)uses of baseline data in clinical trials
Assmann, S. F., Pocock, S. J., Enos, L. E., & Kasten, L. E. (2000) · The Lancet · doi:10.1016/s0140-6736(00)02039-0 · Cited by 1,167 · see citing works
Yusuf, S. (1991) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1991.03470010097038 · Cited by 1,014 · see citing works
A consumer's guide to subgroup analyses
Oxman, A. D., & Guyatt, G. H. (1992) · Annals of Internal Medicine · doi:10.7326/0003-4819-116-1-78 · Cited by 914 · see citing works
Self-citationAspirin for the Primary Prevention of Cardiovascular Events in Women and Men
Berger, J. S., Roncaglioni, M. C., Avanzini, F., Pangrazzi, I., Tognoni, G., & Brown, D. (2006) · JAMA · doi:10.1001/jama.295.3.306 · Cited by 781 · see citing works
IX: Summary of Meta-Analyses of Therapies for Postmenopausal Osteoporosis
Cranney, A., Guyatt, G. H., Griffith, L. E., Wells, G., Tugwell, P., Rosen, C. J., & The Osteoporosis Methodology Group (2002) · Endocrine Reviews · doi:10.1210/er.2001-9002 · Cited by 597 · see citing works
Self-citationRandomized Trial of Reamed and Unreamed Intramedullary Nailing of Tibial Shaft Fractures
Bhandari, M., Guyatt, G. H., Walter, S. D., Tornetta, P., Schemitsch, E. H., Swiontkowski, M. F., & Sanders, D. (2008) · Journal of Bone and Joint Surgery · doi:10.2106/jbjs.g.01694 · Cited by 453 · see citing works
Self-citationPrimary Prevention of Cardiovascular Diseases With Statin Therapy
Thavendiranathan, P. (2006) · Archives of Internal Medicine · doi:10.1001/archinte.166.21.2307 · Cited by 349 · see citing works
Freemantle, N. (2001) · BMJ · doi:10.1136/bmj.322.7292.989 · Cited by 168 · see citing works
Hatala, R. A. (2005) · Canadian Medical Association Journal · doi:10.1503/cmaj.1031920 · Cited by 152 · see citing works
Bias, Spin, and Misreporting: Time for Full Access to Trial Protocols and Results
Chan, A.-W. (2008) · PLoS Medicine · doi:10.1371/journal.pmed.0050230 · Cited by 144 · see citing works
Horwitz, R. I., Singer, B. H., Makuch, R. W., & Viscoli, C. M. (1996) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(95)00058-5 · Cited by 116 · see citing works
The Safety of Long-Acting β-Agonists among Patients with Asthma Using Inhaled Corticosteroids
Jaeschke, R., O’Byrne, P. M., Mejza, F., et al. (2008) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.200804-494oc · Cited by 109 · see citing works
Meta-Analyses of Therapies for Postmenopausal Osteoporosis
The Osteoporosis Methodology Group (2002) · Endocrine Reviews · doi:10.1210/er.2001-1002 · Cited by 97 · see citing works
The benefits of chemotherapy in patient subgroups with unresectable non-small-cell lung cancer
Billingham, L. J., & Cullen, M. H. (2001) · Annals of Oncology · doi:10.1023/a:1013582618920 · Cited by 90 · see citing works
The Problem of Cogent Subgroups: A Clinicostatistical Tragedy
Feinstein, A. R. (1998) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(98)00004-3 · Cited by 85 · see citing works
Akl, E. A. (2008) · Archives of Internal Medicine · doi:10.1001/archinte.168.12.1261 · Cited by 82 · see citing works
Incommunicable Knowledge? Interpreting and Applying the Results of Clinical Trials and Meta-Analyses
Smith, G. D., & Egger, M. (1998) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(97)00293-x · Cited by 55 · see citing works
Naturopathic Care for Chronic Low Back Pain: A Randomized Trial
Szczurko, O., Cooley, K., Busse, J. W., et al. (2007) · PLoS ONE · doi:10.1371/journal.pone.0000919 · Cited by 51 · see citing works
Cooling for Neonatal Hypoxic Ischemic Encephalopathy: Do We Have the Answer?
Kirpalani, H. M., Barks, J., Thorlund, K., & Guyatt, G. H. (2007) · PEDIATRICS · doi:10.1542/peds.2006-2776 · Cited by 42 · see citing works
Self-citationMisuse of Baseline Comparison Tests and Subgroup Analyses in Surgical Trials
Bhandari, M., Devereaux, P. J., Li, P., Mah, D. Y., Lim, K. H., Schünemann, H. J., & Tornetta, P. (2006) · Clinical Orthopaedics and Related Research · doi:10.1097/01.blo.0000218736.23506.fe · Cited by 38 · see citing works
Senn, S. J., & Harrell, F. E. (1997) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(97)00023-1 · Cited by 36 · see citing works
Within Trial Variation—A False Trail?
Altman, D. G. (1998) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(98)00005-5 · Cited by 30 · see citing works
Clinical versus Statistical Considerations in the Design and Analysis of Clinical Research
Horwitz, R. I., Singer, B. H., Makuch, R. W., et al. (1998) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(98)00006-7 · Cited by 23 · see citing works
Horwitz, R. I., Singer, B. H., Makuch, R., & Viscoli, C. M. (1997) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(97)00025-5 · Cited by 17 · see citing works
Use of Aspirin as Primary Prevention of Cardiovascular Events
Dalen, J. E. (2006) · JAMA · doi:10.1001/jama.296.4.391-b · Cited by 7 · see citing works
Interpreting Subgroup Analyses
Bundy, D. G., Berkoff, M. C., Ito, K., et al. (2004) · Archives of Pediatrics and Adolescent Medicine · doi:10.1001/archpedi.158.5.469 · Cited by 5 · see citing works
Necessary cautions when considering digoxin in heart failure
Heckman, G. A., & McKelvie, R. S. (2007) · Canadian Medical Association Journal · doi:10.1503/cmaj.061309 · Cited by 4 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials
Chan, A.-W., Tetzlaff, J. M., Gotzsche, P. C., Altman, D. G., Mann, H., Berlin, J. A., Dickersin, K., Hrobjartsson, A., Schulz, K. F., Parulekar, W. R., et al. (2013) · BMJ · doi:10.1136/bmj.e7586 · Cited by 6,251 · see citing works
Maher, C., Underwood, M., & Buchbinder, R. (2017) · The Lancet · doi:10.1016/s0140-6736(16)30970-9 · Cited by 2,535 · see citing works
Martineau, A. R., Jolliffe, D. A., Hooper, R. L., Greenberg, L., Aloia, J. F., Bergman, P., Dubnov-Raz, G., Esposito, S., Ganmaa, D., Ginde, A. A., et al. (2017) · BMJ · doi:10.1136/bmj.i6583 · Cited by 2,040 · see citing works
Husereau, D., Drummond, M., Petrou, S., Carswell, C., Moher, D., Greenberg, D., Augustovski, F., Briggs, A. H., Mauskopf, J., & Loder, E. (2013) · Value in Health · doi:10.1016/j.jval.2013.02.002 · Cited by 1,998 · see citing works
Progress in evidence-based medicine: a quarter century on
Djulbegovic, B., & Guyatt, G. H. (2017) · The Lancet · doi:10.1016/s0140-6736(16)31592-6 · Cited by 1,022 · see citing works
Self-citationSchandelmaier, S., Briel, M., Varadhan, R., Schmid, C. H., Devasenapathy, N., Hayward, R. A., Gagnier, J., Borenstein, M., van der Heijden, G. J. M. G., Dahabreh, I. J., et al. (2020) · Canadian Medical Association Journal · doi:10.1503/cmaj.200077 · Cited by 749 · see citing works
Self-citationHusereau, D., Drummond, M., Augustovski, F., de Bekker-Grob, E., Briggs, A. H., Carswell, C., Caulley, L., Chaiyakunapruk, N., Greenberg, D., Loder, E., et al. (2022) · Value in Health · doi:10.1016/j.jval.2021.10.008 · Cited by 847 · see citing works
“interpretation of results, such as biological plausibility of hypotheses and prespecification of subgroup testing.(78)”
MethodProbiotics for the prevention of Clostridium difficile-associated diarrhea in adults and children
Goldenberg, J. Z., Ma, S. S., Saxton, J. D., Martzen, M. R., Vandvik, P. O., Thorlund, K., Guyatt, G. H., & Johnston, B. C. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006095.pub3 · Cited by 741 · see citing works
Self-citationHow to Use a Subgroup Analysis: Users’ Guide to the Medical Literature
Sun, X., Ioannidis, J. P. A., Agoritsas, T., Alba, A. C., & Guyatt, G. (2014) · JAMA · doi:10.1001/jama.2013.285063 · Cited by 550 · see citing works
Self-citationRangan, A., Handoll, H., Brealey, S., Jefferson, L., Keding, A., Martin, B. C., Goodchild, L., Chuang, L.-H., Hewitt, C., & Torgerson, D. (2015) · JAMA · doi:10.1001/jama.2015.1629 · Cited by 556 · see citing works
Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal
Kent, D. M., Rothwell, P. M., Ioannidis, J. P., Altman, D. G., & Hayward, R. A. (2010) · Trials · doi:10.1186/1745-6215-11-85 · Cited by 504 · see citing works
Hughes, D., Judge, C., Murphy, R., Loughlin, E., Costello, M., Whiteley, W., Bosch, J., O’Donnell, M. J., & Canavan, M. (2020) · JAMA · doi:10.1001/jama.2020.4249 · Cited by 472 · see citing works
Primary and Secondary Prevention of Cardiovascular Disease
Vandvik, P. O., Lincoff, A. M., Gore, J. M., Gutterman, D. D., Sonnenberg, F. A., Alonso-Coello, P., Akl, E. A., Lansberg, M. G., Guyatt, G. H., & Spencer, F. A. (2012) · Chest · doi:10.1378/chest.11-2306 · Cited by 476 · see citing works
Self-citationBaxter, A. J., Tweed, E. J., Katikireddi, S. V., & Thomson, H. (2019) · Journal of Epidemiology and Community Health · doi:10.1136/jech-2018-210981 · Cited by 380 · see citing works
Johnston, B. C., Ma, S. S. Y., Goldenberg, J. Z., Thorlund, K., Vandvik, P. O., Loeb, M., & Guyatt, G. H. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-12-201212180-00563 · Cited by 367 · see citing works
Self-citationThree simple rules to ensure reasonably credible subgroup analyses
Burke, J. F., Sussman, J. B., Kent, D. M., & Hayward, R. A. (2015) · BMJ · doi:10.1136/bmj.h5651 · Cited by 312 · see citing works
Tutorial in biostatistics: data‐driven subgroup identification and analysis in clinical trials
Lipkovich, I., Dmitrienko, A., & B., R. (2017) · Statistics in Medicine · doi:10.1002/sim.7064 · Cited by 291 · see citing works
Hoffmann, T. C., Oxman, A. D., Ioannidis, J. P., Moher, D., Lasserson, T. J., Tovey, D. I., Stein, K., Sutcliffe, K., Ravaud, P., Altman, D. G., et al. (2017) · BMJ · doi:10.1136/bmj.j2998 · Cited by 194 · see citing works
Dorresteijn, J. A. N., Visseren, F. L. J., Ridker, P. M., Wassink, A. M. J., Paynter, N. P., Steyerberg, E. W., van der Graaf, Y., & Cook, N. R. (2011) · BMJ · doi:10.1136/bmj.d5888 · Cited by 177 · see citing works
Micronized vaginal progesterone to prevent miscarriage: a critical evaluation of randomized evidence
Coomarasamy, A., Devall, A. J., Brosens, J. J., Quenby, S., Stephenson, M. D., Sierra, S., Christiansen, O. B., Small, R., Brewin, J., Roberts, T. E., et al. (2020) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2019.12.006 · Cited by 163 · see citing works
Science, M., Johnstone, J., Roth, D. E., Guyatt, G., & Loeb, M. (2012) · Canadian Medical Association Journal · doi:10.1503/cmaj.111990 · Cited by 157 · see citing works
Light therapy for non-seasonal depression: systematic review and meta-analysis
Perera, S., Eisen, R., Bhatt, M., Bhatnagar, N., de Souza, R., Thabane, L., & Samaan, Z. (2016) · BJPsych Open · doi:10.1192/bjpo.bp.115.001610 · Cited by 153 · see citing works
SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials
Hróbjartsson, A., Boutron, I., Hopewell, S., Moher, D., Schulz, K. F., Collins, G. S., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081660 · Cited by 171 · see citing works
Efficacy of vitamin D supplementation in depression in adults: a systematic review
Li, G., Mbuagbaw, L., Samaan, Z., Falavigna, M., Zhang, S., Adachi, J. D., Cheng, J., Papaioannou, A., & Thabane, L. (2014) · The Journal of Clinical Endocrinology & Metabolism · doi:10.1210/jc.2013-3450 · Cited by 144 · see citing works
Crane, D., Garnett, C., Michie, S., West, R., & Brown, J. (2018) · Scientific Reports · doi:10.1038/s41598-018-22420-8 · Cited by 118 · see citing works
Neuromuscular blockade in patients with ARDS: a rapid practice guideline
Alhazzani, W., Belley-Cote, E., Møller, M. H., Angus, D. C., Papazian, L., Arabi, Y. M., Citerio, G., Connolly, B., Denehy, L., Fox-Robichaud, A., et al. (2020) · Intensive Care Medicine · doi:10.1007/s00134-020-06227-8 · Cited by 123 · see citing works
Low intensity pulsed ultrasound for bone healing: systematic review of randomized controlled trials
Schandelmaier, S., Kaushal, A., Lytvyn, L., Heels-Ansdell, D., Siemieniuk, R. A. C., Agoritsas, T., Guyatt, G. H., Vandvik, P. O., Couban, R., Mollon, B., et al. (2017) · BMJ · doi:10.1136/bmj.j656 · Cited by 128 · see citing works
Bennett, K., Manassis, K., Walter, S. D., Cheung, A., Wilansky-Traynor, P., Diaz-Granados, N., Duda, S., Rice, M., Baer, S., Barrett, P., et al. (2013) · Depression and Anxiety · doi:10.1002/da.22099 · Cited by 104 · see citing works
Self-citationNeumann, I., Rada, G., Claro, J. C., Carrasco-Labra, A., Thorlund, K., Akl, E. A., Bates, S. M., & Guyatt, G. H. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-156-10-201205150-00421 · Cited by 94 · see citing works
Self-citationWelch, V. A., Akl, E. A., Pottie, K., Ansari, M. T., Briel, M., Christensen, R., Dans, A., Dans, L., Eslava-Schmalbach, J., Guyatt, G., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.01.015 · Cited by 88 · see citing works
Self-citationWallach, J. D., Sullivan, P. G., Trepanowski, J. F., Steyerberg, E. W., & Ioannidis, J. P. A. (2016) · BMJ · doi:10.1136/bmj.i5826 · Cited by 89 · see citing works
A structured approach to hypotheses involving continuous exposures over the life course
Smith, A. D. A. C., Hardy, R., Heron, J., Joinson, C. J., Lawlor, D. A., Macdonald-Wallis, C., & Tilling, K. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyw164 · Cited by 59 · see citing works
McFarland, L. V. (2013) · World Journal of Meta-Analysis · doi:10.13105/wjma.v1.i3.102 · Cited by 33 · see citing works
Gao, Y., Bai, Y., Mu, X., & Pang, X. (2026) · Frontiers in Pharmacology · doi:10.3389/fphar.2026.1736821 · Cited by 4 · see citing works
“An interaction p -value < 0.01 was considered significant (Sun et al., 2010).”
InfluentialO’Hara, N. N., Heels-Ansdell, D., Bzovsky, S., Dodds, S., Thabane, L., Bhandari, M., Guyatt, G., Devereaux, P. J., Slobogean, G. P., Sprague, S., et al. (2022) · Trials · doi:10.1186/s13063-022-06541-0 · Cited by 1 · see citing works
Self-citation
- Authors
- Sun, X., Briel, M., Walter, S. D., & Guyatt, G. H.
- Affiliations
- Sichuan University (CN)
- University of Basel (CH)
- McMaster University (CA)
- Corresponding author
- Guyatt, G. H.
- Venue
- BMJ
- Publisher
- BMJ
- Published
- 2010-03-30
- Volume
- 340
- Issue
- mar30 3
- Pages
- c117-c117
- ISSN
- 0007-1447 (ISSN-L)
- 0959-8138
- 1468-5833
- OpenAlex
- W1980472079
- Cited by
- 808
- References
- 34
- Citations by year
- 2026: 64 · 2025: 50 · 2024: 53 · 2023: 34 · 2022: 53 · 2021: 74 · 2020: 74 · 2019: 52 · 2018: 30 · 2017: 55 · 2016: 47 · 2015: 44 · 2014: 39 · 2013: 44 · 2012: 38
- FWCI
- 45.63
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- BMJ (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Keywords
- subgroup analysis, credibility assessment, heterogeneity, checklists, meta-analysis
- MeSH terms
- Data Interpretation, Statistical, Random Allocation, Research Design, Randomized Controlled Trials as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review