Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials
Open accessAbstract
Importance: Many published randomized clinical trials (RCTs) make claims for subgroup differences. Objective: To evaluate how often subgroup claims reported in the abstracts of RCTs are actually supported by statistical evidence (P < .05 from an interaction test) and corroborated by subsequent RCTs and meta-analyses. Data Sources: This meta-epidemiological survey examines data sets of trials with at least 1 subgroup claim, including Subgroup Analysis of Trials Is Rarely Easy (SATIRE) articles and Discontinuation of Randomized Trials (DISCO) articles. We used Scopus (updated July 2016) to search for English-language articles citing each of the eligible index articles with at least 1 subgroup finding in the abstract. Study Selection: Articles with a subgroup claim in the abstract with or without evidence of statistical heterogeneity (P < .05 from an interaction test) in the text and articles attempting to corroborate the subgroup findings. Data Extraction and Synthesis: Study characteristics of trials with at least 1 subgroup claim in the abstract were recorded. Two reviewers extracted the data necessary to calculate subgroup-level effect sizes, standard errors, and the P values for interaction. For individual RCTs and meta-analyses that attempted to corroborate the subgroup findings from the index articles, trial characteristics were extracted. Cochran Q test was used to reevaluate heterogeneity with the data from all available trials. Main Outcomes and Measures: The number of subgroup claims in the abstracts of RCTs, the number of subgroup claims in the abstracts of RCTs with statistical support (subgroup findings), and the number of subgroup findings corroborated by subsequent RCTs and meta-analyses. Results: Sixty-four eligible RCTs made a total of 117 subgroup claims in their abstracts. Of these 117 claims, only 46 (39.3%) in 33 articles had evidence of statistically significant heterogeneity from a test for interaction. In addition, out of these 46 subgroup findings, only 16 (34.8%) ensured balance between randomization groups within the subgroups (eg, through stratified randomization), 13 (28.3%) entailed a prespecified subgroup analysis, and 1 (2.2%) was adjusted for multiple testing. Only 5 (10.9%) of the 46 subgroup findings had at least 1 subsequent pure corroboration attempt by a meta-analysis or an RCT. In all 5 cases, the corroboration attempts found no evidence of a statistically significant subgroup effect. In addition, all effect sizes from meta-analyses were attenuated toward the null. Conclusions and Relevance: A minority of subgroup claims made in the abstracts of RCTs are supported by their own data (ie, a significant interaction effect). For those that have statistical support (P < .05 from an interaction test), most fail to meet other best practices for subgroup tests, including prespecification, stratified randomization, and adjustment for multiple testing. Attempts to corroborate statistically significant subgroup differences are rare; when done, the initially observed subgroup differences are not reproduced.
AI snapshot
Written from the paper’s full text.
TL;DR
This meta-epidemiological survey examined subgroup claims in randomized trial abstracts and found that only a minority were backed by a significant interaction test. Subsequent attempts to corroborate those supported findings were rare and did not reproduce the original subgroup differences.
Key findings
- Among 169 trials with subgroup claims anywhere in the text, 64 (37.9%) made at least one claim in the abstract, yielding 117 abstract-level subgroup claims.
- Only 46 of 117 claims (39.3%) had statistical support from a significant interaction test; among these 46 findings, 16 (34.8%) used a randomization stratification variable, 13 (28.3%) were prespecified, and 1 (2.2%) adjusted for multiple testing.
- Just 5 of the 46 supported findings (10.9%) had at least one pure corroboration attempt by a later trial or meta-analysis; none of these attempts found a significant subgroup interaction, and available effect estimates moved toward the null.
- Modified corroboration attempts were identified for 4 findings in 3 index articles, often using different subgroup definitions or measurements.
Methods
- Design: meta-epidemiological survey using trials with subgroup claims from the SATIRE and DISCO study groups; 169 index articles were screened, and 64 with abstract subgroup claims were included.
- Data sources: Scopus searches updated July 2016 for English-language articles citing eligible index articles; two reviewers extracted data, with three additional reviewers arbitrating discrepancies.
- Analysis: subgroup-level effect sizes and standard errors were calculated where possible, interaction P values were assessed, and heterogeneity was re-evaluated with Cochran Q tests using R, metafor, RevMan, and DerSimonian-Laird random-effects models; P values were two-tailed.
Limitations
- Paper-stated: the follow-up window may have been too short, since SATIRE articles appeared in 2007 and DISCO articles with subgroup findings appeared between 2002 and 2012, and 3 to 4 years may not allow a new trial to publish a corroboration attempt.
- Paper-stated: the authors relied on reported data and the effect measures chosen by original authors, and interaction tests can be sensitive to the effect measure used, which may have affected classification of some findings.
- Paper-stated: the Scopus search may have missed corroboration attempts, particularly trials that did not cite the original articles making similar subgroup claims.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
0 of 34 cited sources checked · 0 supported · 0 flagged
- Findingown data
Of 169 articles with a subgroup effect claimed anywhere in the text, 64 (37.9%) had at least one abstract-level subgroup claim.
“Among the 169 articles with a subgroup effect claimed anywhere in the text, there were 64 articles (37.9%) with at least 1 subgroup claim made in the abstract.”
Quote in the paperNot a substantial claim - Findingown data
Two subgroup findings from the same RCTs had subsequent RCTs investigating similar treatment by insulin interaction status.
“There were 2 findings from the same RCTs that had subsequent RCTs investigating similar treatment by insulin interaction status.”
Quote in the paperNot a substantial claim - Findingown data
Among the 33 claims with a reported interaction P value, 24 (72.7%) were statistically significant.
“We found that most (24 of 33 [72.7%]) of the claims with a reported interaction P value were statistically significant”
- Also stated in: § Discussion · plain
“When interaction P values were reported, they were often (24 of 33 [69.7%]) statistically significant”
Quote in the paperSubstantial claim - Also stated in: § Discussion · plain
- Findingcitation
Two index articles with subgroup findings mentioned and cited previous RCTs that had evaluated similar comparisons, outcomes, and subgroup levels and had statistically significant interaction.
“Two index articles with subgroup findings30,31 mentioned and cited previous RCTs that had evaluated subgroup analyses for similar comparisons, outcomes, and subgroup levels and also had evidence of statistically significant interaction.”
Quote in the paperNot a substantial claim - Findingown data
Of the 117 abstract subgroup claims, 33 (28.2%) had a corresponding interaction P value reported anywhere in the text.
“Among the 117 subgroup claims, there were 33 (28.2%) with a corresponding P value from a statistical test for interaction anywhere in the text”
- Also stated in: § Discussion · plain
“We found that less than one-third (33 of 117 [28.2%]) of the 117 abstract-level subgroup claims had a corresponding P value anywhere in the text.”
Quote in the paperSubstantial claim - Also stated in: § Discussion · plain
- Findingown data
The 64 index articles made 117 individual subgroup claims in their abstracts.
“In these 64 articles, a total of 117 individual abstract subgroup claims were made (Figure).”
Quote in the paperNot a substantial claim - Findingown data
No subsequent trial measured insulin status using insulin concentration at 30 minutes.
“However, none of these RCTs used insulin concentration at 30 minutes as the measurement of insulin status (eTable in the Supplement).”
Quote in the paperNot a substantial claim - Findingown data
Two index articles had citing meta-analyses that attempted corroboration using different subgroup definitions, changing from 3 subgroup levels to 2.
“Two index articles30,32 had citing meta-analyses; in both cases, the corroboration attempts used different subgroup definitions (changing from 3 subgroup levels to 2 subgroup levels).”
Quote in the paperNot a substantial claim - Findingown data
More than half (83 of 117, 70.9%) of the abstract subgroup claims concerned primary outcomes.
“We found that more than half of the subgroup claims made (83 [70.9%]) pertained to primary outcomes.”
Quote in the paperNot a substantial claim - Findingown data
Of the 46 subgroup findings, 13 (28.3%) were prespecified in the abstract, methods, or results of the corresponding RCT.
“For 13 (28.3%) of the 46 subgroup findings, the analyses were listed as prespecified in the abstract, methods, or results sections of the corresponding RCTs.”
Quote in the paperSubstantial claim - Findingown data
Only 1 of the 46 subgroup findings was adjusted for multiple comparisons, using the Bonferroni-Holm step-down procedure.
“Only 1 subgroup finding was adjusted for multiple comparisons (the Bonferroni-Holm step-down procedure).”
Quote in the paperSubstantial claim - Findingcitation
In one of those cases, the outcome of mortality was assessed at a different time point (day 28 instead of day 140).
“However, in one case, the outcome of mortality had been assessed at a different time point (day 28 instead of day 140).33”
Quote in the paperNot a substantial claim - Findingcitation
When effect sizes were available (n=3), the effect sizes were attenuated toward the null.
“When effect sizes were available (n = 3), we found that the effect sizes were attenuated toward the null.23–25”
Quote in the paperNot a substantial claim - Findingown data
Among the 47 claims for which data were extracted to calculate an interaction P value, 18 (38.3%) were statistically significant.
“and that less than half (18 of 47 [38.3%]) of the claims for which data were extracted to calculate a P value for interaction were statistically significant.”
- Also stated in: § Discussion · plain
“but when it was necessary to extract data to evaluate statistical interaction, only a minority (18 of 47 [38.3%]) of these claims were statistically significant.”
Quote in the paperSubstantial claim - Also stated in: § Discussion · plain
- Findingown data
Among the 46 subgroup findings, 5 (10.9%) had at least one subsequent pure corroboration attempt by a meta-analysis or RCT.
“Among the 46 subgroup findings, only 5 (10.9%) had at least 1 subsequent pure corroboration attempt by a meta-analysis or an RCT (Table 3).”
Quote in the paperSubstantial claim - Findingown data
Of the 117 abstract subgroup claims, 47 (40.2%) had extractable data allowing assessment of statistical interaction.
“and 47 (40.2%) with data that could be extracted to assess whether there was statistical interaction.”
Quote in the paperSubstantial claim - Findingown data
Of the 46 subgroup findings, 16 (34.8%) used the subgroup variable as a stratification variable during randomization.
“For 16 of the 46 subgroup findings (34.8%), the subgroup variable was used as a stratification variable during randomization.”
Quote in the paperNot a substantial claim - Findingown data
Subgroup-level effect estimates from meta-analyzed data were generally attenuated toward the null compared with the index article.
“the subgroup-level effect estimates based on meta-analyzed data were generally attenuated toward the null (relative risk, odds ratio, or hazard ratio of 1.0) compared with the index article.”
Quote in the paperSubstantial claim - Findingown data
No corroboration attempt yielded a statistically significant interaction result (P < .05).
“None of the corroboration attempts had P < .05 from an interaction test”
Quote in the paperSubstantial claim - Findingown data
Modified corroboration attempts (different subgroup levels, interventions, outcomes, or population) were found for 4 subgroup findings in 3 index articles.
“We also found modified corroboration attempts (ie, different subgroup levels, interventions, outcomes, or study population) for 4 subgroup findings in 3 index articles.30–32”
- Also stated in: § Limitations · plain
“We also found examples of subsequent studies claiming to be corroboration attempts for subgroup findings from previous studies but which actually performed modified corroboration attempts.”
Quote in the paperSubstantial claim - Also stated in: § Limitations · plain
- Premisecitation
A formal test of interaction is the appropriate statistical approach for establishing a subgroup test.
“The presence of a statistical effect in one subgroup but not the other does not constitute evidence of a subgroup effect, as many authors mistakenly believe; rather, the appropriate statistical approach for establishing a subgroup test is a formal test of interaction.12”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [12] Sainani K Misleading comparisons: the fallacy of comparing statistical significance. PM&R. 2010; 2 (6): 559– 562. 20630442
Not checked in the free run.
- Interpretationown data
Attempts to corroborate statistically significant subgroup findings in subsequent trials and meta-analyses are rare.
“Rarely are attempts made to corroborate statistically significant subgroup findings in subsequent trials and meta-analyses.”
- Also stated in: § Conclusions · hedged
“Furthermore, our research indicates that subsequent meta-analyses and RCTs may rarely attempt to corroborate the subgroup findings prominently reported in RCTs.”
Quote in the paperSubstantial claim - Also stated in: § Conclusions · hedged
- Interpretationown data
The paper provides additional evidence that most subgroup findings reported in abstracts of RCTs are not subsequently corroborated.
“Herein, we provide additional evidence that most subgroup findings reported in abstracts of RCTs are not subsequently corroborated.”
Quote in the paperSubstantial claim - Interpretationown data
Abstract-level subgroup claims in RCTs are often unclear regarding prespecification.
“Subgroup claims reported in the abstracts of RCTs are often vague, unaccompanied by information pertinent to a test for a significant interaction effect, and unclear regardingprespeci-fication.”
Quote in the paperSubstantial claim - Premisecitation
Performing numerous nonprespecified or unadjusted subgroup analyses increases the probability of false-positive findings.
“Second, trials often perform numerous subgroup analyses that are not prespecified or adjusted for multiple testing, which increases the probability of false-positive findings.5,6,9”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [5] Wang R, Lagakos SW, Ware JH, Hunter DJ, Drazen JM. Statistics in medicine: reporting of subgroup analyses in clinical trials. N Engl J Med. 2007; 357(21): 2189– 2194. 18032770
Not checked in the free run.
- cites [6] Assmann SF, Pocock SJ, Enos LE, Kasten LE. Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet. 2000; 355(9209): 1064– 1069. 10744093
Not checked in the free run.
- cites [9] Sun X, Briel M, Busse JW, Credibility of claims ofsubgroup effects in randomised controlled trials: systematic review. BMJ. 2012; 344: e1553. doi: 10.1136/bmj.e1553 22422832
Not checked in the free run.
- Interpretationown data
Interaction testing is not routinely performed or reported.
“Formal testing for interactions is not done (or reported) routinely.”
Quote in the paperSubstantial claim - Interpretationown data
Many abstract-reported subgroup findings do not satisfy other recommended subgroup practices such as prespecification, stratified randomization, and multiple-testing adjustment.
“In addition, most subgroup findings reported in the abstracts of RCTs fail to meet other best practices for subgroup tests, including prespecification, stratified randomization, and adjustment for multiple testing.”
Quote in the paperSubstantial claim - Interpretationown data
Individual subgroup analyses are often spurious.
“Our results support the notion that individual subgroup analyses are often spurious and should be considered hypothesis generating.”
Quote in the paperSubstantial claim - Premisecitation
In most RCT subgroup analyses, basic statistical support for the claims is lacking.
“First, most subgroup analyses in RCTs fail to provide basic statistical support for their claims.5,8–11”
Quote in the paperSubstantial claim0 of 5 sources checked- cites [5] Wang R, Lagakos SW, Ware JH, Hunter DJ, Drazen JM. Statistics in medicine: reporting of subgroup analyses in clinical trials. N Engl J Med. 2007; 357(21): 2189– 2194. 18032770
Not checked in the free run.
- cites [8] Kasenda B, Schandelmaier S, Sun X,; DISCO Study Group. Subgroup analyses in randomised controlled trials: cohort study on trial protocols and journal publications. BMJ. 2014; 349: g4539. 25030633 PMC4100616
Not checked in the free run.
- cites [9] Sun X, Briel M, Busse JW, Credibility of claims ofsubgroup effects in randomised controlled trials: systematic review. BMJ. 2012; 344: e1553. doi: 10.1136/bmj.e1553 22422832
Not checked in the free run.
- cites [10] Rothwell PM. Treating individuals, 2: subgroup analysis in randomised controlled trials: importance, indications, and interpretation. Lancet. 2005; 365(9454): 176– 186. 15639301
Not checked in the free run.
- cites [11] Oxman AD, Guyatt GH. A consumer’s guide to subgroup analyses. Ann Intern Med. 1992; 116(1): 78– 84. 1530753
Not checked in the free run.
- Interpretationnone given
Abstract-level subgroup claims in RCTs are frequently vague.
“Subgroup claims reported in the abstracts of RCTs are often vague, unaccompanied by information pertinent to a test for a significant interaction effect, and unclear regardingprespeci-fication.”
Quote in the paperSubstantial claim - Premisecitation
Most trials do not use within-subgroup randomization, such as stratified randomization.
“Third, most trials fail to randomize participants within subgroups (eg, stratified randomization),5,8,9 which leaves more room for imbalanced confounders between treatment and control arms within subgroups.”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [5] Wang R, Lagakos SW, Ware JH, Hunter DJ, Drazen JM. Statistics in medicine: reporting of subgroup analyses in clinical trials. N Engl J Med. 2007; 357(21): 2189– 2194. 18032770
Not checked in the free run.
- cites [8] Kasenda B, Schandelmaier S, Sun X,; DISCO Study Group. Subgroup analyses in randomised controlled trials: cohort study on trial protocols and journal publications. BMJ. 2014; 349: g4539. 25030633 PMC4100616
Not checked in the free run.
- cites [9] Sun X, Briel M, Busse JW, Credibility of claims ofsubgroup effects in randomised controlled trials: systematic review. BMJ. 2012; 344: e1553. doi: 10.1136/bmj.e1553 22422832
Not checked in the free run.
- Premisecitation
Failure to randomize within subgroups leaves more room for imbalanced confounders between treatment and control arms within subgroups.
“Third, most trials fail to randomize participants within subgroups (eg, stratified randomization),5,8,9 which leaves more room for imbalanced confounders between treatment and control arms within subgroups.”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [5] Wang R, Lagakos SW, Ware JH, Hunter DJ, Drazen JM. Statistics in medicine: reporting of subgroup analyses in clinical trials. N Engl J Med. 2007; 357(21): 2189– 2194. 18032770
Not checked in the free run.
- cites [8] Kasenda B, Schandelmaier S, Sun X,; DISCO Study Group. Subgroup analyses in randomised controlled trials: cohort study on trial protocols and journal publications. BMJ. 2014; 349: g4539. 25030633 PMC4100616
Not checked in the free run.
- cites [9] Sun X, Briel M, Busse JW, Credibility of claims ofsubgroup effects in randomised controlled trials: systematic review. BMJ. 2012; 344: e1553. doi: 10.1136/bmj.e1553 22422832
Not checked in the free run.
- Interpretationown data
Abstract-level subgroup claims in RCTs are often unaccompanied by information pertinent to a test for significant interaction.
“Subgroup claims reported in the abstracts of RCTs are often vague, unaccompanied by information pertinent to a test for a significant interaction effect, and unclear regardingprespeci-fication.”
Quote in the paperSubstantial claim - Premisecitation
Finding a significant effect in one subgroup and not another is not evidence of a subgroup effect.
“The presence of a statistical effect in one subgroup but not the other does not constitute evidence of a subgroup effect, as many authors mistakenly believe; rather, the appropriate statistical approach for establishing a subgroup test is a formal test of interaction.12”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [12] Sainani K Misleading comparisons: the fallacy of comparing statistical significance. PM&R. 2010; 2 (6): 559– 562. 20630442
Not checked in the free run.
- Study design and conductreasoning
The study's Scopus search may not have identified all subgroup corroboration attempts.
“Third, our Scopus search may not have identified all subgroup corroboration attempts.”
Quote in the paperNot a substantial claim
- Classified as probably not a substantial claim. 9
- May combine two claims. 8
- The quote only partly states this claim. 1
Probably not a substantial claim: may not pass S3 (load-bearing).
Findingown dataThe language used to describe prespecification and nonprespecification of subgroup analyses was inconsistent across studies.
“it was evident that the language used to discuss prespecification (ie, preplanned, a priori, previously suggested, planned, and prespecified) and nonprespecification (ie, secondary, explanatory, preliminary, and post hoc) lacked consistency across studies.”
Quote in the paperNot a substantial claimMay combine two claims.
Findingown dataOverall, 46 of 117 (39.3%) abstract subgroup claims had statistical support, defined as a significant interaction P value; equivalently, 71 (60.7%) did not.
“Overall, of the 117 subgroup claims evaluated, only 46 (39.3%) had statistical support (ie, a significant P value for interaction).”
- Also stated in: § Discussion · plain
“Our empirical evaluation of subgroup claims from the abstracts of RCTs revealed that most claims (71 [60.7%] of 117) failed to have underlying evidence of statistical significance based on a test for interaction.”
Quote in the paperSubstantial claim- Also stated in: § Discussion · plain
May combine two claims.
FindingcitationIn the other case, a different study population and different dietary interventions were involved.
“In the other one, a different study population and different dietary interventions were involved.34”
Quote in the paperNot a substantial claimMay combine two claims. The quote only partly states this claim.
Findingown dataNone of the subsequent meta-analyses or individual RCTs successfully corroborated the subgroup findings; when corroboration was attempted, the initially observed subgroup differences were not demonstrated again.
“Moreover, none of the subsequent meta-analyses or individual RCTs successfully corroborated the subgroup findings.”
- Also stated in: § Conclusions · plain
“Moreover, when subgroup corroborations are attempted, the initially observed subgroup differences are not demonstrated again.”
Quote in the paperSubstantial claim- Also stated in: § Conclusions · plain
May combine two claims.
PremisecitationPrevious studies have assessed the credibility of subgroup differences reported anywhere in the text of RCTs but have not focused on those reported in abstracts.
“Previous studies3,8,9 have assessed the credibility of subgroup differences reported anywhere in the text of RCTs but have not focused on those most likely to be credible (ie, those reported in the articles’ abstracts).”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [3] Sun X, Briel M, Busse JW, The influence of study characteristics on reporting of subgroup analyses in randomised controlled trials: systematic review. BMJ. 2011; 342: d1569. 21444636 PMC6173170
Not checked in the free run.
- cites [8] Kasenda B, Schandelmaier S, Sun X,; DISCO Study Group. Subgroup analyses in randomised controlled trials: cohort study on trial protocols and journal publications. BMJ. 2014; 349: g4539. 25030633 PMC4100616
Not checked in the free run.
- cites [9] Sun X, Briel M, Busse JW, Credibility of claims ofsubgroup effects in randomised controlled trials: systematic review. BMJ. 2012; 344: e1553. doi: 10.1136/bmj.e1553 22422832
Not checked in the free run.
May combine two claims.
PremisecitationPrior work indicates that subgroup analyses are frequently conducted and reported poorly.
“Previous research suggests that subgroup analyses are often poorly conducted and reported.5–9”
Quote in the paperSubstantial claim0 of 5 sources checked- cites [5] Wang R, Lagakos SW, Ware JH, Hunter DJ, Drazen JM. Statistics in medicine: reporting of subgroup analyses in clinical trials. N Engl J Med. 2007; 357(21): 2189– 2194. 18032770
Not checked in the free run.
- cites [6] Assmann SF, Pocock SJ, Enos LE, Kasten LE. Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet. 2000; 355(9209): 1064– 1069. 10744093
Not checked in the free run.
- cites [7] Pocock SJ, Assmann SE, Enos LE, Kasten LE. Subgroup analysis, covariate adjustment and baseline comparisons in clinical trial reporting: current practice and problems. StatMed. 2002; 21 (19): 2917– 2930. 12325108
Not checked in the free run.
- cites [8] Kasenda B, Schandelmaier S, Sun X,; DISCO Study Group. Subgroup analyses in randomised controlled trials: cohort study on trial protocols and journal publications. BMJ. 2014; 349: g4539. 25030633 PMC4100616
Not checked in the free run.
- cites [9] Sun X, Briel M, Busse JW, Credibility of claims ofsubgroup effects in randomised controlled trials: systematic review. BMJ. 2012; 344: e1553. doi: 10.1136/bmj.e1553 22422832
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationRecent assessments indicate that nearly 50% of RCT publications include subgroup analyses.
“Recent evaluations of RCTs have found that almost half of the publications report subgroup analyses.8,9”
Quote in the paperNot a substantial claim0 of 2 sources checked- cites [8] Kasenda B, Schandelmaier S, Sun X,; DISCO Study Group. Subgroup analyses in randomised controlled trials: cohort study on trial protocols and journal publications. BMJ. 2014; 349: g4539. 25030633 PMC4100616
Not checked in the free run.
- cites [9] Sun X, Briel M, Busse JW, Credibility of claims ofsubgroup effects in randomised controlled trials: systematic review. BMJ. 2012; 344: e1553. doi: 10.1136/bmj.e1553 22422832
Not checked in the free run.
May combine two claims.
PremisecitationRCTs commonly conduct many subgroup analyses without prespecification or multiple-testing adjustment.
“Second, trials often perform numerous subgroup analyses that are not prespecified or adjusted for multiple testing, which increases the probability of false-positive findings.5,6,9”
Quote in the paperSubstantial claim0 of 3 sources checked- cites [5] Wang R, Lagakos SW, Ware JH, Hunter DJ, Drazen JM. Statistics in medicine: reporting of subgroup analyses in clinical trials. N Engl J Med. 2007; 357(21): 2189– 2194. 18032770
Not checked in the free run.
- cites [6] Assmann SF, Pocock SJ, Enos LE, Kasten LE. Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet. 2000; 355(9209): 1064– 1069. 10744093
Not checked in the free run.
- cites [9] Sun X, Briel M, Busse JW, Credibility of claims ofsubgroup effects in randomised controlled trials: systematic review. BMJ. 2012; 344: e1553. doi: 10.1136/bmj.e1553 22422832
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationPrior work indicates that over a third of published RCT reanalyses reach conclusions different from the original articles.
“Previous research suggests that more than one-third of published reanalyses of RCTs lead to different conclusions than those presented by original articles.35”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [35] Ebrahim S, Sohani ZN, Montoya L, Reanalyses ofrandomized clinical trial data. JAMA. 2014; 312(10): 1024– 1032. 25203082
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
PremisecitationReplication studies are uncommon in the biomedical literature.
“While reproducibility efforts are essential to ensure that trial findings are complete and unbiased, replication studies across the biomedical literature are rare.14”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [14] Iqbal SA, Wallach JD, Khoury MJ, Schully SD, loannidis JP. Reproducible research practices and transparency across the biomedical literature. PLoS Biol. 2016; 14(1): e1002333. PMC4699702 26726926
Not checked in the free run.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductreasoningThe window of opportunity for corroboration may have been too short for some index articles.
“First, it is possible that the window of opportunity for corroboration was too short for some index articles.”
Quote in the paperNot a substantial claimProbably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductnone givenThree additional reviewers arbitrated all potential discrepancies.
“Three additional reviewers arbitrated all potential discrepancies (J.F.T., K.L.S., and J.P.A.I.).”
Quote in the paperNot a substantial claimMay combine two claims.
Study design and conductcitationTests for interaction are influenced by the effect measures considered, which may have influenced the classification of certain subgroup findings as statistically significant.
“Because tests for interaction are influenced by the effect measures considered, this limitation may have influenced the classification of certain subgroup findings as to their statistical significance.36”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [36] Knol MJ, VanderWeele TJ. Recommendations for presenting analyses of effect modification and interaction. IntJ Epidemiol. 2012; 41(2): 514– 520. 22253321 PMC3324457
Not checked in the free run.
Classified as probably not a substantial claim.
Study design and conductown dataTwo reviewers independently screened all index articles to determine the subset with abstract subgroup claims.
“Two reviewers (J.D.W. and P.G.S.) independently screened all index articles (n = 169) to determine the subset of the articles that made subgroup claims in the abstract.”
Quote in the paperSubstantial claimClassified as probably not a substantial claim.
PremisecitationOne-third of RCTs that claimed a subgroup effect for a primary outcome reported a corresponding interaction P value or information allowing calculation of the P value for the primary outcome.
“Furthermore, one-third of RCTs that claimed a subgroup effect for a primary out-come reported a corresponding interaction P value or information that allowed for calculation of the P value for the primary outcome.9”
Quote in the paperSubstantial claim0 of 1 sources checked- cites [9] Sun X, Briel M, Busse JW, Credibility of claims ofsubgroup effects in randomised controlled trials: systematic review. BMJ. 2012; 344: e1553. doi: 10.1136/bmj.e1553 22422832
Not checked in the free run.
May combine two claims.
InterpretationreasoningModifying subgroup analyses in subsequent studies increases the chances of spurious findings and can lead consumers to believe subgroup claims that lack adequate support.
“When subsequent studies modify subgroup analyses, they increase the chances of spurious findings and lead research consumers to believe subgroup claims that actually lack adequate support.”
Quote in the paperSubstantial claim- Rests on an assumption: Interaction tests are sensitive to subgroup definitions, the effect estimates used, and the exact measurements used for subgroup or outcome variables.
Probably not a substantial claim: may not pass S3 (load-bearing).
Study design and conductcitationA third investigator reviewed all subgroup claim classifications and reevaluated tests for interactions applying the Altman and Bland method.
“A third investigator (K.L.S.) reviewed all subgroup claim classifications and reevaluated the test for interactions applying the method by Altman and Bland.21”
Quote in the paperNot a substantial claim0 of 1 sources checked- cites [21] Altman DG, Bland JM. Interaction revisited: the difference between two estimates. BMJ. 2003; 326(7382): 219. 12543843 PMC1125071
Not checked in the free run.
The authors state this as an assumption.
Authors are more careful and selective about reporting subgroup differences in abstracts.
“Presumably, authors are more careful and selective about reporting subgroup differences in abstracts because these claims are most visible to the research community.”
The authors state this as an assumption.
No previous studies have evaluated the credibility of subgroup findings by checking whether they are corroborated.
“Furthermore, to our knowledge, no previous studies have attempted to evaluate the credibility of subgroup findings by checking to see if they are corroborated (eg, new studies producing the same results with the same experimental methods).”
The authors state this as an assumption.
The Scopus search strategy captured most corroboration attempts that could have occurred after the index articles' publication.
“We believe that our search strategy was able to capture most corroboration attempts that could have occurred after the publication date of the index articles.”
The paper relies on this without support.
Interaction tests are sensitive to subgroup definitions, the effect estimates used, and the exact measurements used for subgroup or outcome variables.
“Interaction tests are sensitive to subgroup definitions (ie, 3 group levels or 2 group levels), the effect estimates used (ie, risk ratios or odds ratios), and the exact measurements used for the subgroup or outcome variables.”
The paper relies on this without support.
Subgroup differences reported in abstracts are the most visible ones.
“We focused on subgroup differences that appeared in the abstract of an article, which are the most visible ones.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Subgroup effect claims have a notoriously poor record of validation across medicine.
“However, subgroup effect claims have a notoriously poor record of validation across medicine 246,247 .”
From Preventive psychiatry: a blueprint for improving the mental health of young people (2021) · cites this paper as [247]
Well documented subgroup differences tend to be far less common than claims made for such differences.
“Well documented subgroup differences tend to be far less common than claims made for such differences.443”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [443]
In 2017, Wallach et al demonstrated that 61% of RCTs that claimed subgroup heterogeneity in their abstracts were not supported by their results.
“In 2017, Wallach et al 5 demonstrated that 61% of RCTs that claimed subgroup heterogeneity already in their abstracts (assuming these were most credible) were not supported by their results.”
From Understanding of interaction (subgroup) analysis in clinical trials (2019) · cites this paper as [5]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Incorrect percentage for significant reported interaction tests
MinorThe discussion reports 24 of 33 as 69.7%, but that fraction is 72.7%, as stated earlier in the results.
Reported 69.7 · recomputed 72.727273
“When interaction P values were reported, they were often (24 of 33 [69.7%]) statistically significant”
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505 · see citing works
Self-citationA New Initiative on Precision Medicine
Collins, F. S., & Varmus, H. (2015) · New England Journal of Medicine · doi:10.1056/nejmp1500523 · Cited by 5,202 · see citing works
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · see citing works
Self-citationInteraction revisited: the difference between two estimates.
Altman, D. G., & Bland, J. M. (2003) · BMJ · doi:10.1136/bmj.326.7382.219 · Cited by 3,203 · see citing works
Meta-analysis in clinical trials revisited
DerSimonian, R., & Laird, N. (2015) · Contemporary Clinical Trials · doi:10.1016/j.cct.2015.09.002 · Cited by 2,677 · see citing works
The Path to Personalized Medicine
Hamburg, M. A., & Collins, F. S. (2010) · New England Journal of Medicine · doi:10.1056/nejmp1006304 · Cited by 1,970 · see citing works
Increasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597 · see citing works
Self-citationStatistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials
Wang, R., Lagakos, S. W., Ware, J. H., Hunter, D. J., & Drazen, J. M. (2007) · New England Journal of Medicine · doi:10.1056/nejmsr077003 · Cited by 1,445 · see citing works
Recommendations for presenting analyses of effect modification and interaction
Knol, M. J., & VanderWeele, T. J. (2012) · International Journal of Epidemiology · doi:10.1093/ije/dyr218 · Cited by 1,436 · cited 1× · see citing works
“When the choice was unclear, we calculatedrelativeeffectmeasuresbecausemultiplicativescale interactions are more often assessed and reported based on logistic or Cox proportional hazards regression models in RCTs.(17-19)Anonlinedigitizer (WebPlotDigitizer;http://arohatgi .”
MethodSubgroup analysis and other (mis)uses of baseline data in clinical trials
Assmann, S. F., Pocock, S. J., Enos, L. E., & Kasten, L. E. (2000) · The Lancet · doi:10.1016/s0140-6736(00)02039-0 · Cited by 1,167 · see citing works
Pocock, S. J., Assmann, S. E., Enos, L. E., & Kasten, L. E. (2002) · Statistics in Medicine · doi:10.1002/sim.1296 · Cited by 1,077 · see citing works
Subgroup analysis in randomised controlled trials: importance, indications, and interpretation
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(05)17709-5 · Cited by 942 · see citing works
A consumer's guide to subgroup analyses
Oxman, A. D., & Guyatt, G. H. (1992) · Annals of Internal Medicine · doi:10.7326/0003-4819-116-1-78 · Cited by 914 · see citing works
High-Dose Daunorubicin in Older Patients with Acute Myeloid Leukemia
Löwenberg, B., Ossenkoppele, G. J., van Putten, W., Schouten, H. C., Graux, C., Ferrant, A., Sonneveld, P., Maertens, J. A., Jongen‐Lavrencic, M., von Lilienfeld‐Toal, M., et al. (2009) · New England Journal of Medicine · doi:10.1056/nejmoa0901409 · Cited by 828 · see citing works
BNP-Guided vs Symptom-Guided Heart Failure Therapy
Pfisterer, M. E., Buser, P. T., Rickli, H., Gutmann, M., Erné, P., Rickenbacher, P. R., Vuillomenet, A. G., Jeker, U., Dubach, P., Beer, J. H., et al. (2009) · JAMA · doi:10.1001/jama.2009.2 · Cited by 653 · see citing works
Kim, F., Nichol, G., Maynard, C. C., Hallstrom, A. P., Kudenchuk, P. J., Rea, T. D., Copass, M. K., Carlbom, D., Deem, S. A., Longstreth, W. T., et al. (2013) · JAMA · doi:10.1001/jama.2013.282173 · Cited by 605 · see citing works
Efficacy and Safety of Epoetin Alfa in Critically Ill Patients
Corwin, H. L., Gettinger, A., Fabian, T. C., May, A. K., Pearl, R. G., Heard, S. O., An, R., Bowers, P., Burton, P. B. J., Klausner, M. A., et al. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa071533 · Cited by 546 · see citing works
Efficacy of Recombinant Human Erythropoietin in Critically Ill Patients
Corwin, H. L., Gettinger, A., Pearl, R. G., Fink, M. P., Levy, M. M., Shapiro, M. J., Corwin, M. J., Colton, T., & for the EPO Critical Care Trials Group (2002) · JAMA · doi:10.1001/jama.288.22.2827 · Cited by 494 · see citing works
On the distinction between interaction and effect modification
VanderWeele, T. J. (2009) · Epidemiology · doi:10.1097/ede.0b013e3181ba333c · Cited by 450 · see citing works
Kim, F., Olsufka, M., Longstreth, W. T., et al. (2007) · Circulation · doi:10.1161/circulationaha.106.655480 · Cited by 415 · see citing works
Intra-Arrest Transnasal Evaporative Cooling
Castrén, M. K., Nordberg, P., Svensson, L. A., et al. (2010) · Circulation · doi:10.1161/circulationaha.109.931691 · Cited by 411 · see citing works
Credibility of claims of subgroup effects in randomised controlled trials: systematic review
Sun, X., Briel, M., Busse, J. W., You, J. J., Akl, E. A., Mejza, F., Bała, M. M., Bassler, D., Mertz, D., Diaz-Granados, N., et al. (2012) · BMJ · doi:10.1136/bmj.e1553 · Cited by 394 · see citing works
Reproducible Research Practices and Transparency across the Biomedical Literature
Iqbal, S. A., Wallach, J. D., Khoury, M. J., Schully, S. D., & Ioannidis, J. P. A. (2016) · PLOS Biology · doi:10.1371/journal.pbio.1002333 · Cited by 310 · see citing works
Self-citationSpiegel, D., Butler, L. D., Giese‐Davis, J., Koopman, C., Miller, E., DiMiceli, S., Classen, C. C., Fobair, P., Carlson, R. W., & Kraemer, H. C. (2007) · Cancer · doi:10.1002/cncr.22890 · Cited by 290 · see citing works
Effects of a Low–Glycemic Load vs Low-Fat Diet in Obese Young Adults: A Randomized Trial
Ebbeling, C. B., Leidig, M. M., Feldman, H. A., Lovesky, M. M., & Ludwig, D. S. (2007) · JAMA · doi:10.1001/jama.297.19.2092 · Cited by 284 · see citing works
Dexamethasone in Vietnamese Adolescents and Adults with Bacterial Meningitis
Hoang, N. T., Chau, T. T. H., Thwaites, G. E., Chuong, L. V., Sinh, D. X., Nghia, H. D. T., Tuan, P. Q., Phong, N. D., Phu, N. H., Diep, T., et al. (2007) · New England Journal of Medicine · doi:10.1056/nejmoa070852 · Cited by 231 · see citing works
Four Types of Effect Modification
VanderWeele, T. J., & Robins, J. M. (2007) · Epidemiology · doi:10.1097/ede.0b013e318127181b · Cited by 205 · see citing works
Kissane, D. W., & Li, Y. (2007) · Cancer · doi:10.1002/cncr.23179 · Cited by 198 · see citing works
Reanalyses of Randomized Clinical Trial Data
Ebrahim, S., Sohani, Z. N., Montoya, L., Agarwal, A., Thorlund, K., Mills, E. J., & Ioannidis, J. P. A. (2014) · JAMA · doi:10.1001/jama.2014.9646 · Cited by 186 · see citing works
Self-citationPittas, A. G., Das, S. K., Hajduk, C. L., Golden, J., Saltzman, E., Stark, P. C., Greenberg, A. S., & Roberts, S. B. (2005) · Diabetes Care · doi:10.2337/diacare.28.12.2939 · Cited by 163 · see citing works
Subgroup analyses in therapeutic cardiovascular clinical trials: Are most of them misleading?
Hernández, A. V., Boersma, E., Murray, G. D., Habbema, J. D. F., & Steyerberg, E. W. (2006) · American Heart Journal · doi:10.1016/j.ahj.2005.04.020 · Cited by 131 · see citing works
Self-citationKasenda, B., Schandelmaier, S., Sun, X., von Elm, E. B., You, J., Blümle, A., Tomonaga, Y., Saccilotto, R. T., Amstutz, A., Bengough, T., et al. (2014) · BMJ · doi:10.1136/bmj.g4539 · Cited by 102 · see citing works
Sun, X., Briel, M., Busse, J. W., You, J. J., Akl, E. A., Mejza, F., Bała, M. M., Bassler, D., Mertz, D., Diaz-Granados, N., et al. (2011) · BMJ · doi:10.1136/bmj.d1569 · Cited by 98 · see citing works
Sun, X., Briel, M., Busse, J. W., Akl, E. A., You, J. J., Mejza, F., Bała, M. M., Diaz-Granados, N., Bassler, D., Mertz, D., et al. (2009) · Trials · doi:10.1186/1745-6215-10-101 · Cited by 60 · see citing works
Misleading Comparisons: The Fallacy of Comparing Statistical Significance
Sainani, K. L. (2010) · PM&R · doi:10.1016/j.pmrj.2010.04.016 · Cited by 31 · see citing works
Self-citationHuang, F., Huang, B., Wang, P., et al. (2015) · Resuscitation · doi:10.1016/j.resuscitation.2015.08.005 · Cited by 27 · see citing works
Quantification of Treatment Effect Modification on Both an Additive and Multiplicative Scale
Girerd, N., Rabilloud, M., Pîbarot, P., et al. (2016) · PLoS ONE · doi:10.1371/journal.pone.0153010 · Cited by 19 · see citing works
Hunter, B. R., O’Donnell, D., Allgood, K. L., & Seupaul, R. A. (2014) · Academic Emergency Medicine · doi:10.1111/acem.12342 · Cited by 17 · see citing works
Flagged first, then by citations
6 of the 46 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Schandelmaier, S., Briel, M., Varadhan, R., Schmid, C. H., Devasenapathy, N., Hayward, R. A., Gagnier, J., Borenstein, M., van der Heijden, G. J. M. G., Dahabreh, I. J., et al. (2020) · Canadian Medical Association Journal · doi:10.1503/cmaj.200077 · Cited by 749 · see citing works
Self-citationPreventive psychiatry: a blueprint for improving the mental health of young people
Fusar‐Poli, P., Correll, C. U., Arango, C., Berk, M., Patel, V., & Ioannidis, J. P. A. (2021) · World Psychiatry · doi:10.1002/wps.20869 · Cited by 497 · see citing works
Self-citationPersonalized evidence based medicine: predictive approaches to heterogeneous treatment effects
Kent, D. M., Steyerberg, E. W., & Klaveren, D. V. (2018) · BMJ · doi:10.1136/bmj.k4245 · Cited by 487 · see citing works
Self-citationEndovascular Thrombus Removal for Acute Iliofemoral Deep Vein Thrombosis
Comerota, A. J., Kearon, C., Gu, C., Julian, J. A., Goldhaber, S. Z., Kahn, S. R., Jaff, M. R., Razavi, M. K., Kindzelski, A., Bashir, R., et al. (2019) · Circulation · doi:10.1161/circulationaha.118.037425 · Cited by 321 · see citing works
Kent, D. M., Klaveren, D. V., Paulus, J. K., D’Agostino, R. B., Goodman, S. N., Hayward, R. A., Ioannidis, J. P. A., Patrick‐Lake, B., Morton, S. C., Pencina, M. J., et al. (2019) · Annals of Internal Medicine · doi:10.7326/m18-3668 · Cited by 177 · see citing works
Self-citationRiley, R. D., Debray, T. P. A., Fisher, D. J., Hattle, M., Marlin, N., Hoogland, J., Gueyffier, F., Staessen, J. A., Wang, J., Moons, K. G. M., et al. (2020) · Statistics in Medicine · doi:10.1002/sim.8516 · Cited by 179 · see citing works
SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials
Hróbjartsson, A., Boutron, I., Hopewell, S., Moher, D., Schulz, K. F., Collins, G. S., Tunn, R., Aggarwal, R., Berkwits, M., Berlin, J. A., et al. (2025) · BMJ · doi:10.1136/bmj-2024-081660 · Cited by 171 · see citing works
Self-citationReducing bias, increasing transparency and calibrating confidence with preregistration
Hardwicke, T. E., & Wagenmakers, E. (2023) · Nature Human Behaviour · doi:10.1038/s41562-022-01497-2 · Cited by 190 · see citing works
Understanding of interaction (subgroup) analysis in clinical trials
Brankovic, M., Kardys, I., Steyerberg, E. W., Lemeshow, S., Markovic, M., Rizopoulos, D., & Boersma, E. (2019) · European Journal of Clinical Investigation · doi:10.1111/eci.13145 · Cited by 130 · see citing works
Self-citationDennis, J. (2020) · Diabetes · doi:10.2337/dbi20-0002 · Cited by 120 · see citing works
Klaveren, D. V., Balan, T. A., Steyerberg, E. W., & Kent, D. M. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.05.029 · Cited by 88 · see citing works
Self-citationCalibrating the Scientific Ecosystem Through Meta-Research
Hardwicke, T. E., Serghiou, S., Janiaud, P., Danchev, V., Crüwell, S., Goodman, S. N., & Ioannidis, J. P. A. (2020) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-031219-041104 · Cited by 98 · see citing works
Self-citationPembrolizumab plus Chemotherapy in Lung Cancer
Gil-Sierra, M. D., Fenix-Caballero, S., & Rey, E. J. A. D. (2018) · New England Journal of Medicine · doi:10.1056/nejmc1808567 · Cited by 87 · see citing works
Fustolo‐Gunnink, S. F., Fijnvandraat, K., Klaveren, D. V., Stanworth, S. J., Curley, A., Onland, W., Steyerberg, E. W., de Kort, E. H. M., d’Haens, E. J., Hulzebos, C. V., et al. (2019) · Blood · doi:10.1182/blood.2019000899 · Cited by 76 · see citing works
Self-citationBasu, S., Raghavan, S., Wexler, D. J., & Berkowitz, S. A. (2017) · Diabetes Care · doi:10.2337/dc17-2252 · Cited by 73 · see citing works
Basu, S., Sussman, J. B., Rigdon, J., Steimle, L., Denton, B. T., & Hayward, R. A. (2017) · PLOS Medicine · doi:10.1371/journal.pmed.1002410 · Cited by 77 · see citing works
Donevant, S. B., Dawson, R. M., Culley, J. M., Habing, B., & Adams, S. A. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocy104 · Cited by 64 · see citing works
Kearon, C., Gu, C., Julian, J. A., et al. (2019) · Thrombosis and Haemostasis · doi:10.1055/s-0039-1677795 · Cited by 69 · see citing works
An urgent call to raise the bar in oncology
Schnog, J. B., Samson, M. J., Gans, R. O. B., & Duits, A. J. (2021) · British Journal of Cancer · doi:10.1038/s41416-021-01495-7 · Cited by 63 · see citing works
Sinha, P., Spicer, A. B., Delucchi, K., McAuley, D. F., Calfee, C. S., & Churpek, M. M. (2021) · EBioMedicine · doi:10.1016/j.ebiom.2021.103697 · Cited by 58 · see citing works
Reducing bias, increasing transparency, and calibrating confidence with preregistrationPreprint
Hardwicke, T. E., & Wagenmakers, E. (2021) · doi:10.31222/osf.io/d7bcu · Cited by 55 · see citing works
Ioannidis, J. P. A. (2018) · Social Science & Medicine · doi:10.1016/j.socscimed.2018.04.029 · Cited by 54 · see citing works
Self-citationSex-related differences in the pharmacological treatment of heart failure
Tamargo, J. L., Caballero, R., & Delpón, E. (2021) · Pharmacology & Therapeutics · doi:10.1016/j.pharmthera.2021.107891 · Cited by 44 · see citing works
Griffin, S. J. (2022) · Diabetologia · doi:10.1007/s00125-022-05782-7 · Cited by 42 · see citing works
How effects on health equity are assessed in systematic reviews of interventions
Welch, V., Dewidar, O., Ghogomu, E. T., Abdisalam, S., Ameer, A. A., Barbeau, V. I., Brand, K., Kebedom, K., Benkhalti, M., Kristjansson, E., et al. (2022) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000028.pub3 · Cited by 65 · see citing works
Liang, F., Wu, Z., Mo, M., Zhou, C., Shen, J., Wang, Z., & Zheng, Y. (2019) · European Journal of Cancer · doi:10.1016/j.ejca.2019.08.006 · Cited by 36 · see citing works
Puljak, L., Makarić, Z. L., Buljan, I., & Pieper, D. (2020) · Journal of Comparative Effectiveness Research · doi:10.2217/cer-2019-0201 · Cited by 32 · see citing works
Does evidence support the high expectations placed in precision medicine? A bibliographic review
Cortés, J., González, J. A., Medina, M. N., Vogler, M., Vilaró, M., Elmore, M., Senn, S. J., Campbell, M. J., & Cobo, E. (2019) · F1000Research · doi:10.12688/f1000research.13490.5 · Cited by 31 · see citing works
Fan, J., Song, F., & Bachmann, M. O. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.12.009 · Cited by 31 · see citing works
Schuit, E., Li, A. H., & Ioannidis, J. P. A. (2019) · International Journal of Epidemiology · doi:10.1093/ije/dyy239 · Cited by 27 · see citing works
Self-citationSafety and efficacy of probiotic administration to preterm infants: ten common questions
Underwood, M. A., Umberger, E., & Patel, R. M. (2020) · Pediatric Research · doi:10.1038/s41390-020-1080-6 · Cited by 33 · see citing works
Checklist for clinical applicability of subgroup analysis
Gil‐Sierra, M. D., Fenix-Caballero, S., Martín, L. A., Fraga‐Fuentes, M. D., Sánchéz‐Hidalgo, M., de la Lastra‐Romero, C. A., & Rey, E. J. A. D. (2019) · Journal of Clinical Pharmacy and Therapeutics · doi:10.1111/jcpt.13102 · Cited by 30 · see citing works
Lipkovich, I. A., Svensson, D. J., Ratitch, B., & Dmitrienko, A. A. (2024) · Statistics in Medicine · doi:10.1002/sim.10167 · Cited by 33 · see citing works
A chronicle of the 17-alpha hydroxyprogesterone caproate story to prevent recurrent preterm birth
Nelson, D. B., McIntire, D. D., & Leveno, K. J. (2020) · American Journal of Obstetrics and Gynecology · doi:10.1016/j.ajog.2020.09.045 · Cited by 26 · see citing works
Clinical data mining: challenges, opportunities, and recommendations for translational applications
Qiao, H., Chen, Y., Qian, C., & Guo, Y. (2024) · Journal of Translational Medicine · doi:10.1186/s12967-024-05005-0 · Cited by 27 · see citing works
Schandelmaier, S., Chang, Y., Devasenapathy, N., Devji, T., Kwong, J. S. W., Colunga‐Lozano, L. E., Lee, Y., Agarwal, A., Bhatnagar, N., Ewald, H., et al. (2019) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.05.014 · Cited by 24 · see citing works
Lawson, D. O., Leenus, A., & Mbuagbaw, L. C. E. (2020) · Pilot and Feasibility Studies · doi:10.1186/s40814-019-0544-0 · Cited by 22 · see citing works
Jaspers, N. E. M., Ridker, P. M., Dorresteijn, J. A. N., & Visseren, F. L. J. (2018) · Current Opinion in Lipidology · doi:10.1097/mol.0000000000000554 · Cited by 24 · see citing works
Lipkovich, I. A., Svensson, D. J., Ratitch, B., & Dmitrienko, A. A. (2023) · Clinical Trials · doi:10.1177/17407745231174544 · Cited by 21 · see citing works
Kent, D. M. (2023) · Clinical Trials · doi:10.1177/17407745231171897 · Cited by 21 · see citing works
Leisman, D. E., Handisides, D. R., Chawla, L. S., Albertson, T. E., Busse, L. W., Boldt, D. W., Deane, A. M., Gong, M. N., Ham, K. R., Khanna, A. K., et al. (2023) · Annals of Intensive Care · doi:10.1186/s13613-023-01227-5 · Cited by 23 · see citing works
Access to data from clinical trials in the COVID-19 crisis: open, flexible, and time-sensitive
Ewers, M., Ioannidis, J. P. A., & Plesnila, N. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.10.008 · Cited by 22 · see citing works
Self-citationNew strategies for the development of lipid-lowering therapies to reduce cardiovascular risk
Graham, I. M., Shear, C. L., de Graeff, P., Boulton, C., Catapano, A. L., Stough, W. G., Carlsson, S. C., Backer, G. D., Emmerich, J., Greenfeder, S. A., et al. (2017) · European Heart Journal - Cardiovascular Pharmacotherapy · doi:10.1093/ehjcvp/pvx031 · Cited by 21 · see citing works
Research, regulatory, and clinical decision-making: the importance of scientific integrity
Wallach, J. D., Gonsalves, G., & Ross, J. S. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.08.021 · Cited by 20 · see citing works
Self-citationMultiplicity corrections in life sciences: challenges and consequences
Menyhárt, O., & Győrffy, B. (2025) · International Journal of Epidemiology · doi:10.1093/ije/dyaf098 · Cited by 24 · see citing works
Raghavan, S., Josey, K., Bahn, G., Reda, D. J., Basu, S., Berkowitz, S. A., Emanuele, N., Reaven, P. D., & Ghosh, D. (2021) · Annals of Epidemiology · doi:10.1016/j.annepidem.2021.07.003 · Cited by 22 · see citing works
Djulbegović, B., & Ioannidis, J. P. A. (2018) · European Journal of Clinical Investigation · doi:10.1111/eci.13031 · Cited by 20 · see citing works
Self-citationBiałek, M., & Fugelsang, J. A. (2019) · Journal of Multilingual and Multicultural Development · doi:10.1080/01434632.2018.1547072 · Cited by 20 · see citing works
Does evidence support the high expectations placed in precision medicine? A bibliographic review
Cortés, J., González, J. A., Medina, M. N., Vogler, M., Vilaró, M., Elmore, M., Senn, S. J., Campbell, M. J., & Cobo, E. (2018) · F1000Research · doi:10.12688/f1000research.13490.1 · Cited by 19 · see citing works
Erhel, F., Scanff, A., & Naudet, F. (2020) · Epidemiology and Psychiatric Sciences · doi:10.1017/s2045796020000359 · Cited by 19 · see citing works
Sledge, G. W., & Frenzel, M. (2020) · JAMA Oncology · doi:10.1001/jamaoncol.2020.1518 · Cited by 17 · see citing works
A systematic review of subgroup analyses in randomised clinical trials in cardiovascular disease
Brand, K., Hapfelmeier, A., & Haller, B. (2021) · Clinical Trials · doi:10.1177/1740774520984866 · Cited by 19 · see citing works
Expanding our perspectives on research in musculoskeletal science and practice
Kerry, R. (2017) · Musculoskeletal Science and Practice · doi:10.1016/j.msksp.2017.10.004 · Cited by 16 · see citing works
Age-treatment subgroup analyses in Cochrane intervention reviews: a meta-epidemiological study
Liu, P., Ioannidis, J. P. A., Ross, J. S., Dhruva, S. S., Luxkaranayagam, A. T., Vasiliou, V., & Wallach, J. D. (2019) · BMC Medicine · doi:10.1186/s12916-019-1420-8 · Cited by 18 · see citing works
Self-citationGLP-1RA precision medicine in people with type 2 diabetes: current insights and future prospects
Cardoso, P., Dennis, J., & Pearson, E. R. (2026) · Journal of Clinical Investigation · doi:10.1172/jci194742 · Cited by 16 · see citing works
Seidu, S., Sathanapally, H., Kunutsor, S. K., & Khunti, K. K. (2023) · Journal of the Royal Society of Medicine · doi:10.1177/01410768231156997 · Cited by 16 · see citing works
Emilien, C. H., Hsu, W. H., & Hollis, J. (2020) · Nutrients · doi:10.3390/nu12113341 · Cited by 13 · see citing works
Pemetrexed in Nonsquamous Non–Small Cell Lung Cancer
Gyawali, B., & Prasad, V. K. (2017) · JAMA Oncology · doi:10.1001/jamaoncol.2017.1944 · Cited by 13 · see citing works
Calibrating the scientific ecosystem through meta-research
(2019) · doi:10.31222/osf.io/krb58 · Cited by 97 · see citing works
Self-citationSame Old Challenges in Subgroup Analysis—Should We Do More About Methods Implementation?
Schandelmaier, S., & Guyatt, G. H. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.3339 · Cited by 14 · see citing works
Does evidence support the high expectations placed in precision medicine? A bibliographic review
Cortés, J., González, J. A., Medina, M. N., Vogler, M., Vilaró, M., Elmore, M., Senn, S. J., Campbell, M., & Cobo, E. (2019) · F1000Research · doi:10.12688/f1000research.13490.4 · Cited by 13 · see citing works
Investigación en el enfermo crítico. Dificultades y perspectivas
Latour-Pérez, J. (2017) · Medicina Intensiva · doi:10.1016/j.medin.2017.07.008 · Cited by 10 · see citing works
How Not to Fool Ourselves About Heterogeneity of Treatment Effects
von Hippel, P. V., & Schuetze, B. A. (2025) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459241304347 · Cited by 20 · see citing works
Re-Evaluating Evidence for Best Practice in Paediatric Speech-Language Pathology
Dodd, B. (2021) · Folia Phoniatrica et Logopaedica · doi:10.1159/000505265 · Cited by 12 · see citing works
Assessing heterogeneity of treatment effect in multiple sclerosis trials
Sormani, M. P., Chataway, J., Kent, D. M., & Marrie, R. A. (2023) · Multiple Sclerosis Journal · doi:10.1177/13524585231189673 · Cited by 12 · see citing works
Dual Immune Checkpoint Inhibition in Melanoma and PD-L1 Expression: The Jury Is Still Out
Schmitt, A. M., Larkin, J., & Patel, S. P. (2024) · Journal of Clinical Oncology · doi:10.1200/jco-24-01572 · Cited by 11 · see citing works
The Challenges of Generating Evidence to Support Precision Medicine
Pletcher, M. J., & McCulloch, C. E. (2017) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2016.9138 · Cited by 11 · see citing works
Buckley, P. R., Gust, C. J., Coffin, S. G., Aikawa, S. M., Steeger, C. M., & Pampel, F. C. (2025) · Prevention Science · doi:10.1007/s11121-025-01765-3 · Cited by 10 · see citing works
Zeraatkar, D., Ling, M., Kirsh, S., Jassal, T., Pitre, T., Chakraborty, S. P., Turner, T., Turkstra, L. S., McIntyre, R. S., Izcovich, A., et al. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-086407 · Cited by 13 · see citing works
The Myth of the Placebo Response
Jonas, W. B. (2019) · Frontiers in Psychiatry · doi:10.3389/fpsyt.2019.00577 · Cited by 10 · see citing works
Bryce, Y. C. D., Katzen, B. T., Patel, P. J., Moreira, C., Fakorede, F. A., Arya, S., D’Andrea, M., Mustapha, J. A., Rowe, V. L., Rosenfield, K., et al. (2022) · Journal of Vascular and Interventional Radiology · doi:10.1016/j.jvir.2022.02.010 · Cited by 9 · see citing works
Evidence-based Urology: Subgroup Analysis in Randomized Controlled Trials
Kilpeläinen, T. P., Tikkinen, K. A. O., Guyatt, G. H., & Vernooij, R. W. M. (2021) · European Urology Focus · doi:10.1016/j.euf.2021.10.001 · Cited by 9 · see citing works
Wolf, J. M., Koopmeiners, J. S., & Vock, D. M. (2022) · Clinical Trials · doi:10.1177/17407745221095855 · Cited by 9 · see citing works
Clinical research in critical care. Difficulties and perspectives
Latour-Pérez, J. (2018) · Medicina Intensiva (English Edition) · doi:10.1016/j.medine.2017.07.001 · Cited by 9 · see citing works
Raghavan, S., Liu, W. G., Saxon, D. R., Grunwald, G. K., Maddox, T. M., Reusch, J. E. B., Berkowitz, S. A., & Caplan, L. (2018) · BMJ Open Diabetes Research & Care · doi:10.1136/bmjdrc-2018-000516 · Cited by 7 · see citing works
Does evidence support the high expectations placed in precision medicine? A bibliographic review
Cortés, J., González, J. A., Medina, M. N., Vogler, M., Vilaró, M., Elmore, M., Senn, S. J., Campbell, M. J., & Cobo, E. (2018) · F1000Research · doi:10.12688/f1000research.13490.3 · Cited by 7 · see citing works
Klap, R. S., & Humphreys, K. N. (2019) · Women s Health Issues · doi:10.1016/j.whi.2019.05.002 · Cited by 7 · see citing works
Carland, C., Hansra, B., Parsons, C., Lyubarova, R., & Khandelwal, A. (2022) · American Heart Journal Plus Cardiology Research and Practice · doi:10.1016/j.ahjo.2022.100155 · Cited by 6 · see citing works
Hannigan, L. J., Phillippo, D., Hanlon, P., Moss, L., Butterly, E. W., Hawkins, N., Dias, S., Welton, N. J., & McAllister, D. (2022) · White Rose Research Online (University of Leeds, The University of Sheffield, University of York) · Cited by 6 · see citing works
Tewarie, I. A., Hulsbergen, A. F. C., Volovici, V., & Broekman, M. L. D. (2020) · Neurosurgical FOCUS · doi:10.3171/2020.8.focus20597 · Cited by 6 · see citing works
Kastrati, L., Farina, S., Gris, A. V., Raeisi‐Dehkordi, H., Llanaj, E., Quezada‐Pinedo, H. G., Bally, L., Muka, T., & Ioannidis, J. P. A. (2025) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2024-113359 · Cited by 5 · see citing works
Self-citationWallach, J. D., Glick, L. R., Gueorguieva, R. V., & O’Malley, S. S. (2023) · Alcoholism Clinical and Experimental Research · doi:10.1111/acer.15229 · Cited by 5 · see citing works
Self-citationSubgroup discovery in non‐inferiority trials
(2021) · doi:10.1002/sim.9118 · Cited by 5 · see citing works
Shetty, V. U., Brotherton, B. J., Achilleos, A., Akrami, K., Barros, L. M., Checkley, W., Cobb, N. L., Maximous, S. I., Misango, D., Park, C., et al. (2020) · American Journal of Tropical Medicine and Hygiene · doi:10.4269/ajtmh.20-1106 · Cited by 4 · see citing works
The credibility of subgroup analyses reported in stroke trials is low: A systematic review
Ademola, A., Thabane, L., Adekanye, J., Okikiolu, A., Babatunde, S., Almekhlafi, M., Menon, B. K., Hill, M. D., Hildebrand, K. A., & Sajobi, T. T. (2023) · International Journal of Stroke · doi:10.1177/17474930231168517 · Cited by 5 · see citing works
(2017) · doi:10.1016/j.ajog.2017.07.013 · Cited by 4 · see citing works
How health equity is reported and analyzed in randomized trials
Petkovic, J., Welch, V. A., Jull, J., Petticrew, M. P., Kristjansson, E., Rader, T., Yoganathan, M., McGowan, J., Lyddiatt, A., Grimshaw, J., et al. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000046 · Cited by 4 · see citing works
Tamargo, J. L., Aggarwal, N. R., & Tamargo, M. (2021) · Elsevier eBooks · doi:10.1016/b978-0-12-819369-3.00020-4 · Cited by 4 · see citing works
Subgroup Analysis in Pulmonary Hypertension-Specific Therapy Clinical Trials: A Systematic Review
Ramallo, H. R., Gutiérrez, N. B., Otero, R., & Martín, L. A. (2022) · Journal of Personalized Medicine · doi:10.3390/jpm12060863 · Cited by 3 · see citing works
Preventable sources of bias in subgroup analyses and secondary outcomes of randomized trials
Núñez, I., & Belaunzarán-Zamudio, P. F. (2024) · Contemporary Clinical Trials · doi:10.1016/j.cct.2024.107641 · Cited by 3 · see citing works
Andersson, J., Smidt, M. F. D., Sundström, A. E., Nordin, S., Wennberg, P., & Näslund, U. (2025) · BMC Public Health · doi:10.1186/s12889-025-24076-2 · Cited by 4 · see citing works
Sechidis, K., Sun, S. B., Chen, Y., Lu, J., Zhang, C., Baillie, M., Ohlssen, D. I., Vandemeulebroecke, M., Hemmings, R., Ruberg, S. J., et al. (2024) · Pharmaceutical Statistics · doi:10.1002/pst.2463 · Cited by 4 · see citing works
Gu, B., Zhou, Y., Nie, Y., Wang, L., Liang, L., Liao, Z., Wen, J., Guan, X., Chen, M., Wu, J., et al. (2025) · Frontiers in Cellular and Infection Microbiology · doi:10.3389/fcimb.2025.1673959 · Cited by 3 · see citing works
- Authors
- Wallach, J. D., Sullivan, P. G., Trepanowski, J. F., Sainani, K. L., Steyerberg, E. W., & Ioannidis, J. P. A.
- Affiliations
- Stanford Medicine (US)
- Erasmus MC (NL)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- JAMA Internal Medicine
- Publisher
- American Medical Association (AMA)
- Published
- 2017-04-01
- Volume
- 177
- Issue
- 4
- Pages
- 554
- ISSN
- 2168-6106 (ISSN-L)
- PMC
- PMC6657347
- OpenAlex
- W2595252694
- Semantic Scholar
- 283fdd6ebbe8bc714a69dd1198f26c06bc2631bc
- Cited by
- 148
- References
- 46
- Citations by year
- 2026: 9 · 2025: 17 · 2024: 13 · 2023: 9 · 2022: 14 · 2021: 13 · 2020: 21 · 2019: 20 · 2018: 18 · 2017: 14
- FWCI
- 17.86
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- JAMA Internal Medicine (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Funders
- National Institutes of Health (UL1TR001085)
- National Heart, Lung, and Blood Institute (T32HL007034)
- National Institute of Neurological Disorders and Stroke (U01NS086294)
- National Center for Advancing Translational Sciences
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Life Sciences › Pharmacology, Toxicology and Pharmaceutics › Pharmacology › Pharmaceutical industry and healthcare
- Keywords
- subgroup analysis, interaction testing, statistical heterogeneity, meta-epidemiology, randomized controlled trials, stratified randomization, multiple testing correction, publication bias, meta-analysis, trial discontinuation, satire, disco
- MeSH terms
- Data Interpretation, Statistical, Diagnosis-Related Groups, Humans, Publishing, Randomized Controlled Trials as Topic, Evidence-Based Medicine, Intention to Treat Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-06.
Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-06.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
