The DOI matches its Crossref record; 40 of the 46 references it cites were found in a registry or an index; none of the 34 sources its claims cite is checked for support yet.
Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials
Open accessAbstract
Importance: Many published randomized clinical trials (RCTs) make claims for subgroup differences. Objective: To evaluate how often subgroup claims reported in the abstracts of RCTs are actually supported by statistical evidence (P < .05 from an interaction test) and corroborated by subsequent RCTs and meta-analyses. Data Sources: This meta-epidemiological survey examines data sets of trials with at least 1 subgroup claim, including Subgroup Analysis of Trials Is Rarely Easy (SATIRE) articles and Discontinuation of Randomized Trials (DISCO) articles. We used Scopus (updated July 2016) to search for English-language articles citing each of the eligible index articles with at least 1 subgroup finding in the abstract. Study Selection: Articles with a subgroup claim in the abstract with or without evidence of statistical heterogeneity (P < .05 from an interaction test) in the text and articles attempting to corroborate the subgroup findings. Data Extraction and Synthesis: Study characteristics of trials with at least 1 subgroup claim in the abstract were recorded. Two reviewers extracted the data necessary to calculate subgroup-level effect sizes, standard errors, and the P values for interaction. For individual RCTs and meta-analyses that attempted to corroborate the subgroup findings from the index articles, trial characteristics were extracted. Cochran Q test was used to reevaluate heterogeneity with the data from all available trials. Main Outcomes and Measures: The number of subgroup claims in the abstracts of RCTs, the number of subgroup claims in the abstracts of RCTs with statistical support (subgroup findings), and the number of subgroup findings corroborated by subsequent RCTs and meta-analyses. Results: Sixty-four eligible RCTs made a total of 117 subgroup claims in their abstracts. Of these 117 claims, only 46 (39.3%) in 33 articles had evidence of statistically significant heterogeneity from a test for interaction. In addition, out of these 46 subgroup findings, only 16 (34.8%) ensured balance between randomization groups within the subgroups (eg, through stratified randomization), 13 (28.3%) entailed a prespecified subgroup analysis, and 1 (2.2%) was adjusted for multiple testing. Only 5 (10.9%) of the 46 subgroup findings had at least 1 subsequent pure corroboration attempt by a meta-analysis or an RCT. In all 5 cases, the corroboration attempts found no evidence of a statistically significant subgroup effect. In addition, all effect sizes from meta-analyses were attenuated toward the null. Conclusions and Relevance: A minority of subgroup claims made in the abstracts of RCTs are supported by their own data (ie, a significant interaction effect). For those that have statistical support (P < .05 from an interaction test), most fail to meet other best practices for subgroup tests, including prespecification, stratified randomization, and adjustment for multiple testing. Attempts to corroborate statistically significant subgroup differences are rare; when done, the initially observed subgroup differences are not reproduced.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
This meta-epidemiological survey examined subgroup claims in randomized trial abstracts and found that only a minority were backed by a significant interaction test. Subsequent attempts to corroborate those supported findings were rare and did not reproduce the original subgroup differences.
Key findings
- Among 169 trials with subgroup claims anywhere in the text, 64 (37.9%) made at least one claim in the abstract, yielding 117 abstract-level subgroup claims.
- Only 46 of 117 claims (39.3%) had statistical support from a significant interaction test; among these 46 findings, 16 (34.8%) used a randomization stratification variable, 13 (28.3%) were prespecified, and 1 (2.2%) adjusted for multiple testing.
- Just 5 of the 46 supported findings (10.9%) had at least one pure corroboration attempt by a later trial or meta-analysis; none of these attempts found a significant subgroup interaction, and available effect estimates moved toward the null.
- Modified corroboration attempts were identified for 4 findings in 3 index articles, often using different subgroup definitions or measurements.
Methods
- Design: meta-epidemiological survey using trials with subgroup claims from the SATIRE and DISCO study groups; 169 index articles were screened, and 64 with abstract subgroup claims were included.
- Data sources: Scopus searches updated July 2016 for English-language articles citing eligible index articles; two reviewers extracted data, with three additional reviewers arbitrating discrepancies.
- Analysis: subgroup-level effect sizes and standard errors were calculated where possible, interaction P values were assessed, and heterogeneity was re-evaluated with Cochran Q tests using R, metafor, RevMan, and DerSimonian-Laird random-effects models; P values were two-tailed.
Limitations
- Paper-stated: the follow-up window may have been too short, since SATIRE articles appeared in 2007 and DISCO articles with subgroup findings appeared between 2002 and 2012, and 3 to 4 years may not allow a new trial to publish a corroboration attempt.
- Paper-stated: the authors relied on reported data and the effect measures chosen by original authors, and interaction tests can be sensitive to the effect measure used, which may have affected classification of some findings.
- Paper-stated: the Scopus search may have missed corroboration attempts, particularly trials that did not cite the original articles making similar subgroup claims.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
0 of 34 cited sources checked · 0 supported · 0 flagged
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among the 169 articles with a subgroup effect claimed anywhere in the text, there were 64 articles (37.9%) with at least 1 subgroup claim made in the abstract.”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Results > Search Findings
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“There were 2 findings from the same RCTs that had subsequent RCTs investigating similar treatment by insulin interaction status.”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Results > Corroboration of Subgroup Findings
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that most (24 of 33 [72.7%]) of the claims with a reported interaction P value were statistically significant”
- Also stated in: § Discussion · plain
“When interaction P values were reported, they were often (24 of 33 [69.7%]) statistically significant”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Results > Frequency and Characteristics of Subgroup Claims
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Also stated in: § Discussion · plain
None of the sources cited for this claim have been checked yet.
In the paper
“Two index articles with subgroup findings30,31 mentioned and cited previous RCTs that had evaluated subgroup analyses for similar comparisons, outcomes, and subgroup levels and also had evidence of statistically significant interaction.”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Results > Corroboration of Subgroup Findings
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Among the 117 subgroup claims, there were 33 (28.2%) with a corresponding P value from a statistical test for interaction anywhere in the text”
- Also stated in: § Discussion · plain
“We found that less than one-third (33 of 117 [28.2%]) of the 117 abstract-level subgroup claims had a corresponding P value anywhere in the text.”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Results > Frequency and Characteristics of Subgroup Claims
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes- Also stated in: § Discussion · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In these 64 articles, a total of 117 individual abstract subgroup claims were made (Figure).”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Results > Search Findings
Strength:plain· not yet calibratedn = 117Tests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, none of these RCTs used insulin concentration at 30 minutes as the measurement of insulin status (eTable in the Supplement).”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Results > Corroboration of Subgroup Findings
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Two index articles30,32 had citing meta-analyses; in both cases, the corroboration attempts used different subgroup definitions (changing from 3 subgroup levels to 2 subgroup levels).”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Results > Corroboration of Subgroup Findings
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We found that more than half of the subgroup claims made (83 [70.9%]) pertained to primary outcomes.”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Results > Frequency and Characteristics of Subgroup Claims
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesA result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For 13 (28.3%) of the 46 subgroup findings, the analyses were listed as prespecified in the abstract, methods, or results sections of the corresponding RCTs.”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Results > Frequency and Characteristics of Subgroup Findings
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passes
- Probably not a substantial claim. 9
- It may state two separate claims as one. 8
- The quote only partly states this claim. 1
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“it was evident that the language used to discuss prespecification (ie, preplanned, a priori, previously suggested, planned, and prespecified) and nonprespecification (ie, secondary, explanatory, preliminary, and post hoc) lacked consistency across studies.”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Results > Frequency and Characteristics of Subgroup Findings
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, of the 117 subgroup claims evaluated, only 46 (39.3%) had statistical support (ie, a significant P value for interaction).”
- Also stated in: § Discussion · plain
“Our empirical evaluation of subgroup claims from the abstracts of RCTs revealed that most claims (71 [60.7%] of 117) failed to have underlying evidence of statistical significance based on a test for interaction.”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Results > Frequency and Characteristics of Subgroup Claims
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Also stated in: § Discussion · plain
None of the sources cited for this claim have been checked yet.
In the paper
“In the other one, a different study population and different dietary interventions were involved.34”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Results > Corroboration of Subgroup Findings
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, none of the subsequent meta-analyses or individual RCTs successfully corroborated the subgroup findings.”
- Also stated in: § Conclusions · plain
“Moreover, when subgroup corroborations are attempted, the initially observed subgroup differences are not demonstrated again.”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Discussion
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one. The quote only partly states this claim.
- Also stated in: § Conclusions · plain
None of the sources cited for this claim have been checked yet.
In the paper
“Previous studies3,8,9 have assessed the credibility of subgroup differences reported anywhere in the text of RCTs but have not focused on those most likely to be credible (ie, those reported in the articles’ abstracts).”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Abstract > CONCLUSIONS AND RELEVANCE
What the sources say
- Not checked yetcites [3] Sun X, Briel M, Busse JW, The influence of study characteristics on reporting of subgroup analyses in randomised controlled trials: systematic review. BMJ. 2011; 342: d1569. 21444636 PMC6173170
- Not checked yetcites [8] Kasenda B, Schandelmaier S, Sun X,; DISCO Study Group. Subgroup analyses in randomised controlled trials: cohort study on trial protocols and journal publications. BMJ. 2014; 349: g4539. 25030633 PMC4100616
- Not checked yetcites [9] Sun X, Briel M, Busse JW, Credibility of claims ofsubgroup effects in randomised controlled trials: systematic review. BMJ. 2012; 344: e1553. doi: 10.1136/bmj.e1553 22422832
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Previous research suggests that subgroup analyses are often poorly conducted and reported.5–9”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Abstract > CONCLUSIONS AND RELEVANCE
What the sources say
- Not checked yetcites [5] Wang R, Lagakos SW, Ware JH, Hunter DJ, Drazen JM. Statistics in medicine: reporting of subgroup analyses in clinical trials. N Engl J Med. 2007; 357(21): 2189– 2194. 18032770
- Not checked yetcites [6] Assmann SF, Pocock SJ, Enos LE, Kasten LE. Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet. 2000; 355(9209): 1064– 1069. 10744093
- Not checked yetcites [7] Pocock SJ, Assmann SE, Enos LE, Kasten LE. Subgroup analysis, covariate adjustment and baseline comparisons in clinical trial reporting: current practice and problems. StatMed. 2002; 21 (19): 2917– 2930. 12325108
- Not checked yetcites [8] Kasenda B, Schandelmaier S, Sun X,; DISCO Study Group. Subgroup analyses in randomised controlled trials: cohort study on trial protocols and journal publications. BMJ. 2014; 349: g4539. 25030633 PMC4100616
- Not checked yetcites [9] Sun X, Briel M, Busse JW, Credibility of claims ofsubgroup effects in randomised controlled trials: systematic review. BMJ. 2012; 344: e1553. doi: 10.1136/bmj.e1553 22422832
Load-bearingStrength:hedged· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
None of the sources cited for this claim have been checked yet.
In the paper
“Recent evaluations of RCTs have found that almost half of the publications report subgroup analyses.8,9”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Discussion
What the sources say
- Not checked yetcites [8] Kasenda B, Schandelmaier S, Sun X,; DISCO Study Group. Subgroup analyses in randomised controlled trials: cohort study on trial protocols and journal publications. BMJ. 2014; 349: g4539. 25030633 PMC4100616
- Not checked yetcites [9] Sun X, Briel M, Busse JW, Credibility of claims ofsubgroup effects in randomised controlled trials: systematic review. BMJ. 2012; 344: e1553. doi: 10.1136/bmj.e1553 22422832
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
None of the sources cited for this claim have been checked yet.
In the paper
“Second, trials often perform numerous subgroup analyses that are not prespecified or adjusted for multiple testing, which increases the probability of false-positive findings.5,6,9”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Abstract > CONCLUSIONS AND RELEVANCE
What the sources say
- Not checked yetcites [5] Wang R, Lagakos SW, Ware JH, Hunter DJ, Drazen JM. Statistics in medicine: reporting of subgroup analyses in clinical trials. N Engl J Med. 2007; 357(21): 2189– 2194. 18032770
- Not checked yetcites [6] Assmann SF, Pocock SJ, Enos LE, Kasten LE. Subgroup analysis and other (mis)uses of baseline data in clinical trials. Lancet. 2000; 355(9209): 1064– 1069. 10744093
- Not checked yetcites [9] Sun X, Briel M, Busse JW, Credibility of claims ofsubgroup effects in randomised controlled trials: systematic review. BMJ. 2012; 344: e1553. doi: 10.1136/bmj.e1553 22422832
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
In the paper
“Previous research suggests that more than one-third of published reanalyses of RCTs lead to different conclusions than those presented by original articles.35”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Discussion
What the sources say
- Not checked yetcites [35] Ebrahim S, Sohani ZN, Montoya L, Reanalyses ofrandomized clinical trial data. JAMA. 2014; 312(10): 1024– 1032. 25203082
Strength:hedged· not yet calibratedCausal:direct· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“While reproducibility efforts are essential to ensure that trial findings are complete and unbiased, replication studies across the biomedical literature are rare.14”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Discussion
What the sources say
- Not checked yetcites [14] Iqbal SA, Wallach JD, Khoury MJ, Schully SD, loannidis JP. Reproducible research practices and transparency across the biomedical literature. PLoS Biol. 2016; 14(1): e1002333. PMC4699702 26726926
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“First, it is possible that the window of opportunity for corroboration was too short for some index articles.”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Discussion > Limitations
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Three additional reviewers arbitrated all potential discrepancies (J.F.T., K.L.S., and J.P.A.I.).”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Methods > Identification of RCTs With Subgroup Claims
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Because tests for interaction are influenced by the effect measures considered, this limitation may have influenced the classification of certain subgroup findings as to their statistical significance.36”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Discussion > Limitations
What the sources say
- Not checked yetcites [36] Knol MJ, VanderWeele TJ. Recommendations for presenting analyses of effect modification and interaction. IntJ Epidemiol. 2012; 41(2): 514– 520. 22253321 PMC3324457
Strength:hedged· not yet calibratedCausal:conditional· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes? Load-bearing: unclear✓ Based: passesIt may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Two reviewers (J.D.W. and P.G.S.) independently screened all index articles (n = 169) to determine the subset of the articles that made subgroup claims in the abstract.”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Methods > Identification of RCTs With Subgroup Claims
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
In the paper
“Furthermore, one-third of RCTs that claimed a subgroup effect for a primary out-come reported a corresponding interaction P value or information that allowed for calculation of the P value for the primary outcome.9”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Discussion
What the sources say
- Not checked yetcites [9] Sun X, Briel M, Busse JW, Credibility of claims ofsubgroup effects in randomised controlled trials: systematic review. BMJ. 2012; 344: e1553. doi: 10.1136/bmj.e1553 22422832
Load-bearingStrength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesProbably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When subsequent studies modify subgroup analyses, they increase the chances of spurious findings and lead research consumers to believe subgroup claims that actually lack adequate support.”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Discussion > Limitations
Tests✓ Asserted: passes✓ Checkable: passes✓ Load-bearing: passes✓ Based: passesIt may state two separate claims as one.
- Rests on an assumption: Interaction tests are sensitive to subgroup definitions, the effect estimates used, and the exact measurements used for subgroup or outcome variables.
In the paper
“A third investigator (K.L.S.) reviewed all subgroup claim classifications and reevaluated the test for interactions applying the method by Altman and Bland.21”
§ Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials > Methods > Statistical Analysis
What the sources say
- Not checked yetcites [21] Altman DG, Bland JM. Interaction revisited: the difference between two estimates. BMJ. 2003; 326(7382): 219. 12543843 PMC1125071
Strength:plain· not yet calibratedTests✓ Asserted: passes✓ Checkable: passes✗ Load-bearing: fails✓ Based: passesProbably not a substantial claim. The paper’s conclusions may not depend on it.
The authors state this as an assumption.
Authors are more careful and selective about reporting subgroup differences in abstracts.
“Presumably, authors are more careful and selective about reporting subgroup differences in abstracts because these claims are most visible to the research community.”
The authors state this as an assumption.
No previous studies have evaluated the credibility of subgroup findings by checking whether they are corroborated.
“Furthermore, to our knowledge, no previous studies have attempted to evaluate the credibility of subgroup findings by checking to see if they are corroborated (eg, new studies producing the same results with the same experimental methods).”
The authors state this as an assumption.
The Scopus search strategy captured most corroboration attempts that could have occurred after the index articles' publication.
“We believe that our search strategy was able to capture most corroboration attempts that could have occurred after the publication date of the index articles.”
The paper relies on this without support.
Interaction tests are sensitive to subgroup definitions, the effect estimates used, and the exact measurements used for subgroup or outcome variables.
“Interaction tests are sensitive to subgroup definitions (ie, 3 group levels or 2 group levels), the effect estimates used (ie, risk ratios or odds ratios), and the exact measurements used for the subgroup or outcome variables.”
The paper relies on this without support.
Subgroup differences reported in abstracts are the most visible ones.
“We focused on subgroup differences that appeared in the abstract of an article, which are the most visible ones.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“However, subgroup effect claims have a notoriously poor record of validation across medicine 246,247 .”
From Preventive psychiatry: a blueprint for improving the mental health of young people (2021) · cites this paper as [247]
In the citing paper
“Well documented subgroup differences tend to be far less common than claims made for such differences.443”
From SPIRIT 2025 explanation and elaboration: updated guideline for protocols of randomised trials (2025) · cites this paper as [443]
In the citing paper
“In 2017, Wallach et al 5 demonstrated that 61% of RCTs that claimed subgroup heterogeneity already in their abstracts (assuming these were most credible) were not supported by their results.”
From Understanding of interaction (subgroup) analysis in clinical trials (2019) · cites this paper as [5]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Incorrect percentage for significant reported interaction tests
MinorThe discussion reports 24 of 33 as 69.7%, but that fraction is 72.7%, as stated earlier in the results.
Reported 69.7 · recomputed 72.727273
“When interaction P values were reported, they were often (24 of 33 [69.7%]) statistically significant”
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationA New Initiative on Precision Medicine
Collins, F. S., & Varmus, H. (2015) · New England Journal of Medicine · doi:10.1056/nejmp1500523 · Cited by 5,202
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702
Self-citationInteraction revisited: the difference between two estimates.
Altman, D. G., & Bland, J. M. (2003) · BMJ · doi:10.1136/bmj.326.7382.219 · Cited by 3,203
Meta-analysis in clinical trials revisited
DerSimonian, R., & Laird, N. (2015) · Contemporary Clinical Trials · doi:10.1016/j.cct.2015.09.002 · Cited by 2,677
The Path to Personalized Medicine
Hamburg, M. A., & Collins, F. S. (2010) · New England Journal of Medicine · doi:10.1056/nejmp1006304 · Cited by 1,970
Increasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597
Self-citationRecommendations for presenting analyses of effect modification and interaction
Knol, M. J., & VanderWeele, T. J. (2012) · International Journal of Epidemiology · doi:10.1093/ije/dyr218 · Cited by 1,494 · cited 1×
“When the choice was unclear, we calculatedrelativeeffectmeasuresbecausemultiplicativescale interactions are more often assessed and reported based on logistic or Cox proportional hazards regression models in RCTs.(17-19)Anonlinedigitizer (WebPlotDigitizer;http://arohatgi .”
MethodStatistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials
Wang, R., Lagakos, S. W., Ware, J. H., Hunter, D. J., & Drazen, J. M. (2007) · New England Journal of Medicine · doi:10.1056/nejmsr077003 · Cited by 1,445
Subgroup analysis and other (mis)uses of baseline data in clinical trials
Assmann, S. F., Pocock, S. J., Enos, L. E., & Kasten, L. E. (2000) · The Lancet · doi:10.1016/s0140-6736(00)02039-0 · Cited by 1,167
Flagged first, then by citations
6 of the 46 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Items 101–149 of 149
Subgroup analysis in randomized controlled trials: Useful or misleading?
Tritschler, T., Sadeghipour, P., & Bikdeli, B. (2022) · Thrombosis Research · doi:10.1016/j.thromres.2022.10.009 · Cited by 2
Heterogeneity in associations between food insecurity and diabetes outcomes
Berkowitz, S. A., Gao, M., Ochoa, A., Kuhn, M. L., Dankovchik, J., Donovan, J., LaPoint, M., Basu, S., Hudgens, M. G., & Gold, R. B. (2025) · Journal of Epidemiology & Community Health · doi:10.1136/jech-2025-224037 · Cited by 2
Kastrati, L., Farina, S., Gris, A. V., Raeisi‐Dehkordi, H., Llanaj, E., Quezada‐Pinedo, H. G., Bally, L., Muka, T., & Ioannidis, J. P. A. (2024) · medRxiv · doi:10.1101/2024.07.04.24309572 · Cited by 2
Self-citationSaver, J. L., & Hankey, G. J. (2020) · Cambridge University Press eBooks · doi:10.1017/9781316286234.003 · Cited by 2
Nordbeck, P., Göker-Alpan, Ö., Bernat, J. A., Germain, D. P., Giraldo, P., Jovanović, A., Kimonis, V. E., Nicholls, K., Greenberg, C. R., Schiffmann, R., et al. (2026) · Nature Communications · doi:10.1038/s41467-025-68256-5 · Cited by 10
Herman, K. C., Wiedermann, W., & Reinke, W. M. (2025) · Journal of School Psychology · doi:10.1016/j.jsp.2025.101505 · Cited by 3
Clinical and pharmacoeconomic impact of subgroup analysis in onco-hematological patients
Gil‐Sierra, M. D., Briceño-Casado, M. D. P., & Sánchéz‐Hidalgo, M. (2022) · Supportive Care in Cancer · doi:10.1007/s00520-022-06823-y · Cited by 3
An Evolutionary Medicine Perspective on Treatment of Pediatric Functional Abdominal Pain
Leontiadis, G. I., & Longstreth, G. F. (2020) · The American Journal of Gastroenterology · doi:10.14309/ajg.0000000000001024 · Cited by 2
Interaction Tests With Covariate‐Adaptive Randomization
Zhang, L., & Ma, W. (2025) · Statistical Analysis and Data Mining The ASA Data Science Journal · doi:10.1002/sam.70003 · Cited by 2
Gianfrancesco, M. A., Trupin, L. S., McCulloch, C. E., SHIBOSKI, S. C., Schmajuk, G., & Yazdany, J. (2019) · ACR Open Rheumatology · doi:10.1002/acr2.1017 · Cited by 2
Understanding heterogeneity of pragmatic trial results and subgroup analyses: The FIAT example
Raymond, J., & Darsaut, T. E. (2025) · Neurochirurgie · doi:10.1016/j.neuchi.2025.101683 · Cited by 2
Ali, M., Dekker, L., Ali, M., van Zwet, E. W., Hofmeijer, J., Nederkoorn, P. J., Majoie, C. B. L. M., van Es, A. C., Uyttenboogaart, M., van der Meij, A., et al. (2024) · Journal of NeuroInterventional Surgery · doi:10.1136/jnis-2023-021086 · Cited by 1
Gil‐Sierra, M. D., Briceño-Casado, M. D. P., & Fenix-Caballero, S. (2020) · JAMA Oncology · doi:10.1001/jamaoncol.2020.1516 · Cited by 2
Shan, G., & Cummings, J. L. (2025) · Alzheimer s & Dementia Translational Research & Clinical Interventions · doi:10.1002/trc2.70186 · Cited by 2
No evidence for decreased foreign language effect in highly proficient and acculturated bilingualsPreprint
Białek, M., & Fugelsang, J. A. (2018) · doi:10.31234/osf.io/zwyhf · Cited by 1
Empirical Bayes estimators in hierarchical models with mixture priors
Rosenkranz, G. (2018) · doi:10.1080/02664763.2018.1450364 · Cited by 1
Does evidence support the high expectations placed in precision medicine? A bibliographic review
Cortés, J., González, J. A., Medina, M. N., Vogler, M., Vilaró, M., Elmore, M., Senn, S. J., Campbell, M. J., & Cobo, E. (2018) · F1000Research · doi:10.12688/f1000research.13490.2 · Cited by 1
(2017) · doi:10.3399/bjgp17x690365 · Cited by 1
Salvage Treatment After Radical Prostatectomy
Parker, C. C., & Sydes, M. R. (2017) · European Urology · doi:10.1016/j.eururo.2017.07.025 · Cited by 1
Peat, G., Jordan, K. P., Wilkie, R., van der Windt, D., Yu, D., Narle, G., & Ali, N. (2022) · Journal of Public Health · doi:10.1093/pubmed/fdac014 · Cited by 1
Zeraatkar, D., Ling, M., Kirsh, S., et al. (2024) · medRxiv · doi:10.1101/2024.03.15.24304168 · Cited by 1
Martínez-Puig, P., Gutiérrez, N. B., Ramallo, H. R., Martín, L. A., & Otero, R. (2025) · PLoS ONE · doi:10.1371/journal.pone.0318739 · Cited by 1
Shamliyan, T., Avanesova, A. A., & Aronow, W. S. (2020) · Vessel Plus · doi:10.20517/2574-1209.2020.34
Robbins-Welty, G. A., & Riordan, P. A. (2022) · Springer eBooks · doi:10.1007/978-3-030-94960-0_41
Furukawa, Y., Siafis, S., Schneider‐Thoma, J., Nomura, N., Fares‐Otero, N. E., Illing, S., Storosum, B. W. C., Zantvoord, J. B., Schoretsanitis, G., Schwarz, C., et al. (2026) · The Lancet Psychiatry · doi:10.1016/s2215-0366(26)00280-4
Treatment Effect Moderation with Small Subgroups: An Incremental Subgroup Analysis Approach
Liu, X., Eddy, J. M., & Martínez, C. R. (2026) · Multivariate Behavioral Research · doi:10.1080/00273171.2026.2634993
Schandelmaier, S. (2019) · MacSphere (McMaster University) · doi:10.71548/14968
Credible Subgroup Effects in Randomized Clinical Trials
de Grooth, H., Yehya, N., & Harhay, M. O. (2025) · American Journal of Respiratory and Critical Care Medicine · doi:10.1164/rccm.202502-0411ed
Riley, R. D., Tierney, J. F., & Stewart, L. A. (2021) · Wiley eBooks · doi:10.1002/9781119333784.refs2
Suen, S., Scheinker, D., & Enns, E. A. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108872188.004
Schandelmaier, S., Schmitt, A. M., Herbrand, A. K., Glinz, D., Ewald, H., Briel, M., Guyatt, G. H., Hemkens, L. G., & Kasenda, B. (2020) · BMJ Open · doi:10.1136/bmjopen-2019-034565
Duan, T., & Basu, S. (2022) · Cambridge University Press eBooks · doi:10.1017/9781108872188.006
Clinical Trials: Handling the Data
Swords, D. S., & Brooke, B. S. (2020) · Success in academic surgery · doi:10.1007/978-3-030-35488-6_10
Buckley, P. R., Gust, C. J., Coffin, S. G., Aikawa, S. M., Steeger, C. M., & Pampel, F. C. (2024) · Research Square · doi:10.21203/rs.3.rs-4784450/v1
Understanding the Distributional Aspects of Microcredit ExpansionsPreprint
Weeks, M., & Christiansen, T. G. (2020) · RePEc: Research Papers in Economics · doi:10.48550/arxiv.2011.10509
Rosenkranz, G. (2019) · Wiley eBooks · doi:10.1002/9781119536734.ref
Dupuis, C., Perrin, H., Morel, E., & Martin, T. (2025) · International Journal Of Pharmaceutical Research And Allied Sciences · doi:10.51847/2vx9iuavw1
Comparing Methods to Assess Treatment Effect Heterogeneity in General Parametric Regression Models
Chen, Y., Sun, S. B., Sechidis, K., Zhang, C., Hothorn, T., & Bornkamp, B. (2026) · Statistics in Medicine · doi:10.1002/sim.70381
Statistical innovation: Transforming pharmaceutical research and development
Kulkarni, P. M., Alemayehu, D., Qu, Y., Backenroth, D., Millen, B. A., Fu, H., der Elst, W. V., Roychoudhury, S., Price, K., Chen, T., et al. (2025) · Handbook of statistics · doi:10.1016/bs.host.2025.04.005
Packer, M., Schiattarella, G. G., Petrie, M. C., Burkhoff, D., Butler, J., Lam, C. S. P., Zannad, F. M., Vaduganathan, M., & Borlaug, B. A. (2026) · Circulation · doi:10.1161/circulationaha.126.079235 · Cited by 1
Can the Analysis Bias the Findings?
Crombie, I. (2021) · Wiley eBooks · doi:10.1002/9781394322183.ch4
Donevant, S. B. (2018) · Scholar Commons (University of South Carolina)
Subgroup Analysis in Pulmonary Hypertension-specific Therapy Trials: a Systematic ReviewPreprint
Ramallo, H. R., Gutiérrez, N. B., Candelera, R. O., Flores-Moreno, S., & Martín, L. A.-K. (2021) · Research Square · doi:10.21203/rs.3.rs-879986/v1
Metadata
- Authors
- Wallach, J. D., Sullivan, P. G., Trepanowski, J. F., Sainani, K. L., Steyerberg, E. W., & Ioannidis, J. P. A.
- Affiliations
- Stanford Medicine (US)
- Erasmus MC (NL)
- Corresponding author
- Ioannidis, John P. A.
- Venue
- JAMA Internal Medicine
- Publisher
- American Medical Association (AMA)
- Published
- 2017-04-01
- Volume
- 177
- Issue
- 4
- Pages
- 554
- ISSN
- 2168-6106 (ISSN-L)
- PMC
- PMC6657347
- OpenAlex
- W2595252694
- Semantic Scholar
- 283fdd6ebbe8bc714a69dd1198f26c06bc2631bc
- Cited by
- 148
- References
- 46
- Citations by year
- 2026: 9 · 2025: 17 · 2024: 13 · 2023: 9 · 2022: 14 · 2021: 13 · 2020: 21 · 2019: 20 · 2018: 18 · 2017: 14
- FWCI
- 17.86
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- JAMA Internal Medicine (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion)
- EUR Research Repository (Erasmus University Rotterdam) (submittedVersion)
- Funders
- National Institutes of Health (UL1TR001085)
- National Heart, Lung, and Blood Institute (T32HL007034)
- National Institute of Neurological Disorders and Stroke (U01NS086294)
- National Center for Advancing Translational Sciences
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Health Sciences › Nursing › Issues, ethics and legal aspects › Pharmaceutical industry and healthcare
- Keywords
- subgroup analysis, interaction testing, statistical heterogeneity, meta-epidemiology, randomized controlled trials, stratified randomization, multiple testing correction, publication bias, meta-analysis, trial discontinuation, satire, disco
- MeSH terms
- Data Interpretation, Statistical, Diagnosis-Related Groups, Humans, Publishing, Randomized Controlled Trials as Topic, Evidence-Based Medicine, Intention to Treat Analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
