Assessing Heterogeneity of Treatment Effects: Are Authors Misinterpreting Their Results?
Open accessThe DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Abstract
Objective. To determine whether investigations of heterogeneity of treatment effects (HTE) in randomized‐controlled trials (RCTs) are prespecified and whether authors' interpretations of their analyses are consistent with the objective evidence. Data Sources/Study Setting. Trials published in Annals of Internal Medicine, British Medical Journal, Journal of the American Medical Association, Lancet, and New England Journal of Medicine in 1994, 1999, and 2004. Study Design. We reviewed 87 RCTs that reported formal tests for statistical interaction or heterogeneity (HTE analyses), derived from a probability sample of 541 articles. Data Collection/Extraction. We recorded reasons for performing HTE analysis; an objective classification of evidence for HTE (termed “clinicostatistical divergence” [CSD]); and authors' interpretations of findings. Authors' interpretations, compared with CSD, were coded as understated, overstated, or adequately stated. Principle Findings. Fifty‐three RCTs (61 percent) claimed prespecified covariates for HTE analyses. Trials showed strong (6), moderate (11), weak (25), or negligible (16) evidence for CSD (29 could not be classified due to inadequate information). Authors stated that evidence for HTE was sufficient to support differential treatment in subgroups (10); warranted more research (31); was absent (21); or provided no interpretation (25). HTE was overstated in 22 trials, adequately stated in 57 trials, and understated in 8 trials. Conclusions. Inconsistencies in performance and reporting may limit the potential of HTE analysis as a tool for identifying HTE and individualizing care in diverse populations. Recommendations for future studies on the reporting and interpretation of HTE analyses are provided.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The number needed to treat: a clinically useful measure of treatment effect
Cook, R. J., & Sackett, D. L. (1995) · BMJ · doi:10.1136/bmj.310.6977.452 · Cited by 1,885 · cited 10×
“After triallevel inclusion and exclusion criteria were applied, we identified 319 eligible RCTs reported in 303 articles among the initial random sample of 379 articles.”
MethodInfluentialStatistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials
Wang, R., Lagakos, S. W., Ware, J. H., Hunter, D. J., & Drazen, J. M. (2007) · New England Journal of Medicine · doi:10.1056/nejmsr077003 · Cited by 1,445
Drakulovic, M. B., Torres, A., Bauer, T. T., Nicolas, J. M., Nogué, S., & Ferrer, M. (1999) · The Lancet · doi:10.1016/s0140-6736(98)12251-1 · Cited by 1,243 · cited 2×
“As an example of this classification scheme, consider the trial reported by Drakulovic et al. (1999), a comparison of semirecumbent (intervention group) versus supine (control or usual care group) body position for prevention of clinically suspected nosocomial pneumonia (primary outcome) in…”
Pocock, S. J., Assmann, S. E., Enos, L. E., & Kasten, L. E. (2002) · Statistics in Medicine · doi:10.1002/sim.1296 · Cited by 1,077
Yusuf, S. (1991) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1991.03470010097038 · Cited by 1,014 · cited 5×
“Although recommendations for the design, reporting, and interpretation of analyses for HTE have been published (Yusuf et al. 1991; Altman et al. 2001; Lu et al. 2005; Rothwell 2005, 2007; Wang et al. 2007), actual practice does not reflect the recommendations (Assmann et al. 2000; Parker and Naylor…”
InfluentialSubgroup analysis in randomised controlled trials: importance, indications, and interpretation
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(05)17709-5 · Cited by 942
Subgroup analyses in randomized trials: risks of subgroup-specific analyses;
Brookes, S. T., Whitely, E., Egger, M., Smith, G. D., Mulheran, P. A., & Peters, T. J. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2003.08.009 · Cited by 775
Calculating the number needed to treat for trials where the outcome is time to an event
Altman, D. G., & Andersen, P. K. (1999) · BMJ · doi:10.1136/bmj.319.7223.1492 · Cited by 710 · cited 1×
“…acknowledging the possible need for a clinically relevant and standardized measure of effect size when comparing different RCTs (Cook and Sackett 1995; Altman and Andersen 1999; Kraemer and Kupfer 2006), we attempted to calculate number needed to treat (NNT) for each trial based on reported data.”
Brookes, S. T., Whitley, E., Peters, T. J., Mulheran, P. A., Egger, M., & Davey Smith, G. (2001) · Health Technology Assessment · doi:10.3310/hta5330 · Cited by 580
The Challenge of Subgroup Analyses — Reporting without Distorting
Lagakos, S. W. (2006) · New England Journal of Medicine · doi:10.1056/nejmp068070 · Cited by 504 · cited 1×
“…of an explicit rationale); appropriate statistical analysis (through use of formal tests for interaction or heterogeneity); and balanced interpretation of the evidence (Bulpitt 1988; Yusuf et al. 1991; Altman et al. 2001; Lu et al. 2005; Rothwell 2005, 2007; Lagakos 2006; Wang et al. 2007).”
Cited by
‘Spin’ in published biomedical literature: A methodological systematic review
Chiu, K., Grundy, Q., & Bero, L. (2017) · PLOS Biology · doi:10.1371/journal.pbio.2002173 · Cited by 317
Carter, G. C., Cantrell, R. A., Zarotsky, V., Haynes, V. S., Phillips, G., Alatorre, C. I., Goetz, I., Paczkowski, R., & Marangell, L. B. (2012) · Depression and Anxiety · doi:10.1002/da.21918 · Cited by 109
Personalizing behavioral interventions through single-patient (N-of-1) trials
Davidson, K. W., Peacock, J., Kronish, I. M., & Edmondson, D. (2014) · Social and Personality Psychology Compass · doi:10.1111/spc3.12121 · Cited by 86
Müller, S. R., Chen, X., Peters, H., Chaintreau, A., & Matz, S. C. (2021) · Scientific Reports · doi:10.1038/s41598-021-93087-x · Cited by 51
“…field of machine learning 43,44 , because it allows predictive models to leverage the unique patterns between input and output variables observed for each subsample, thereby avoiding issues arising from generalization bias across groups (see 45 for a review of subgroup effects in clinical trials).”
Background
Metadata
- Authors
- Fernandez y Garcia, E., Nguyen, H., Duan, N., Gabler, N. B., & Kravitz, R. L.
- Venue
- Health Services Research
- Publisher
- Wiley
- Published
- 2010-01-08
- Volume
- 45
- Issue
- 1
- Pages
- 283-301
- ISSN
- 0017-9124 (ISSN-L)
- 1475-6773
- OpenAlex
- W2160540948
- Cited by
- 23
- References
- 12
- FWCI
- 0.80
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Assessing Heterogeneity of Treatment Effects: Are Authors Misinterpreting Their Results? https://app.cochat.ai/papers/10.1111/j.1475-6773.2009.01064.x
- The paper
- Fernandez y Garcia, E., Nguyen, H., Duan, N., Gabler, N. B., & Kravitz, R. L. (2010). Assessing Heterogeneity of Treatment Effects: Are Authors Misinterpreting Their Results? Health Services Research. https://doi.org/10.1111/j.1475-6773.2009.01064.x
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.