The DOI matches its Crossref record; 75 of the 75 references it cites were found in a registry or an index.
Assessing and reporting heterogeneity in treatment effects in clinical trials: a proposal
Open accessAbstract
Mounting evidence suggests that there is frequently considerable variation in the risk of the outcome of interest in clinical trial populations. These differences in risk will often cause clinically important heterogeneity in treatment effects (HTE) across the trial population, such that the balance between treatment risks and benefits may differ substantially between large identifiable patient subgroups; the "average" benefit observed in the summary result may even be non-representative of the treatment effect for a typical patient in the trial. Conventional subgroup analyses, which examine whether specific patient characteristics modify the effects of treatment, are usually unable to detect even large variations in treatment benefit (and harm) across risk groups because they do not account for the fact that patients have multiple characteristics simultaneously that affect the likelihood of treatment benefit. Based upon recent evidence on optimal statistical approaches to assessing HTE, we propose a framework that prioritizes the analysis and reporting of multivariate risk-based HTE and suggests that other subgroup analyses should be explicitly labeled either as primary subgroup analyses (well-motivated by prior evidence and intended to produce clinically actionable results) or secondary (exploratory) subgroup analyses (performed to inform future research). A standardized and transparent approach to HTE assessment and reporting could substantially improve clinical trial utility and interpretability.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“RCTs provide estimates of the between-subject treatment response (the average difference between the two groups) but does not provide sufficient data on how a specific individual responds to a given treatment i.e. due to heterogeneity in treatment effects [21].”
The cited source says what the claim attributes to it.
In the citing paper
“RCTs provide estimates of the between-subject treatment response (the average difference between the two groups) but does not provide sufficient data on how a specific individual responds to a given treatment i.e. due to heterogeneity in treatment effects [21].”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020124 · Cited by 10,505
Self-citationValidation of a combined comorbidity index
Charlson, M., Szatrowski, T. P., Peterson, J., & Gold, J. (1994) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)90129-5 · Cited by 6,587
Gage, B. F., Waterman, A. D., Shannon, W., Boechler, M., Rich, M. W., & Radford, M. J. (2001) · JAMA · doi:10.1001/jama.285.22.2864 · Cited by 4,832
Why Most Published Research Findings Are False
Ioannidis, J. P. A. (2005) · CHANCE · doi:10.1080/09332480.2005.10722754 · Cited by 4,702 · cited 1×
“Additionally, because conventional subgroup analyses involve multiple comparisons and involve splitting the overall sample to smaller sub-samples, they are both under-powered for detecting genuine subgroup effects (prone to false-negatives), and even more commonly they are prone to false positive findings [25-31].”
BackgroundSelf-citationThe TIMI Risk Score for Unstable Angina/Non–ST Elevation MI
Antman, E. M., Cohen, M. C., Bernink, P. J. L. M., McCabe, C. H., Horacek, T., Papuchis, G., Mautner, B., Corbalán, R., Radley, D. R., & Braunwald, E. (2000) · JAMA · doi:10.1001/jama.284.7.835 · Cited by 3,221
Adjusting for multiple testing—when and how?
Bender, R., & Lange, S. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00314-0 · Cited by 3,168
Yusuf, S., Zucker, D. M., Passamani, E. R., Peduzzi, P. N., Takaro, T., Fisher, L. D., Kennedy, J. W., Davis, K. B., Killip, T., Norris, R. M., et al. (1994) · The Lancet · doi:10.1016/s0140-6736(94)91963-1 · Cited by 2,125
Cannon, C. P., Weintraub, W. S., Demopoulos, L. A., Vicari, R. M., Frey, M. J., Lakkis, N. M., Neumann, F., Robertson, D. H., DeLucca, P. T., DiBattiste, P. M., et al. (2001) · New England Journal of Medicine · doi:10.1056/nejm200106213442501 · Cited by 2,093
(2005) · The Lancet · doi:10.1016/s0140-6736(05)67660-x · Cited by 1,858
Statistics in Medicine — Reporting of Subgroup Analyses in Clinical Trials
Wang, R., Lagakos, S. W., Ware, J. H., Hunter, D. J., & Drazen, J. M. (2007) · New England Journal of Medicine · doi:10.1056/nejmsr077003 · Cited by 1,445
Subgroup analysis and other (mis)uses of baseline data in clinical trials
Assmann, S. F., Pocock, S. J., Enos, L. E., & Kasten, L. E. (2000) · The Lancet · doi:10.1016/s0140-6736(00)02039-0 · Cited by 1,167
Pocock, S. J., Assmann, S. E., Enos, L. E., & Kasten, L. E. (2002) · Statistics in Medicine · doi:10.1002/sim.1296 · Cited by 1,077
Yusuf, S. (1991) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1991.03470010097038 · Cited by 1,014
Subgroup analysis in randomised controlled trials: importance, indications, and interpretation
Rothwell, P. M. (2005) · The Lancet · doi:10.1016/s0140-6736(05)17709-5 · Cited by 942
Self-citationEarly versus delayed invasive intervention in acute coronary syndromes
Mehta, S. R., Granger, C. B., Boden, W. E., Steg, P. G., Bassand, J.-P., Faxon, D. P., Afzal, R., Chrolavicius, S., Jolly, S. S., Widimsky, P., et al. (2009) · New England Journal of Medicine · doi:10.1056/nejmoa0807986 · Cited by 925
A consumer's guide to subgroup analyses
Oxman, A. D., & Guyatt, G. H. (1992) · Annals of Internal Medicine · doi:10.7326/0003-4819-116-1-78 · Cited by 914
Drotrecogin Alfa (Activated) for Adults with Severe Sepsis and a Low Risk of Death
Abraham, E. H., Laterre, P., Garg, R., Levy, H. A., Talwar, D., Trzaskoma, B. L., François, B., Guy, J. S., Brückmann, M., Réa-Neto, Á., et al. (2005) · New England Journal of Medicine · doi:10.1056/nejmoa050935 · Cited by 887
Routine vs Selective Invasive Strategies in Patients With Acute Coronary Syndromes
Mehta, S. R., Cannon, C. P., Fox, K. A. A., Wallentin, L. C., Boden, W. E., Spacek, R., Widimský, P., McCullough, P. A., Hunt, D. U., Braunwald, E., et al. (2005) · JAMA · doi:10.1001/jama.293.23.2908 · Cited by 821
Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses
Sun, X., Briel, M., Walter, S. D., & Guyatt, G. H. (2010) · BMJ · doi:10.1136/bmj.c117 · Cited by 808
Telmisartan to Prevent Recurrent Stroke and Cardiovascular Events
Yusuf, S., Diener, H.-C., Sacco, R. L., Cotton, D., Ôunpuu, S., Lawton, W. A., Palesch, Y., Martin, R. H., Albers, G. W., Bath, P., et al. (2008) · New England Journal of Medicine · doi:10.1056/nejmoa0804593 · Cited by 785
Subgroup analyses in randomized trials: risks of subgroup-specific analyses;
Brookes, S. T., Whitely, E., Egger, M., Smith, G. D., Mulheran, P. A., & Peters, T. J. (2004) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2003.08.009 · Cited by 775
Gage, B. F., van Walraven, C., Pearce, L., Hart, R. G., Koudstaal, P. J., Boode, B. S. P., & Petersen, P. (2004) · Circulation · doi:10.1161/01.cir.0000145172.55640.93 · Cited by 749
Problems in the “Evidence” of “Evidence-Based Medicine”
Feinstein, A. R., & Horwitz, R. I. (1997) · The American Journal of Medicine · doi:10.1016/s0002-9343(97)00244-1 · Cited by 653
Evidence‐Based Medicine, Heterogeneity of Treatment Effects, and the Trouble with Averages
Kravitz, R. L., Duan, N., & Braslow, J. T. (2004) · Milbank Quarterly · doi:10.1111/j.0887-378x.2004.00327.x · Cited by 613
Brookes, S. T., Whitley, E., Peters, T. J., Mulheran, P. A., Egger, M., & Davey Smith, G. (2001) · Health Technology Assessment · doi:10.3310/hta5330 · Cited by 580
Limitations of Applying Summary Results of Clinical Trials to Individual Patients
Kent, D. M., & Hayward, R. A. (2007) · JAMA · doi:10.1001/jama.298.10.1209 · Cited by 502
Self-citationWhy Most Published Research Findings Are False
Ioannidis, J. P. A. (2019) · CHANCE · doi:10.1080/09332480.2019.1579573 · Cited by 474
Self-citationPrediction of benefit from carotid endar terectomy in individual patients: a risk-modelling study
Rothwell, P. M., & Warlow, C. P. (1999) · The Lancet · doi:10.1016/s0140-6736(98)11415-0 · Cited by 409
Self-citationAn evidence based approach to individualising treatment
Glasziou, P., & Irwig, L. (1995) · BMJ · doi:10.1136/bmj.311.7016.1356 · Cited by 389
Furukawa, T. A., Guyatt, G. H., & Griffith, L. E. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.1.72 · Cited by 318
Can overall results of clinical trials be applied to all patients?
Rothwell, P. M. (1995) · The Lancet · doi:10.1016/s0140-6736(95)90120-5 · Cited by 314
Self-citationModerators of Treatment Outcomes: Clinical, Research, and Policy Importance
Kraemer, H. C., Frank, E., & Kupfer, D. J. (2006) · JAMA · doi:10.1001/jama.296.10.1286 · Cited by 310
Assessment of claims of improved prediction beyond the Framingham risk score.
Tzoulaki, I., Liberopoulos, G., & Ioannidis, J. (2009) · Journal of the American Medical Association (JAMA) · doi:10.1001/jama.2009.1757 · Cited by 281
From subgroups to individuals: general principles and the example of carotid endarterectomy
Rothwell, P. M., Mehta, Z., Howard, S. C., Gutnikov, S. A., & Warlow, C. P. (2005) · The Lancet · doi:10.1016/s0140-6736(05)17746-0 · Cited by 281
Self-citationDevelopment of a Contemporary Bleeding Risk Model for Elderly Warfarin Recipients
Shireman, T. I., Mahnken, J. D., Howard, P. A., Kresowik, T. F., Hou, Q., & Ellerbeck, E. F. (2006) · CHEST Journal · doi:10.1378/chest.130.5.1390 · Cited by 271
Beyond the Randomized Clinical Trial
Nallamothu, B. K., Hayward, R. A., & Bates, E. (2008) · Circulation · doi:10.1161/circulationaha.107.703579 · Cited by 253
Self-citationGurwitz, J. H., Gore, J. M., Goldberg, R. J., Barron, H. V., Breen, T., Rundle, A. C., Sloan, M. A., French, W., & Rogers, W. J. (1998) · Annals of Internal Medicine · doi:10.7326/0003-4819-129-8-199810150-00002 · Cited by 230
Practical Guidelines for Multiplicity Adjustment in Clinical Trials
Proschan, M. A., & Waclawiw, M. A. (2000) · Controlled Clinical Trials · doi:10.1016/s0197-2456(00)00106-9 · Cited by 226
Morrow, D. A. (2002) · European Heart Journal · doi:10.1053/euhj.2001.2738 · Cited by 225
From subgroups to individuals: general principles and the example of carotid endarterectomy
Rothwell, P. M., Mehta, Z., Howard, S. P., et al. (2005) · The Lancet · doi:10.1016/s0140-6736(05)70156-2 · Cited by 207
Self-citationHayward, R. A., Kent, D. M., Vijan, S. S., & Hofer, T. P. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-18 · Cited by 196
Self-citationGreenfield, S., Billimek, J., Pellegrini, F., Franciosi, M., Berardis, G. D., Nicolucci, A., & Kaplan, S. H. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-12-200912150-00005 · Cited by 172
Kent, D. M., Jafar, T. H., Hayward, R. A., Tighiouart, H., Landa, M., de Jong, P., de Zeeuw, D., Remuzzi, G., Kamper, A.-L., & Levey, A. S. (2007) · Journal of the American Society of Nephrology · doi:10.1681/asn.2006101081 · Cited by 162
Self-citationThune, J. J., Hoefsten, D. E., Lindholm, M. G., Mortensen, L. S., Andersen, H., Nielsen, T. T., Køber, L., & Kelbæk, H. (2005) · Circulation · doi:10.1161/circulationaha.105.558676 · Cited by 139
Subgroup analyses in therapeutic cardiovascular clinical trials: Are most of them misleading?
Hernández, A. V., Boersma, E., Murray, G. D., Habbema, J. D. F., & Steyerberg, E. W. (2006) · American Heart Journal · doi:10.1016/j.ahj.2005.04.020 · Cited by 131
Yusuf, S. (1991) · JAMA · doi:10.1001/jama.266.1.93 · Cited by 129
Heterogeneity Is Not Always Noise
Davidoff, F. (2009) · JAMA · doi:10.1001/jama.2009.1845 · Cited by 117
Dealing with heterogeneity of treatment effects: is the literature up to the challenge?
Gabler, N. B., Duan, N., Liao, D., Elmore, J. G., Ganiats, T. G., & Kravitz, R. L. (2009) · Trials · doi:10.1186/1745-6215-10-43 · Cited by 114
The impact of high-risk patients on the results of clinical trials
Ioannidis, J. P. A., & Lau, J. (1997) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(97)00149-2 · Cited by 108
Self-citationHeterogeneity of Treatment Effects: Implications for Guidelines, Payment, and Quality Assessment
Greenfield, S., Kravitz, R. L., Duan, N., & Kaplan, S. H. (2007) · The American Journal of Medicine · doi:10.1016/j.amjmed.2007.02.002 · Cited by 104
Assessment of prognosis with the total illness burden index for prostate cancer
Litwin, M. S., Greenfield, S., Elkin, E. P., Lubeck, D. P., Broering, J. M., & Kaplan, S. H. (2007) · Cancer · doi:10.1002/cncr.22615 · Cited by 100
Reporting Clinical Trial Results To Inform Providers, Payers, And Consumers
Hayward, R. A., Kent, D. M., Vijan, S. S., & Hofer, T. P. (2005) · Health Affairs · doi:10.1377/hlthaff.24.6.1571 · Cited by 93
Self-citationTannock, I. F. (1996) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/88.3-4.206 · Cited by 92
Evidence based medicine: concerns of a clinical neurologist
Caplan, L. R. (2001) · Journal of Neurology Neurosurgery & Psychiatry · doi:10.1136/jnnp.71.5.569 · Cited by 88
Selection of thrombolytic therapy for individual patients: Development of a clinical model
Califf, R. M., Woodlief, L. H., Harrell, F. E., Lee, K. L., White, H. D., Guerci, A. D., Barbash, G. I., Simes, R. J., Weaver, W. D., Simoons, M. L., et al. (1997) · American Heart Journal · doi:10.1016/s0002-8703(97)70164-9 · Cited by 82
What Makes a Good Predictor?: The Evidence Applied to Coronary Artery Calcium Score
Ioannidis, J. P. A., & Tzoulaki, I. (2010) · JAMA · doi:10.1001/jama.2010.503 · Cited by 81
Self-citationFeiveson, A. H. (2002) · The Stata Journal Promoting communications on statistics and Stata · doi:10.1177/1536867x0200200201 · Cited by 72
loannidis, J. P. A., & Lau, J. (1998) · American Journal of Epidemiology · doi:10.1093/oxfordjournals.aje.a009590 · Cited by 72
Kent, D. M., Hayward, R. A., Griffith, J. L., Vijan, S. S., Beshansky, J. R., Califf, R. M., & Selker, H. P. (2002) · The American Journal of Medicine · doi:10.1016/s0002-9343(02)01160-9 · Cited by 62
Self-citationBlack, D. A. K. (1998) · PubMed · Cited by 57
Subgroups, treatment effects, and baseline risks: Some lessons from major cardiovascular trials
Parker, A. B., & Naylor, C. D. (2000) · American Heart Journal · doi:10.1067/mhj.2000.106610 · Cited by 56
A Framework for Tailoring Clinical Guidelines to Comorbidity at the Point of Care
Braithwaite, R. S., Concato, J., Chang, C., Roberts, M. S., & Justice, A. C. (2007) · Archives of Internal Medicine · doi:10.1001/archinte.167.21.2361 · Cited by 54
Kent, D. M., Ruthazer, R., & Selker, H. P. (2003) · Stroke · doi:10.1161/01.str.0000051506.43212.8b · Cited by 53
Self-citationCompeting risk and heterogeneity of treatment effect in clinical trials
Kent, D. M., Alsheikh-Ali, A. A. A., & Hayward, R. A. (2008) · Trials · doi:10.1186/1745-6215-9-30 · Cited by 52
Self-citationPredictive modeling and heterogeneity of baseline risk in meta-analysis of individual patient data
Trikalinos, T. A., & Ioannidis, J. P. A. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(00)00311-5 · Cited by 47
Self-citationCd, F., & Rp, B. (1983) · PubMed · Cited by 44
Vijan, S. S., Kent, D. M., & Hayward, R. A. (2000) · Diabetologia · doi:10.1007/s001250050017 · Cited by 43
Self-citationKent, D. M., Ruthazer, R., Griffith, J. L., Beshansky, J. R., Grines, C. L., Aversano, T. R., Concannon, T. W., Zalenski, R. J., & Selker, H. P. (2007) · The American Journal of Cardiology · doi:10.1016/j.amjcard.2006.12.068 · Cited by 32
Self-citationMore on Subgroup Analyses in Clinical Trials
Pocock, S., & Lubsen, J. C. (2008) · New England Journal of Medicine · doi:10.1056/nejmc0800616 · Cited by 29
A Multivariate Test of Interaction for Use in Clinical Trials
Follmann, D. A., & Proschan, M. A. (1999) · Biometrics · doi:10.1111/j.0006-341x.1999.01151.x · Cited by 22
Optimal Management of Acute Coronary Syndromes
Hillis, L. D., & Lange, R. A. (2009) · New England Journal of Medicine · doi:10.1056/nejme0902632 · Cited by 20
Assessing Treatment Effect Heterogeneity in Clinical Trials with Blocked Binary Outcomes
Albert, J. M., Gadbury, G. L., & Mascha, E. J. (2005) · Biometrical Journal · doi:10.1002/bimj.200510157 · Cited by 18
The ‘number need to treat’: does it help clinical decision making?
Ebrahim, S. B. J., & Smith, G. D. (1999) · Journal of Human Hypertension · doi:10.1038/sj.jhh.1000919 · Cited by 17
Kent, D. M., Ruthazer, R., Griffith, J. L., Beshansky, J. R., Concannon, T. W., Aversano, T. R., Grines, C. L., Zalenski, R. J., & Selker, H. P. (2008) · The American Journal of Cardiology · doi:10.1016/j.amjcard.2007.10.050 · Cited by 9
Self-citation
Cited by
Sorted by citations
What Is the Evidence for Physical Therapy Poststroke? A Systematic Review and Meta-Analysis
Veerbeek, J. M., van Wegen, E., van Peppen, R., van der Wees, P. J., Hendriks, E., Rietberg, M., & Kwakkel, G. (2014) · PLoS ONE · doi:10.1371/journal.pone.0087987 · Cited by 1,203
A catalogue of reporting guidelines for health research
Simera, I., Moher, D., Hoey, J., Schulz, K. F., & Altman, D. G. (2010) · European Journal of Clinical Investigation · doi:10.1111/j.1365-2362.2009.02234.x · Cited by 1,047
Self-citationA Randomized Trial of Rectal Indomethacin to Prevent Post-ERCP Pancreatitis
Elmunzer, B. J., Scheiman, J. M., Lehman, G. A., Chak, A., Mosler, P., Higgins, P., Hayward, R. A., Romagnuolo, J., Elta, G. H., Sherman, S., et al. (2012) · New England Journal of Medicine · doi:10.1056/nejmoa1111103 · Cited by 741
Self-citationFollow-up of glycemic control and cardiovascular outcomes in type 2 diabetes
Hayward, R. A., Reaven, P. D., Wiitala, W. L., Bahn, G. D., Reda, D. J., Ge, L., McCarren, M., Duckworth, W. C., & Emanuele, N. V. (2015) · New England Journal of Medicine · doi:10.1056/nejmoa1414266 · Cited by 668
Self-citation
Metadata
- Authors
- Kent, D. M., Rothwell, P. M., Ioannidis, J. P., Altman, D. G., & Hayward, R. A.
- Affiliations
- Tufts Medical Center (US)
- John Radcliffe Hospital (GB)
- University of Oxford (GB)
- University of Michigan (US)
- Corresponding author
- Kent, David M
- Venue
- Trials
- Published
- 2010-08-12
- Volume
- 11
- Issue
- 1
- Pages
- 85-85
- ISSN
- 1745-6215 (ISSN-L)
- PMC
- PMC2928211
- OpenAlex
- W2128780239
- Cited by
- 504
- References
- 72
- Citations by year
- 2026: 32 · 2025: 20 · 2024: 33 · 2023: 26 · 2022: 25 · 2021: 43 · 2020: 32 · 2019: 42 · 2018: 36 · 2017: 37 · 2016: 41 · 2015: 31 · 2014: 39 · 2013: 15 · 2012: 28
- FWCI
- 17.70
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Trials (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Pfizer
- National Institutes of Health (P60DK20572)
- Quality Enhancement Research Initiative
- National Institute of Diabetes and Digestive and Kidney Diseases (P60DK020572, P60 DK 20572)
- National Institute of Neurological Disorders and Stroke (R01NS062153)
- National Center for Research Resources (U54 RR023562)
- Topics
- Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
- Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- treatment effect heterogeneity, subgroup analysis, clinical trial reporting, treatment risks and benefits, risk stratification
- MeSH terms
- Clinical Trials as Topic, Humans, Risk Factors, Therapeutics, Reproducibility of Results, Multivariate Analysis, Risk Assessment, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
