CoChat
Sign Up
Cited by 35Meta-analysis and systematic reviewsStatistical Methods in Clinical TrialsStatistical Methods in Epidemiology

Assessing the robustness of results from clinical trials and meta-analyses with the fragility index

Open access
Lin, Lifeng, Xing, Aiwen, Chu, Haitao, Murad, M. Hassan, Xu, Chang, Baer, Benjamin R., Wells, Martin T., Sanchez-Ramos, Luis·2023·American Journal of Obstetrics and Gynecology
DOI 10.1016/j.ajog.2022.08.053PMC PMC9974556
Open this paper in CoChat — free account

Abstract

The fragility index has been increasingly used to assess the robustness of the results of clinical trials since 2014. It aims at finding the smallest number of event changes that could alter originally statistically significant results. Despite its popularity, some researchers have expressed several concerns about the validity and usefulness of the fragility index. It offers a comprehensive review of the fragility index's rationale, calculation, software, and interpretation, with emphasis on application to studies in obstetrics and gynecology. This article presents the fragility index in the settings of individual clinical trials, standard pairwise meta-analyses, and network meta-analyses. Moreover, this article provides worked examples to demonstrate how the fragility index can be appropriately calculated and interpreted. In addition, the limitations of the traditional fragility index and some solutions proposed in the literature to address these limitations were reviewed. In summary, the fragility index is recommended to be used as a supplemental measure in the reporting of clinical trials and a tool to communicate the robustness of trial results to clinicians. Other considerations that can aid in the fragility index's interpretation include the loss to follow-up and the likelihood of data modifications that achieve the loss of statistical significance.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.ajog.2022.08.053

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The risk of bias in RCTs was assessed using the Cochrane recommended tool, which includes seven specific domains.

    “The risk of bias in RCTs was assessed using the Cochrane recommended tool, 26 which includes seven specific domains”

    From Glucagon‐like peptide‐1 receptor agonists and risk of thyroid cancer: A systematic review and meta‐analysis of randomized controlled trials (2024) · cites this paper as [26]

    The section on the fragility index for pairwise meta-analysis discusses RCT data-extraction errors, not a method for assessing risk of bias. Searches for the Cochrane risk-of-bias tool and its seven domains found no passage in our copy addressing either point; page numbers were not available. (LLM, conf=0.72)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 55

  1. [1]Benjamin DJ, Berger JO, Johannesson M, Redefine statistical significance. Nature Human Behaviour 2018; 2: 6– 10. 30980045
  2. [2]Amrhein V, Greenland S, McShane B. Scientists rise up against statistical significance. Nature 2019; 567: 305– 07. 30894741
  3. [5]Abaid LN, Grimes DA, Schulz KF. Reducing publication bias through trial registration. Obstetrics & Gynecology 2007; 109: 1434– 37. 17540818
  4. [6]Bibens ME, Chong AB, Vassar M. Utilization of clinical trials registries in obstetrics and gynecology systematic reviews. Obstetrics & Gynecology 2016; 127: 248– 53. 26942350
  5. [7]Lin L, Chu H. Quantifying publication bias in meta-analysis. Biometrics 2018; 74: 785– 94. 29141096 PMC5953768
  6. [8]Lin L, Chu H, Murad MH, Empirical comparison of publication bias tests in meta-analysis. Journal of General Internal Medicine 2018; 33: 1260– 67. 29663281 PMC6082203
  7. [9]Turner EH, Matthews AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its influence on apparent efficacy. New England Journal of Medicine 2008; 358: 252– 60. 18199864
  8. [10]Eyding D, Lelgemann M, Grouven U, Reboxetine for acute treatment of major depression: systematic review and meta-analysis of published and unpublished placebo and selective serotonin reuptake inhibitor controlled trials. BMJ 2010; 341: c4737. 20940209 PMC2954275
  9. [11]Sanchez-Ramos L. Intrapartum amnioinfusion for meconium-stained amniotic fluid: a systematic review of randomised controlled trials. BJOG: An International Journal of Obstetrics & Gynaecology 2008; 115: 409– 10. 18190380
  10. [12]Walsh M, Srinathan SK, McAuley DF, The statistical significance of randomized controlled trial results is frequently fragile: a case for a Fragility Index. Journal of Clinical Epidemiology 2014; 67: 622– 28. 24508144
  11. [13]Feinstein AR. The unit fragility index: an additional appraisal of "statistical significance" for a contrast of two proportions. Journal of Clinical Epidemiology 1990; 43: 201– 09. 2303850
  12. [14]Walter SD. Statistical significance and fragility criteria for assessing a difference of two proportions. Journal of Clinical Epidemiology 1991; 44: 1373– 78. 1753268
  13. [15]Ho AK. The fragility index for assessing the robustness of the statistically significant results of experimental clinical studies. Journal of General Internal Medicine 2022; 37: 206– 11. 34357573 PMC8739402
  14. [16]Pundir J, Achilli C, Bhide P, Risk of foetal harm with letrozole use in fertility treatment: a systematic review and meta-analysis. Human Reproduction Update 2020; 27: 474– 85. 33374012
  15. [17]Pascoal E, Liu M, Lin L, Luketic L. The fragility of statistically significant results in gynaecologic surgery: a systematic review. Journal of Obstetrics and Gynaecology Canada 2022; 44: 508– 14. 34954411
  16. [18]Sanchez-Ramos L, Lin L. Cerclage placement in twin pregnancies with short or dilated cervix does not prevent preterm birth: a fragility index assessment. American Journal of Obstetrics & Gynecology 2022; 227: 338– 39. 35447146
  17. [19]Sigurdardottir T, Steingrimsdottir T, Geirsson RT, Halldorsson TI, Aspelund T, Bø K. Can postpartum pelvic floor muscle training reduce urinary and anal incontinence?: An assessor-blinded randomized controlled trial. American Journal of Obstetrics and Gynecology 2020; 222: 247.e1– 47.e8. 31526791
  18. [20]Lin L Factors that impact fragility index and their visualizations. Journal of Evaluation in Clinical Practice 2021; 27: 356– 64. 32524768 PMC7725889
  19. [21]Baer BR, Gaudino M, Charlson M, Fremes SE, Wells MT. Fragility indices for only sufficiently likely modifications. Proceedings of the National Academy of Sciences 2021; 118: e2105254118. PMC8670515 34848537
  20. [26]Lin L, Chu H. Assessing and visualizing fragility of clinical results with binary outcomes in R using the fragility package. PLOS ONE 2022; 17: e0268754. 35648746 PMC9159630
  21. [27]Carter RE, McKie PM, Storlie CB. The fragility index: a P-value in sheep's clothing? European Heart Journal 2016; 38: 346– 48. 28417139
  22. [28]Condon TM, Sexton RW, Wells AJ, To M-S. The weakness of fragility index exposed in an analysis of the traumatic brain injury management guidelines: a meta-epidemiological and simulation study. PLOS ONE 2020; 15: e0237879. 32810192 PMC7433866
  23. [29]Niforatos JD, Zheutlin AR, Chaitoff A, Pescatore RM. The fragility index of practice changing clinical trials is low and highly correlated with P-values. Journal of Clinical Epidemiology 2020; 119: 140– 42. 31711911
  24. [30]Porco TC, Lietman TM. A fragility index: handle with care. Ophthalmology 2018; 125: 649. 29681292
  25. [31]Li J, O’Connell PJ. The fragility index: the P-value by another name? Transplantation 2022; 106: 239– 40. 33966027
  26. [32]Potter GE. Dismantling the Fragility Index: a demonstration of statistical reasoning. Statistics in Medicine 2020; 39: 3720– 31. 32781488
  27. [33]Schröder A, Muensterer OJ, Oetzmann von Sochaczewski C. Paediatric surgical trials, their fragility index, and why to avoid using it to evaluate results. Pediatric Surgery International 2022; 38: 1057– 66. 35524787 PMC9162995
  28. [34]Ahmed W, Fowler RA, McCredie VA. Does sample size matter when interpreting the fragility index? Critical Care Medicine 2016; 44: e1142– e43. 27755081
  29. [35]Acuna SA, Sue-Chue-Lam C, Dossa F. The fragility index—P values reimagined, flaws and all. JAMA Surgery 2019; 154: 674– 74. 30994897
  30. [36]Walter SD, Thabane L, Briel M. The fragility of trial results involves more than statistical significance alone. Journal of Clinical Epidemiology 2020; 124: 34– 41. 32298777
  31. [37]Baer BR, Fremes SE, Gaudino M, Charlson M, Wells MT. On clinical trial fragility due to patients lost to follow up. BMC Medical Research Methodology 2021; 21: 254. 34800976 PMC8606097
  32. [38]Shochet LR, Kerr PG, Polkinghorne KR. The fragility of significant results underscores the need of larger randomized controlled trials in nephrology. Kidney International 2017; 92: 1469– 75. 28754551
  33. [39]Murad MH, Kara Balla A, Khan MS, Shaikh A, Saadi S, Wang Z. Thresholds for interpreting the fragility index derived from sample of randomised controlled trials in cardiology: a meta-epidemiologic study. BMJ Evidence-Based Medicine 2022:In press. 35264405
  34. [40]Boutron I, Chaimani A, Meerpohl JJ, The COVID-NMA project: building an evidence ecosystem for the COVID-19 pandemic. Annals of Internal Medicine 2020; 173: 1015– 17. 32931326 PMC7518109
  35. [41]Lotfi T, Stevens A, Akl EA, Getting trustworthy guidelines into the hands of decision-makers and supporting their consideration of contextual factors for implementation globally: recommendation mapping of COVID-19 guidelines. Journal of Clinical Epidemiology 2021; 135: 182– 86. 33836255 PMC8022513
  36. [42]Bomze D, Meirson T. A critique of the fragility index. The Lancet Oncology 2019; 20: e551. 31578993
  37. [43]Bomze D, Asher N, Hasan Ali O, Survival-inferred fragility index of phase 3 clinical trials evaluating immune checkpoint inhibitors. JAMA Network Open 2020; 3: e2017675. 33095247 PMC7584930
  38. [44]Caldwell J-ME, Youssefzadeh K, Limpisvasti O. A method for calculating the fragility index of continuous outcomes. Journal of Clinical Epidemiology 2021; 136: 20– 25. 33684509
  39. [46]Baer BR, Gaudino M, Fremes SE, Charlson M, Wells MT. The fragility index can be used for sample size calculations in clinical trials. Journal of Clinical Epidemiology 2021; 139: 199– 209. 34403756 PMC8665025
  40. [47]Khan MS, Fonarow GC, Friede T, Application of the reverse fragility index to statistically nonsignificant randomized clinical trial results. JAMA Network Open 2020; 3: e2012469. 32756927 PMC7407075
  41. [48]Niforatos JD, Weaver M, Johansen ME. Assessment of publication trends of systematic reviews and randomized clinical trials, 1995 to 2017. JAMA Internal Medicine 2019; 179: 1593– 94. 31355871 PMC6664379
  42. [49]Ioannidis JPA. The mass production of redundant, misleading, and conflicted systematic reviews and meta-analyses. The Milbank Quarterly 2016; 94: 485– 514. 27620683 PMC5020151
  43. [50]Hacke C, Nunan D. Discrepancies in meta-analyses answering the same clinical question were hard to explain: a meta-epidemiological study. Journal of Clinical Epidemiology 2020; 119: 47– 56. 31783099
  44. [51]Palpacuer C, Hammas K, Duprez R, Laviolle B, Ioannidis JPA, Naudet F. Vibration of effects from diverse inclusion/exclusion criteria and analytical choices: 9216 different ways to perform an indirect comparison meta-analysis. BMC Medicine 2019; 17: 174. 31526369 PMC6747755
  45. [52]Xu H, Hofmeyr J, Roy C, Fraser W. Intrapartum amnioinfusion for meconium-stained amniotic fluid: a systematic review of randomised controlled trials. BJOG: An International Journal of Obstetrics & Gynaecology 2007; 114: 383– 90. 17378813
  46. [53]Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLOS Medicine 2009; 6: e1000097. 19621072 PMC2707599
  47. [56]Gøtzsche PC, Hróbjartsson A, Marić K, Tendal B. Data extraction errors in meta-analyses that use standardized mean differences. JAMA 2007; 298: 430– 37. 17652297
  48. [59]Atal I, Porcher R, Boutron I, Ravaud P. The statistical significance of meta-analyses is frequently fragile: definition of a fragility index for meta-analyses. Journal of Clinical Epidemiology 2019; 111: 32– 40. 30940600
  49. [61]Di Mascio D, Saccone G, Bellussi F, Delayed versus immediate pushing in the second stage of labor in women with neuraxial analgesia: a systematic review and meta-analysis of randomized controlled trials. American Journal of Obstetrics & Gynecology 2020; 223: 189– 203. 32067972
  50. [62]Baer BR, Fremes SE, Charlson M, Gaudino M, Wells MT. Fragility measures for typical cases. arXiv preprint arXiv:2201.07093, 2022. · arXiv
  51. [64]Lu G, Ades AE. Combination of direct and indirect evidence in mixed treatment comparisons. Statistics in Medicine 2004; 23: 3105– 24. 15449338
  52. [65]Zhang J, Carlin BP, Neaton JD, Network meta-analysis of randomized clinical trials: reporting the proper summaries. Clinical Trials 2014; 11: 246– 62. 24096635 PMC3972291
  53. [66]Naudet F, Schuit E, Ioannidis JPA. Overlapping network meta-analyses on the same topic: survey of published studies. International Journal of Epidemiology 2017; 46: 1999– 2008. 29040566
  54. [67]Xing A, Chu H, Lin L. Fragility index of network meta-analysis with application to smoking cessation data. Journal of Clinical Epidemiology 2020; 127: 29– 39. 32659361 PMC7669670
  55. [68]Nguyen HT, Thavorncharoensap M, Phung TL, Comparative efficacy and safety of pharmacologic interventions to prevent mother-to-child transmission of hepatitis B virus: a systematic review and network meta-analysis. American Journal of Obstetrics & Gynecology 2022; 227: 163– 72. 35263648

Cited by

Sorted by citations

Authors
Lin, L., Xing, A., Chu, H., Murad, M. H., Xu, C., Baer, B. R., Wells, M. T., & Sanchez-Ramos, L.
Affiliations
  • Florida State University (US)
  • University of Minnesota (US)
  • Mayo Clinic (US)
  • Anhui Medical University (CN)
  • University of Rochester (US)
  • Cornell University (US)
Corresponding author
Lin, Lifeng
ORCID
Venue
American Journal of Obstetrics and Gynecology
Publisher
Elsevier BV
Published
2023-03
Volume
228
Issue
3
Pages
276-282
ISSN
0002-9378 (ISSN-L)
DOI
10.1016/j.ajog.2022.08.053
PMC
PMC9974556
OpenAlex
W4294740949
Cited by
35
References
7
Citations by year
2026: 5 · 2025: 13 · 2024: 12 · 2023: 4 · 2022: 1
FWCI
2.47
Type
journal article
Language
en
Open access
Licence
Locations
  • American Journal of Obstetrics and Gynecology (acceptedVersion)
  • PubMed (publishedVersion)
  • PubMed Central (acceptedVersion)
Funders
  • National Institutes of Health
  • National Institute of Mental Health (R03 MH128727)
  • U.S. National Library of Medicine (R01 LM012982)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Clinical Trials
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods in Epidemiology
Keywords
fragility index, clinical trial reporting, meta-analysis, network meta-analysis, loss to follow-up, data modification, obstetrics and gynecology
MeSH terms
Network Meta-Analysis, Network Meta-Analysis as Topic, Network Meta-Analysis As Topic, Clinical Trials as Topic, Humans, Probability, Meta-Analysis as Topic

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review