Proposed triggers for retiring a living systematic review
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
Living systematic reviews (LSRs) are systematic reviews that are continually updated, incorporating relevant new evidence as it becomes available. LSRs are critical for decision-making in topics where the evidence continues to evolve. It is not feasible to continue to update LSRs indefinitely; however, guidance on when to retire LSRs from the living mode is not clear. We propose triggers for making such a decision. The first trigger is to retire LSRs when the evidence becomes conclusive for the outcomes that are required for decision-making. Conclusiveness of evidence is best determined based on the GRADE certainty of evidence construct, which is more comprehensive than solely relying on statistical considerations. The second trigger to retire LSRs is when the question becomes less pertinent for decision-making as determined by relevant stakeholders, including people affected by the problem, healthcare professionals, policymakers and researchers. LSRs can also be retired from a living mode when new studies are not anticipated to be published on the topic and when resources become unavailable to continue updating. We describe examples of retired LSRs and apply the proposed approach using one LSR about adjuvant tyrosine kinase inhibitors in high-risk renal cell carcinoma that we retired from a living mode and published its last update.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Murad and colleagues proposed a number of triggers that might lead to an LSR being retired. These include when the evidence becomes conclusive, which can be determined based on certainty of the evidence or statistical methods; when the topic becomes less relevant to stakeholders; when new studies are not expected to be published; when required resources become unavailable 33 ; however, other”
From Extension of the PRISMA 2020 statement for living systematic reviews (PRISMA-LSR): checklist and explanation (2024) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Grading quality of evidence and strength of recommendations
(2004) · BMJ · doi:10.1136/bmj.328.7454.1490 · Cited by 8,477 · cited 4×
“15 17 While guidance for reporting LSRs is still under development, 31 32 it is important for the LSR to report on triggers for retirement from the living mode.”
BackgroundInfluentialGRADE guidelines 6. Rating the quality of evidence—imprecision
Guyatt, G. H., Oxman, A. D., Kunz, R., Brozek, J., Alonso-Coello, P., Rind, D., Devereaux, P. J., Montori, V. M., Freyschuss, B., Vist, G., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.01.012 · Cited by 2,693
Self-citationTrial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis
Wetterslev, J., Thorlund, K., Brok, J., & Gluud, C. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.03.013 · Cited by 1,881 · cited 2×
“19 20 The available statistical methods to determine conclusiveness of evidence are based on the null effect.”
BackgroundMethodThe GRADE Working Group clarifies the construct of certainty of evidence
Hultcrantz, M., Rind, D., Akl, E. A., Treweek, S., Mustafa, R. A., Iorio, A., Alper, B. S., Meerpohl, J. J., Murad, M. H., Ansari, M. T., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.05.006 · Cited by 853 · cited 1×
“22 23 Thus, future novel statistical methods that incorporate specific thresholds are needed.”
BackgroundSelf-citationDrug treatments for covid-19: living systematic review and network meta-analysis
Siemieniuk, R. A., Bartoszko, J. J., Zeraatkar, D., Kum, E., Qasim, A., Díaz Martinez, J. P., Izcovich, A., Rochwerg, B., Lamontagne, F., Han, M. A., et al. (2020) · BMJ · doi:10.1136/bmj.m2980 · Cited by 846
How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis
Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., & Moher, D. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-4-200708210-00179 · Cited by 838
Living systematic review: 1. Introduction-the why, what, when, and how
Elliott, J. H., Synnot, A., Turner, T., Simmonds, M., Akl, E. A., McDonald, S., Salanti, G., Meerpohl, J., MacLehose, H., Hilton, J., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.08.010 · Cited by 655
Self-citationWhen and how to update systematic reviews: consensus and checklist
Garner, P., Hopewell, S., Chandler, J., MacLehose, H., Schünemann, H. J., Akl, E. A., Beyene, J., Chang, S., Churchill, R., Dearness, K., et al. (2016) · BMJ · doi:10.1136/bmj.i3507 · Cited by 426 · cited 1×
“Finally, it is plausible that moderate certainty evidence may be sufficient in some cases to make the decision that the efforts of keeping a review in the living mode are not warranted, rather, such review can be updated when certain triggers or conditions are met, 14 but it would not be a true LSR.”
BackgroundSelf-citationPogue, J. M., & Yusuf, S. (1997) · Controlled Clinical Trials · doi:10.1016/s0197-2456(97)00051-2 · Cited by 415 · cited 1×
“18 Simmonds et al present four possible methods that can be applied in a meta-analysis.”
MethodZeng, L., Brignardello-Petersen, R., Hultcrantz, M., Siemieniuk, R. A. C., Santesso, N., Traversy, G., Izcovich, A., Sadeghirad, B., Alexander, P. E., Devji, T., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.03.026 · Cited by 294 · cited 1×
“22 23 Thus, future novel statistical methods that incorporate specific thresholds are needed.”
BackgroundSelf-citationLiving systematic reviews: 3. Statistical methods for updating meta-analyses
Simmonds, M. C., Salanti, G., McKenzie, J. E., Elliott, J., Agoritsas, T., Hilton, J., Perron, C., Akl, E. A., Hodder, R. K., Pestridge, C., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.08.008 · Cited by 163 · cited 4×
“19 20 The available statistical methods to determine conclusiveness of evidence are based on the null effect.”
BackgroundMethodInfluentialSelf-citationGandhi, G. Y., Kovalaske, M., Kudva, Y., Walsh, K., Elamin, M. B., Beers, M., Coyle, C., Goalen, M., Murad, M. S., Erwin, P. J., et al. (2011) · Journal of Diabetes Science and Technology · doi:10.1177/193229681100500419 · Cited by 134 · cited 1×
“24 In the current pandemic, LSRs evaluating hydroxychloroquine can be terminated since the use of this drug, which was a hot topic in 2020, is not of interest to physicians or the public anymore.”
BackgroundSelf-citationKashour, Z., Riaz, M., Garbati, M. A., AlDosary, O., Tlayjeh, H., Gerberi, D., Murad, M. H., Sohail, M. R., Kashour, T., & Tleyjeh, I. M. (2021) · Journal of Antimicrobial Chemotherapy · doi:10.1093/jac/dkaa403 · Cited by 133 · cited 1×
“…interest in hydroxy-chloroquine may relate to studies that suggested lack of efficacy, increased harms, other treatments showing a clear net benefit, or may relate to other political and social factors 25 26 and may also reflect our certainty in its effect (the previous trigger based on certainty).”
BackgroundSelf-citationMackey, K., King, V. J., Gurley, S., Kiefer, M., Liederbauer, E., Vela, K., Sonnen, P., & Kansagara, D. (2020) · Annals of Internal Medicine · doi:10.7326/m20-1515 · Cited by 129
Schünemann, H. J., Khabsa, J., Solo, K., Khamis, A. M., Brignardello-Petersen, R., El-Harakeh, A., Darzi, A., Hajizadeh, A., Bognanni, A., Bak, A., et al. (2020) · Annals of Internal Medicine · doi:10.7326/m20-2306 · Cited by 128
Self-citationA Framework for the Development of Living Practice Guidelines in Health Care
El Mikati, I. K., Khabsa, J., Harb, T., Khamis, M., Agarwal, A., Pardo-Hernandez, H., Farran, S., Khamis, A. M., El Zein, O., El-Khoury, R., et al. (2022) · Annals of Internal Medicine · doi:10.7326/m22-0514 · Cited by 99 · cited 1×
“In addition, LSRs, and similar to living guidelines, 28 require ongoing conflict of interest management, quality control and stakeholder engagement.”
BackgroundSelf-citationRiaz, I. B., He, H., Ryu, A. J., Siddiqi, R. L., Naqvi, S. A. A., Yao, Y., Husnain, M., Narasimhulu, D. M., Mathew, J., Sipra, Q. U. A. R., et al. (2021) · European Urology · doi:10.1016/j.eururo.2021.03.016 · Cited by 86
Self-citationMartins‐Filho, P. R., Ferreira, L. C., Heimfarth, L., Araújo, A. A. D. S., & Quintans‐Júnior, L. J. (2021) · The Lancet Regional Health - Americas · doi:10.1016/j.lana.2021.100062 · Cited by 69 · cited 1×
“…interest in hydroxy-chloroquine may relate to studies that suggested lack of efficacy, increased harms, other treatments showing a clear net benefit, or may relate to other political and social factors 25 26 and may also reflect our certainty in its effect (the previous trigger based on certainty).”
BackgroundKahale, L. A., Elkhoury, R., El Mikati, I., Pardo-Hernandez, H., Khamis, A. M., Schünemann, H. J., Haddaway, N. R., & Akl, E. A. (2022) · F1000Research · doi:10.12688/f1000research.51723.3 · Cited by 48
Self-citationIannizzi, C., Dorando, E., Burns, J., Weibel, S., Dooley, C., Wakeford, H., Estcourt, L. J., Skoetz, N., & Piechotta, V. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.09.013 · Cited by 45 · cited 2×
“27 LSRs about the pandemic that synthesised observational studies or evaluated surrogate outcomes can be terminated (or modified) as randomised trials using patient-important outcomes have become available.”
BackgroundPRISMA flow diagrams for living systematic reviews: a methodological survey and a proposal
Kahale, L. A., Elkhoury, R., El Mikati, I., Pardo-Hernandez, H., Khamis, A. M., Schünemann, H. J., Haddaway, N. R., & Akl, E. A. (2021) · F1000Research · doi:10.12688/f1000research.51723.1 · Cited by 33 · cited 1×
“15 17 While guidance for reporting LSRs is still under development, 31 32 it is important for the LSR to report on triggers for retirement from the living mode.”
BackgroundSelf-citationRiaz, I. B., Fuentes, H. E., Naqvi, S. A. A., He, H., Sipra, Q.-U.-A. R., Tafur, A. J., Padranos, L., Wysokiński, W. E., Marshall, A. L., Vandvik, P. O., et al. (2021) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2020.10.041 · Cited by 27
Self-citationMurad, M. H., Balla, A. K., Khan, M. S., Shaikh, A., Saadi, S. M., & Wang, Z. (2022) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2021-111858 · Cited by 27 · cited 1×
“16 17 Type 1 and 2 errors (false-positive and false-negative conclusions, respectively) increase every time a trial is added to a meta-analysis, and the conventional CI that does not account for error inflation may be too narrow.”
BackgroundSelf-citationExtension of the PRISMA 2020 statement for living systematic reviews (LSRs): protocol
Kahale, L. A., Piechotta, V., McKenzie, J. E., Dorando, E., Iannizzi, C., Barker, J. M., Page, M. J., Skoetz, N., & Akl, E. A. (2022) · F1000Research · doi:10.12688/f1000research.75449.1 · Cited by 24 · cited 1×
“15 17 While guidance for reporting LSRs is still under development, 31 32 it is important for the LSR to report on triggers for retirement from the living mode.”
BackgroundSelf-citationBabić, A., Peričić, T. P., Pieper, D., & Puljak, L. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1451 · Cited by 17 · cited 2×
“11 The justification was deemed insufficient in 49% of the reviews, 11 and reanalysis of a sample of 40 reviews demonstrated that there was no clear algorithm or consistent approach to make such decision.”
BackgroundPartially systematic thoughts on the history of systematic reviews
Clarke, M. J. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0833-3 · Cited by 15
Rochwerg, B. N., Solo, K., Darzi, A., Chen, G., Khamis, A. M., Borowiack, E., Morgano, G. P., Hajizadeh, A., Chen, C., Zhao, H., et al. (2020) · Annals of Internal Medicine · doi:10.7326/l20-0944 · Cited by 15
Self-citationRiaz, I. B., Siddiqi, R. L., Islam, M., He, H., Riaz, A., Asghar, N., Naqvi, S. A. A., Warner, J. L., Murad, M. H., & Kohli, M. (2021) · JCO Clinical Cancer Informatics · doi:10.1200/cci.21.00035 · Cited by 13 · cited 1×
“9”
Self-citationMuti-Schüenemann, G. E. U., Szczeklik, W., Solo, K., Khabsa, J., Thomas, R., Borowiack, E., Khamis, A. M., Hneiny, L., Darzi, A., Harrison, L., et al. (2021) · Annals of Internal Medicine · doi:10.7326/l21-0424 · Cited by 12
Self-citationBabić, A., Peričić, T. P., Pieper, D., & Puljak, L. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1556 · Cited by 12 · cited 1×
“12 In this proposal, we aim to identify potential situations in which LSRs can be retired from the living mode and illustrate these situations through a published example of an LSR that has published its last update and was retired from the living mode.”
BackgroundRiaz, I. B., Sipra, Q. U. A. R., Naqvi, S. A. A., He, H., Siddiqi, R. L., Islam, M., Asghar, N., Ikram, W., Xu, W. V., Liu, H., et al. (2022) · Critical Reviews in Oncology/Hematology · doi:10.1016/j.critrevonc.2022.103706 · Cited by 7
Self-citationVentilation Techniques and Risk for Transmission of Coronavirus Disease, Including COVID-19
Rochwerg, B. N., Schünemann, H. J., & Akl, E. A. (2020) · Annals of Internal Medicine · doi:10.7326/l20-1179
Self-citation
Cited by
Systematic reviews of the literature: an introduction to current methods
Brignardello‐Petersen, R., Santesso, N. A., & Guyatt, G. H. (2024) · American Journal of Epidemiology · doi:10.1093/aje/kwae232 · Cited by 254
“16 This makes living reviews particularly helpful when new potentially practice-changing evidence continues to emerge.”
Akl, E. A., Khabsa, J., Iannizzi, C., Piechotta, V., Kahale, L. A., Barker, J. M., McKenzie, J. E., Page, M. J., & Skoetz, N. (2024) · BMJ · doi:10.1136/bmj-2024-079183 · Cited by 103
“…conclusive, which can be determined based on certainty of the evidence or statistical methods; when the topic becomes less relevant to stakeholders; when new studies are not expected to be published; when required resources become unavailable 33 ; however, other triggers might also be possible.”
Self-citationHasan, B., Saadi, S., Rajjoub, N. S., Hegazi, M., Al-Kordi, M., Fleti, F., Farah, M., Riaz, I. B., Banerjee, I., Wang, Z., et al. (2024) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2023-112597 · Cited by 67
Self-citationLuo, J., Chen, Z., Liu, D., Li, H., He, S., Zeng, L., Yang, M., Liu, Z., Xiao, X., & Zhang, L. (2023) · BMC Medical Research Methodology · doi:10.1186/s12874-023-01980-y · Cited by 27
Zeraatkar, D., Ling, M., Kirsh, S., Jassal, T., Pitre, T., Chakraborty, S. P., Turner, T., Turkstra, L. S., McIntyre, R. S., Izcovich, A., et al. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-086407 · Cited by 18
“130 At this time, given the timeline of planned trials and the funding available to us, we intend to continue to maintain the living systematic review for 3 years.”
Metadata
- Authors
- Murad, M. H., Wang, Z., Chu, H., Lin, L., El Mikati, I. K., Khabsa, J., Akl, E. A., Nieuwlaat, R., Schuenemann, H. J., & Riaz, I. B.
- Affiliations
- Mayo Clinic (US)
- University of Minnesota (US)
- Banner - University Medical Center Tucson (US)
- American University of Beirut (LB)
- McMaster University (CA)
- Humanitas University (IT)
- Mayo Clinic Hospital (US)
- Corresponding author
- Murad, Mohammad Hassan
- ORCID
- Venue
- BMJ Evidence-Based Medicine
- Publisher
- BMJ
- Published
- 2023-03-08
- Volume
- 28
- Issue
- 5
- Pages
- 348-352
- ISSN
- 1356-5524 (ISSN-L)
- 2515-446X
- 2515-4478
- PMC
- PMC10579491
- OpenAlex
- W4323542543
- Cited by
- 17
- References
- 32
- Citations by year
- 2026: 4 · 2025: 5 · 2024: 6 · 2023: 2
- FWCI
- 1.21
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- BMJ Evidence-Based Medicine (publishedVersion, cc-by-nc)
- PubMed Central (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- HighWire Press Open Archive (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- living systematic review, GRADE certainty of evidence, stakeholder relevance, resource constraints, high-risk renal cell carcinoma
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
