Methodological challenges for living systematic reviews conducted during the COVID-19 pandemic: A concept paper
Open accessAbstract
Background A living systematic review (LSR) is an emerging review type that makes use of continual updating. In the COVID-19 pandemic, we were confronted with a shifting epidemiological landscape, clinical uncertainties and evolving evidence. These unexpected challenges compelled us to amend standard LSR methodology.Objective and outline Our primary objective is to discuss some challenges faced when conducting LSRs in the context of the COVID-19 pandemic, and to provide methodological guidance for others doing similar work. Based on our experience and lessons learned from two Cochrane LSRs and challenges identified in several non-Cochrane LSRs, we highlight methodological considerations, particularly with regards to the study design, interventions and comparators, changes in outcome measure, and the search strategy. We discuss when to update, or rather when not to update the review, and the importance of transparency when reporting changes.Lessons learned and conclusion We learned that a LSR is a very suitable review type for the pandemic context, even in the face of new methodological and clinical challenges. Our experience showed that the decision for updating a LSR depends not only on the evolving disease or emerging evidence, but also on the individual review question and the review teams' resources.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
For some covid-19 related LSRs, authors initially included non-randomised studies, but later excluded these studies and included clinical trials.
“For example, for some covid-19 related LSRs, authors initially included non-randomised studies, but later excluded these studies and included clinical trials. 19”
From Extension of the PRISMA 2020 statement for living systematic reviews (PRISMA-LSR): checklist and explanation (2024) · cites this paper as [19]
In “Relevant design of studies? – a choice based on new conditions” (page unavailable), the paper says some COVID-19 LSRs included observational studies early because RCTs were scarce. It does not say those LSRs later excluded the observational studies and added clinical trials; for the authors’ own review, it instead describes a later decision to include additional safety data from controlled and uncontrolled studies. (LLM, conf=0.82)
Experimental — accuracy not yet measuredLiving systematic reviews face major challenges beyond adequate reporting, including the vast workload needed to maintain an LSR and publication platforms not being fit for purpose.
“We acknowledge that LSRs face major challenges beyond adequate reporting, including the vast workload needed to maintain an LSR, 7 19 as well as the publication platforms not being fit for purpose.”
From Extension of the PRISMA 2020 statement for living systematic reviews (PRISMA-LSR): checklist and explanation (2024) · cites this paper as [19]
The Funding section says LSR updates depend on time and personnel, and the Developing the search strategy section describes the continuing work of maintaining searches. These passages support the workload challenge, but our copy does not say that publication platforms are not fit for purpose. (LLM, conf=0.84)
Experimental — accuracy not yet measuredMost covid-19 related LSRs considered by a concept paper addressing methodological challenges for LSRs ran their search weekly or monthly.
“Most covid-19 related LSRs considered by a concept paper addressing methodological challenges for LSRs ran their search weekly or monthly. 19”
From Extension of the PRISMA 2020 statement for living systematic reviews (PRISMA-LSR): checklist and explanation (2024) · cites this paper as [19]
A challenge reported for reviews conducted during the covid-19 pandemic was the dynamic nature of electronic databases.
“A challenge reported for reviews conducted during the covid-19 pandemic was the dynamic nature of electronic databases. 19”
From Extension of the PRISMA 2020 statement for living systematic reviews (PRISMA-LSR): checklist and explanation (2024) · cites this paper as [19]
The Centers for Disease Control and Prevention covid-19 Research Articles Downloadable Database was discontinued in mid-2020, but was later completely covered by the World Health Organization covid-19 Global literature on coronavirus disease database.
“For example, the Centers for Disease Control and Prevention covid-19 Research Articles Downloadable Database, an early and comprehensive source of preprint articles, was discontinued in mid-2020, but was later completely covered by the World Health Organization covid-19 Global literature on coronavirus disease database. 19”
From Extension of the PRISMA 2020 statement for living systematic reviews (PRISMA-LSR): checklist and explanation (2024) · cites this paper as [19]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · see citing works
Allotey, J., Fernandez, S., Bonet, M., Stallings, E., Yap, M., Kew, T., Zhou, D., Coomar, D., Sheikh, J., Lawson, H., et al. (2020) · BMJ · doi:10.1136/bmj.m3320 · Cited by 2,304 · see citing works
Murad, M. H., Asi, N., Alsawas, M., & Alahdab, F. (2016) · Evidence Based Medicine · doi:10.1136/ebmed-2016-110401 · Cited by 1,388 · see citing works
Living systematic review: 1. Introduction-the why, what, when, and how
Elliott, J. H., Synnot, A., Turner, T., Simmonds, M., Akl, E. A., McDonald, S., Salanti, G., Meerpohl, J., MacLehose, H., Hilton, J., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.08.010 · Cited by 655 · see citing works
Self-citationWhen and how to update systematic reviews: consensus and checklist
Garner, P., Hopewell, S., Chandler, J., MacLehose, H., Schünemann, H. J., Akl, E. A., Beyene, J., Chang, S., Churchill, R., Dearness, K., et al. (2016) · BMJ · doi:10.1136/bmj.i3507 · Cited by 426 · see citing works
Zhang, B., Liu, S., Tan, T., Huang, W., Dong, Y., Chen, L., Chen, Q., Zhang, L., Zhong, Q., Zhang, X., et al. (2020) · CHEST Journal · doi:10.1016/j.chest.2020.03.039 · Cited by 421 · see citing works
Piechotta, V., Chai, K. L., Valk, S. J., Doree, C. J., Monsef, I., Wood, E. M., Lamikanra, A., Kimber, C., McQuilten, Z., So‐Osman, C., et al. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013600.pub2 · Cited by 365 · see citing works
Self-citationLiving systematic reviews: 2. Combining human and machine effort
Thomas, J., Noel-Storr, A., Marshall, I., Wallace, B., McDonald, S., Mavergames, C., Glasziou, P., Shemilt, I., Synnot, A., Turner, T., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.08.011 · Cited by 355 · see citing works
Self-citationAhn, J. Y., Sohn, Y., Lee, S. H., Cho, Y., Hyun, J. H., Baek, Y. J., Jeong, S. J., Kim, J. H., Ku, N. S., Yeom, J., et al. (2020) · Journal of Korean Medical Science · doi:10.3346/jkms.2020.35.e149 · Cited by 352 · see citing works
Living systematic reviews: 4. Living guideline recommendations.
Akl, E. A., Meerpohl, J. J., Elliott, J., Kahale, L. A., Schünemann, H. J., Agoritsas, T., Hilton, J., Perron, C., Akl, E., Hodder, R., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.08.009 · Cited by 328 · see citing works
Self-citationHydroxychloroquine or Chloroquine for Treatment or Prophylaxis of COVID-19
Hernández, A. V., Roman, Y. M., Pasupuleti, V., Barboza, J. J., & White, C. M. (2020) · Annals of Internal Medicine · doi:10.7326/m20-2496 · Cited by 221 · see citing works
John, A., Eyles, E., Webb, R. T., Okolie, C., Schmidt, L., Arensman, E., Hawton, K., O'Connor, R. C., Kapur, N., Moran, P., et al. (2021) · F1000Research · doi:10.12688/f1000research.25522.2 · Cited by 189 · see citing works
The impact of the COVID-19 pandemic on self-harm and suicidal behaviour: a living systematic review
John, A., Okolie, C., Eyles, E., Webb, R. T., Schmidt, L., McGuinness, L. A., Olorisade, B. K., Arensman, E., Hawton, K., Kapur, N., et al. (2020) · F1000Research · doi:10.12688/f1000research.25522.1 · Cited by 178 · see citing works
Living systematic reviews: 3. Statistical methods for updating meta-analyses
Simmonds, M. C., Salanti, G., McKenzie, J. E., Elliott, J., Agoritsas, T., Hilton, J., Perron, C., Akl, E. A., Hodder, R. K., Pestridge, C., et al. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2017.08.008 · Cited by 145 · see citing works
Self-citationVentilation Techniques and Risk for Transmission of Coronavirus Disease, Including COVID-19
Schünemann, H. J., Khabsa, J., Solo, K., Khamis, A. M., Brignardello‐Petersen, R., El‐Harakeh, A., Darzi, A. J., Hajizadeh, A., Bognanni, A., Bąk, A., et al. (2020) · Annals of Internal Medicine · doi:10.7326/m20-2306 · Cited by 126 · see citing works
International travel-related control measures to contain the COVID-19 pandemic: a rapid review
Burns, J., Movsisyan, A., Stratil, J. M., Biallas, R. L., Coenen, M., Emmert-Fees, K. M. F., Geffert, K., Hoffmann, S., Horstick, O., Laxy, M., et al. (2021) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013717.pub2 · Cited by 122 · see citing works
Self-citationTravel-related control measures to contain the COVID-19 pandemic: a rapid review
Burns, J., Movsisyan, A., Stratil, J. M., Coenen, M., Emmert-Fees, K. M. F., Geffert, K., Hoffmann, S., Horstick, O., Laxy, M., Pfadenhauer, L. M., et al. (2020) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd013717 · Cited by 108 · see citing works
Self-citationParenteral anticoagulation in ambulatory patients with cancer
Akl, E. A., Kahale, L. A., Hakoum, M. B., Matar, C. F., Sperati, F., Barba, M., Yosuico, V. E. D., Terrenato, I., Synnot, A., & Schünemann, H. J. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006652.pub5 · Cited by 103 · see citing works
Juul, S., Nielsen, E. E., Feinberg, J. B., Siddiqui, F., Kamp, C. B., Barot, E., Holgersson, J., Nielsen, N., Bentzer, P., Veroniki, A. A., et al. (2021) · PLoS ONE · doi:10.1371/journal.pone.0248132 · Cited by 59 · see citing works
PRISMA flow diagrams for living systematic reviews: a methodological survey and a proposal
Kahale, L. A., El-Khoury, R., Mikati, I. E., Hernandez, H. P., Khamis, A. M., Schünemann, H. J., Haddaway, N., & Akl, E. A. (2021) · F1000Research · doi:10.12688/f1000research.51723.1 · Cited by 46 · see citing works
When and how to update systematic reviews: consensus and checklist
(2016) · BMJ · doi:10.1136/bmj.i4853 · Cited by 38 · see citing works
Metzendorf, M., & Featherstone, R. M. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1501 · Cited by 25 · see citing works
Kahale, L. A., El-Khoury, R., Mikati, I. E., Hernandez, H. P., Khamis, A. M., Schünemann, H. J., Haddaway, N., & Akl, E. A. (2021) · F1000Research · doi:10.12688/f1000research.51723.2 · Cited by 15 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [25]Annals Information for Authors Annals of Internal Medicine 2021
Cited by
Sorted by citations
Akl, E. A., Khabsa, J., Iannizzi, C., Piechotta, V., Kahale, L. A., Barker, J. M., McKenzie, J. E., Page, M. J., & Skoetz, N. (2024) · BMJ · doi:10.1136/bmj-2024-079183 · Cited by 71 · see citing works
Self-citationExtension of the PRISMA 2020 statement for living systematic reviews (LSRs): protocol
Kahale, L. A., Piechotta, V., McKenzie, J. E., Dorando, E., Iannizzi, C., Barker, J. M., Page, M. J., Skoetz, N., & Akl, E. A. (2022) · F1000Research · doi:10.12688/f1000research.75449.1 · Cited by 24 · see citing works
Self-citationHow to update a living systematic review and keep it alive during a pandemic: a practical guide
Heron, L., Buitrago-Garcia, D., Ipekci, A. M., Baumann, R., Imeri, H., Salanti, G., Counotte, M. J., & Low, N. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02325-y · Cited by 26 · see citing works
“Guidance about methods for several aspects of the conduct, reporting, and publication of living systematic reviews is available [1, 16–19] or in development [20, 21].”
BackgroundMethodProposed triggers for retiring a living systematic review
Murad, M. H., Wang, Z., Chu, H., Lin, L., El Mikati, I. K., Khabsa, J., Akl, E. A., Nieuwlaat, R., Schuenemann, H. J., & Riaz, I. B. (2023) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2022-112100 · Cited by 17 · see citing works
- Authors
- Iannizzi, C., Dorando, E., Burns, J., Weibel, S., Dooley, C., Wakeford, H., Estcourt, L. J., Skoetz, N., & Piechotta, V.
- Affiliations
- University of Cologne (DE)
- Institut für Medizinische Informationsverarbeitung, Biometrie und Epidemiologie (DE)
- Universitätsklinikum Würzburg (DE)
- Cochrane (GB)
- NHS Blood and Transplant (GB)
- Corresponding author
- Iannizzi, Claire
- ORCID
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2022-01
- Volume
- 141
- Pages
- 82-89
- ISSN
- 0895-4356 (ISSN-L)
- PMC
- PMC8435072
- OpenAlex
- W3201287655
- Cited by
- 45
- References
- 23
- Citations by year
- 2026: 3 · 2025: 3 · 2024: 11 · 2023: 17 · 2022: 11
- FWCI
- 0.41
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Kölner Universitäts PublikationsServer (Universität zu Köln) (publishedVersion)
- Funders
- National Institute for Health and Care Research (COVID-19-RECPLAS)
- Bundesministerium für Bildung und Forschung (01KX2021)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Decision Sciences › Information Systems and Management › Academic Publishing and Open Access
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- living systematic review, COVID-19 pandemic, methodological issues
- MeSH terms
- Systematic Reviews as Topic, COVID-19, Humans, Research Design, Pandemics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
