Evaluation of the interim Cochrane rapid review methods guidance—A mixed‐methods study on the understanding of and adherence to the guidance
Open accessAbstract
The Cochrane Rapid Review Methods Group (RRMG) first released interim guidance in March 2020 to support authors in conducting rapid reviews (RRs). The objective of this mixed‐methods study was to assess the adherence and investigate authors' understanding of the RRMG guidance. We identified all documents citing the Interim Cochrane RRMG guidance up to February 17, 2022 and performed an exploratory adherence analysis. We interviewed 20 RR authors to assess the recommendations' comprehensibility and reasons for any deviations. Further, we surveyed nine authors of COVID‐19‐related Cochrane reviews for their reasons for not conducting a RR. We analyzed 128 RRs (111 non‐Cochrane, 17 Cochrane) that cited the RRMG guidance. Several recommendations were not followed by a large proportion of RR authors such as stepwise approach to study design inclusion or peer review of search strategies, whereas others were exceeded, for example, dual independent screening of abstracts/full texts. The most reported reasons for deviating from the guidance were time constraints, unclarities in the recommended approach, or inapplicability to the specific RR. Overall, the guidance was viewed as user‐friendly; however, without pre‐existing knowledge of systematic review (SR) conduct, the application was perceived as difficult. The main reasons for conducting a full SR over a RR were late availability of the guidance, preset mandate to conduct a SR, uncertainty regarding methodological distinctions between SR and RR, and inapplicability to the evidence base. Clarifications are warranted throughout the Interim Cochrane RRMG guidance to ensure that users with various experience levels can understand and apply its recommendations accordingly.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The evaluation analysed the text of 128 rapid reviews (17 from Cochrane, 111 not) and conducted 20 in-depth qualitative interviews.
“This evaluation included the text analysis of 128 rapid reviews (17 Cochrane and 111 non-Cochrane) and 20 in-depth qualitative interviews.”
From Updated recommendations for the Cochrane rapid review methods guidance for rapid reviews of effectiveness (2024) · cites this paper as [7]
The update drew on a formal evaluation that examined how well authors followed and understood the interim guidance, and how comprehensible, usable, and useful it was.
“We have also integrated findings from a formal evaluation conducted to assess authors' adherence to and understanding of the interim Cochrane guidance and the guidance's comprehensibility, usability, and usefulness.”
From Updated recommendations for the Cochrane rapid review methods guidance for rapid reviews of effectiveness (2024) · cites this paper as [7]
A number of authors failed to follow some recommendations, for example the stepwise approach to including studies or peer review of search strategies.
“Main findings suggested that many authors did not follow certain recommendations, such as the stepwise approach to study inclusion or peer review of search strategies.”
From Updated recommendations for the Cochrane rapid review methods guidance for rapid reviews of effectiveness (2024) · cites this paper as [7]
Frequent reasons given for non-adherence were lack of time, unclear recommendations, or recommendations not fitting particular rapid reviews.
“Common reasons for not adhering to the guidance included time constraints, unclear recommended approaches, or inapplicability to specific rapid reviews.”
From Updated recommendations for the Cochrane rapid review methods guidance for rapid reviews of effectiveness (2024) · cites this paper as [7]
For other recommendations, such as dual independent screening of abstracts or full texts, adherence went beyond what was asked.
“However, some recommendations, such as dual independent screening of abstracts or full texts, were exceeded.”
From Updated recommendations for the Cochrane rapid review methods guidance for rapid reviews of effectiveness (2024) · cites this paper as [7]
Users found the guidance easy to use, yet hard to apply without prior experience of doing systematic reviews.
“Overall, the guidance was considered user friendly, but it was perceived as challenging to apply without experience of conducting systematic reviews.”
From Updated recommendations for the Cochrane rapid review methods guidance for rapid reviews of effectiveness (2024) · cites this paper as [7]
The guidance is uniquely based on empirical evidence about rapid review methods, supplemented by expert input and feedback from a formal evaluation of how it was used in practice.
“Although these other guides are also evidence informed and we encourage their use, our guide is uniquely grounded in empirical evidence evaluating methods of rapid reviews, 12 further enriched by expert input [13][14][15][16] and feedback from a formal evaluation of its real world use.7”
From Updated recommendations for the Cochrane rapid review methods guidance for rapid reviews of effectiveness (2024) · cites this paper as [7]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414 · see citing works
Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L., François, R., Grolemund, G., Hayes, A., Henry, L., Hester, J., et al. (2019) · Journal of Open Source Software · doi:10.21105/joss.01686 · Cited by 22,138 · see citing works
Standards for Reporting Qualitative Research: A Synthesis of Recommendations
O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., & Cook, D. A. (2014) · Academic Medicine · doi:10.1097/acm.0000000000000388 · Cited by 12,211 · see citing works
Designing and Conducting Mixed Methods Research
นิลพันธุ์, ม. (2010) · Silpakorn Educational Research Journal · Cited by 9,350 · see citing works
Eysenbach, G. (2004) · Journal of Medical Internet Research · doi:10.2196/jmir.6.3.e34 · Cited by 6,480 · see citing works
Qualitative data analysis for applied policy research
Ritchie, J., & Spencer, L. (2010) · Analyzing qualitative data · doi:10.4324/9780203413081_chapter_9 · Cited by 4,800 · see citing works
A scoping review of rapid review methods
Tricco, A. C., Antony, J., Zarin, W., Strifler, L., Ghassemi, M., Ivory, J., Perrier, L., Hutton, B., Moher, D., & Straus, S. E. (2015) · BMC Medicine · doi:10.1186/s12916-015-0465-6 · Cited by 1,146 · see citing works
Cochrane Rapid Reviews Methods Group offers evidence-informed guidance to conduct rapid reviews
Garritty, C., Gartlehner, G., Nussbaumer-Streit, B., King, V. J., Hamel, C., Kamel, C., Affengruber, L., & Stevens, A. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.10.007 · Cited by 1,141 · see citing works
Self-citationEpidemiology and reporting characteristics of systematic reviews.
Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M., & Altman, D. G. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040078 · Cited by 891 · see citing works
Expediting systematic reviews: methods and implications of rapid reviews
Ganann, R., Ciliska, D., & Thomas, H. (2010) · Implementation Science · doi:10.1186/1748-5908-5-56 · Cited by 752 · see citing works
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737 · see citing works
Hamel, C., Michaud, A., Thuku, M., Skidmore, B., Stevens, A., Nussbaumer-Streit, B., & Garritty, C. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.09.041 · Cited by 384 · see citing works
Shemilt, I., Khan, N., Park, S., & Thomas, J. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0315-4 · Cited by 168 · see citing works
Kelly, S., Moher, D., & Clifford, T. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0258-9 · Cited by 144 · see citing works
Redefining rapid reviews: a flexible framework for restricted systematic reviews
Plüddemann, A., Aronson, J. K., Onakpoya, I., Heneghan, C. J., & Mahtani, K. R. (2018) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2018-110990 · Cited by 144 · see citing works
Rapid reviews: the pros and cons of an accelerated review process
Moons, P., Goossens, E., & Thompson, D. R. (2021) · European Journal of Cardiovascular Nursing · doi:10.1093/eurjcn/zvab041 · Cited by 139 · see citing works
Thinking Aloud: Insights Into Information Processing
Payne, J. W. (1994) · Psychological Science · doi:10.1111/j.1467-9280.1994.tb00620.x · Cited by 130 · see citing works
Resource use during systematic review production varies widely: a scoping review
Nussbaumer-Streit, B., Ellen, M., Klerings, I., Sfetcu, R., Riva, N., Mahmić-Kaknjo, M., Poulentzas, G., Martinez, P., Baladia, E., Ziganshina, L. E., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.05.019 · Cited by 98 · see citing works
Self-citationFit for purpose: perspectives on rapid reviews from end-user interviews
Hartling, L., Guise, J.-M., Hempel, S., Featherstone, R., Mitchell, M. D., Motu’apuaka, M. L., Robinson, K. A., Schoelles, K., Totten, A., Whitlock, E., et al. (2017) · Systematic Reviews · doi:10.1186/s13643-017-0425-7 · Cited by 65 · see citing works
Thigpen, S., Puddy, R. W., Singer, H. H., & Hall, D. M. (2012) · American Journal of Community Psychology · doi:10.1007/s10464-012-9537-3 · Cited by 48 · see citing works
Do policy-makers find commissioned rapid reviews useful?
Moore, G., Redman, S., Rudge, S., & Haynes, A. (2018) · Health Research Policy and Systems · doi:10.1186/s12961-018-0293-1 · Cited by 41 · see citing works
The use of rapid review methods for the U.S. Preventive services task force
Patnode, C. D., Eder, M. L., Walsh, E. S., Viswanathan, M., & Lin, J. S. (2018) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2017.07.024 · Cited by 40 · see citing works
The use of rapid review methods in health technology assessments: 3 case studies
Kaltenthaler, E., Cooper, K. L., Pandor, A., James, M. M., Chatters, R., & Wong, R. (2016) · BMC Medical Research Methodology · doi:10.1186/s12874-016-0216-1 · Cited by 38 · see citing works
Methodological guidance for rapid reviews in healthcare: A scoping review
Speckemeier, C., Niemann, A., Wasem, J., Buchberger, B., & Neusser, S. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1555 · Cited by 34 · see citing works
Hunter, J., Arentz, S., Goldenberg, J. Z., Yang, G., Beardsley, J., Lee, M. S., & Myers, S. P. (2020) · Integrative Medicine Research · doi:10.1016/j.imr.2020.100484 · Cited by 20 · see citing works
Neilson, C. J. (2021) · Health Information & Libraries Journal · doi:10.1111/hir.12367 · Cited by 16 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Garritty, C., Hamel, C., Trivella, M., Gartlehner, G., Nussbaumer-Streit, B., Devane, D., Kamel, C., Griebler, U., & King, V. J. (2024) · BMJ · doi:10.1136/bmj-2023-076335 · Cited by 260 · see citing works
Self-citation
- Authors
- Griebler, U., Dobrescu, A., Ledinger, D., Klingenstein, P., Sommer, I., Emprechtinger, R., Persad, E., Gadinger, A., Trivella, M., Klerings, I., & Nussbaumer‐Streit, B.
- Affiliations
- Universität für Weiterbildung Krems (AT)
- Department for Evidence‐based Medicine and Evaluation University of Krems Krems an der Donau Austria
- Corresponding author
- Griebler, Ursula
- ORCID
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2023-07-21
- Volume
- 14
- Issue
- 6
- Pages
- 824-846
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W4385074669
- Cited by
- 5
- References
- 27
- Citations by year
- 2026: 1 · 2025: 1 · 2024: 3
- FWCI
- 0.93
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Research Synthesis Methods (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Social Sciences › Sociology and Political Science › Delphi Technique in Research
- Keywords
- rapid review, methodological recommendations
- MeSH terms
- Systematic Reviews as Topic, COVID-19, Humans, Research Design, Review Literature as Topic, Guidelines as Topic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
