Trading certainty for speed - how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey
Open accessThe DOI matches its Crossref record; 27 of the 27 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Decisionmakers and guideline developers demand rapid syntheses of the evidence when time sensitive evidence-informed decisions are required. A potential trade-off of such rapid reviews is that their results can have less reliability than results of systematic reviews that can lead to an increased risk of making incorrect decisions or recommendations. We sought to determine how much incremental uncertainty about the correctness of an answer guideline developers and health policy decisionmakers are willing to accept in exchange for a rapid evidence-synthesis. METHODS: Employing a purposive sample, we conducted an international web-based, anonymous survey of decisionmakers and guideline developers. Based on a clinical treatment, a public health, and a clinical prevention scenario, participants indicated the maximum risk of getting an incorrect answer from a rapid review that they would be willing to accept. We carefully reviewed data and performed descriptive statistical analyses. RESULTS: In total, 325 (58.5%) of 556 participants completed our survey and were eligible for analysis. The median acceptable incremental risk for getting an incorrect answer from a rapid review across all three scenarios was 10.0% (interquartile range [IQR] 5.0-15.0). Acceptable risks were similar for the clinical treatment (n = 313, median 10.0% [IQR 5.0-15.0]) and the public health scenarios (n = 320, median 10.0% [IQR 5.0-15.0]) and lower for the clinical prevention scenario (n = 312, median 6.5% [IQR 5.0-10.5]). CONCLUSIONS: Findings suggest that decisionmakers are willing to accept some trade-off in validity in exchange for a rapid review. Nevertheless, they expect the validity of rapid reviews to come close to that of systematic reviews.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“A survey of stakeholders (e.g., policy-makers, healthcare providers) reported that the median acceptable incremental risk of getting an incorrect answer from a rapid review is 10% (interquartile range of 5–15%) [35].”
From An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes (2020) · cites this paper as [35]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Eysenbach, G. (2004) · Journal of Medical Internet Research · doi:10.2196/jmir.6.3.e34 · Cited by 6,480 · cited 1×
“Throughout the manuscript we adhere to the Checklist for Reporting Results of Internet E-Surveys (CHERRIES) (Additional file 1) [16].”
MethodQualitative Evaluation and Research Methods
Welch, J. K., & Patton, M. Q. (1992) · The Modern Language Journal · doi:10.2307/330063 · Cited by 3,306
Systematic Reviews: Synthesis of Best Evidence for Clinical Decisions
Cook, D. J., Mulrow, C. D., & Haynes, R. B. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-126-5-199703010-00006 · Cited by 2,184 · cited 1×
“Well conducted systematic reviews employ high methodological standards to address clinical or healthcare policy questions by identifying, appraising, and summarizing primary research [2].”
BackgroundEvidence summaries: the evolution of a rapid review approach
Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-10 · Cited by 1,232 · cited 1×
“As a consequence, rapid reviews have become a pragmatic alternative to systematic reviews [4].”
BackgroundEgger, M., Jüni, P., Bartlett, C., Holenstein, F., & Sterne, J. (2003) · Health Technology Assessment · doi:10.3310/hta7010 · Cited by 1,181
A scoping review of rapid review methods
Tricco, A. C., Antony, J., Zarin, W., Strifler, L., Ghassemi, M., Ivory, J., Perrier, L., Hutton, B., Moher, D., & Straus, S. E. (2015) · BMC Medicine · doi:10.1186/s12916-015-0465-6 · Cited by 1,146
Kulkarni, A. V. (2009) · JAMA · doi:10.1001/jama.2009.1307 · Cited by 817
Expediting systematic reviews: methods and implications of rapid reviews
Ganann, R., Ciliska, D., & Thomas, H. (2010) · Implementation Science · doi:10.1186/1748-5908-5-56 · Cited by 752 · cited 1×
“A drawback of systematic reviews, however, is that the amount of time they require to complete (a minimum of 6 months to 1 year) can be too long for the timesensitive needs of decisionmakers [3].”
BackgroundVarieties of Uncertainty in Health Care: A Conceptual Taxonomy
Han, P. K. J., Klein, W. M. P., & Arora, N. K. (2011) · Medical Decision Making · doi:10.1177/0272989x10393976 · Cited by 747 · cited 2×
“Han and colleagues distinguish among three sources of uncertainty in healthcare, namely probability (the likelihood of a future event), ambiguity (possible different states for a single event), and complexity (the multiplicity of causal factors and interpretive cues that make a phenomenon difficult to comprehend) [15].”
BackgroundVarieties of Uncertainty in Health Care: A Conceptual Taxonomy
Han, P. K. J., Klein, W. M. P., & Arora, N. K. (2011) · Medical Decision Making · doi:10.1177/0272989x11393976 · Cited by 589
Cited by
Cooper, C., Booth, A., Varley-Campbell, J., Britten, N., & Garside, R. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0545-3 · Cited by 561
“It may be that decision makers and users of systematic reviews are willing to trade the certainty from a comprehensive literature search and systematic review in exchange for different approaches to evidence synthesis [58], and that comprehensive literature searches are not necessarily a marker of literature search qu…”
BackgroundNussbaumer-Streit, B., Klerings, I., Dobrescu, A. I., Persad, E., Stevens, A., Garritty, C., Kamel, C., Affengruber, L., King, V. J., & Gartlehner, G. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2019.10.011 · Cited by 440
Self-citationGartlehner, G., Affengruber, L., Titscher, V., Noel-Storr, A., Dooley, G., Ballarini, N., & König, F. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.005 · Cited by 206
Self-citationPrioritising references for systematic reviews with RobotAnalyst: A user study
Przybyła, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M., McNaught, J., von Elm, E., Nolan, K., & Ananiadou, S. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1311 · Cited by 148
“In the case of a rapid review, 82 it may not be as essential to finish screening once enough relevant evidence has been identified.”
BackgroundRedefining rapid reviews: a flexible framework for restricted systematic reviews
Plüddemann, A., Aronson, J. K., Onakpoya, I., Heneghan, C., & Mahtani, K. R. (2018) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2018-110990 · Cited by 143
Metadata
- Authors
- Wagner, G., Nussbaumer-Streit, B., Greimel, J., Ciapponi, A., & Gartlehner, G.
- Affiliations
- Universität für Weiterbildung Krems (AT)
- University of Hohenheim (DE)
- Instituto de Efectividad Clínica y Sanitaria (AR)
- Corresponding author
- Wagner, Gernot
- Venue
- BMC Medical Research Methodology
- Published
- 2017-08-14
- Volume
- 17
- Issue
- 1
- Pages
- 121-121
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC5557322
- OpenAlex
- W2747096592
- Cited by
- 55
- References
- 27
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 10 · 2023: 1 · 2022: 9 · 2021: 5 · 2020: 14 · 2019: 7 · 2018: 6
- FWCI
- 6.61
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Americanae (AECID Library) (publishedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- LA Referencia (Red Federada de Repositorios Institucionales de Publicaciones Científicas) (submittedVersion, cc-by-nc-sa)
- Funders
- This work was supported by internal funds of Cochrane Austria.
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Keywords
- rapid review, evidence synthesis, guideline developers, acceptable risk, systematic reviews, clinical prevention, evidence-informed decision-making
- MeSH terms
- Adult, Aged, Decision Making, Female, Humans, Male, Middle Aged, Surveys and Questionnaires, Review Literature as Topic, Reproducibility of Results, Practice Guidelines as Topic, Evidence-Based Medicine, Uncertainty, Young Adult
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Trading certainty for speed - how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey. https://app.cochat.ai/papers/10.1186/s12874-017-0406-5
- The paper
- Wagner, G., Nussbaumer-Streit, B., Greimel, J., Ciapponi, A., & Gartlehner, G. (2017). Trading certainty for speed - how much uncertainty are decisionmakers and guideline developers willing to accept when using rapid reviews: an international survey. BMC Medical Research Methodology. https://doi.org/10.1186/s12874-017-0406-5
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

