CoChat
Sign Up
Cited by 93Meta-analysis and systematic reviewsClinical practice guidelines implementationDelphi Technique in Research

A system for rating the stability and strength of medical evidence

Open access
Treadwell, Jonathan R, Tregear, Stephen J, Reston, James T, Turkelson, Charles M·2006·BMC Medical Research Methodology
DOI 10.1186/1471-2288-6-52PMC PMC1624842
Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Methods for describing one's confidence in the available evidence are useful for end-users of evidence reviews. Analysts inevitably make judgments about the quality, quantity consistency, robustness, and magnitude of effects observed in the studies identified. The subjectivity of these judgments in several areas underscores the need for transparency in judgments. DISCUSSION: This paper introduces a new system for rating medical evidence. The system requires explicit judgments and provides explicit rules for balancing these judgments. Unlike other systems for rating the strength of evidence, our system draws a distinction between two types of conclusions: quantitative and qualitative. A quantitative conclusion addresses the question, "How well does it work?", whereas a qualitative conclusion addresses the question, "Does it work?" In our system, quantitative conclusions are tied to stability ratings, and qualitative conclusions are tied to strength ratings. Our system emphasizes extensive a priori criteria for judgments to reduce the potential for bias. Further, the system makes explicit the impact of heterogeneity testing, meta-analysis, and sensitivity analyses on evidence ratings. This article provides details of our system, including graphical depictions of how the numerous judgments that an analyst makes can be combined. We also describe two worked examples of how the system can be applied to both interventional and diagnostic technologies. SUMMARY: Although explicit judgments and formal combination rules are two important steps on the path to a comprehensive system for rating medical evidence, many additional steps must also be taken. Foremost among these are the distinction between quantitative and qualitative conclusions, an extensive set of a priori criteria for making judgments, and the direct impact of analytic results on evidence ratings. These attributes form the basis for a logically consistent system that can improve the usefulness of evidence reviews.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/1471-2288-6-52

Cites

29 of 31 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The minimum number of studies for a meta-analysis is three.

    “the minimum number of studies for a meta -analysis is only three (Treadwell et al., 2006)”

    From Cross-age peer mentoring for youth: A meta-analysis (2022) · cites this paper as Treadwell et al., 2006

    In “How the System Works,” the paper explicitly says to perform a random-effects meta-analysis when there are only two studies with calculable effect sizes. Its three-study thresholds concern particular quantitative estimates or conclusions, not the minimum number for any meta-analysis. (LLM, conf=0.90)

    Checked by a reasoning model — may be wrong.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 2

  1. Levels of evidence and grades of recommendation. [internet] Oxford Centre for Evidence-Based Medicine, University Department of Psychiatry, Warneford Hospital 4 [cited 2003 Apr 8]
  2. Olkin I Diagnostic statistical procedures in medical meta-analysis Stat Med 18 2331 41 1999 Sep 15; 10474143

Cited by

Sorted by citations

Related

Authors
Treadwell, J. R., Tregear, S. J., Reston, J. T., & Turkelson, C. M.
Affiliations
  • ECRI Institute (US)
  • American Academy of Orthopaedic Surgeons (US)
Corresponding author
Treadwell, Jonathan R
ORCID
Venue
BMC Medical Research Methodology
Publisher
Springer Science and Business Media LLC
Published
2006-10-19
Volume
6
Issue
1
Pages
52-52
ISSN
1471-2288 (ISSN-L)
DOI
10.1186/1471-2288-6-52
PMC
PMC1624842
OpenAlex
W1862433166
Cited by
93
References
29
Citations by year
2026: 1 · 2025: 2 · 2024: 3 · 2023: 1 · 2021: 9 · 2020: 3 · 2019: 4 · 2018: 3 · 2017: 2 · 2016: 6 · 2015: 3 · 2013: 4 · 2012: 10
FWCI
7.62
Type
journal article
Language
en
Open access
Licence
Locations
  • BMC Medical Research Methodology (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
  • Social Sciences › Sociology and Political Science › Delphi Technique in Research
Keywords
evidence rating, quality of evidence, combination rules, heterogeneity testing, meta-analysis, sensitivity analysis, bias mitigation, diagnostic technologies
MeSH terms
Evaluation Studies as Topic, Humans, Judgment, Sensitivity and Specificity, Technology Assessment, Biomedical, Meta-Analysis as Topic, Practice Guidelines as Topic, Risk Assessment, Evidence-Based Medicine, Qualitative Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review