CoChat

Two methods provide similar signals for the need to update systematic reviews

Open access
Chung, Mei, Newberry, Sydne J., Ansari, Mohammed T., Yu, Winifred W., Wu, Helen, Lee, Jounghee, Suttorp, Marika, Gaylor, James M., Motala, Aneesa, Moher, David, Balk, Ethan M., Shekelle, Paul G. · 2012 · Journal of Clinical Epidemiology
Cited by 42

The DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.

Abstract

Objective Apply and compare two methods that identify signals for the need to update systematic reviews, using three Evidence-based Practice Center reports on omega-3 fatty acids as test cases. Study Design and Setting We applied the RAND method, which uses domain (subject matter) expert guidance, and a modified Ottawa method, which uses quantitative and qualitative signals. For both methods, we conducted focused electronic literature searches of recent studies using the key terms from the original reports. We assessed the agreement between the methods and qualitatively assessed the merits of each system. Results Agreement between the two methods was “substantial” or better (kappa > 0.62) in three of the four systematic reviews. Overall agreement between the methods was “substantial” (kappa = 0.64, 95% confidence interval [CI] 0.45–0.83). Conclusion The RAND and modified Ottawa methods appear to provide similar signals for the possible need to update systematic reviews in this pilot study. Future evaluation with a broader range of clinical topics and eventual comparisons between signals to update reports and the results of full evidence review updates will be needed. We propose a hybrid approach combining the best features of both methods, which should allow efficient review and assessment of the need to update.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1016/j.jclinepi.2011.12.004

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 9

  1. Whitlock, 2010, AHRQ series paper 3: identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the effective health-care program, J Clin Epidemiol, 63, 491, 10.1016/j.jclinepi.2009.03.008 · DOI
  2. Garritty, 2010, Updating systematic reviews: an international survey, PLoS ONE, 5, e9914, 10.1371/journal.pone.0009914 · DOI · Paper page
  3. The Cochrane Collaboration. Maintaining your review.Accessed Jan 10, 2011. Available at http://www.cochrane-net.org/openlearning/html/mod19-2.htm. 2002.
  4. French, 2005, Investing in updating: how do conclusions change when Cochrane systematic reviews are updated?, BMC Med Res Methodol, 5, 33, 10.1186/1471-2288-5-33 · DOI · Paper page
  5. Shekelle, 2001, When should clinical guidelines be updated?, BMJ, 323, 155, 10.1136/bmj.323.7305.155 · DOI · Paper page
  6. Shekelle, 2001, Validity of the Agency for Healthcare Research and Quality Clinical Practice Guidelines: how quickly do guidelines become outdated?, JAMA, 286, 1461, 10.1001/jama.286.12.1461 · DOI · Paper page
  7. Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, Sampson M, et al. When and how to update systematic reviews. Cochrane Database Syst Rev 2008;MR000023. · DOI · Paper page
  8. Balk, 2007, A collaborative effort to apply the evidence-based review process to the field of nutrition: challenges, benefits, and lessons learned, Am J Clin Nutr, 85, 1448, 10.1093/ajcn/85.6.1448 · DOI
  9. Landis, 1977, The measurement of observer agreement for categorical data, Biometrics, 33, 159, 10.2307/2529310 · DOI · Paper page

Cited by

See all 42

Metadata

Authors
Chung, M., Newberry, S. J., Ansari, M. T., Yu, W. W., Wu, H., Lee, J., Suttorp, M., Gaylor, J. M., Motala, A., Moher, D., Balk, E. M., & Shekelle, P. G.
Venue
Journal of Clinical Epidemiology
Publisher
Elsevier BV
Published
2012-06
Volume
65
Issue
6
Pages
660-668
ISSN
0895-4356 (ISSN-L)
OpenAlex
W1987451225
Cited by
42
References
4
FWCI
2.12
Type
journal article
Language
en
Open access
green
Licence
www.elsevier.com/tdm/userlicense/1.0/ (tdm)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.