CoChat

A surveillance system to assess the need for updating systematic reviews

Open access

The DOI matches its Crossref record; 51 of the 56 references it cites were found in a registry or an index.

Abstract

BACKGROUND: Systematic reviews (SRs) can become outdated as new evidence emerges over time. Organizations that produce SRs need a surveillance method to determine when reviews are likely to require updating. This report describes the development and initial results of a surveillance system to assess SRs produced by the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) Program. METHODS: Twenty-four SRs were assessed using existing methods that incorporate limited literature searches, expert opinion, and quantitative methods for the presence of signals triggering the need for updating. The system was designed to begin surveillance six months after the release of the original review, and then ceforth every six months for any review not classified as being a high priority for updating. The outcome of each round of surveillance was a classification of the SR as being low, medium or high priority for updating. RESULTS: Twenty-four SRs underwent surveillance at least once, and ten underwent surveillance a second time during the 18 months of the program. Two SRs were classified as high, five as medium, and 17 as low priority for updating. The time lapse between the searches conducted for the original reports and the updated searches (search time lapse - STL) ranged from 11 months to 62 months: The STL for the high priority reports were 29 months and 54 months; those for medium priority reports ranged from 19 to 62 months; and those for low priority reports ranged from 11 to 33 months. Neither the STL nor the number of new relevant articles was perfectly associated with a signal for updating. Challenges of implementing the surveillance system included determining what constituted the actual conclusions of an SR that required assessing; and sometimes poor response rates of experts. CONCLUSION: In this system of regular surveillance of 24 systematic reviews on a variety of clinical interventions produced by a leading organization, about 70% of reviews were determined to have a low priority for updating. Evidence suggests that the time period for surveillance is yearly rather than the six months used in this project.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/2046-4053-2-104

Cites

51 of 51 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 The process of surveillance assessment for a systematic review (SR). Figure 1 portrays the overall process of surveillance assessment for an SR that mainly includes: 1) literature search, 2) contacting experts, and 3) obtaining safety alerts from various sources sent by ECRI (one of the AHRQ evidence-based centers). The number of hits identified by literature search would be transferred to Reference Manager database and then will be screened by: 1) title and abstract, and 2) full text. The data was extracted from the number of studies that were deemed eligible for inclusion. Next, the extracted data was assessed for identifying qualitative and quantitative signals. Then, the findings from literature, expert opinion, and safety alerts were collated and assessed for updating priority status (high, medium or low). If an SR was deemed as ‘high’ priority for assessment, it was referred to AHRQ for updating. If an SR was deemed as ‘medium’ or ‘high’ priority for updating, it was re-assessed six months after the completion of the first assessment.
Figure 1 The process of surveillance assessment for a systematic review (SR). Figure 1 portrays the overall process of surveillance assessment for an SR that mainly includes: 1) literature search, 2) contacting experts, and 3) obtaining safety alerts from various sources sent by ECRI (one of the AHRQ evidence-based centers). The number of hits identified by literature search would be transferred to Reference Manager database and then will be screened by: 1) title and abstract, and 2) full text. The data was extracted from the number of studies that were deemed eligible for inclusion. Next, the extracted data was assessed for identifying qualitative and quantitative signals. Then, the findings from literature, expert opinion, and safety alerts were collated and assessed for updating priority status (high, medium or low). If an SR was deemed as ‘high’ priority for assessment, it was referred to AHRQ for updating. If an SR was deemed as ‘medium’ or ‘high’ priority for updating, it was re-assessed six months after the completion of the first assessment.CC BY · PubMed Central
Figure 2 The process of surveillance assessment for a Systematic Review. (a) Time elapsed since the search date in the original review. Green color: low priority for updating; Yellow color: medium priority for updating; red color: high priority for updating. (b) Number of new relevant articles identified. Green color: low priority for updating; Yellow color: medium priority for updating; red color: high priority for updating.
Figure 2 The process of surveillance assessment for a Systematic Review. (a) Time elapsed since the search date in the original review. Green color: low priority for updating; Yellow color: medium priority for updating; red color: high priority for updating. (b) Number of new relevant articles identified. Green color: low priority for updating; Yellow color: medium priority for updating; red color: high priority for updating.CC BY · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 51 references

Research works printed but not linked: 6

  1. Warren Z, Veenstra-Vanderweele J, Stone W, Bruzek JL, Nahmias AS, Foss-Feig JH, Jerome RN, Krishnaswami S, Sathe NA, Glasser AM, Surawicz T, McPheeters ML: Therapies for children with autism spectrum disorders. [ http://www.ncbi.nlm.nih.gov/books/NBK56343/pdf/TOC.pdf ]
  2. Guillamondegui OD, Montgomery SA, Phibbs FT, McPheeters ML, Alexander PT, Jerome RN, McKoy JN, Seroogy JJ, Eicken JJ, Krishnaswami S, Salomon RM, Hartmann KE: Traumatic brain injury and depression. [ http://www.ncbi.nlm.nih.gov/books/NBK62061/pdf/TOC.pdf ]
  3. Uhlig K, Balk EM, Patel K, Ip S, Kitsios GD, Obadan NO, Haynes SM, Stefan M, Rao M, Kong W, Chang L, Gaylor J, Iovin RC: Self-measured blood pressure monitoring: comparative effectiveness. [ http://www.ncbi.nlm.nih.gov/books/NBK84604/ ]
  4. Balk EM, Moorthy D, Obadan NO, Patel K, Ip S, Chung M, Bannuru RR, Kitsios GD, Sen S, Iovin RC, Gaylor JM, D'Ambrosio C, Lau J: Diagnosis and treatment of obstructive sleep apnea in adults. [ http://www.ncbi.nlm.nih.gov/books/NBK63560/ ]
  5. Seida JC, Schouten JR, Mousavi SS, Hamm M, Beaith A, Vandermeer B, Dryden DM, Boylan K, Newton AS, Carrey N: First & second generation antipsychotics for children and young adults. [ http://www.ncbi.nlm.nih.gov/books/NBK84643/ ]
  6. Charach A, Dashti B, Carson P, Booker L, Lim CG, Lillie E, Yeung E, Ma J, Raina P, Schachar R: Attention Deficit Hyperactivity Disorder (ADHD): effectiveness of treatment in at-risk preschoolers; long-term effectiveness in all ages; and variability in prevalence, diagnosis, and treatment. [ http://www.ncbi.nlm.nih.gov/books/NBK82368/ ]

Cited by

See all 33

Metadata

Authors
Ahmadzai, N., Newberry, S. J., Maglione, M. A., Tsertsvadze, A., Ansari, M. T., Hempel, S., Motala, A., Tsouros, S., Schneider Chafen, J. J., Shanman, R., Moher, D., & Shekelle, P. G.
Affiliations
  • Ottawa Hospital Research Institute (CA)
  • RAND Corporation (US)
Corresponding author
Ahmadzai, Nadera
Venue
Systematic Reviews
Published
2013-11-14
Volume
2
Issue
1
Pages
104-104
ISSN
2046-4053 (ISSN-L)
OpenAlex
W2163687711
Cited by
33
References
56
Citations by year
2025: 1 · 2024: 4 · 2022: 3 · 2021: 2 · 2020: 2 · 2018: 4 · 2017: 4 · 2016: 4 · 2015: 6 · 2014: 3
FWCI
0.89
Type
journal article
Language
en
Open access
Licence
  • cc-by
  • creativecommons.org/licenses/by/2.0 (tdm)
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • Europe PMC (PubMed Central) (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • uO Research (University of Ottawa) (submittedVersion, public-domain)
  • eScholarship (California Digital Library) (submittedVersion)
  • eScholarship (California Digital Library) (submittedVersion, cc-by)
Funders
  • U.S. Department of Health and Human Services
  • University of Ottawa
  • Agency for Healthcare Research and Quality
  • Health Canada
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
  • Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
Keywords
systematic review updating, surveillance systems, evidence-based practice centers, Agency for Healthcare Research and Quality, expert opinion
MeSH terms
Data Collection, Humans, Periodicals as Topic, Quality Control, Review Literature as Topic, Time Factors, United States, Information Storage and Retrieval, United States Agency for Healthcare Research and Quality, Evidence-Based Medicine

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: A surveillance system to assess the need for updating systematic reviews. https://app.cochat.ai/papers/10.1186/2046-4053-2-104
The paper
Ahmadzai, N., Newberry, S. J., Maglione, M. A., Tsertsvadze, A., Ansari, M. T., Hempel, S., Motala, A., Tsouros, S., Schneider Chafen, J. J., Shanman, R., Moher, D., & Shekelle, P. G. (2013). A surveillance system to assess the need for updating systematic reviews. Systematic Reviews. https://doi.org/10.1186/2046-4053-2-104

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.