A surveillance system to assess the need for updating systematic reviews
Open accessThe DOI matches its Crossref record; 51 of the 56 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Systematic reviews (SRs) can become outdated as new evidence emerges over time. Organizations that produce SRs need a surveillance method to determine when reviews are likely to require updating. This report describes the development and initial results of a surveillance system to assess SRs produced by the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) Program. METHODS: Twenty-four SRs were assessed using existing methods that incorporate limited literature searches, expert opinion, and quantitative methods for the presence of signals triggering the need for updating. The system was designed to begin surveillance six months after the release of the original review, and then ceforth every six months for any review not classified as being a high priority for updating. The outcome of each round of surveillance was a classification of the SR as being low, medium or high priority for updating. RESULTS: Twenty-four SRs underwent surveillance at least once, and ten underwent surveillance a second time during the 18 months of the program. Two SRs were classified as high, five as medium, and 17 as low priority for updating. The time lapse between the searches conducted for the original reports and the updated searches (search time lapse - STL) ranged from 11 months to 62 months: The STL for the high priority reports were 29 months and 54 months; those for medium priority reports ranged from 19 to 62 months; and those for low priority reports ranged from 11 to 33 months. Neither the STL nor the number of new relevant articles was perfectly associated with a signal for updating. Challenges of implementing the surveillance system included determining what constituted the actual conclusions of an SR that required assessing; and sometimes poor response rates of experts. CONCLUSION: In this system of regular surveillance of 24 systematic reviews on a variety of clinical interventions produced by a leading organization, about 70% of reviews were determined to have a low priority for updating. Evidence suggests that the time period for surveillance is yearly rather than the six months used in this project.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“SRs conducted expeditiously may also be relevant if an existing SR is in need of updating or when the available resources are limited [51, 52].”
From How to conduct systematic reviews more expeditiously? (2015) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Systematic review and meta-analysis
(2019) · Providing Information for Health · doi:10.1201/9781315377315-19 · Cited by 2,659
Radical Prostatectomy versus Observation for Localized Prostate Cancer
Wilt, T. J., Brawer, M. K., Jones, K. M., Barry, M. J., Aronson, W. J., Fox, S., Gingrich, J. R., Wei, J. T., Gilhooly, P., Grob, B. M., et al. (2012) · New England Journal of Medicine · doi:10.1056/nejmoa1113162 · Cited by 1,770
How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis
Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., & Moher, D. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-4-200708210-00179 · Cited by 838 · cited 8×
“A study by Shojania and colleagues assessed 100 systematic reviews to determine how quickly they go out-ofdate, but this study limited its sample to meta-analyses that produced a summary estimate of outcome, and then further limited the analysis to only one outcome per study [7].”
BackgroundMethodInfluentialSelf-citationA Systematic Review of Early Intensive Intervention for Autism Spectrum Disorders
Warren, Z., McPheeters, M. L., Sathe, N., Foss-Feig, J. H., Glasser, A., & Veenstra-VanderWeele, J. (2011) · Pediatrics · doi:10.1542/peds.2011-0426 · Cited by 790
Shekelle, P. G., Ortiz, E., Rhodes, S., Morton, S. C., Eccles, M. P., Grimshaw, J. M., & Woolf, S. H. (2001) · JAMA · doi:10.1001/jama.286.12.1461 · Cited by 604
Self-citationBruening, W., Fontanarosa, J., Tipton, K., Treadwell, J. R., Launders, J., & Schoelles, K. (2010) · Annals of Internal Medicine · doi:10.7326/0003-4819-152-1-201001050-00190 · Cited by 251
Comparative Effectiveness of Pain Management Interventions for Hip Fracture: A Systematic Review
Abou-Setta, A. M., Beaupre, L. A., Rashiq, S., Dryden, D. M., Hamm, M. P., Sadowski, C. A., Menon, M. R. G., Majumdar, S. R., Wilson, D. M., Karkhaneh, M., et al. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-4-201108160-00346 · Cited by 250
Juusola, J. L. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-156-8-201204170-00004 · Cited by 244
Yank, V., Tuohy, C. V., Logan, A. C., Bravata, D. M., Staudenmayer, K., Eisenhut, R., Sundaram, V., McMahon, D., Olkin, I., McDonald, K. M., et al. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-154-8-201104190-00004 · Cited by 240
Systematic Review: Nonoperative and Operative Treatments for Rotator Cuff Tears
Seida, J. C., LeBlanc, C., Schouten, J. R., Mousavi, S. S., Hartling, L., Vandermeer, B., Tjosvold, L., & Sheps, D. M. (2010) · Annals of Internal Medicine · doi:10.7326/0003-4819-153-4-201008170-00263 · Cited by 164
Research works printed but not linked: 6
- Warren Z, Veenstra-Vanderweele J, Stone W, Bruzek JL, Nahmias AS, Foss-Feig JH, Jerome RN, Krishnaswami S, Sathe NA, Glasser AM, Surawicz T, McPheeters ML: Therapies for children with autism spectrum disorders. [ http://www.ncbi.nlm.nih.gov/books/NBK56343/pdf/TOC.pdf ]
- Guillamondegui OD, Montgomery SA, Phibbs FT, McPheeters ML, Alexander PT, Jerome RN, McKoy JN, Seroogy JJ, Eicken JJ, Krishnaswami S, Salomon RM, Hartmann KE: Traumatic brain injury and depression. [ http://www.ncbi.nlm.nih.gov/books/NBK62061/pdf/TOC.pdf ]
- Uhlig K, Balk EM, Patel K, Ip S, Kitsios GD, Obadan NO, Haynes SM, Stefan M, Rao M, Kong W, Chang L, Gaylor J, Iovin RC: Self-measured blood pressure monitoring: comparative effectiveness. [ http://www.ncbi.nlm.nih.gov/books/NBK84604/ ]
- Balk EM, Moorthy D, Obadan NO, Patel K, Ip S, Chung M, Bannuru RR, Kitsios GD, Sen S, Iovin RC, Gaylor JM, D'Ambrosio C, Lau J: Diagnosis and treatment of obstructive sleep apnea in adults. [ http://www.ncbi.nlm.nih.gov/books/NBK63560/ ]
- Seida JC, Schouten JR, Mousavi SS, Hamm M, Beaith A, Vandermeer B, Dryden DM, Boylan K, Newton AS, Carrey N: First & second generation antipsychotics for children and young adults. [ http://www.ncbi.nlm.nih.gov/books/NBK84643/ ]
- Charach A, Dashti B, Carson P, Booker L, Lim CG, Lillie E, Yeung E, Ma J, Raina P, Schachar R: Attention Deficit Hyperactivity Disorder (ADHD): effectiveness of treatment in at-risk preschoolers; long-term effectiveness in all ages; and variability in prevalence, diagnosis, and treatment. [ http://www.ncbi.nlm.nih.gov/books/NBK82368/ ]
Cited by
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
Self-citationThe COMET Handbook: version 1.0
Williamson, P. R., Altman, D. G., Bagley, H., Barnes, K. L., Blazeby, J. M., Brookes, S. T., Clarke, M., Gargon, E., Gorst, S., Harman, N., et al. (2017) · Trials · doi:10.1186/s13063-017-1978-4 · Cited by 2,011
“However, as with the updating of systematic reviews [196], there are uncertainties about the frequency with”
BackgroundOkoth, K., Chandan, J. S., Marshall, T., Thangaratinam, S., Thomas, G. N., Nirantharakumar, K., & Adderley, N. J. (2020) · BMJ · doi:10.1136/bmj.m3502 · Cited by 416
Thielen, F., Van Mastrigt, G., Burgers, L., Bramer, W., Majoie, H., Evers, S., & Kleijnen, J. (2016) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2016.1246962 · Cited by 132
How to conduct systematic reviews more expeditiously?
Tsertsvadze, A., Chen, Y.-F., Moher, D., Sutcliffe, P., & McCarthy, N. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0147-7 · Cited by 104
“SRs conducted expeditiously may also be relevant if an existing SR is in need of updating or when the available resources are limited [51, 52].”
BackgroundSelf-citation
Metadata
- Authors
- Ahmadzai, N., Newberry, S. J., Maglione, M. A., Tsertsvadze, A., Ansari, M. T., Hempel, S., Motala, A., Tsouros, S., Schneider Chafen, J. J., Shanman, R., Moher, D., & Shekelle, P. G.
- Affiliations
- Ottawa Hospital Research Institute (CA)
- RAND Corporation (US)
- Corresponding author
- Ahmadzai, Nadera
- ORCID
- Venue
- Systematic Reviews
- Published
- 2013-11-14
- Volume
- 2
- Issue
- 1
- Pages
- 104-104
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC3874670
- OpenAlex
- W2163687711
- Cited by
- 33
- References
- 56
- Citations by year
- 2025: 1 · 2024: 4 · 2022: 3 · 2021: 2 · 2020: 2 · 2018: 4 · 2017: 4 · 2016: 4 · 2015: 6 · 2014: 3
- FWCI
- 0.89
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/2.0 (tdm)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- uO Research (University of Ottawa) (submittedVersion, public-domain)
- eScholarship (California Digital Library) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- Funders
- U.S. Department of Health and Human Services
- University of Ottawa
- Agency for Healthcare Research and Quality
- Health Canada
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Keywords
- systematic review updating, surveillance systems, evidence-based practice centers, Agency for Healthcare Research and Quality, expert opinion
- MeSH terms
- Data Collection, Humans, Periodicals as Topic, Quality Control, Review Literature as Topic, Time Factors, United States, Information Storage and Retrieval, United States Agency for Healthcare Research and Quality, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A surveillance system to assess the need for updating systematic reviews. https://app.cochat.ai/papers/10.1186/2046-4053-2-104
- The paper
- Ahmadzai, N., Newberry, S. J., Maglione, M. A., Tsertsvadze, A., Ansari, M. T., Hempel, S., Motala, A., Tsouros, S., Schneider Chafen, J. J., Shanman, R., Moher, D., & Shekelle, P. G. (2013). A surveillance system to assess the need for updating systematic reviews. Systematic Reviews. https://doi.org/10.1186/2046-4053-2-104
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.