A surveillance system to assess the need for updating systematic reviews
Open accessThe DOI matches its Crossref record; 51 of the 56 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Systematic reviews (SRs) can become outdated as new evidence emerges over time. Organizations that produce SRs need a surveillance method to determine when reviews are likely to require updating. This report describes the development and initial results of a surveillance system to assess SRs produced by the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) Program. METHODS: Twenty-four SRs were assessed using existing methods that incorporate limited literature searches, expert opinion, and quantitative methods for the presence of signals triggering the need for updating. The system was designed to begin surveillance six months after the release of the original review, and then ceforth every six months for any review not classified as being a high priority for updating. The outcome of each round of surveillance was a classification of the SR as being low, medium or high priority for updating. RESULTS: Twenty-four SRs underwent surveillance at least once, and ten underwent surveillance a second time during the 18 months of the program. Two SRs were classified as high, five as medium, and 17 as low priority for updating. The time lapse between the searches conducted for the original reports and the updated searches (search time lapse - STL) ranged from 11 months to 62 months: The STL for the high priority reports were 29 months and 54 months; those for medium priority reports ranged from 19 to 62 months; and those for low priority reports ranged from 11 to 33 months. Neither the STL nor the number of new relevant articles was perfectly associated with a signal for updating. Challenges of implementing the surveillance system included determining what constituted the actual conclusions of an SR that required assessing; and sometimes poor response rates of experts. CONCLUSION: In this system of regular surveillance of 24 systematic reviews on a variety of clinical interventions produced by a leading organization, about 70% of reviews were determined to have a low priority for updating. Evidence suggests that the time period for surveillance is yearly rather than the six months used in this project.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“SRs conducted expeditiously may also be relevant if an existing SR is in need of updating or when the available resources are limited [51, 52].”
From How to conduct systematic reviews more expeditiously? (2015) · cites this paper as [52]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Systematic review and meta-analysis
(2019) · Providing Information for Health · doi:10.1201/9781315377315-19 · Cited by 2,659
Radical Prostatectomy versus Observation for Localized Prostate Cancer
Wilt, T. J., Brawer, M. K., Jones, K. M., Barry, M. J., Aronson, W. J., Fox, S., Gingrich, J. R., Wei, J. T., Gilhooly, P., Grob, B. M., et al. (2012) · New England Journal of Medicine · doi:10.1056/nejmoa1113162 · Cited by 1,770
How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis
Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., & Moher, D. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-4-200708210-00179 · Cited by 838 · cited 8×
“A study by Shojania and colleagues assessed 100 systematic reviews to determine how quickly they go out-ofdate, but this study limited its sample to meta-analyses that produced a summary estimate of outcome, and then further limited the analysis to only one outcome per study [7].”
BackgroundMethodInfluentialSelf-citationA Systematic Review of Early Intensive Intervention for Autism Spectrum Disorders
Warren, Z., McPheeters, M. L., Sathe, N., Foss-Feig, J. H., Glasser, A., & Veenstra-VanderWeele, J. (2011) · Pediatrics · doi:10.1542/peds.2011-0426 · Cited by 790
Shekelle, P. G., Ortiz, E., Rhodes, S., Morton, S. C., Eccles, M. P., Grimshaw, J. M., & Woolf, S. H. (2001) · JAMA · doi:10.1001/jama.286.12.1461 · Cited by 604
Self-citationBruening, W., Fontanarosa, J., Tipton, K., Treadwell, J. R., Launders, J., & Schoelles, K. (2010) · Annals of Internal Medicine · doi:10.7326/0003-4819-152-1-201001050-00190 · Cited by 251
Comparative Effectiveness of Pain Management Interventions for Hip Fracture: A Systematic Review
Abou-Setta, A. M., Beaupre, L. A., Rashiq, S., Dryden, D. M., Hamm, M. P., Sadowski, C. A., Menon, M. R. G., Majumdar, S. R., Wilson, D. M., Karkhaneh, M., et al. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-4-201108160-00346 · Cited by 250
Juusola, J. L. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-156-8-201204170-00004 · Cited by 244
Yank, V., Tuohy, C. V., Logan, A. C., Bravata, D. M., Staudenmayer, K., Eisenhut, R., Sundaram, V., McMahon, D., Olkin, I., McDonald, K. M., et al. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-154-8-201104190-00004 · Cited by 240
Systematic Review: Nonoperative and Operative Treatments for Rotator Cuff Tears
Seida, J. C., LeBlanc, C., Schouten, J. R., Mousavi, S. S., Hartling, L., Vandermeer, B., Tjosvold, L., & Sheps, D. M. (2010) · Annals of Internal Medicine · doi:10.7326/0003-4819-153-4-201008170-00263 · Cited by 164
Fink, H. A., Ishani, A., Taylor, B. C., Greer, N. L., MacDonald, R., Rossini, D., Sadiq, S., Lankireddy, S., Kane, R. L., & Wilt, T. J. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-156-8-201204170-00008 · Cited by 154
Terasawa, T., Balk, E. M., Chung, M., Garlitski, A. C., Alsheikh-Ali, A. A. A., Lau, J., & Ip, S. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-3-200908040-00131 · Cited by 148
Ip, S., D'Ambrosio, C., Patel, K., Obadan, N., Kitsios, G. D., Chung, M., & Balk, E. M. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-20 · Cited by 148 · cited 1×
“44 (KQ1); 1 (KQ2); 2 (KQ3); 11 (KQ4); 173 (KQ5); 6 (KQ6); 18 (KQ7) (Systematic review: auto-titrating versus fixed continuous positive airway pressure for the treatment of obstructive sleep apnea: a systematic review with meta-analyses) [41] 11 months 19 months”
BackgroundWilt, T. J., Shamliyan, T., Taylor, B. C., MacDonald, R. H., & Kane, R. L. (2008) · The Journal of Urology · doi:10.1016/j.juro.2008.05.010 · Cited by 146
Comparative Effectiveness of Clostridium difficile Treatments: A Systematic Review
Drekonja, D. M., Butler, M., MacDonald, R., Bliss, D., Filice, G. A., Rector, T. S., & Wilt, T. J. (2011) · Annals of Internal Medicine · doi:10.7326/0003-4819-155-12-201112200-00007 · Cited by 123
Antipsychotics for Children and Young Adults: A Comparative Effectiveness Review
Seida, J. C., Schouten, J. R., Boylan, K., Newton, A. S., Mousavi, S. S., Beaith, A., Vandermeer, B., Dryden, D. M., & Carrey, N. (2012) · PEDIATRICS · doi:10.1542/peds.2011-2158 · Cited by 110
Systematic Review: Comparative Effectiveness of Medications to Reduce Risk for Primary Breast Cancer
Nelson, H. D., Fu, R., Griffin, J. C., Nygren, P., Smith, M. E. B., & Humphrey, L. (2009) · Annals of Internal Medicine · doi:10.7326/0000605-200911170-00147 · Cited by 100
Baker, W. L., Coleman, C. I., Kluger, J., Reinhart, K. M., Talati, R., Quercia, R., Phung, O. J., & White, C. M. (2009) · PubMed · doi:10.7326/0003-4819-151-12-200912150-00162 · Cited by 91
Sharma, M., Ansari, M. T., Abou-Setta, A. M., Soares-Weiser, K., Ooi, T. C., Sears, M., Yazdi, F., Tsertsvadze, A., & Moher, D. (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-9-200911030-00144 · Cited by 79 · cited 1×
“(Systematic review: comparative effectiveness and harms of combination therapy and monotherapy for dyslipidemia) [47] 4 months 29 months”
BackgroundSelf-citationBaker, W. L., Coleman, C. I., Kluger, J., Reinhart, K. M., Talati, R., Quercia, R., Phung, O. J., & White, C. M. (2009) · Annals of Internal Medicine · doi:10.7326/0000605-200912150-00162 · Cited by 78
Updating Systematic Reviews: An International Survey
Garritty, C., Tsertsvadze, A., Tricco, A. C., Sampson, M., & Moher, D. (2010) · PLoS ONE · doi:10.1371/journal.pone.0009914 · Cited by 62
Self-citationComparative effectiveness of therapies for clinically localized prostate cancer
(2009) · Clinical Governance An International Journal · doi:10.1108/cgij.2009.24814bae.009 · Cited by 52
AHRQ Series Paper 2: Principles for developing guidance: AHRQ and the Effective Health-Care Program
Helfand, M., & Balshem, H. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.05.005 · Cited by 51
AHRQ Series Paper 1: Comparing medical interventions: AHRQ and the Effective Health-Care Program
Slutsky, J. R., Atkins, D. C., Chang, S., & Sharp, B. A. C. (2008) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.06.009 · Cited by 49
Updating comparative effectiveness reviews: Current efforts in AHRQ's Effective Health Care Program
Tsertsvadze, A., Maglione, M., Chou, R., Garritty, C., Coleman, C. I., Lux, L. J., Bass, E. B., Balshem, H., & Moher, D. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.03.011 · Cited by 46
Self-citationTwo methods provide similar signals for the need to update systematic reviews
Chung, M., Newberry, S. J., Ansari, M. T., Yu, W. W., Wu, H., Lee, J., Suttorp, M., Gaylor, J. M., Motala, A., Moher, D., et al. (2012) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.12.004 · Cited by 42
Self-citationRecombinant Human Growth Hormone in the Treatment of Patients With Cystic Fibrosis
Phung, O. J., Coleman, C. I., Baker, E. L., Scholle, J. M., Girotto, J. E., Makanji, S. S., Chen, W. T., Talati, R., Kluger, J., & White, C. M. (2010) · Pediatrics · doi:10.1542/peds.2010-2007 · Cited by 39
Nonpharmacologic Interventions for Treatment-Resistant Depression in Adults
Gaynes, B. N., Lux, L. J., Lloyd, S., Hansen, R. A., Gartlehner, G., Keener, P., Brode, S., Evans, T. S., Jonas, D. E., Crotty, K., et al. (2011) · Cited by 34
Systematic Review of Therapies for Noncyclic Chronic Pelvic Pain in Women
Yunker, A., Sathe, N. A., Reynolds, W. M., Likis, F. E., & Andrews, J. (2012) · Obstetrical & Gynecological Survey · doi:10.1097/ogx.0b013e31825cecb3 · Cited by 30
Pain Management Interventions for Hip Fracture
Abou-Setta, A. M., Beaupré, L. A., Jones, C. A., Rashiq, S., Hamm, M. P., Sadowski, C. A., Menon, M. R., Majumdar, S. R., Wilson, D. M., Karkhaneh, M., et al. (2011) · Cited by 27
Response from the Cochrane Collaboration
Clarke, M. J., Tharyan, P., & Green, S. (2008) · The Lancet · doi:10.1016/s0140-6736(08)60196-8 · Cited by 24
Noncyclic Chronic Pelvic Pain Therapies for Women: Comparative Effectiveness
Andrews, J., Yunker, A., Reynolds, W. M., Likis, F. E., Sathe, N. A., & Jerome, R. N. (2012) · PubMed · Cited by 21
Effectiveness of early diagnosis, prevention, and treatment of Clostridium difficile infection
Butler, M. E., Bliss, D. Z., Drekonja, D. M., FILICE, G. A., Rector, T. S., MacDonald, R. H., & Wilt, T. J. (2011) · Agency for Healthcare Research and Quality (US) eBooks · Cited by 21
Bruening, W., Schoelles, K. M., Treadwell, J., Launders, J. H., Fontanarosa, J., & Tipton, K. N. (2009) · Cited by 19
Chronic Kidney Disease Stages 1–3
Ha, F., Ishani, A., Taylor, B. C., Nl, G., MacDonald, R. H., Rossini, D., Sadiq, S., Lankireddy, S., Kane, R. L., & Wilt, T. J. (2012) · Europe PMC (PubMed Central) · Cited by 18
Decisions to update comparative drug effectiveness reviews vary based on type of new evidence
Peterson, K., McDonagh, M. S., & Fu, R. F. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.11.019 · Cited by 17
Gaudet, L., Singh, K., Weeks, L., Skidmore, B., Tsertsvadze, A., & Ansari, M. (2012) · PLoS ONE · doi:10.1371/journal.pone.0031679 · Cited by 15
Self-citationShekelle, P., Newberry, S. J., Maglione, M., Shanman, R., Johnsen, B., Carter, J. M., Motala, A., Hulley, B., Wang, Z., Bravata, D. M., et al. (2009) · Cited by 14
Self-citationComparative effectiveness of management strategies for renal artery stenosis
Balk, E. M., Raman, G., Chung, M. C., Ip, S., Tatsioni, A., Alonso, Á., Kupelnick, B., Chew, P. W., DeVine, D., Gilbert, S. F., et al. (2006) · Annals of Internal Medicine · Cited by 14
White, M., & Greene, L. (2011) · Journal of Managed Care Pharmacy · doi:10.18553/jmcp.2011.17.s5.1 · Cited by 14
Comparative Effectiveness and Safety of Radiotherapy Treatments for Head and Neck Cancer
Samson, D. J., Ratko, T. A., Rothenberg, B. M., Brown, H. M., Bonnell, C. J., Ziegler, K. M., & Aronson, N. (2010) · Europe PMC (PubMed Central) · Cited by 14
Comparative Effectiveness of Radiofrequency Catheter Ablation for Atrial Fibrillation
Ip, S., Terasawa, T., Balk, E., Chung, M. K., Alshelkh-Ali, A., Garlitski, A., & Lau, J. S. M. (2009) · Cited by 13
Comparative Effectiveness of Nonoperative and Operative Treatments for Rotator Cuff Tears
Seida, J. C., Schouten, J. R., Mousavi, S. S., Tjosvold, L., Vandermeer, B., Milne, A., Bond, K., Hartling, L., LeBlanc, C. M. A., & Sheps, D. M. (2010) · Europe PMC (PubMed Central) · Cited by 9
Coleman, C. I., Wl, B., Kluger, J., Reinhart, K. M., Talati, R., Quercia, R., Mather, J. F., Giovenale, S., & White, C. M. (2009) · Agency for Healthcare Research and Quality (US) eBooks · Cited by 9
Comparative Effectiveness of Lipid-Modifying Agents
Sharma, M., Ansari, M., Soares‐Weiser, K., Abou-Setta, A., Tc, O., Sears, M., Yazdi, F., Tsertsvadze, A., & Moher, D. (2009) · Cited by 9
Self-citationComparative Effectiveness of Medications To Reduce Risk of Primary Breast Cancer in Women
Hd, N., Ren, F., Humphrey, L. L., Meb, S., Jc, G., & Nygren, P. (2009) · Europe PMC (PubMed Central) · Cited by 8
Comparative Effectiveness of Management Strategies for Renal Artery Stenosis: 2007 Update
Balk, E. M., & Raman, G. (2007) · Cited by 7
Hematopoietic Stem-Cell Transplantation in the Pediatric Population
Ratko, T. A., Belinson, S., Brown, H. M., Noorani, H. Z., Chopra, R., Marbella, A. M., Samson, D. J., Bonnell, C. J., Ziegler, K. M., & Aronson, N. (2012) · Cited by 7
Phung, O. J., Coleman, C. I., Baker, E. L., Scholle, J. M., Girotto, J. E., Makanji, S. S., Chen, W. T., Talati, R., Kluger, J., Quercia, R., et al. (2010) · Europe PMC (PubMed Central) · Cited by 5
Comparative Effectiveness of Recombinant Factor VIIa for Off-label Indications Versus Usual Care
Yank, V. A., Tuohy, V. K., Logan, A. C., Bravata, D. M., Staudenmayer, K., H, V. S. M. P., McMahon, D., Stave, C. D., Zehnder, J. L., Olkin, I., et al. (2010) · Cited by 3
Terbutaline Pump for the Prevention of Preterm Birth
Gaudet, L., Singh, K., Weeks, L., Skidmore, B., Tsouros, S., Tsertsvadze, A., Daniel, R., Doucette, S. P., WALKER, M. C., & Ansari, M. (2011) · Cited by 2
Self-citation
Research works printed but not linked: 6
- 13. Warren Z, Veenstra-Vanderweele J, Stone W, Bruzek JL, Nahmias AS, Foss-Feig JH, Jerome RN, Krishnaswami S, Sathe NA, Glasser AM, Surawicz T, McPheeters ML. Therapies for children with autism spectrum disorders. [ http://www.ncbi.nlm.nih.gov/books/NBK56343/pdf/TOC.pdf]
- 15. Guillamondegui OD, Montgomery SA, Phibbs FT, McPheeters ML, Alexander PT, Jerome RN, McKoy JN, Seroogy JJ, Eicken JJ, Krishnaswami S, Salomon RM, Hartmann KE. Traumatic brain injury and depression. [ http://www.ncbi.nlm.nih.gov/books/NBK62061/pdf/TOC.pdf]
- 24. Uhlig K, Balk EM, Patel K, Ip S, Kitsios GD, Obadan NO, Haynes SM, Stefan M, Rao M, Kong W, Chang L, Gaylor J, Iovin RC. Self-measured blood pressure monitoring: comparative effectiveness. [ http://www.ncbi.nlm.nih.gov/books/NBK84604/]
- 27. Balk EM, Moorthy D, Obadan NO, Patel K, Ip S, Chung M, Bannuru RR, Kitsios GD, Sen S, Iovin RC, Gaylor JM, D'Ambrosio C, Lau J. Diagnosis and treatment of obstructive sleep apnea in adults. [ http://www.ncbi.nlm.nih.gov/books/NBK63560/]
- 32. Seida JC, Schouten JR, Mousavi SS, Hamm M, Beaith A, Vandermeer B, Dryden DM, Boylan K, Newton AS, Carrey N. First & second generation antipsychotics for children and young adults. [ http://www.ncbi.nlm.nih.gov/books/NBK84643/]
- 33. Charach A, Dashti B, Carson P, Booker L, Lim CG, Lillie E, Yeung E, Ma J, Raina P, Schachar R. Attention Deficit Hyperactivity Disorder (ADHD): effectiveness of treatment in at-risk preschoolers; long-term effectiveness in all ages; and variability in prevalence, diagnosis, and treatment. [ http://www.ncbi.nlm.nih.gov/books/NBK82368/]
Cited by
Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., Stewart, L. A., & the PRISMA-P Group (2015) · BMJ · doi:10.1136/bmj.g7647 · Cited by 13,120
Self-citationThe COMET Handbook: version 1.0
Williamson, P. R., Altman, D. G., Bagley, H., Barnes, K. L., Blazeby, J. M., Brookes, S. T., Clarke, M., Gargon, E., Gorst, S., Harman, N., et al. (2017) · Trials · doi:10.1186/s13063-017-1978-4 · Cited by 2,011
“However, as with the updating of systematic reviews [196], there are uncertainties about the frequency with”
BackgroundOkoth, K., Chandan, J. S., Marshall, T., Thangaratinam, S., Thomas, G. N., Nirantharakumar, K., & Adderley, N. J. (2020) · BMJ · doi:10.1136/bmj.m3502 · Cited by 416
Thielen, F., Van Mastrigt, G., Burgers, L., Bramer, W., Majoie, H., Evers, S., & Kleijnen, J. (2016) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2016.1246962 · Cited by 132
How to conduct systematic reviews more expeditiously?
Tsertsvadze, A., Chen, Y.-F., Moher, D., Sutcliffe, P., & McCarthy, N. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0147-7 · Cited by 104
“SRs conducted expeditiously may also be relevant if an existing SR is in need of updating or when the available resources are limited [51, 52].”
BackgroundSelf-citation
Metadata
- Authors
- Ahmadzai, N., Newberry, S. J., Maglione, M. A., Tsertsvadze, A., Ansari, M. T., Hempel, S., Motala, A., Tsouros, S., Schneider Chafen, J. J., Shanman, R., Moher, D., & Shekelle, P. G.
- Affiliations
- Ottawa Hospital Research Institute (CA)
- RAND Corporation (US)
- Corresponding author
- Ahmadzai, Nadera
- ORCID
- Venue
- Systematic Reviews
- Published
- 2013-11-14
- Volume
- 2
- Issue
- 1
- Pages
- 104-104
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC3874670
- OpenAlex
- W2163687711
- Cited by
- 33
- References
- 56
- Citations by year
- 2025: 1 · 2024: 4 · 2022: 3 · 2021: 2 · 2020: 2 · 2018: 4 · 2017: 4 · 2016: 4 · 2015: 6 · 2014: 3
- FWCI
- 0.89
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/2.0 (tdm)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- uO Research (University of Ottawa) (submittedVersion, public-domain)
- eScholarship (California Digital Library) (submittedVersion)
- eScholarship (California Digital Library) (submittedVersion, cc-by)
- Funders
- U.S. Department of Health and Human Services
- University of Ottawa
- Agency for Healthcare Research and Quality
- Health Canada
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Keywords
- systematic review updating, surveillance systems, evidence-based practice centers, Agency for Healthcare Research and Quality, expert opinion
- MeSH terms
- Data Collection, Humans, Periodicals as Topic, Quality Control, Review Literature as Topic, Time Factors, United States, Information Storage and Retrieval, United States Agency for Healthcare Research and Quality, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.