CoChat

Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed

Open access
Tóth, Barbara, Berek, László, Gulácsi, László, Péntek, Márta, Zrubka, Zsombor · 2024 · Systematic Reviews
Cited by 48

The DOI matches its Crossref record; 68 of the 68 references it cites were found in a registry or an index.

Abstract

Background: The demand for high-quality systematic literature reviews (SRs) for evidence-based medical decision-making is growing. SRs are costly and require the scarce resource of highly skilled reviewers. Automation technology has been proposed to save workload and expedite the SR workflow. We aimed to provide a comprehensive overview of SR automation studies indexed in PubMed, focusing on the applicability of these technologies in real world practice. Methods: In November 2022, we extracted, combined, and ran an integrated PubMed search for SRs on SR automation. Full-text English peer-reviewed articles were included if they reported studies on SR automation methods (SSAM), or automated SRs (ASR). Bibliographic analyses and knowledge-discovery studies were excluded. Record screening was performed by single reviewers, and the selection of full text papers was performed in duplicate. We summarized the publication details, automated review stages, automation goals, applied tools, data sources, methods, results, and Google Scholar citations of SR automation studies. Results: From 5321 records screened by title and abstract, we included 123 full text articles, of which 108 were SSAM and 15 ASR. Automation was applied for search (19/123, 15.4%), record screening (89/123, 72.4%), full-text selection (6/123, 4.9%), data extraction (13/123, 10.6%), risk of bias assessment (9/123, 7.3%), evidence synthesis (2/123, 1.6%), assessment of evidence quality (2/123, 1.6%), and reporting (2/123, 1.6%). Multiple SR stages were automated by 11 (8.9%) studies. The performance of automated record screening varied largely across SR topics. In published ASR, we found examples of automated search, record screening, full-text selection, and data extraction. In some ASRs, automation fully complemented manual reviews to increase sensitivity rather than to save workload. Reporting of automation details was often incomplete in ASRs. Conclusions: Automation techniques are being developed for all SR stages, but with limited real-world adoption. Most SR automation tools target single SR stages, with modest time savings for the entire SR process and varying sensitivity and specificity across studies. Therefore, the real-world benefits of SR automation remain uncertain. Standardizing the terminology, reporting, and metrics of study reports could enhance the adoption of SR automation techniques in real-world practice.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1186/s13643-024-02592-3

Cites

68 of 68 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 68 references

Research works printed but not linked: 94

  1. Rubin A, Bellamy J. Practitioner’s guide to using research for evidence-based practice. Hoboken: Wiley; 2012.
  2. Melissa LR, Tara JB, Carrie P, David M, Lex MB, Jamie JK, et al. Systematic review search strategies are poorly described and not reproducible: a cross-sectional meta-research study. J Clin Epidemiol. 2024;166:111229. · DOI
  3. Holgyesi A, Toth B, Kozlovszky M, Kuti J, Weszl M, Balazs G, et al. Epidemiology and patients’ self-reported knowledge of implantable medical devices: results of a cross-sectional survey in Hungary. PLoS One. 2023;18(4):e0284577. · DOI
  4. Zah V, Burrel A, Asche C, Zrubka Z. Paying for digital health interventions – what evidence is needed? Acta Polytech Hung. 2022;19(9):179–99. · DOI
  5. Moga K, Holgyesi A, Zrubka Z, Pentek M, Haidegger T. Augmented or mixed reality enhanced head-mounted display navigation for in vivo spine surgery: a systematic review of clinical outcomes. J Clin Med. 2023;12(11):3788. · doi:10.3390/jcm12113788
  6. Czere JT, Gulacsi L, Zrubka Z, Pentek M, editors. Quality assessment of clinical trials with artificial intelligence based chatbots in healthcare: points to consider in the protocol development for a systematic literature review. In: SACI 2022 - IEEE 16th international symposium on applied computational intelligence and informatics, proceedings. 2022. · doi:10.1109/saci55618.2022.9919543
  7. Motahari-Nezhad H, Fgaier M, Mahdi Abid M, Pentek M, Gulacsi L, Zrubka Z. Digital biomarker-based studies: scoping review of systematic reviews. JMIR Mhealth Uhealth. 2022;10(10):e35722. · DOI
  8. Motahari-Nezhad H, Al-Abdulkarim H, Fgaier M, Abid MM, Péntek M, Gulácsi L, et al. Digital biomarker–based interventions: systematic review of systematic reviews. J Med Internet Res. 2022;24(12):e41042. · DOI
  9. Méndez-Gurrola II, Ramírez-Reyes A, Barranco-Gutiérrez AI. A review and perspective on the main machine learning methods applied to physical sciences. Acta Polytech Hung. 2022;19(10):205–20. · DOI
  10. Tran TA, Péntek M, Motahari-Nezhad H, Abonyi J, Kovács L, Gulácsi L, et al. Heart rate variability measurement to assess acute work-content-related stress of workers in industrial manufacturing environment—a systematic scoping review. IEEE Trans Syst Man Cybern Syst. 2023;53(11):6685–92. · DOI
  11. Varga J. Defining the economic role and benefits of micro, small and medium-sized enterprises in the 21st century with a systematic review of the literature. Acta Polytech Hung. 2021;18(11):209–28. · DOI
  12. Llewellyn A, Whittington C, Stewart G, Higgins JP, Meader N. The use of Bayesian networks to assess the quality of evidence from research synthesis: 2. Inter-rater reliability and comparison with standard GRADE Assessment. PLoS One. 2015;10(12):e0123511. · DOI
  13. Stewart GB, Higgins JP, Schunemann H, Meader N. The use of Bayesian networks to assess the quality of evidence from research synthesis: 1. PLoS One. 2015;10(3):e0114497. · DOI
  14. Burgard T, Bittermann A. Reducing literature screening workload with machine learning. Z Psychol. 2023;231(1):3–15.
  15. Abdelkader W, Navarro T, Parrish R, Cotoi C, Germini F, Iorio A, et al. Machine learning approaches to retrieve high-quality, clinically relevant evidence from the biomedical literature: systematic review. JMIR Med Inform. 2021;9(9):e30401. · DOI
  16. Krnic Martinic M, Pieper D, Glatt A, Puljak L. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol. 2019;19(1):203. · DOI · Paper page
  17. Oertelt-Prigione S, Wiedmann S, Endres M, Nolte CH, Regitz-Zagrosek V, Heuschmann P. Stroke and myocardial infarction: a comparative systematic evaluation of gender-specific analysis, funding and authorship patterns in cardiovascular research. Cerebrovasc Dis. 2011;31(4):373–81. · DOI
  18. O’Mara-Eves A, Brunton G, McDaid D, Kavanagh J, Oliver S, Thomas J. Techniques for identifying cross-disciplinary and ‘hard-to-detect’ evidence for systematic review. Res Synth Methods. 2014;5(1):50–9. · DOI
  19. Thompson J, Davis J, Mazerolle L. A systematic method for search term selection in systematic reviews. Res Synth Methods. 2014;5(2):87–97. · DOI
  20. Bui DD, Jonnalagadda S, Del Fiol G. Automatically finding relevant citations for clinical guideline development. J Biomed Inform. 2015;57:436–45. · DOI · Paper page
  21. Mwachui MA, Crump L, Hartskeerl R, Zinsstag J, Hattendorf J. Environmental and behavioural determinants of leptospirosis transmission: a systematic review. PLoS Negl Trop Dis. 2015;9(9):e0003843. · doi:10.1371/journal.pntd.0003843
  22. Sampson M, de Bruijn B, Urquhart C, Shojania K. Complementary approaches to searching MEDLINE may be sufficient for updating systematic reviews. J Clin Epidemiol. 2016;78:108–15. · DOI
  23. Rathbone J, Albarqouni L, Bakhit M, Beller E, Byambasuren O, Hoffmann T, et al. Expediting citation screening using PICo-based title-only screening for identifying studies in scoping searches and rapid reviews. Syst Rev. 2017;6(1):233. · DOI · Paper page
  24. Lanera C, Minto C, Sharma A, Gregori D, Berchialla P, Baldi I. Extending PubMed searches to ClinicalTrials.gov through a machine learning approach for systematic reviews. J Clin Epidemiol. 2018;103:22–30. · DOI
  25. Schmitz T, Bukowski M, Koschmieder S, Schmitz-Rode T, Farkas R. Potential Technologies Review: a hybrid information retrieval framework to accelerate demand-pull innovation in biomedical engineering. Res Synth Methods. 2019;10(3):420–39. · DOI
  26. Alharbi A, Stevenson M. Refining Boolean queries to identify relevant studies for systematic review updates. J Am Med Inform Assoc. 2020;27(11):1658–66. · DOI
  27. Brunskill A. A Microsoft Excel approach to reduce errors and increase efficiency in systematic searching. Med Ref Serv Q. 2020;39(1):15–26. · DOI
  28. Thiabaud A, Triulzi I, Orel E, Tal K, Keiser O. Social, behavioral, and cultural factors of HIV in Malawi: semi-automated systematic review. J Med Internet Res. 2020;22(8):e18747. · doi:10.2196/18747
  29. Paynter R, Fiordalisi C, Stoeger E, Erinoff E, Featherstone R, Voisin C, et al. A prospective comparison of evidence synthesis search strategies developed with and without text-mining tools. Rockville: Agency for Healthcare Research and Quality (US); 2021. Contract no.: report no.: 21-EHC008. · DOI
  30. Vaghela U, Rabinowicz S, Bratsos P, Martin G, Fritzilas E, Markar S, et al. Using a secure, continually updating, web source processing pipeline to support the real-time data synthesis and analysis of scientific literature: development and validation study. J Med Internet Res. 2021;23(5):e25714. · DOI
  31. O’Keefe H, Rankin J, Wallace SA, Beyer F. Investigation of text-mining methodologies to aid the construction of search strategies in systematic reviews of diagnostic test accuracy-a case study. Res Synth Methods. 2023;14(1):79–98. · DOI
  32. Yu W, Clyne M, Dolan SM, Yesupriya A, Wulf A, Liu T, et al. GAPscreener: an automatic tool for screening human genetic association literature in PubMed using the support vector machine technique. BMC Bioinformatics. 2008;9:205. · DOI · Paper page
  33. Bekhuis T, Demner-Fushman D. Towards automating the initial screening phase of a systematic review. Stud Health Technol Inform. 2010;160(Pt 1):146–50.
  34. Frunza O, Inkpen D, Matwin S, Klement W, O’Blenis P. Exploiting the systematic review protocol for classification of medical abstracts. Artif Intell Med. 2011;51(1):17–25. · DOI
  35. Shao W, Adams CE, Cohen AM, Davis JM, McDonagh MS, Thakurta S, et al. Aggregator: a machine learning approach to identifying MEDLINE articles that derive from the same underlying clinical trial. Methods. 2015;74:65–70. · DOI · Paper page
  36. Mo Y, Kontonatsios G, Ananiadou S. Supporting systematic reviews using LDA-based document representations. Syst Rev. 2015;4:172. · DOI · Paper page
  37. Almeida H, Meurs MJ, Kosseim L, Tsang A. Data sampling and supervised learning for HIV literature screening. IEEE Trans Nanobioscience. 2016;15(4):354–61. · DOI
  38. Cawley M, Beardslee R, Beverly B, Hotchkiss A, Kirrane E, Sams R 2nd, et al. Novel text analytics approach to identify relevant literature for human health risk assessments: a pilot study with health effects of in utero exposures. Environ Int. 2020;134:105228. · DOI
  39. Lanera C, Berchialla P, Sharma A, Minto C, Gregori D, Baldi I. Screening PubMed abstracts: is class imbalance always a challenge to machine learning? Syst Rev. 2019;8(1):317. · DOI
  40. Odintsova VV, Roetman PJ, Ip HF, Pool R, Van der Laan CM, Tona KD, et al. Genomics of human aggression: current state of genome-wide studies and an automated systematic review tool. Psychiatr Genet. 2019;29(5):170–90. · DOI · Paper page
  41. Ambalavanan AK, Devarakonda MV. Using the contextual language model BERT for multi-criteria classification of scientific articles. J Biomed Inform. 2020;112:103578. · DOI
  42. Dhrangadhariya A, Hilfiker R, Schaer R, Muller H. Machine learning assisted citation screening for systematic reviews. Stud Health Technol Inform. 2020;270:302–6.
  43. Aum S, Choe S. srBERT: automatic article classification model for systematic review using BERT. Syst Rev. 2021;10(1):285. · DOI · Paper page
  44. van Altena AJ, Spijker R, Leeflang MMG, Olabarriaga SD. Training sample selection: impact on screening automation in diagnostic test accuracy reviews. Res Synth Methods. 2021;12(6):831–41. · DOI
  45. Ajiji P, Cottin J, Picot C, Uzunali A, Ripoche E, Cucherat M, et al. Feasibility study and evaluation of expert opinion on the semi-automated meta-analysis and the conventional meta-analysis. Eur J Clin Pharmacol. 2022;78(7):1177–84. · DOI
  46. Ishankulov T, Danilov G, Kotik K, Orlov Y, Shifrin M, Potapov A. The classification of scientific abstracts using text statistical features. Stud Health Technol Inform. 2022;290:263–7.
  47. Li X, Zhang A, Al-Zaidy R, Rao A, Baral S, Bao L, et al. Automating document classification with distant supervision to increase the efficiency of systematic reviews: a case study on identifying studies with HIV impacts on female sex workers. PLoS One. 2022;17(6):e0270034. · DOI
  48. Proescholdt R, Hsiao TK, Schneider J, Cohen AM, McDonagh MS, Smalheiser NR. Testing a filtering strategy for systematic reviews: evaluating work savings and recall. AMIA Jt Summits Transl Sci Proc. 2022;2022:406–13.
  49. Schneider J, Hoang L, Kansara Y, Cohen AM, Smalheiser NR. Evaluation of publication type tagging as a strategy to screen randomized controlled trial articles in preparing systematic reviews. JAMIA Open. 2022;5(1):ooac015. · DOI
  50. Viner R, Russell S, Saulle R, Croker H, Stansfield C, Packer J, et al. School closures during social lockdown and mental health, health behaviors, and well-being among children and adolescents during the first COVID-19 wave: a systematic review. JAMA Pediatr. 2022;176(4):400–9. · DOI · Paper page
  51. Hempel S, Shetty KD, Shekelle PG, Rubenstein LV, Danz MS, Johnsen B, et al. AHRQ Methods for Effective Health Care. Machine learning methods in systematic reviews: identifying quality improvement intervention evaluations. Rockville: Agency for Healthcare Research and Quality (US); 2012.
  52. Dunn AG, Arachi D, Bourgeois FT. Identifying clinical study types from PubMed metadata: the active (machine) learning approach. Stud Health Technol Inform. 2015;216:867–71.
  53. Hashimoto K, Kontonatsios G, Miwa M, Ananiadou S. Topic detection using paragraph vectors to support active learning in systematic reviews. J Biomed Inform. 2016;62:59–65. · DOI · Paper page
  54. Ji X, Ritter A, Yen PY. Using ontology-based semantic similarity to facilitate the article screening process for systematic reviews. J Biomed Inform. 2017;69:33–42. · DOI
  55. Kontonatsios G, Brockmeier AJ, Przybyla P, McNaught J, Mu T, Goulermas JY, et al. A semi-supervised approach using label propagation to support citation screening. J Biomed Inform. 2017;72:67–76. · DOI · Paper page
  56. Norman CR, Leeflang MMG, Porcher R, Neveol A. Measuring the impact of screening automation on meta-analyses of diagnostic test accuracy. Syst Rev. 2019;8(1):243. · DOI · Paper page
  57. Giummarra MJ, Lau G, Gabbe BJ. Evaluation of text mining to reduce screening workload for injury-focused systematic reviews. Inj Prev. 2020;26(1):55–60. · DOI · Paper page
  58. Li Z, Gurgel H, Dessay N, Hu L, Xu L, Gong P. Semi-supervised text classification framework: an overview of dengue landscape factors and satellite earth observation. Int J Environ Res Public Health. 2020;17(12):4509. · DOI
  59. Yamada T, Yoneoka D, Hiraike Y, Hino K, Toyoshiba H, Shishido A, et al. Deep neural network for reducing the screening workload in systematic reviews for clinical guidelines: algorithm validation study. J Med Internet Res. 2020;22(12):e22422. · DOI · Paper page
  60. Gaskins NJ, Bray E, Hill JE, Doherty PJ, Harrison A, Connell LA. Factors influencing implementation of aerobic exercise after stroke: a systematic review. Disabil Rehabil. 2021;43(17):2382–96. · DOI
  61. Pham B, Jovanovic J, Bagheri E, Antony J, Ashoor H, Nguyen TT, et al. Text mining to support abstract screening for knowledge syntheses: a semi-automated workflow. Syst Rev. 2021;10(1):156. · DOI · Paper page
  62. Carey N, Harte M, Mc CL. A text-mining tool generated title-abstract screening workload savings: performance evaluation versus single-human screening. J Clin Epidemiol. 2022;149:53–9. · DOI
  63. Yang JJ, Cohen AM, McDonagh MS. SYRIAC: the systematic review information automated collection system a data warehouse for facilitating automated biomedical text classification. AMIA Annu Symp Proc. 2008;2008:825–9.
  64. Surian D, Dunn AG, Orenstein L, Bashir R, Coiera E, Bourgeois FT. A shared latent space matrix factorisation method for recommending new trial evidence for systematic review updates. J Biomed Inform. 2018;79:32–40. · DOI
  65. Xiong Z, Liu T, Tse G, Gong M, Gladding PA, Smaill BH, et al. A machine learning aided systematic review and meta-analysis of the relative risk of atrial fibrillation in patients with diabetes mellitus. Front Physiol. 2018;9:835. · DOI · Paper page
  66. Currie GL, Angel-Scott HN, Colvin L, Cramond F, Hair K, Khandoker L, et al. Animal models of chemotherapy-induced peripheral neuropathy: a machine-assisted systematic review and meta-analysis. PLoS Biol. 2019;17(5):e3000243. · DOI
  67. Norman CR, Gargon E, Leeflang MMG, Neveol A, Williamson PR. Evaluation of an automatic article selection method for timelier updates of the Comet Core Outcome Set database. Database (Oxford). 2019;2019:baz109. · DOI
  68. Reddy SM, Patel S, Weyrich M, Fenton J, Viswanathan M. Comparison of a traditional systematic review approach with review-of-reviews and semi-automation as strategies to update the evidence. Syst Rev. 2020;9(1):243. · DOI · Paper page
  69. Stansfield C, Stokes G, Thomas J. Applying machine classifiers to update searches: analysis from two case studies. Res Synth Methods. 2022;13(1):121–33. · DOI
  70. Surian D, Bourgeois FT, Dunn AG. The automation of relevant trial registration screening for systematic review updates: an evaluation study on a large dataset of ClinicalTrials.gov registrations. BMC Med Res Methodol. 2021;21(1):281. · DOI
  71. Dalal SR, Shekelle PG, Hempel S, Newberry SJ, Motala A, Shetty KD. AHRQ Methods for Effective Health Care. A pilot study using machine learning and domain knowledge to facilitate comparative effectiveness review updating. Rockville: Agency for Healthcare Research and Quality (US); 2012.
  72. Qin X, Liu J, Wang Y, Liu Y, Deng K, Ma Y, et al. Natural language processing was effective in assisting rapid title and abstract screening when updating systematic reviews. J Clin Epidemiol. 2021;133:121–9. · DOI · Paper page
  73. Cohen AM. Optimizing feature representation for automated systematic review work prioritization. AMIA Annu Symp Proc. 2008;2008:121–5.
  74. Cohen AM, Ambert K, McDonagh M. A prospective evaluation of an automated classification system to support evidence-based medicine and systematic review. AMIA Annu Symp Proc. 2010;2010:121–5.
  75. Li D, Wang Z, Wang L, Sohn S, Shen F, Murad MH, et al. A text-mining framework for supporting systematic reviews. Am J Inf Manag. 2016;1(1):1–9.
  76. Lewis SJ, Gardner M, Higgins J, Holly JMP, Gaunt TR, Perks CM, et al. Developing the WCRF International/University of Bristol methodology for identifying and carrying out systematic reviews of mechanisms of exposure-cancer associations. Cancer Epidemiol Biomarkers Prev. 2017;26(11):1667–75. · DOI
  77. Lam J, Howard BE, Thayer K, Shah RR. Low-calorie sweeteners and health outcomes: a demonstration of rapid evidence mapping (rEM). Environ Int. 2019;123:451–8. · DOI
  78. Carlson LM, Angrish M, Shirke AV, Radke EG, Schulz B, Kraft A, et al. Systematic evidence map for over one hundred and fifty per- and polyfluoroalkyl substances (PFAS). Environ Health Perspect. 2022;130(5):56001. · DOI · Paper page
  79. Halamoda-Kenzaoui B, Rolland E, Piovesan J, Puertas Gallardo A, Bremer-Hoffmann S. Toxic effects of nanomaterials for health applications: how automation can support a systematic review of the literature? J Appl Toxicol. 2022;42(1):41–51. · DOI · Paper page
  80. Ji X, Yen PY. Using MEDLINE elemental similarity to assist in the article screening process for systematic reviews. JMIR Med Inform. 2015;3(3):e28. · DOI
  81. Mytton J, Ingram J, Manns S, Thomas J. Facilitators and barriers to engagement in parenting programs: a qualitative systematic review. Health Educ Behav. 2014;41(2):127–37. · DOI · Paper page
  82. Trypsteen W, Kiselinova M, Vandekerckhove L, De Spiegelaere W. Diagnostic utility of droplet digital PCR for HIV reservoir quantification. J Virus Erad. 2016;2(3):162–9. · DOI
  83. Gates A, Gates M, Sim S, Elliott S, Pillay J, Hartling L. Creating efficiencies in the extraction of data from randomized trials: a prospective evaluation of a machine learning and text mining tool. Rockville: Agency for Healthcare Research and Quality (US); 2021. Contract no.: report no.: 21-EHC006.
  84. De Menezes A, Nunes AC, Pimenta DN, Lotta G, Nkya T, Krieger MM, et al. Examining the intersection between gender, community health workers, and vector control policies: a text mining literature review. Am J Trop Med Hyg. 2022;106(3):768–74.
  85. Pradhan R, Hoaglin DC, Cornell M, Liu W, Wang V, Yu H. Automatic extraction of quantitative data from ClinicalTrials.gov to conduct meta-analyses. J Clin Epidemiol. 2019;105:92–100. · DOI · Paper page
  86. Cramond F, O’Mara-Eves A, Doran-Constant L, Rice AS, Macleod M, Thomas J. The development and evaluation of an online application to assist in the extraction of data from graphs for use in systematic reviews. Wellcome Open Res. 2018;3:157. · DOI
  87. Jardim PSJ, Rose CJ, Ames HM, Echavez JFM, Van de Velde S, Muller AE. Automating risk of bias assessment in systematic reviews: a real-time mixed methods comparison of human researchers to a machine learning system. BMC Med Res Methodol. 2022;22(1):167. · DOI · Paper page
  88. Wang Q, Liao J, Lapata M, Macleod M. Risk of bias assessment in preclinical literature using natural language processing. Res Synth Methods. 2022;13(3):368–80. · DOI
  89. Sanchez-Graillet O, Witte C, Grimm F, Grautoff S, Ell B, Cimiano P. Synthesizing evidence from clinical trials with dynamic interactive argument trees. J Biomed Semantics. 2022;13(1):16. · DOI
  90. Van der Mierden S, Tsaioun K, Bleich A, Leenaars CHC. Software tools for literature screening in systematic reviews in biomedical research. Altex. 2019;36(3):508–17.
  91. Blaizot A, Veettil SK, Saidoung P, Moreno-Garcia CF, Wiratunga N, Aceves-Martins M, et al. Using artificial intelligence methods for systematic review in health sciences: a systematic review. Res Synth Methods. 2022;13(3):353–62. · DOI · Paper page
  92. Feng Y, Liang S, Zhang Y, Chen S, Wang Q, Huang T, et al. Automated medical literature screening using artificial intelligence: a systematic review and meta-analysis. J Am Med Inform Assoc. 2022;29(8):1425–32. · DOI · Paper page
  93. Santos AOD, da Silva ES, Couto LM, Reis GVL, Belo VS. The use of artificial intelligence for automating or semi-automating biomedical literature analyses: a scoping review. J Biomed Inform. 2023;142:104389. · DOI · Paper page
  94. Burrell A, Zrubka Z, Champion A, Zah V, Vinuesa L, Holtorf A-P, et al. How useful are digital health terms for outcomes research? An ISPOR special interest group report. Value Health. 2022;25(9):1469–79. · DOI

Cited by

See all 48

Metadata

Authors
Tóth, B., Berek, L., Gulácsi, L., Péntek, M., & Zrubka, Z.
Venue
Systematic Reviews
Published
2024-07-08
Volume
13
Issue
1
Pages
174-174
ISSN
2046-4053 (ISSN-L)
OpenAlex
W4400421957
Cited by
48
References
68
FWCI
6.59
Type
journal article
Language
en
Open access
gold
Licence
Funders
  • Nemzeti Kutatási, Fejlesztési és Innovaciós Alap (TKP2021-NKTA-36)
  • Óbuda University

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed. https://app.cochat.ai/papers/10.1186/s13643-024-02592-3
The paper
Tóth, B., Berek, L., Gulácsi, L., Péntek, M., & Zrubka, Z. (2024). Automation of systematic reviews of biomedical literature: a scoping review of studies indexed in PubMed. Systematic Reviews. https://doi.org/10.1186/s13643-024-02592-3

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.