CoChat
Sign Up
Cited by 256Meta-analysis and systematic reviewsArtificial Intelligence in Healthcare and EducationInformation Retrieval and Search Behavior

Artificial intelligence in systematic reviews: promising when appropriately used

Open access
van Dijk, Sanne H B, Brusse-Keizer, Marjolein G J, Bucsán, Charlotte C, van der Palen, Job, Doggen, Carine J M, Lenferink, Anke·2023·BMJ Open
DOI 10.1136/bmjopen-2023-072254PMC PMC10335470
Open this paper in CoChat — free accountOpen PDF

Abstract

Background: Systematic reviews provide a structured overview of the available evidence in medical-scientific research. However, due to the increasing medical-scientific research output, it is a time-consuming task to conduct systematic reviews. To accelerate this process, artificial intelligence (AI) can be used in the review process. In this communication paper, we suggest how to conduct a transparent and reliable systematic review using the AI tool ‘ASReview’ in the title and abstract screening. Methods: Use of the AI tool consisted of several steps. First, the tool required training of its algorithm with several prelabelled articles prior to screening. Next, using a researcher-in-the-loop algorithm, the AI tool proposed the article with the highest probability of being relevant. The reviewer then decided on relevancy of each article proposed. This process was continued until the stopping criterion was reached. All articles labelled relevant by the reviewer were screened on full text. Results: Considerations to ensure methodological quality when using AI in systematic reviews included: the choice of whether to use AI, the need of both deduplication and checking for inter-reviewer agreement, how to choose a stopping criterion and the quality of reporting. Using the tool in our review resulted in much time saved: only 23% of the articles were assessed by the reviewer. Conclusion: The AI tool is a promising innovation for the current systematic reviewing practice, as long as it is appropriately used and methodological quality can be assured. PROSPERO registration number: CRD42022283952.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1136/bmjopen-2023-072254

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Artificial intelligence has emerged as a potential solution to systematic review challenges, with recent studies suggesting its capability to enhance the quality and efficiency of systematic reviews.

    “Artificial intelligence (AI) has emerged as a potential solution to these challenges, with recent studies and evaluations suggesting its capability to enhance the quality and efficiency of systematic reviews (Blaizot et al., 2022; Dijk et al., 2023; Kebede et al., 2023; Mahuli et al., 2023; Moreno-Garcia et al., 2023; Nugroho et al., 2023; Santos et al., 2023).”

    From Can large language models replace humans in systematic reviews? Evaluating <scp>GPT</scp> ‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languages (2024) · cites this paper as Dijk et al., 2023

    The Background and Discussion support AI’s potential to make systematic reviews more efficient, and the Abstract Results report substantial time saved. The paper says methodological quality must be protected when AI is used; it does not show that AI enhances review quality. Page numbers are unavailable in our copy. (LLM, conf=0.86)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 27

  1. [1]Bornmann L, Mutz R. Growth rates of modern science: a bibliometric analysis based on the number of publications and cited references. J Assn Inf Sci Tec 2015; 66: 2215– 22. 10.1002/asi.23329 · DOI · Paper page
  2. [2]Bornmann L, Haunschild R, Mutz R. Growth rates of modern science: a latent piecewise growth curve approach to model publication numbers from established and new literature databases. Humanit Soc Sci Commun 2021; 8: 224. 10.1057/s41599-021-00903-w · DOI
  3. [3]Michels C, Schmoch U. The growth of science and database coverage. Scientometrics 2012; 93: 831– 46. 10.1007/s11192-012-0732-7 · DOI
  4. [4]Haghani M, Abbasi A, Zwack CC, et al. Trends of research productivity across author gender and research fields: a multidisciplinary and multi-country observational study. PLoS ONE 2022; 17: e0271998. 10.1371/journal.pone.0271998 35947579 PMC9365186 · DOI
  5. [5]Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. J Clin Epidemiol 2021; 134: 178– 89. 10.1016/j.jclinepi.2021.03.001 33789819 · DOI · Paper page
  6. [7]Burns PB, Rohrich RJ, Chung KC. The levels of evidence and their role in evidence-based medicine. Plast Reconstr Surg 2011; 128: 305– 10. 10.1097/PRS.0b013e318219c171 21701348 PMC3124652 · DOI · Paper page
  7. [8]Bastian H, Glasziou P, Chalmers I. Seventy-five trials and eleven systematic reviews a day: how will we ever keep up PLoS Med 2010; 7: e1000326. 10.1371/journal.pmed.1000326 20877712 PMC2943439 · DOI · Paper page
  8. [11]Ferdinands G. AI-assisted systematic reviewing: selecting studies to compare Bayesian versus Frequentist SEM for small sample sizes. Multivariate Behav Res 2021; 56: 153– 4. 10.1080/00273171.2020.1853501 33263417 · DOI
  9. [12]Havrlant L, Kreinovich V. A simple probabilistic explanation of term frequency-inverse document frequency (TF-Idf) Heuristic (and variations motivated by this explanation). International Journal of General Systems 2017; 46: 27– 36. 10.1080/03081079.2017.1291635 · DOI
  10. [13]Wang R, Cao Z, Li Y, et al. Utility of N-terminal pro B-type natriuretic peptide and mean platelet volume in differentiating congestive heart failure from chronic obstructive pulmonary disease. Int J Cardiol 2013; 170: e28– 9. 10.1016/j.ijcard.2013.10.048 24169535 · DOI
  11. [14]Ouanes I, Jalloul F, Ayed S, et al. N-terminal proB-type natriuretic peptide levels aid the diagnosis of left ventricular dysfunction in patients with severe acute exacerbations of chronic obstructive pulmonary disease and renal dysfunction. Respirology 2012; 17: 660– 6. 10.1111/j.1440-1843.2012.02153.x 22348741 · DOI
  12. [15]Andrijevic I, Milutinov S, Lozanov Crvenkovic Z, et al. N-terminal prohormone of brain natriuretic peptide (NT-proBNP) as a diagnostic biomarker of left ventricular systolic dysfunction in patients with acute exacerbation of chronic obstructive pulmonary disease (AECOPD). Lung 2018; 196: 583– 90. 10.1007/s00408-018-0137-3 29951921 · DOI
  13. [16]Hawkins NM, Khosla A, Virani SA, et al. B-type natriuretic peptides in chronic obstructive pulmonary disease: a systematic review. BMC Pulm Med 2017; 17: 11. 10.1186/s12890-016-0345-7 28073350 PMC5223538 · DOI
  14. [17]Haddaway NR, Page MJ, Pritchard CC, et al. Prisma2020: an R package and shiny app for producing PRISMA 2020-compliant flow diagrams, with interactivity for optimised digital transparency and open synthesis. Campbell Syst Rev 2022; 18: e1230. 10.1002/cl2.1230 36911350 PMC8958186 · DOI · Paper page
  15. [19]Veritas Health Innovation. Covidence systematic review software. Melbourne, Australia: Veritas Health Innovation, 2022.
  16. [21]Ishibashi H, Hino H. Stopping criterion for active learning based on error stability. ArXiv 2021; 1: 1– 32. Available: http://arxiv.org/abs/2104.01836 · arXiv
  17. [22]Wang W, Cai W, Zhang Y. Stability-based stopping criterion for active learning. 2014 IEEE International Conference on Data Mining (ICDM); Shenzhen, China. ICDM: IEEE, 2014: 1019– 24 10.1109/ICDM.2014.99 · DOI
  18. [23]Blaizot A, Veettil SK, Saidoung P, et al. Using artificial intelligence methods for systematic review in health sciences: a systematic review. Res Synth Methods 2022; 13: 353– 62. 10.1002/jrsm.1553 35174972 · DOI · Paper page
  19. [24]McHugh ML. Interrater reliability: the Kappa statistic. Biochem Med 2012; 22: 276– 82. 10.11613/BM.2012.031 PMC3900052 23092060 · DOI · Paper page
  20. [25]Bernardes RC, Botina LL, Araújo R dos S, et al. Artificial intelligence-aided meta-analysis of toxicological assessment of agrochemicals in bees. Front Ecol Evol 2022; 10: 845608. 10.3389/fevo.2022.845608 · DOI
  21. [26]Silva GFS, Fagundes TP, Teixeira BC, et al. Machine learning for hypertension prediction: a systematic review. Curr Hypertens Rep 2022; 24: 523– 33. 10.1007/s11906-022-01212-6 35731335 · DOI
  22. [27]Miranda L, Paul R, Pütz B, et al. Systematic review of functional MRI applications for psychiatric disease subtyping. Front Psychiatry 2021; 12: 665536. 10.3389/fpsyt.2021.665536 34744805 PMC8569315 · DOI
  23. [28]Schouw HM, Huisman LA, Janssen YF, et al. Targeted optical fluorescence imaging: a meta-narrative review and future perspectives. Eur J Nucl Med Mol Imaging 2021; 48: 4272– 92. 10.1007/s00259-021-05504-y 34633509 PMC8566445 · DOI
  24. [29]Bakkum L, Schuengel C, Sterkenburg PS, et al. People with intellectual disabilities living in care facilities engaging in virtual social contact: a systematic review of the feasibility and effects on well-being. J Appl Res Intellect Disabil 2022; 35: 60– 74. 10.1111/jar.12926 34245077 PMC9291764 · DOI
  25. [30]Huang Y, Procházková M, Lu J, et al. Family related variables’ influences on adolescents’ health based on health behaviour in school-aged children database, an AI-assisted scoping review, and narrative synthesis. Front Psychol 2022; 13: 871795. 10.3389/fpsyg.2022.871795 36033089 PMC9400839 · DOI · Paper page
  26. [31]Zhang W, Huang S, Lam L, et al. Cyberbullying definitions and measurements in children and adolescents: summarizing 20 years of global efforts. Front Public Health 2022; 10: 1000504. 10.3389/fpubh.2022.1000504 36388377 PMC9642089 · DOI · Paper page
  27. [33]Muthu S. The efficiency of machine learning-assisted platform for article screening in systematic reviews in orthopaedics. Int Orthop 2023; 47: 551– 6. 10.1007/s00264-022-05672-y 36562816 · DOI · Paper page

Cited by

Sorted by citations

Authors
van Dijk, S. H. B., Brusse-Keizer, M. G. J., Bucsán, C. C., van der Palen, J., Doggen, C. J. M., & Lenferink, A.
Affiliations
  • Medisch Spectrum Twente (NL)
  • Centre for Health Technology and Services Research (PT)
Corresponding author
Lenferink, Anke
ORCID
Venue
BMJ Open
Publisher
BMJ
Published
2023-07
Volume
13
Issue
7
Pages
e072254
ISSN
2044-6055 (ISSN-L)
DOI
10.1136/bmjopen-2023-072254
PMC
PMC10335470
OpenAlex
W4383554209
Cited by
256
References
7
Citations by year
2026: 75 · 2025: 109 · 2024: 65 · 2023: 7
FWCI
16.02
Type
journal article
Language
en
Open access
Licence
Locations
  • BMJ Open (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • University of Twente Research Information (publishedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • HighWire Press Open Archive (submittedVersion)
Funders
European Union’s Horizon 2020 research and innovation programme (965315)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
Keywords
artificial intelligence, systematic reviews, ASReview, title and abstract screening, algorithm training, stopping rules, inter-reviewer agreement, data deduplication, methodological quality, reporting quality, time efficiency
MeSH terms
Systematic Reviews as Topic, Algorithms, Artificial Intelligence, Communication, Humans, Biomedical Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review