CoChat

Artificial intelligence for literature reviews: opportunities and challenges

Open access
Bolaños, Francisco, Salatino, Angelo, Osborne, Francesco, Motta, Enrico · 2024 · Artificial Intelligence Review

The DOI matches its Crossref record; 217 of the 218 references it cites were found in a registry or an index; 62 of the 115 sources its claims cite were checked for support.

Abstract

This paper presents a comprehensive review of the use of Artificial Intelligence (AI) in Systematic Literature Reviews (SLRs). A SLR is a rigorous and organised methodology that assesses and integrates prior research on a given topic. Numerous tools have been developed to assist and partially automate the SLR process. The increasing role of AI in this field shows great potential in providing more effective support for researchers, moving towards the semi-automatic creation of literature reviews. Our study focuses on how AI techniques are applied in the semi-automation of SLRs, specifically in the screening and extraction phases. We examine 21 leading SLR tools using a framework that combines 23 traditional features with 11 AI features. We also analyse 11 recent tools that leverage large language models for searching the literature and assisting academic writing. Finally, the paper discusses current trends in the field, outlines key research challenges, and suggests directions for future research. We highlight three primary research challenges: integrating advanced AI solutions, such as large language models and knowledge graphs, improving usability, and developing a standardised evaluation framework. We also propose best practices to ensure more robust evaluations in terms of performance, usability, and transparency. Overall, this review offers a detailed overview of AI-enhanced SLR tools for researchers and practitioners, providing a foundation for the development of next-generation AI solutions in this field.

Overview

CoChat Summary

TL;DR

This paper reviews AI use in systematic literature reviews, analyzing 21 SLR tools with 23 general and 11 AI-specific features plus 11 recent LLM-based tools. It finds most SLR tools rely on older AI methods and have usability/evaluation gaps, and identifies three main challenges: advanced AI integration, usability, and standardised evaluation.

Key findings

  • Of the 21 SLR tools, 19 support screening and 4 support extraction; 19 are web applications and 2 are desktop (SWIFT-Review, ASReview); only 4 release code under an open license (Colandr, ASReview, FAST2, RobotReviewer/RobotSearch). Most use basic classifiers and bag-of-words; SVM is the most adopted approach, with Abstrackr, FAST2, Rayyan, and RobotAnalyst relying exclusively on SVM. Minimum seed requirements vary: ASReview, SWIFT-Active Screener, and SWIFT-Review require 1 relevant and 1 irrelevant paper; Covidence 2; Rayyan 5; Colandr 10; SysRev 30.
  • Only 8 of 21 tools implement at least 70% of the 23 general features. DistillerSR, Nested Knowledge, Dextr, and ExaCT lead at 82%; PICOPortal and Rayyan at 78%; EPPI-Reviewer at 74%; SWIFT-Active Screener at 70%. 13 of 21 are free to use; only LitSuggest supports living reviews; only 4 tools provide PRISMA diagram or protocol templates.
  • The paper identifies 11 recent LLM-based tools: 7 search engines, 3 writing assistants, and Textero.ai fitting both categories. Most use the OpenAI API, often with RAG; output quality varies and may be more useful for master’s students than researchers.
  • Usability and evaluation remain weak: a survey of 81 researchers found discontinuation reasons were poor usability (43%), insufficient functionality (37%), and workflow incompatibility (37%); SUS scores ranged from 66 to 77 (C–B). No standard evaluation framework exists; F2, WSS, and nWSS are proposed metrics, and only 4 of 21 tools operate under open licenses.

Methods

  • PRISMA systematic review. The authors searched Scopus for prior surveys, the Systematic Literature Review Toolbox, CRAN, and performed snowballing via Semantic Scholar; after inclusion/exclusion criteria, they selected 21 SLR tools. They analyzed 34 features (11 AI-specific, 23 general) using literature, official websites, tutorials, and developer interviews.
  • An additional exploratory search on TopAI Tools for LLM-based literature review and writing tools returned 164 tools; 18 candidates were examined, yielding 11 included tools.

Limitations

  • Stated: internal validity threats from subjective inclusion/exclusion decisions and the SLR Toolbox being offline since March 2024; external validity threats from search engines/search strings possibly missing tools and from excluding tools without user interfaces, under maintenance, or not updated in 10 years; construct validity threat that the 34 features may not cover all AI dimensions; conclusion validity threat from dynamic software and rapid AI evolution, making findings a snapshot.
  • Evident from the text: the LLM-tool analysis is described as an initial exploratory study rather than a systematic review, and the proposed best practices/evaluation framework are not empirically validated in the paper.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1007/s10462-024-10902-3

Cites

217 of 218 references verified0 citation errors · 1 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

39 Valid · 20 Not valid · 40 Not checked · 177 Own results and interpretations

91 more claims · 13 with their cited sources checked

Create a free account to see all 276 claims and every source verdict.

See all claims — free account
  • Probably not a substantial claim. 142
  • It may state two separate claims as one. 64
  • The quote only partly states this claim. 4

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Research works printed but not linked: 12

  1. Iansiti M, Lakhani KR (2020) Competing in the age of AI: strategy and leadership when algorithms and networks run the world. Harvard Business Press, Boston
  2. Kusa W, Knoth P, Hanbury A (2022) Evaluation of automated citation screening in systematic literature reviews with work saved over sampling: an analysis. In: 1st workshop on augmented intelligence for technology-assisted reviews systems: evaluation metrics and protocols for ediscovery and systematic review systems, pp 1–7
  3. Le Q, Mikolov T (2014) Distributed representations of sentences and documents. In: International Conference on Machine Learning. PMLR, pp 1188–1196
  4. Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, Küttler H, Lewis M, Yih W-T, Rocktäschel T et al (2020) Retrieval-augmented generation for knowledge-intensive NLP tasks. Adv Neural Inf Process Syst 33:9459–9474
  5. Li Q, Li J, Sheng J, Cui S, Wu J, Hei Y, Peng H, Guo S, Wang L, Beheshti A et al (2022) A survey on deep learning event extraction: approaches and applications. IEEE Trans Neural Netw Learn Syst
  6. Lipscomb CE (2000) Medical subject headings (mesh). Bull Med Libr Assoc 88(3):265
  7. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. Adv Neural Inf Process Syst 26
  8. Minion JT, Egunsola O, Mastikhina L, Farkas B, Hofmeister M, Flanagan J, Salmon C, Clement F (2021) Pico portal. J Can Health Libraries Assoc 42(3):181
  9. Ng A, Jordan M, Weiss Y (2001) On spectral clustering: analysis and an algorithm. Adv Neural Inf Process Syst 14
  10. National Academies of Sciences, Engineering, and Medicine; Policy and Global Affairs; Board on Research Data and Information; Committee on Toward an Open Science Enterprise (2018) Open science by design: realizing a vision for 21st century research. National Academies Press, Washington
  11. Quesenbery W (2014) The five dimensions of usability. Content and complexity. Routledge, New York, pp 93–114
  12. Wang LL, DeYoung J, Wallace B (2022) Overview of mslr2022: a shared task on multi-document summarization for literature reviews. In: Proceedings of the third workshop on scholarly document processing

Cited by

See all 246

Metadata

Authors
Bolaños, F., Salatino, A., Osborne, F., & Motta, E.
Affiliations
The Open University (GB)
Corresponding author
Bolaños, Francisco
Venue
Artificial Intelligence Review
Published
2024-08-17
Volume
57
Issue
10
ISSN
  • 0269-2821 (ISSN-L)
  • 1573-7462
OpenAlex
W4401667275
Cited by
246
References
170
Citations by year
2026: 124 · 2025: 103 · 2024: 19
FWCI
69.13
Type
journal article
Language
en
Open access
Licence
Locations
  • Artificial Intelligence Review (publishedVersion, cc-by)
  • Open Research Online (The Open University) (acceptedVersion, cc-by)
  • BOA (University of Milano-Bicocca) (submittedVersion, cc-by)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Keywords
artificial intelligence, systematic reviews, large language models, literature screening, data extraction, knowledge graphs, academic writing support, usability, transparency

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.