The DOI matches its Crossref record; 216 of the 217 references it cites were found in a registry or an index; 62 of the 115 sources its claims cite were checked for support.
Artificial intelligence for literature reviews: opportunities and challenges
Open accessAbstract
This paper presents a comprehensive review of the use of Artificial Intelligence (AI) in Systematic Literature Reviews (SLRs). A SLR is a rigorous and organised methodology that assesses and integrates prior research on a given topic. Numerous tools have been developed to assist and partially automate the SLR process. The increasing role of AI in this field shows great potential in providing more effective support for researchers, moving towards the semi-automatic creation of literature reviews. Our study focuses on how AI techniques are applied in the semi-automation of SLRs, specifically in the screening and extraction phases. We examine 21 leading SLR tools using a framework that combines 23 traditional features with 11 AI features. We also analyse 11 recent tools that leverage large language models for searching the literature and assisting academic writing. Finally, the paper discusses current trends in the field, outlines key research challenges, and suggests directions for future research. We highlight three primary research challenges: integrating advanced AI solutions, such as large language models and knowledge graphs, improving usability, and developing a standardised evaluation framework. We also propose best practices to ensure more robust evaluations in terms of performance, usability, and transparency. Overall, this review offers a detailed overview of AI-enhanced SLR tools for researchers and practitioners, providing a foundation for the development of next-generation AI solutions in this field.
Overview
CoChat Summary
TL;DR
This paper reviews AI use in systematic literature reviews, analyzing 21 SLR tools with 23 general and 11 AI-specific features plus 11 recent LLM-based tools. It finds most SLR tools rely on older AI methods and have usability/evaluation gaps, and identifies three main challenges: advanced AI integration, usability, and standardised evaluation.
Key findings
- Of the 21 SLR tools, 19 support screening and 4 support extraction; 19 are web applications and 2 are desktop (SWIFT-Review, ASReview); only 4 release code under an open license (Colandr, ASReview, FAST2, RobotReviewer/RobotSearch). Most use basic classifiers and bag-of-words; SVM is the most adopted approach, with Abstrackr, FAST2, Rayyan, and RobotAnalyst relying exclusively on SVM. Minimum seed requirements vary: ASReview, SWIFT-Active Screener, and SWIFT-Review require 1 relevant and 1 irrelevant paper; Covidence 2; Rayyan 5; Colandr 10; SysRev 30.
- Only 8 of 21 tools implement at least 70% of the 23 general features. DistillerSR, Nested Knowledge, Dextr, and ExaCT lead at 82%; PICOPortal and Rayyan at 78%; EPPI-Reviewer at 74%; SWIFT-Active Screener at 70%. 13 of 21 are free to use; only LitSuggest supports living reviews; only 4 tools provide PRISMA diagram or protocol templates.
- The paper identifies 11 recent LLM-based tools: 7 search engines, 3 writing assistants, and Textero.ai fitting both categories. Most use the OpenAI API, often with RAG; output quality varies and may be more useful for master’s students than researchers.
- Usability and evaluation remain weak: a survey of 81 researchers found discontinuation reasons were poor usability (43%), insufficient functionality (37%), and workflow incompatibility (37%); SUS scores ranged from 66 to 77 (C–B). No standard evaluation framework exists; F2, WSS, and nWSS are proposed metrics, and only 4 of 21 tools operate under open licenses.
Methods
- PRISMA systematic review. The authors searched Scopus for prior surveys, the Systematic Literature Review Toolbox, CRAN, and performed snowballing via Semantic Scholar; after inclusion/exclusion criteria, they selected 21 SLR tools. They analyzed 34 features (11 AI-specific, 23 general) using literature, official websites, tutorials, and developer interviews.
- An additional exploratory search on TopAI Tools for LLM-based literature review and writing tools returned 164 tools; 18 candidates were examined, yielding 11 included tools.
Limitations
- Stated: internal validity threats from subjective inclusion/exclusion decisions and the SLR Toolbox being offline since March 2024; external validity threats from search engines/search strings possibly missing tools and from excluding tools without user interfaces, under maintenance, or not updated in 10 years; construct validity threat that the 34 features may not cover all AI dimensions; conclusion validity threat from dynamic software and rapid AI evolution, making findings a snapshot.
- Evident from the text: the LLM-tool analysis is described as an initial exploratory study rather than a systematic review, and the proposed best practices/evaluation framework are not empirically validated in the paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
39 Valid · 20 Not valid · 40 Not checked · 177 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“(Cowie's et al. 2022) conducted the most restricted analysis, considering only one feature ( approach ).”
What the sources say
- Disputedcites Cowie's et al. 2022 sourceExperimental — accuracy not yet measured
The Introduction’s “Feature Analysis of Systematic Review Tools” says the analysis covered the full systematic-review life cycle across six feature classes. The Results’ “Feature Assessment: Breakout by Feature Class” reports multiple features within those classes, contrary to the claim that Cowie et al. considered only one feature. The paper does not establish the comparative label “most restricted.” (LLM, conf=0.92)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“(Burgard and Bittermann 2023) employed only three features: text representation , approach , and output .”
What the sources say
- Disputedcites Burgard and Bittermann 2023 sourceExperimental — accuracy not yet measured
Table 2 in “Applications and Performance of Screening Automation Tools” describes tools across more than three characteristics, including type of screening automation, web availability, fees, open-source status, machine-learning method, and active learning. The paper therefore does not restrict its analysis to the three features named in the claim. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In the context of SLR, these methodologies are predominantly utilised to semi-automate tasks like screening and data extraction (Cowie et al. 2022; Burgard and Bittermann 2023).”
What the sources say
- Partly supportedcites Cowie et al. 2022 sourceExperimental — accuracy not yet measured
The Discussion, under “Automation and Machine Learning,” reports automation for screening in 13 of 24 tools. But under “Data Extraction,” it says automated extraction is underdeveloped and has not achieved partial extraction at sufficient accuracy for adoption. The source therefore supports screening, but not the claim that AI is predominantly used to semi-automate data extraction. (LLM, conf=0.90)
- Partly supportedcites Burgard and Bittermann 2023 sourceExperimental — accuracy not yet measured
In “Need for Automation in Systematic Reviews” (page not available), the paper says tools have been developed to automate review steps including data extraction and reference screening, and it describes semiautomated screening. It does not establish that AI methodologies are predominantly used for those tasks, or that data extraction is predominantly semiautomated. (LLM, conf=0.82)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“We selected Scopus over other alternatives because it is widely recognised as the preferred source for conducting systematic literature reviews due to its high-quality metadata, reliable citation tracking, and extensive coverage of scientific documents, including journals, conference proceedings, and books (Baas et al. 2020; Visser et al. 2021).”
What the sources say
- Partly supportedcites Baas et al. 2020 sourceExperimental — accuracy not yet measured
Section 1, “Scopus as a Bibliometric Data Source” (page not provided), supports the stated reasons for choosing Scopus: it describes extensive journal, conference, and book coverage, quality controls for metadata, and high citation-linking accuracy. It does not establish that Scopus is widely recognised as the preferred source for systematic literature reviews over alternatives. (LLM, conf=0.88)
- Partly supportedcites Visser et al. 2021 sourceExperimental — accuracy not yet measured
The Introduction describes Scopus as a major bibliographic source, and the Conclusions report high precision and recall for its citation links and substantial coverage of journal articles and proceedings papers. The paper does not establish that Scopus is the preferred source for systematic literature reviews or assess its metadata quality as claimed; its conclusions also note strengths of other databases. Page numbers are unavailable in our copy. (LLM, conf=0.88)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Many of these tools have adopted Artificial Intelligence (AI) solutions (van den Bulk et al. 2022; Kebede et al. 2023), particularly for the screening and data extraction phases.”
What the sources say
- Partly supportedcites van den Bulk et al. 2022 sourceExperimental — accuracy not yet measured
The Introduction and Discussion (page numbers unavailable) describe machine-learning tools used to screen literature for systematic reviews. The Discussion describes automatically collecting information from relevant articles only as possible future work, so this paper does not support the claim that tools have already adopted AI for the data-extraction phase. (LLM, conf=0.88)
- Supportedcites Kebede et al. 2023 source
The Introduction says that systematic-review steps including screening and data extraction can be semi-automated with machine-learning algorithms or web-based tools. It also reports a review finding that a majority of studies used AI methods for screening, risk-of-bias assessment, and data extraction. This supports the citing sentence as a broad background statement. (LLM, conf=0.88)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
91 more claims · 13 with their cited sources checked
Create a free account to see all 276 claims and every source verdict.
- Probably not a substantial claim. 142
- It may state two separate claims as one. 64
- The quote only partly states this claim. 4
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ability of LLMs to document the rationale behind each screening and data extraction decision is a major advantage over earlier AI approaches, improving transparency and aiding in the interpretation of the models’ outputs, as well as informing the future development of models and prompts for better performance (Bolaños et al., 2024; de la Torre-López et al., 2023).”
From Analysis of article screening and data extraction performance by an AI systematic literature review platform (2025) · cites this paper as Bolaños et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Researchers’ current lack of trust in the accuracy and quality of SLR automation tools has been identified as a major barrier to uptake, and transparency from the developers and users of these tools has been suggested as a means to overcome this challenge (Bolaños et al., 2024; de la Torre-López et al., 2023; Gates et al., 2019; O’Connor et al., 2019; van Altena et al., 2019).”
From Analysis of article screening and data extraction performance by an AI systematic literature review platform (2025) · cites this paper as Bolaños et al., 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Until recently, most work in this field used natural language processing (NLP) and similar text mining approaches (Blaizot et al., 2022; Bolaños et al., 2024; Cowie et al., 2022; de la Torre-López et al., 2023; van Dinter et al., 2021).”
From Analysis of article screening and data extraction performance by an AI systematic literature review platform (2025) · cites this paper as Bolaños et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“Many of the tasks involved in SLRs are amenable to automation using artificial intelligence (AI) (Beller et al., 2018; Bolaños et al., 2024; Michelson and Reuter, 2019; Shemilt et al., 2016; Tsertsvadze et al., 2015).”
From Analysis of article screening and data extraction performance by an AI systematic literature review platform (2025) · cites this paper as Bolaños et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“Although we have demonstrated the potential utility of ISLaR in rapidly performing SLR tasks to ease researcher workload, there are many challenges and barriers that must be addressed before AI tools can be routinely used in practice (Bolaños et al., 2024; de la Torre-López et al., 2023; Doyal et al., 2023; O’Connor et al., 2019; van Altena et al., 2019).”
From Analysis of article screening and data extraction performance by an AI systematic literature review platform (2025) · cites this paper as Bolaños et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.3 Outstanding SLR tools
“| | Multiple user support | 21 (100%) | 0 (0%) |”
§ 8 Conclusion
“| Mul- tiple user sup- port | 2 | 2 | >1 | >1 | No | >1”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.3 Outstanding SLR tools
“| | Multiple user roles | 21 (100%) | 0 (0%) |”
§ 8 Conclusion
“| Mul- tiple user roles | | | | | | | | Mult. | Sing. | | | Sing. | Sing.”
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 3×
“Finally, Section 4.2.3 describes the full set of 21 SLR tools according to the general features.”
InfluentialRayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · The Lancet · doi:10.1016/s0140-6736(07)61602-x · Cited by 21,320
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769 · cited 1×
“This methodology originally emerged within the realm of Evidence-Based Medicine [3], and it was subsequently adapted and employed in diverse research disciplines including social sciences [4], engineering and technology [5], education [6], environmental sciences [7], and business and management [8].”
Tranfield, D., Denyer, D., & Smart, P. (2003) · British Journal of Management · doi:10.1111/1467-8551.00375 · Cited by 15,386 · cited 1×
“This is due to several factors, such as the duration of the whole process, which can extend beyond a year [9]; the necessity for a multidisciplinary team of experts, often accompanied by specific training [10]; significant financial implications arising from database subscriptions, specialised…”
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 2×
“For this reason, it was suggested to use F2 score [141] over the F1 score.”
Influential
Research works printed but not linked: 11
- Iansiti M, Lakhani KR (2020) Competing in the age of AI: strategy and leadership when algorithms and networks run the world. Harvard Business Press, Boston
- Kusa W, Knoth P, Hanbury A (2022) Evaluation of automated citation screening in systematic literature reviews with work saved over sampling: an analysis. In: 1st workshop on augmented intelligence for technology-assisted reviews systems: evaluation metrics and protocols for ediscovery and systematic review systems, pp 1–7
- Le Q, Mikolov T (2014) Distributed representations of sentences and documents. In: International Conference on Machine Learning. PMLR, pp 1188–1196
- Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, Küttler H, Lewis M, Yih W-T, Rocktäschel T et al (2020) Retrieval-augmented generation for knowledge-intensive NLP tasks. Adv Neural Inf Process Syst 33:9459–9474
- Li Q, Li J, Sheng J, Cui S, Wu J, Hei Y, Peng H, Guo S, Wang L, Beheshti A et al (2022) A survey on deep learning event extraction: approaches and applications. IEEE Trans Neural Netw Learn Syst
- Lipscomb CE (2000) Medical subject headings (mesh). Bull Med Libr Assoc 88(3):265
- Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. Adv Neural Inf Process Syst 26
- Minion JT, Egunsola O, Mastikhina L, Farkas B, Hofmeister M, Flanagan J, Salmon C, Clement F (2021) Pico portal. J Can Health Libraries Assoc 42(3):181
- Ng A, Jordan M, Weiss Y (2001) On spectral clustering: analysis and an algorithm. Adv Neural Inf Process Syst 14
- National Academies of Sciences, Engineering, and Medicine; Policy and Global Affairs; Board on Research Data and Information; Committee on Toward an Open Science Enterprise (2018) Open science by design: realizing a vision for 21st century research. National Academies Press, Washington
- Wang LL, DeYoung J, Wallace B (2022) Overview of mslr2022: a shared task on multi-document summarization for literature reviews. In: Proceedings of the third workshop on scholarly document processing
Cited by
Resnik, D. B., & Hosseini, M. (2025) · AI and Ethics · doi:10.1007/s43681-024-00493-8 · Cited by 228
“• Searching and reviewing the scientific literature [30, 72].”
Bandi, A., Kongari, B., Naguru, R., Pasnoor, S., & Vilipala, S. V. (2025) · Future Internet · doi:10.3390/fi17090404 · Cited by 207
Suchikova, Y., Tsybuliak, N., Teixeira da Silva, J. A., & Nazarovets, S. (2026) · Accountability in Research · doi:10.1080/08989621.2025.2544331 · Cited by 70
Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning
Sušnjak, T., Hwang, P., Reyes, N. H., Barczak, A. L. C., McIntosh, T. R., & Ranathunga, S. (2025) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3715964 · Cited by 142
“In their most recent review, Bolanos et al. [28] found that LLM usage for SLRs is hampered by limitations such as reduced efficacy in domain-specific and narrower subject areas, their propensity for hallucinating and generating misleading information, alongside their opaque decision processes which…”
BackgroundEconomic impacts of AI-augmented R&D
Besiroglu, T., Emery-Xu, N., & Thompson, N. (2024) · Research Policy · doi:10.1016/j.respol.2024.105037 · Cited by 115
Metadata
- Authors
- Bolaños, F., Salatino, A., Osborne, F., & Motta, E.
- Affiliations
- The Open University (GB)
- Corresponding author
- Bolaños, Francisco
- Venue
- Artificial Intelligence Review
- Published
- 2024-08-17
- Volume
- 57
- Issue
- 10
- ISSN
- 0269-2821 (ISSN-L)
- 1573-7462
- OpenAlex
- W4401667275
- Cited by
- 246
- References
- 170
- Citations by year
- 2026: 124 · 2025: 103 · 2024: 19
- FWCI
- 69.13
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Artificial Intelligence Review (publishedVersion, cc-by)
- Open Research Online (The Open University) (acceptedVersion, cc-by)
- BOA (University of Milano-Bicocca) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- artificial intelligence, systematic reviews, large language models, literature screening, data extraction, knowledge graphs, academic writing support, usability, transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Artificial intelligence for literature reviews: opportunities and challenges. https://app.cochat.ai/papers/10.1007/s10462-024-10902-3
- The paper
- Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024). Artificial intelligence for literature reviews: opportunities and challenges. Artificial Intelligence Review. https://doi.org/10.1007/s10462-024-10902-3
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.