Artificial intelligence for literature reviews: opportunities and challenges
Open accessThe DOI matches its Crossref record; 217 of the 218 references it cites were found in a registry or an index; 62 of the 115 sources its claims cite were checked for support.
Abstract
This paper presents a comprehensive review of the use of Artificial Intelligence (AI) in Systematic Literature Reviews (SLRs). A SLR is a rigorous and organised methodology that assesses and integrates prior research on a given topic. Numerous tools have been developed to assist and partially automate the SLR process. The increasing role of AI in this field shows great potential in providing more effective support for researchers, moving towards the semi-automatic creation of literature reviews. Our study focuses on how AI techniques are applied in the semi-automation of SLRs, specifically in the screening and extraction phases. We examine 21 leading SLR tools using a framework that combines 23 traditional features with 11 AI features. We also analyse 11 recent tools that leverage large language models for searching the literature and assisting academic writing. Finally, the paper discusses current trends in the field, outlines key research challenges, and suggests directions for future research. We highlight three primary research challenges: integrating advanced AI solutions, such as large language models and knowledge graphs, improving usability, and developing a standardised evaluation framework. We also propose best practices to ensure more robust evaluations in terms of performance, usability, and transparency. Overall, this review offers a detailed overview of AI-enhanced SLR tools for researchers and practitioners, providing a foundation for the development of next-generation AI solutions in this field.
Overview
CoChat Summary
TL;DR
This paper reviews AI use in systematic literature reviews, analyzing 21 SLR tools with 23 general and 11 AI-specific features plus 11 recent LLM-based tools. It finds most SLR tools rely on older AI methods and have usability/evaluation gaps, and identifies three main challenges: advanced AI integration, usability, and standardised evaluation.
Key findings
- Of the 21 SLR tools, 19 support screening and 4 support extraction; 19 are web applications and 2 are desktop (SWIFT-Review, ASReview); only 4 release code under an open license (Colandr, ASReview, FAST2, RobotReviewer/RobotSearch). Most use basic classifiers and bag-of-words; SVM is the most adopted approach, with Abstrackr, FAST2, Rayyan, and RobotAnalyst relying exclusively on SVM. Minimum seed requirements vary: ASReview, SWIFT-Active Screener, and SWIFT-Review require 1 relevant and 1 irrelevant paper; Covidence 2; Rayyan 5; Colandr 10; SysRev 30.
- Only 8 of 21 tools implement at least 70% of the 23 general features. DistillerSR, Nested Knowledge, Dextr, and ExaCT lead at 82%; PICOPortal and Rayyan at 78%; EPPI-Reviewer at 74%; SWIFT-Active Screener at 70%. 13 of 21 are free to use; only LitSuggest supports living reviews; only 4 tools provide PRISMA diagram or protocol templates.
- The paper identifies 11 recent LLM-based tools: 7 search engines, 3 writing assistants, and Textero.ai fitting both categories. Most use the OpenAI API, often with RAG; output quality varies and may be more useful for master’s students than researchers.
- Usability and evaluation remain weak: a survey of 81 researchers found discontinuation reasons were poor usability (43%), insufficient functionality (37%), and workflow incompatibility (37%); SUS scores ranged from 66 to 77 (C–B). No standard evaluation framework exists; F2, WSS, and nWSS are proposed metrics, and only 4 of 21 tools operate under open licenses.
Methods
- PRISMA systematic review. The authors searched Scopus for prior surveys, the Systematic Literature Review Toolbox, CRAN, and performed snowballing via Semantic Scholar; after inclusion/exclusion criteria, they selected 21 SLR tools. They analyzed 34 features (11 AI-specific, 23 general) using literature, official websites, tutorials, and developer interviews.
- An additional exploratory search on TopAI Tools for LLM-based literature review and writing tools returned 164 tools; 18 candidates were examined, yielding 11 included tools.
Limitations
- Stated: internal validity threats from subjective inclusion/exclusion decisions and the SLR Toolbox being offline since March 2024; external validity threats from search engines/search strings possibly missing tools and from excluding tools without user interfaces, under maintenance, or not updated in 10 years; construct validity threat that the 34 features may not cover all AI dimensions; conclusion validity threat from dynamic software and rapid AI evolution, making findings a snapshot.
- Evident from the text: the LLM-tool analysis is described as an initial exploratory study rather than a systematic review, and the proposed best practices/evaluation framework are not empirically validated in the paper.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
39 Valid · 20 Not valid · 40 Not checked · 177 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“(Cowie's et al. 2022) conducted the most restricted analysis, considering only one feature ( approach ).”
What the sources say
- Disputedcites Cowie's et al. 2022 sourceExperimental — accuracy not yet measured
The Introduction’s “Feature Analysis of Systematic Review Tools” says the analysis covered the full systematic-review life cycle across six feature classes. The Results’ “Feature Assessment: Breakout by Feature Class” reports multiple features within those classes, contrary to the claim that Cowie et al. considered only one feature. The paper does not establish the comparative label “most restricted.” (LLM, conf=0.92)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“(Burgard and Bittermann 2023) employed only three features: text representation , approach , and output .”
What the sources say
- Disputedcites Burgard and Bittermann 2023 sourceExperimental — accuracy not yet measured
Table 2 in “Applications and Performance of Screening Automation Tools” describes tools across more than three characteristics, including type of screening automation, web availability, fees, open-source status, machine-learning method, and active learning. The paper therefore does not restrict its analysis to the three features named in the claim. (LLM, conf=0.86)
Checked by a reasoning model — may be wrong.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In the context of SLR, these methodologies are predominantly utilised to semi-automate tasks like screening and data extraction (Cowie et al. 2022; Burgard and Bittermann 2023).”
What the sources say
- Partly supportedcites Cowie et al. 2022 sourceExperimental — accuracy not yet measured
The Discussion, under “Automation and Machine Learning,” reports automation for screening in 13 of 24 tools. But under “Data Extraction,” it says automated extraction is underdeveloped and has not achieved partial extraction at sufficient accuracy for adoption. The source therefore supports screening, but not the claim that AI is predominantly used to semi-automate data extraction. (LLM, conf=0.90)
- Partly supportedcites Burgard and Bittermann 2023 sourceExperimental — accuracy not yet measured
In “Need for Automation in Systematic Reviews” (page not available), the paper says tools have been developed to automate review steps including data extraction and reference screening, and it describes semiautomated screening. It does not establish that AI methodologies are predominantly used for those tasks, or that data extraction is predominantly semiautomated. (LLM, conf=0.82)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“We selected Scopus over other alternatives because it is widely recognised as the preferred source for conducting systematic literature reviews due to its high-quality metadata, reliable citation tracking, and extensive coverage of scientific documents, including journals, conference proceedings, and books (Baas et al. 2020; Visser et al. 2021).”
What the sources say
- Partly supportedcites Baas et al. 2020 sourceExperimental — accuracy not yet measured
Section 1, “Scopus as a Bibliometric Data Source” (page not provided), supports the stated reasons for choosing Scopus: it describes extensive journal, conference, and book coverage, quality controls for metadata, and high citation-linking accuracy. It does not establish that Scopus is widely recognised as the preferred source for systematic literature reviews over alternatives. (LLM, conf=0.88)
- Partly supportedcites Visser et al. 2021 sourceExperimental — accuracy not yet measured
The Introduction describes Scopus as a major bibliographic source, and the Conclusions report high precision and recall for its citation links and substantial coverage of journal articles and proceedings papers. The paper does not establish that Scopus is the preferred source for systematic literature reviews or assess its metadata quality as claimed; its conclusions also note strengths of other databases. Page numbers are unavailable in our copy. (LLM, conf=0.88)
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Many of these tools have adopted Artificial Intelligence (AI) solutions (van den Bulk et al. 2022; Kebede et al. 2023), particularly for the screening and data extraction phases.”
What the sources say
- Partly supportedcites van den Bulk et al. 2022 sourceExperimental — accuracy not yet measured
The Introduction and Discussion (page numbers unavailable) describe machine-learning tools used to screen literature for systematic reviews. The Discussion describes automatically collecting information from relevant articles only as possible future work, so this paper does not support the claim that tools have already adopted AI for the data-extraction phase. (LLM, conf=0.88)
- Supportedcites Kebede et al. 2023 source
The Introduction says that systematic-review steps including screening and data extraction can be semi-automated with machine-learning algorithms or web-based tools. It also reports a review finding that a majority of studies used AI methods for screening, risk-of-bias assessment, and data extraction. This supports the citing sentence as a broad background statement. (LLM, conf=0.88)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
91 more claims · 13 with their cited sources checked
Create a free account to see all 276 claims and every source verdict.
- Probably not a substantial claim. 142
- It may state two separate claims as one. 64
- The quote only partly states this claim. 4
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ability of LLMs to document the rationale behind each screening and data extraction decision is a major advantage over earlier AI approaches, improving transparency and aiding in the interpretation of the models’ outputs, as well as informing the future development of models and prompts for better performance (Bolaños et al., 2024; de la Torre-López et al., 2023).”
From Analysis of article screening and data extraction performance by an AI systematic literature review platform (2025) · cites this paper as Bolaños et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Researchers’ current lack of trust in the accuracy and quality of SLR automation tools has been identified as a major barrier to uptake, and transparency from the developers and users of these tools has been suggested as a means to overcome this challenge (Bolaños et al., 2024; de la Torre-López et al., 2023; Gates et al., 2019; O’Connor et al., 2019; van Altena et al., 2019).”
From Analysis of article screening and data extraction performance by an AI systematic literature review platform (2025) · cites this paper as Bolaños et al., 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Until recently, most work in this field used natural language processing (NLP) and similar text mining approaches (Blaizot et al., 2022; Bolaños et al., 2024; Cowie et al., 2022; de la Torre-López et al., 2023; van Dinter et al., 2021).”
From Analysis of article screening and data extraction performance by an AI systematic literature review platform (2025) · cites this paper as Bolaños et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“Many of the tasks involved in SLRs are amenable to automation using artificial intelligence (AI) (Beller et al., 2018; Bolaños et al., 2024; Michelson and Reuter, 2019; Shemilt et al., 2016; Tsertsvadze et al., 2015).”
From Analysis of article screening and data extraction performance by an AI systematic literature review platform (2025) · cites this paper as Bolaños et al., 2024
The cited source says what the claim attributes to it.
In the citing paper
“Although we have demonstrated the potential utility of ISLaR in rapidly performing SLR tasks to ease researcher workload, there are many challenges and barriers that must be addressed before AI tools can be routinely used in practice (Bolaños et al., 2024; de la Torre-López et al., 2023; Doyal et al., 2023; O’Connor et al., 2019; van Altena et al., 2019).”
From Analysis of article screening and data extraction performance by an AI systematic literature review platform (2025) · cites this paper as Bolaños et al., 2024
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.3 Outstanding SLR tools
“| | Multiple user support | 21 (100%) | 0 (0%) |”
§ 8 Conclusion
“| Mul- tiple user sup- port | 2 | 2 | >1 | >1 | No | >1”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.3 Outstanding SLR tools
“| | Multiple user roles | 21 (100%) | 0 (0%) |”
§ 8 Conclusion
“| Mul- tiple user roles | | | | | | | | Mult. | Sing. | | | Sing. | Sing.”
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 3×
“Finally, Section 4.2.3 describes the full set of 21 SLR tools according to the general features.”
Influential(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · The Lancet · doi:10.1016/s0140-6736(07)61602-x · Cited by 21,320
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769 · cited 1×
“This methodology originally emerged within the realm of Evidence-Based Medicine [3], and it was subsequently adapted and employed in diverse research disciplines including social sciences [4], engineering and technology [5], education [6], environmental sciences [7], and business and management [8].”
Tranfield, D., Denyer, D., & Smart, P. (2003) · British Journal of Management · doi:10.1111/1467-8551.00375 · Cited by 15,386 · cited 1×
“This is due to several factors, such as the duration of the whole process, which can extend beyond a year [9]; the necessity for a multidisciplinary team of experts, often accompanied by specific training [10]; significant financial implications arising from database subscriptions, specialised…”
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · cited 2×
“For this reason, it was suggested to use F2 score [141] over the F1 score.”
InfluentialBlei, D. M., Ng, A. Y., & Jordan, M. I. (2002) · Advances in Neural Information Processing Systems 14 · doi:10.7551/mitpress/1120.003.0082 · Cited by 12,223
A training algorithm for optimal margin classifiers
Boser, B. E., Guyon, I. M., & Vapnik, V. N. (1992) · Proceedings of the fifth annual workshop on Computational learning theory · doi:10.1145/130385.130401 · Cited by 11,992 · cited 1×
“Finally, RobotReviewer/RobotSearch employs an ensemble classifier, combining multiple CNN models [101] and soft-margin Support Vector Machines [102] in order to categorise articles based on their risk of bias assessment (either low or high/unclear) and concurrently extract sentences that…”
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 2×
“Likewise, several tools continue to employ BoW methods for text representation, although some of the most recent ones [42, 45, 71] have shifted towards adopting word and sentence embedding techniques, such as GloVe [95], ELMo [103], SciBERT [90], and Sentence-BERT [92].”
The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · International Journal of Surgery · doi:10.1016/j.ijsu.2021.105906 · Cited by 11,748
A typology of reviews: an analysis of 14 review types and associated methodologies
Grant, M. J., & Booth, A. (2009) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2009.00848.x · Cited by 10,817 · cited 1×
“Nevertheless, there are some emerging tools, which we will examine in Section 6, that have begun to incorporate LLMs in academic search engines, often within a Retrieval-Augmented Generation (RAG) framework [35].”
SUS: A 'Quick and Dirty' Usability Scale
(1996) · Usability Evaluation In Industry · doi:10.1201/9781498710411-35 · Cited by 8,989
Analyzing the Past to Prepare for the Future: Writing a Literature Review
Webster, J., & Watson, R. T. (2002) · MIS Quarterly · doi:10.2307/4132319 · Cited by 7,449
Hearst, M. A., Dumais, S. T., Osuna, E., Platt, J., & Scholkopf, B. (1998) · IEEE Intelligent Systems and their Applications · doi:10.1109/5254.708428 · Cited by 7,243 · cited 1×
“Lastly, the economic category reflects the financial considerations associated with the tools.”
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218 · cited 1×
“In the second group, Covidence and EPPI-Reviewer adopt a BoW representation, while PICOPortal uses both BoW and the BioBERT embeddings [93].”
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · The Lancet · doi:10.1016/s0140-6736(00)04337-3 · Cited by 5,373
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025 · cited 3×
“This allows the LLMs to rely on verifiable information, thereby reducing inaccuracies or hallucinations in their outputs [107].”
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Experimentation in Software Engineering
Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., & Wesslén, A. (2012) · doi:10.1007/978-3-642-29044-2 · Cited by 4,460 · cited 1×
“To mitigate this constraint, a thorough review of relevant research papers, the tools’ official web-sites, and instructional materials was conducted.”
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.03.001 · Cited by 4,265
Wohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 1×
“This process is iteratively repeated until no new relevant scholarly documents are found [30, 31].”
Systematic reviews in the social sciences: a practical guide
(2006) · Choice Reviews Online · doi:10.5860/choice.43-5664 · Cited by 4,089
Systematic Reviews in the Social Sciences: A Practical Guide
Petticrew, M., & Roberts, H. (2006) · doi:10.1002/9780470754887 · Cited by 3,586
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397 · cited 1×
“This involves conducting in-depth studies to understand the various aspects of usability, such as effectiveness, efficiency, engagement, error tolerance, and ease of learning [157].”
Assessing Risk of Bias in Included Studies
Higgins, J. P., & Altman, D. G. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch8 · Cited by 3,396 · cited 1×
“We then describe the PRISMA methodology [24] employed to identify, filter, and analyse the SLR tools (Section 2.2).”
InfluentialTowards complete and accurate reporting of studies of diagnostic accuracy: The STARD Initiative.
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., de Vet, H. C. W., et al. (2003) · Annals of Internal Medicine · doi:10.7326/0003-4819-138-1-200301070-00010 · Cited by 3,219
SciBERT: A Pretrained Language Model for Scientific Text
Beltagy, I., Lo, K., & Cohan, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1371 · Cited by 3,218 · cited 2×
“For this reason, it was suggested to use F2 score [141] over the F1 score.”
InfluentialTowards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · BMJ · doi:10.1136/bmj.326.7379.41 · Cited by 3,215
Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., Lijmer, J. G., Moher, D., Rennie, D., & de Vet, H. C. W. (2003) · Radiology · doi:10.1148/radiol.2261021292 · Cited by 3,215
Explainable AI: a review of machine learning interpretability methods
Linardatos, P., Papastefanopoulos, V., & Kotsiantis, S. (2020) · Entropy · doi:10.3390/e23010018 · Cited by 2,922
Moher, D., Schulz, K. F., & Altman, D. G. (2001) · BMC Medical Research Methodology · doi:10.1186/1471-2288-1-2 · Cited by 2,755
Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World
Pranckutė, R. (2021) · Publications · doi:10.3390/publications9010012 · Cited by 2,696 · cited 1×
“In snowballing, the researcher examines the references and citations of an initial group of papers (also known as seed papers) to identify additional articles.”
Construction and Evaluation of a User Experience Questionnaire
Laugwitz, B., Held, T., & Schrepp, M. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-89350-9_6 · Cited by 2,114
Baas, J., Schotten, M., Plume, A., Côté, G., & Karimi, R. (2020) · Quantitative Science Studies · doi:10.1162/qss_a_00019 · Cited by 2,093 · cited 1×
“Traditionally, the search phase had not been significantly supported by artificial intelligence techniques [34].”
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2007) · Epidemiology · doi:10.1097/ede.0b013e3181577654 · Cited by 1,991
The System Usability Scale: Past, Present, and Future
Lewis, J. R. (2018) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2018.1455307 · Cited by 1,958 · cited 1×
“In the same study, a set of SLR tools was assessed using the System Usability Scale (SUS) questionnaire [156].”
Can we open the black box of AI?
Castelvecchi, D. (2016) · Nature · doi:10.1038/538020a · Cited by 1,773 · cited 2×
“The full tables describing the 21 tools for SLRs according to the 34 features are available in digital format at: https://angelosalatino. github.io/ai-slr/.”
Bornmann, L., & Mutz, R. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23329 · Cited by 1,576 · cited 1×
“…by specific training [10]; significant financial implications arising from database subscriptions, specialised software utilisation, and personnel remunerations [11]; the growing rate of the number of publication [12], and the periodic need for updates to ensure continued relevance [13].”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387 · cited 1×
“Therefore, the first interesting research direction regards incorporating advanced NLP technologies, particularly the rapidly evolving Large Language Models (LLMs) [104].”
Understanding bag-of-words model: a statistical framework
Zhang, Y., Jin, R., & Zhou, Z.-H. (2010) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-010-0001-0 · Cited by 1,384 · cited 2×
“Text representation - describes the processes employed to convert text into suitable input for the algorithm (e.g., BoW [76], LDA topics [77], word embeddings [78]).”
An Introduction to Systematic Reviews
Gough, D., Oliver, S., & Thomas, J. (2017) · doi:10.4135/9781036234942 · Cited by 1,239
Tree of Thoughts: Deliberate Problem Solving with Large Language Models
Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023) · arXiv · doi:10.48550/arxiv.2305.10601 · Cited by 1,158
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157 · cited 2×
“Likewise, several tools continue to employ BoW methods for text representation, although some of the most recent ones [42, 45, 71] have shifted towards adopting word and sentence embedding techniques, such as GloVe [95], ELMo [103], SciBERT [90], and Sentence-BERT [92].”
Mariani, J. (2018) · HAL · Cited by 1,151
Guidelines for systematic review in Conservation and environmental management
PULLIN, A. S., & STEWART, G. B. (2006) · Conservation Biology · doi:10.1111/j.1523-1739.2006.00485.x · Cited by 1,137 · cited 1×
“This methodology originally emerged within the realm of Evidence-Based Medicine [3], and it was subsequently adapted and employed in diverse research disciplines including social sciences [4], engineering and technology [5], education [6], environmental sciences [7], and business and management [8].”
Bias in data‐driven artificial intelligence systems—An introductory survey
Ntoutsi, E., Fafalios, P., Gadiraju, U., Iosifidis, V., Nejdl, W., Vidal, M., Ruggieri, S., Turini, F., Papadopoulos, S., Krasanakis, E., et al. (2020) · WIREs Data Mining and Knowledge Discovery · doi:10.1002/widm.1356 · Cited by 1,043 · cited 1×
“Textero.ai was the only identified tool fitting into both categories.”
Brynjolfsson, E., Li, D., & Raymond, L. (2023) · National Bureau of Economic Research · doi:10.3386/w31161 · Cited by 995
Reproducibility and Replicability in Science
Committee on Reproducibility and Replicability in Science, Board on Behavioral, C., Committee on National Statistics, Education, D. O. B. A. S. S. A., Nuclear and Radiation Studies Board, Studies, D. O. E. A. L., Board on Mathematical Sciences and Analytics, Committee on Applied and Theoretical Statistics, Sciences, D. O. E. A. P., Board on Research Data and Information, et al. (2019) · doi:10.17226/25303 · Cited by 924 · cited 1×
“The tools in this category allow users to formulate a natural language query and generate a list of relevant research papers sourced from online repositories.”
Visser, M., van Eck, N. J., & Waltman, L. (2021) · Quantitative Science Studies · doi:10.1162/qss_a_00112 · Cited by 865
How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis
Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., & Moher, D. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-4-200708210-00179 · Cited by 838 · cited 1×
“…whole process, which can extend beyond a year [9]; the necessity for a multidisciplinary team of experts, often accompanied by specific training [10]; significant financial implications arising from database subscriptions, specialised software utilisation, and personnel remunerations [11]; the…”
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737 · cited 1×
“This is due to several factors, such as the duration of the whole process, which can extend beyond a year [9]; the necessity for a multidisciplinary team of experts, often accompanied by specific training [10]; significant financial implications arising from database subscriptions, specialised…”
Knowledge Graphs: Opportunities and Challenges
Peng, C., Xia, F., Naseriparsa, M., & Osborne, F. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10465-9 · Cited by 717 · cited 1×
“Knowledge graphs consist of large networks of entities and relationships that provide machine-readable and understandable information about a specific domain following formal semantics [124].”
Self-citationAn Introduction to the Policy Process: Theories, Concepts, and Models of Public Policy Making
Birkland, T. A. (2019) · Routledge eBooks · doi:10.4324/9781351023948 · Cited by 690
Ensemble of keyword extraction methods and classifiers in text classification
Onan, A., Korukoğlu, S., & Bulut, H. (2016) · Expert Systems with Applications · doi:10.1016/j.eswa.2016.03.045 · Cited by 630 · cited 1×
“PICOPortal employs instead an ensemble of machine learning classifiers, which combines both decision trees and neural networks [89].”
Convolutional Neural Networks: A Survey
Krichen, M. (2023) · Computers · doi:10.3390/computers12080151 · Cited by 623 · cited 1×
“Finally, RobotReviewer/RobotSearch employs an ensemble classifier, combining multiple CNN models [101] and soft-margin Support Vector Machines [102] in order to categorise articles based on their risk of bias assessment (either low or high/unclear) and concurrently extract sentences that…”
The Usability Metric for User Experience
Finstad, K. (2010) · Interacting with Computers · doi:10.1016/j.intcom.2010.04.004 · Cited by 604
Graph of Thoughts: Solving Elaborate Problems with Large Language Models
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., et al. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29720 · Cited by 556 · cited 1×
“These techniques include Chain-of-Thought (CoT) [118], Tree of Thoughts (ToT) [119, 120] and Graph of Thoughts (GoT) [121].”
Graph of Thoughts: Solving Elaborate Problems with Large Language Models
Besta, M., Blach, N., Kubicek, A., Gerstenberger, R., Podstawski, M., Gianinazzi, L., Gajda, J., Lehmann, T., Niewiadomski, H., Nyczyk, P., et al. (2023) · arXiv · doi:10.48550/arxiv.2308.09687 · Cited by 556
Deploying an interactive machine learning system in an evidence-based practice center: abstrackr
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012) · Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium · doi:10.1145/2110363.2110464 · Cited by 532
JBI's Systematic Reviews: Data Extraction and Synthesis
Munn, Z., Tufanaru, C., & Aromataris, E. (2014) · AJN, American Journal of Nursing · doi:10.1097/01.naj.0000451683.66447.89 · Cited by 510 · cited 1×
“We then describe the PRISMA methodology [24] employed to identify, filter, and analyse the SLR tools (Section 2.2).”
InfluentialSystematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472 · cited 1×
“[ 55]; IC 2.”
Kohl, C., McIntosh, E. J., Unger, S., Haddaway, N. R., Kecke, S., Schiemann, J., & Wilhelm, R. (2018) · Environmental Evidence · doi:10.1186/s13750-018-0115-5 · Cited by 471 · cited 3×
“Among the remaining 13 tools, eight cover between 50% and 70% of the features, while the last five cover between 35% and 50%.”
Transparency in artificial intelligence
Larsson, S., & Heintz, F. (2020) · Internet Policy Review · doi:10.14763/2020.2.1469 · Cited by 458 · cited 1×
“For instance, the latest iteration of the Computer Science Knowledge Graph (CS-KG) 8 details an impressive array of 24 million methods, tasks, materials, and metrics automatically extracted from approximately 14.5 million scientific articles [129].”
InfluentialHarrison, H., Griffin, S. J., Kuhn, I., & Usher-Smith, J. A. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-0897-3 · Cited by 449 · cited 3×
“Comparing our results with the previous studies in the literature [17, 80, 82, 83], we observed that many SLR tools have undergone significant development and advancements in the last few years.”
Marshall, I. J., Noel‐Storr, A., Kuiper, J., Thomas, J., & Wallace, B. C. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1287 · Cited by 435 · cited 3×
“Likewise, several tools continue to employ BoW methods for text representation, although some of the most recent ones [42, 45, 71] have shifted towards adopting word and sentence embedding techniques, such as GloVe [95], ELMo [103], SciBERT [90], and Sentence-BERT [92].”
Wohlin, C., Kalinowski, M., Romero Felizardo, K., & Mendes, E. (2022) · Information and Software Technology · doi:10.1016/j.infsof.2022.106908 · Cited by 432 · cited 1×
“The hybrid approach is the combination of search strategy and snowballing [32, 33].”
Lawrence, J., & Reed, C. (2020) · Computational Linguistics · doi:10.1162/coli_a_00364 · Cited by 396 · cited 1×
“One important research challenge here is to improve this step by including inter-pretability mechanisms such as fact-checking [115] or argument mining [116] to provide further insights.”
Defining the Review Question and Developing Criteria for Including Studies
O'Connor, D., Green, S., & Higgins, J. P. (2008) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9780470712184.ch5 · Cited by 374 · cited 1×
“We then describe the PRISMA methodology [24] employed to identify, filter, and analyse the SLR tools (Section 2.2).”
InfluentialSentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · arXiv · doi:10.48550/arxiv.1908.10084 · Cited by 343
Large language models for generative information extraction: a survey
Xu, D., Chen, W., Peng, W., Zhang, C., Xu, T., Zhao, X., Wu, X., Zheng, Y., Wang, Y., & Chen, E. (2024) · Frontiers of Computer Science · doi:10.1007/s11704-024-40555-y · Cited by 326 · cited 1×
“This section will discuss some of the key research challenges identified from our analysis that the academic community will need to address in future work.”
Bhattacharya, D. (2021) · Strategic Analysis · doi:10.1080/09700161.2021.1918951 · Cited by 311 · cited 1×
“We then describe the PRISMA methodology [24] employed to identify, filter, and analyse the SLR tools (Section 2.2).”
InfluentialAmpatzoglou, A., Bibi, S., Avgeriou, P., Verbeek, M., & Chatzigeorgiou, A. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.10.006 · Cited by 305 · cited 1×
“This section will discuss some of the key research challenges identified from our analysis that the academic community will need to address in future work.”
Artificial intelligence and the conduct of literature reviews
Wagner, G., Lukyanenko, R., & Paré, G. (2022) · Journal of Information Technology · doi:10.1177/02683962211048201 · Cited by 296 · cited 1×
“This survey aims to address the existing gap by rigorously examining the application of AI techniques in the semi-automation of SLRs, within the two main stages of application, namely screening and extraction .”
Introduction to Systematic Reviews
Clarke, M. (2017) · Veterinary Evidence · doi:10.18849/ve.v2i2.131 · Cited by 276
Thomas, J., McDonald, S., Noel-Storr, A., Shemilt, I., Elliott, J., Mavergames, C., & Marshall, I. J. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.11.003 · Cited by 253 · cited 2×
“Covidence also implements a machine learning classifier based on SVM with a fixed threshold for the identification of RCTs following the Cochrane guidelines [86].”
Putting AI ethics to work: are the tools fit for purpose?
Ayling, J., & Chapman, A. (2022) · AI and Ethics · doi:10.1007/s43681-021-00084-x · Cited by 253
Content‐based citation analysis: The next generation of citation analysis
Ding, Y., Zhang, G., Chambers, T., Song, M., Wang, X., & Zhai, C. (2014) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23256 · Cited by 252
A review of the semantic web field
Hitzler, P. (2021) · Communications of the ACM · doi:10.1145/3397512 · Cited by 250 · cited 1×
“They typically organise information according to a domain ontology, which provides a formalised description of entity types and their relationships [125].”
Named Entity Recognition and Relation Extraction: State-of-the-Art
Nasar, Z., Jaffry, S. W., & Malik, M. K. (2022) · ACM Computing Surveys · doi:10.1145/3445965 · Cited by 245
Generative AI at WorkPreprint
Brynjolfsson, E., Li, D., & Raymond, L. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4426942 · Cited by 235
Automatic classification of sentences to support Evidence Based Medicine.
Kim, S. N., Martinez, D., Cavedon, L., & Yencken, L. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-s2-s5 · Cited by 214 · cited 1×
“We then describe the PRISMA methodology [24] employed to identify, filter, and analyse the SLR tools (Section 2.2).”
InfluentialOpen Research Knowledge Graph: Next Generation Infrastructure for Semantic Scholarly Knowledge
Jaradeh, M. Y., Oelen, A., Farfar, K. E., Prinz, M., D'Souza, J., Kismihók, G., Stocker, M., & Auer, S. (2019) · Proceedings of the 10th International Conference on Knowledge Capture · doi:10.1145/3360901.3364435 · Cited by 212
A systematic review of artificial intelligence impact assessments
Stahl, B. C., Antoniou, J., Bhalla, N., Brooks, L., Jansen, P., Lindqvist, B., Kirichenko, A., Marchal, S., Rodrigues, R., Santiago, N., et al. (2023) · Artificial Intelligence Review · doi:10.1007/s10462-023-10420-8 · Cited by 212
Sumbul, G., de Wall, A., Kreuziger, T., Marcelino, F., Costa, H., Benevides, P., Caetano, M., Demir, B., & Markl, V. (2021) · IEEE Geoscience and Remote Sensing Magazine · doi:10.1109/mgrs.2021.3089174 · Cited by 208 · cited 1×
“For the analysis of this emerging generation of tools, we focused on those that are accessible as online services, excluding those that are only referenced in academic papers and not available as such.”
Nicholson, J. M., Mordaunt, M., Lopez, P., Uppala, A., Rosati, D., Rodrigues, N. P., Grabitz, P., & Rife, S. C. (2021) · bioRxiv (published in Quantitative Science Studies) · doi:10.1101/2021.03.15.435418 · Cited by 202
Developing search strategies for detecting relevant experiments
Dieste, O., Grimán, A., & Juristo, N. (2009) · Empirical Software Engineering · doi:10.1007/s10664-008-9091-7 · Cited by 200 · cited 1×
“Nevertheless, there are some emerging tools, which we will examine in Section 6, that have begun to incorporate LLMs in academic search engines, often within a Retrieval-Augmented Generation (RAG) framework [35].”
SWIFT-Review: a text-mining workbench for systematic review
Howard, B. E., Phillips, J., Miller, K., Tandon, A., Mav, D., Shah, M. R., Holmgren, S., Pelch, K. E., Walker, V., Rooney, A. A., et al. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0263-z · Cited by 196
A survey of word embeddings based on deep learning
Wang, S., Zhou, W., & Jiang, C. (2020) · Computing · doi:10.1007/s00607-019-00768-7 · Cited by 193 · cited 2×
“Text representation - describes the processes employed to convert text into suitable input for the algorithm (e.g., BoW [76], LDA topics [77], word embeddings [78]).”
On behalf of the founding members of the ICASR group, Beller, E., Clark, J., Tsafnat, G., Adams, C., Diehl, H., Lund, H., Ouzzani, M., Thayer, K., Thomas, J., et al. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0740-7 · Cited by 191 · cited 1×
“Finally, we recommend that the research community participates to scientific events and initiatives in this field, such as ICASR 11 [133–136], ALTAR 12 [137], and the MSLR Shared Task 13 [138].”
Reducing systematic review workload through certainty-based screening
Miwa, M., Thomas, J., O’Mara-Eves, A., & Ananiadou, S. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.06.005 · Cited by 190 · cited 1×
“It usually involves employing machine learning classifiers, which are trained on an initial set of user-selected papers and then used to identify additional relevant articles [36].”
Tools to support the automation of systematic reviews: a scoping review.
Khalil, H., Ameen, D., & Zarnegar, A. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.12.005 · Cited by 190 · cited 1×
“This survey aims to address the existing gap by rigorously examining the application of AI techniques in the semi-automation of SLRs, within the two main stages of application, namely screening and extraction .”
Chai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01635-3 · Cited by 190
ExaCT: automatic extraction of clinical trial characteristics from journal publications
Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-56 · Cited by 188 · cited 2×
“ExaCT applies a two-step approach [43].”
The anatomy of a nanopublication
Groth, P., Gibson, A., & Velterop, J. (2010) · Information Services and Use · doi:10.3233/isu-2010-0613 · Cited by 177
Artificial intelligence to automate the systematic review of scientific literature
de la Torre-López, J., Ramírez, A., & Romero, J. R. (2023) · Computing · doi:10.1007/s00607-023-01181-x · Cited by 170
Shemilt, I., Khan, N., Park, S., & Thomas, J. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0315-4 · Cited by 168 · cited 1×
“…by specific training [10]; significant financial implications arising from database subscriptions, specialised software utilisation, and personnel remunerations [11]; the growing rate of the number of publication [12], and the periodic need for updates to ensure continued relevance [13].”
O’Connor, A. M., Tsafnat, G., Thomas, J., Glasziou, P., Gilbert, S. B., & Hutton, B. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1062-0 · Cited by 167 · cited 1×
“…compatibility with other software systems [148, 149], iii) distrust , as there is uncertainty about the reliability and the mechanisms of these tools [150, 151], and iv) financial obstacles , predominantly arising from licensing expenses, along with feature restrictions in trial versions [152].”
Howard, B. E., Phillips, J., Tandon, A., Maharana, A., Elmore, R., Mav, D., Sedykh, A., Thayer, K., Merrick, B. A., Walker, V., et al. (2020) · Environment International · doi:10.1016/j.envint.2020.105623 · Cited by 157 · cited 1×
“…techniques to assist users in manually selecting relevant papers, typically by highlighting key terms or grouping similar papers (e.g., topic maps [65] based on LDA [77], clustering approaches [68]); • Post-screening support - which refers to the application of AI techniques to conduct a final…”
Using machine learning to advance synthesis and use of conservation and environmental evidence
Cheng, S. H., Augustin, C., Bethel, A., Gill, D., Anzaroot, S., Brun, J., DeWilde, B., Minnich, R. C., Garside, R., Masuda, Y. J., et al. (2018) · Conservation Biology · doi:10.1111/cobi.13117 · Cited by 155 · cited 2×
“Finally, Colandr, in addition to the standard identification of relevant papers, enables users to define their own set of categories (e.g., “water management”) and subsequently performs a multi-label classification of articles based on them [60].”
Automatic summarization of scientific articles: A survey
Altmami, N. I., & Menaï, M. E. B. (2020) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2020.04.020 · Cited by 151
Prioritising references for systematic reviews with RobotAnalyst: A user study
Przybyła, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M., McNaught, J., von Elm, E., Nolan, K., & Ananiadou, S. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1311 · Cited by 148 · cited 1×
“…relevant papers, typically by highlighting key terms or grouping similar papers (e.g., topic maps [65] based on LDA [77], clustering approaches [68]); • Post-screening support - which refers to the application of AI techniques to conduct a final review of the screened papers (e.g.,…”
Systematic literature review of validation methods for AI systems
Myllyaho, L., Raatikainen, M., Männistö, T., Mikkonen, T., & Nurminen, J. K. (2021) · Journal of Systems and Software · doi:10.1016/j.jss.2021.111050 · Cited by 145 · cited 1×
“These guidelines should offer clear and actionable recommendations for designing user-friendly interfaces and functionalities in future tools.”
Introducing Systematic Reviews
Gough, D., Oliver, S. R., & Thomas, J. (2017) · SAGE Publications Ltd eBooks · doi:10.4135/9781036234942.n1 · Cited by 125
Development of a search filter to identify reports of controlled clinical trials within CINAHL Plus
Glanville, J., Dooley, G., Wisniewski, S., Foxlee, R., & Noel‐Storr, A. (2019) · Health Information & Libraries Journal · doi:10.1111/hir.12251 · Cited by 121 · cited 1×
“Search strategies are typically implemented by creating a query based on a combination of terms using boolean operators [28, 29].”
Sysrev: A FAIR Platform for Data Curation and Systematic Evidence Review
Bozada, T., Borden, J., Workman, J., Del Cid, M., Malinowski, J., & Luechtefeld, T. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.685298 · Cited by 118
O’Connor, A. M., Tsafnat, G., Gilbert, S. B., Thayer, K. A., Shemilt, I., Thomas, J., Glasziou, P., & Wolfe, M. S. (2019) · Systematic Reviews · doi:10.1186/s13643-019-0975-y · Cited by 110
Moher, D., Tsertsvadze, A., Tricco, A. C., Eccles, M., Grimshaw, J., Sampson, M., & Barrowman, N. J. (2007) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2007.03.008 · Cited by 108 · cited 1×
“…by specific training [10]; significant financial implications arising from database subscriptions, specialised software utilisation, and personnel remunerations [11]; the growing rate of the number of publication [12], and the periodic need for updates to ensure continued relevance [13].”
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Present and future of semantic web technologies: a research statement
Patel, A., & Jain, S. (2021) · International Journal of Computers and Applications · doi:10.1080/1206212x.2019.1570666 · Cited by 105
Software Tools for Literature Screening in Systematic Reviews in Biomedical Research
van der Mierden, S. (2019) · ALTEX · doi:10.14573/altex.1902131 · Cited by 99
Automating Biomedical Evidence Synthesis: RobotReviewer
Marshall, I., Kuiper, J., Banner, E., & Wallace, B. C. (2017) · Proceedings of ACL 2017, System Demonstrations · doi:10.18653/v1/p17-4002 · Cited by 96
Scientific Fact-Checking: A Survey of Resources and Approaches
Vladika, J., & Matthes, F. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.387 · Cited by 96
Research, policy and practice: why developing countries are different
Young, J. T. (2005) · Journal of International Development · doi:10.1002/jid.1235 · Cited by 95 · cited 1×
“These guidelines should offer clear and actionable recommendations for designing user-friendly interfaces and functionalities in future tools.”
Identifying Barriers to the Systematic Literature Review Process
Carver, J. C., Hassler, E., Hernandes, E., & Kraft, N. A. (2013) · 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement · doi:10.1109/esem.2013.28 · Cited by 90 · cited 1×
“While it exists an extensive body of literature examining earlier iterations of SLR tools, it appears that these studies have largely overlooked the role of AI embedded within such tools [17–23].”
Investigating and Designing for Trust in AI-powered Code Generation Tools
Wang, R., Cheng, R., Ford, D., & Zimmermann, T. (2024) · The 2024 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3630106.3658984 · Cited by 87
Cierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01805-4 · Cited by 85
Finding better active learners for faster literature reviews
Yu, Z., Kraft, N. A., & Menzies, T. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9587-0 · Cited by 83 · cited 1×
“Existing literature includes various evaluations of SLR tools that focus on individual phases of the SLR process [21, 139, 140].”
Scott, A. M., Forbes, C., Clark, J., Carter, M., Glasziou, P., & Munn, Z. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.06.030 · Cited by 81 · cited 1×
“…issues, in addition to several other relevant factors: i) steep learning curve , as researchers may be unfamiliar with the tools’ functionalities [147], ii) misalignment with user requirements , as many of these software deviate from the guidelines set forth by SLR protocols and exhibit limited…”
FAST2: An intelligent assistant for finding relevant papers
Yu, Z., & Menzies, T. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.11.021 · Cited by 80
Robledo, S., Grisales Aguirre, A. M., Hughes, M., & Eggers, F. (2023) · Journal of Small Business Management · doi:10.1080/00472778.2021.1955125 · Cited by 80 · cited 6×
“Output - represents the outcome generated by the trained algorithm, and it has been analysed in three studies [20–22].”
InfluentialDeep learning in citation recommendation models survey
Ali, Z., Kefalas, P., Muhammad, K., Ali, B., & Imran, M. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.113790 · Cited by 79
Usage of automation tools in systematic reviews
van Altena, A. J., Spijker, R., & Olabarriaga, S. D. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1335 · Cited by 79 · cited 2×
“Recent studies [146], suggest that this limited usage primarily stems from usability issues, in addition to several other relevant factors: i) steep learning curve , as researchers may be unfamiliar with the tools’ functionalities [147], ii) misalignment with user requirements , as many of these…”
Towards Complete and Accurate Reporting of Studies of Diagnostic Accuracy: The STARD Initiative
Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., et al. (2003) · Clinical Chemistry and Laboratory Medicine · doi:10.1515/cclm.2003.012 · Cited by 78
Quality assessment of systematic reviews in software engineering: a tertiary study
Zhou, Y., Zhang, H., Huang, X., Yang, S., Babar, M. A., & Tang, H. (2015) · Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2745802.2745815 · Cited by 77
Investigating the Use of a Hybrid Search Strategy for Systematic Reviews
Mourao, E., Kalinowski, M., Murta, L., Mendes, E., & Wohlin, C. (2017) · 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) · doi:10.1109/esem.2017.30 · Cited by 77
Text-Mining Techniques and Tools for Systematic Literature Reviews: A Systematic Literature Review
Feng, L., Chiam, Y. K., & Lo, S. K. (2017) · 2017 24th Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2017.10 · Cited by 70 · cited 1×
“This survey aims to address the existing gap by rigorously examining the application of AI techniques in the semi-automation of SLRs, within the two main stages of application, namely screening and extraction .”
O’Connor, A. M., Tsafnat, G., Gilbert, S. B., Thayer, K. A., & Wolfe, M. S. (2018) · Systematic Reviews · doi:10.1186/s13643-017-0667-4 · Cited by 69 · cited 1×
“In summary, while advancements have been made in this field, there remains a significant need for further research, especially in the creation of robust benchmarks and evaluation frameworks that would allow the scientific community to objectively compare different tools and methodologies.”
Cowie, K., Rahmatullah, A., Hardy, N., Holub, K., & Kallmes, K. (2022) · JMIR Medical Informatics · doi:10.2196/33219 · Cited by 68 · cited 9×
“In the context of SLR, these methodologies are predominantly utilised to semi-automate tasks like screening and data extraction [17, 21].”
InfluentialA Systematic Review of Fairness, Accountability, Transparency, and Ethics in Information Retrieval
Bernard, N., & Balog, K. (2023) · ACM Computing Surveys · doi:10.1145/3637211 · Cited by 67
Structured information extraction from complex scientific text with fine-tuned large language models
Dunn, A., Dagdelen, J., Walker, N., Lee, S., Rosen, A. S., Ceder, G., Persson, K., & Jain, A. (2022) · arXiv · doi:10.48550/arxiv.2212.05238 · Cited by 67
Integrating Conversational Agents and Knowledge Graphs Within the Scholarly Domain
Meloni, A., Angioni, S., Salatino, A. A., Osborne, F., Recupero, D. R., & Motta, E. (2023) · IEEE Access · doi:10.1109/access.2023.3253388 · Cited by 66 · cited 1×
“A possible solution to these issues is the integration of LLMs with different types of knowledge bases that can provide verifiable factual information [108].”
Self-citationA Computational Inflection for Scientific Discovery
Hope, T., Downey, D., Weld, D. S., Etzioni, O., & Horvitz, E. (2023) · Communications of the ACM · doi:10.1145/3576896 · Cited by 65 · cited 1×
“These AI-enabled research assistants could support researchers by performing a variety of crucial tasks such as generating comprehensive literature reviews [168], identifying new scientific hypotheses [169], and fostering crucial innovation in research practices [170].”
LitSuggest: a web-based system for literature recommendation and curation using machine learning
Allot, A., Lee, K., Chen, Q., Luo, L., & Lu, Z. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab326 · Cited by 61
Noel-Storr, A., Dooley, G., Affengruber, L., & Gartlehner, G. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.09.024 · Cited by 61
CS-KG: A Large-Scale Knowledge Graph of Research Entities and Claims in Computer Science
Dessì, D., Osborne, F., Recupero, D. R., Buscaldi, D., & Motta, E. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19433-7_39 · Cited by 59
Self-citationTools to support systematic reviews in software engineering: a feature analysis
Marshall, C., Brereton, P., & Kitchenham, B. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601270 · Cited by 56
On the use of computer‐assistance to facilitate systematic mapping
Haddaway, N., Callaghan, M., Collins, A., Lamb, W. F., Minx, J. C., Thomas, J., & John, D. (2020) · Campbell Systematic Reviews · doi:10.1002/cl2.1129 · Cited by 55 · cited 1×
“…compatibility with other software systems [148, 149], iii) distrust , as there is uncertainty about the reliability and the mechanisms of these tools [150, 151], and iv) financial obstacles , predominantly arising from licensing expenses, along with feature restrictions in trial versions [152].”
Content and Complexity: information Design in Technical Communication
(2014) · Routledge eBooks · doi:10.4324/9781410607409 · Cited by 54
Tulk Jesso, S., Kelliher, A., Sanghavi, H., Martin, T., & Henrickson Parker, S. (2022) · Frontiers in Psychology · doi:10.3389/fpsyg.2022.830345 · Cited by 54
Hassler, E., Carver, J. C., Kraft, N. A., & Hale, D. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601274 · Cited by 53 · cited 1×
“This suggests that usability and accessibility should be prioritised in the design process to encourage wider adoption of these tools [153–155].”
Kebede, M. M., Le Cornet, C., & Fortner, R. T. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1589 · Cited by 50 · cited 1×
“Several of these tools adopted Artificial Intelligence (AI) solutions [14, 15] in order to support SLRs, in particular for the screening and data extraction phases.”
Dawes, M. G., Pluye, P., Shea, L., Grad, R. M., Greenberg, A., & Nie, J. (2007) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v15i1.640 · Cited by 49
Kung, J. (2023) · Journal of the Canadian Health Libraries Association / Journal de l'Association des bibliothèques de la santé du Canada · doi:10.29173/jchla29657 · Cited by 48
Citation prediction by leveraging transformers and natural language processing heuristics
Buscaldi, D., Dessì, D., Motta, E., Murgia, M., Osborne, F., & Recupero, D. R. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103583 · Cited by 48
Self-citationDillon, M. P., Fatone, S., & Quigley, M. (2016) · Systematic Reviews · doi:10.1186/s13643-015-0171-7 · Cited by 48
Garousi, V., & Felderer, M. (2017) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3084226.3084238 · Cited by 47 · cited 1×
“Following the extraction, the data is aggregated and summarised [40, 41].”
Marshall, C., Brereton, P., & Kitchenham, B. (2015) · Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2745802.2745827 · Cited by 46 · cited 1×
“Most researchers continue to depend on manual methods, often supported by software like Microsoft Excel, or reference management tools [145] such as Zotero 14 and Mendeley 15 .”
Schall, M. C., Cullen, L., Pennathur, P., Chen, H., Burrell, K., & Matthews, G. (2017) · CIN: Computers, Informatics, Nursing · doi:10.1097/cin.0000000000000325 · Cited by 46
Spök, A., Racovita, M., Karner, S., Adduci, G., Smets, G., Rüdelsheim, P., Kohl, C., Wilhelm, R., & Schiemann, J. (2018) · Environmental Evidence · doi:10.1186/s13750-017-0113-z · Cited by 45
Thomas, J. (2013) · OA Evidence-Based Medicine · doi:10.13172/2053-2636-1-2-1109 · Cited by 44 · cited 1×
“…, as many of these software deviate from the guidelines set forth by SLR protocols and exhibit limited compatibility with other software systems [148, 149], iii) distrust , as there is uncertainty about the reliability and the mechanisms of these tools [150, 151], and iv) financial obstacles ,…”
Systematic review toolbox: a catalogue of tools to support systematic reviews
Marshall, C., & Brereton, P. (2015) · Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2745802.2745824 · Cited by 44
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2023) · F1000Research · doi:10.12688/f1000research.51117.2 · Cited by 44
AIDA: A knowledge graph about research dynamics in academia and industry
Angioni, S., Salatino, A. A., Osborne, F., Recupero, D. R., & Motta, E. (2021) · Quantitative Science Studies · doi:10.1162/qss_a_00162 · Cited by 43 · cited 1×
“In summary, while advancements have been made in this field, there remains a significant need for further research, especially in the creation of robust benchmarks and evaluation frameworks that would allow the scientific community to objectively compare different tools and methodologies.”
Self-citationLiu, J., Timsina, P., & El-Gayar, O. (2016) · Information Systems Frontiers · doi:10.1007/s10796-016-9724-0 · Cited by 41 · cited 1×
“Existing literature includes various evaluations of SLR tools that focus on individual phases of the SLR process [21, 139, 140].”
The views of health guideline developers on the use of automation in health evidence synthesis
Arno, A., Elliott, J., Wallace, B., Turner, T., & Thomas, J. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01569-2 · Cited by 40 · cited 1×
“…as many of these software deviate from the guidelines set forth by SLR protocols and exhibit limited compatibility with other software systems [148, 149], iii) distrust , as there is uncertainty about the reliability and the mechanisms of these tools [150, 151], and iv) financial obstacles ,…”
Usability Metric for User Experience
Finstad, K. (2010) · PsycTESTS Dataset · doi:10.1037/t81952-000 · Cited by 40
Automatic classification of literature in systematic reviews on food safety using machine learning
van den Bulk, L. M., Bouzembrak, Y., Gavai, A., Liu, N., van den Heuvel, L. J., & Marvin, H. J. P. (2022) · Current Research in Food Science · doi:10.1016/j.crfs.2021.12.010 · Cited by 38
Identification of SLR tool needs – results of a community workshop
Hassler, E., Carver, J. C., Hale, D., & Al-Zubidy, A. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2015.10.011 · Cited by 35
A Survey on Text SimplificationPreprint
Sikka, P., & Mago, V. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.08612 · Cited by 35
Bio-SIEVE: Exploring Instruction Tuning Large Language Models for Systematic Review Automation
Robinson, A., Thorne, W., Wu, B. P., Pandor, A., Essat, M., Stevenson, M., & Song, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.06610 · Cited by 31
Salatino, A. A., Osborne, F., & Motta, E. (2021) · International Journal on Digital Libraries · doi:10.1007/s00799-021-00305-y · Cited by 30 · cited 1×
“A third promising research direction involves the use of semantic technologies [122], particularly knowledge graphs, to enhance the characterisation and classification of research papers [123].”
Self-citationImproving Editorial Workflow and Metadata Quality at Springer Nature
Salatino, A. A., Osborne, F., Birukou, A., & Motta, E. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-30796-7_31 · Cited by 30
Self-citationData extraction methods for systematic review (semi)automation: A living review protocol.
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2020) · F1000Research · doi:10.12688/f1000research.22781.2 · Cited by 30
Burgard, T., & Bittermann, A. (2023) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000509 · Cited by 29 · cited 7×
“Furthermore, most SLR tools are evaluated using small, custom datasets [21], which, although informative, may not accurately reflect real-world usage scenarios.”
InfluentialO’Connor, A. M., Glasziou, P., Taylor, M., Thomas, J., Spijker, R., & Wolfe, M. S. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01351-4 · Cited by 29 · cited 1×
“Finally, we recommend that the research community participates to scientific events and initiatives in this field, such as ICASR 11 [133–136], ALTAR 12 [137], and the MSLR Shared Task 13 [138].”
Completing Scientific Facts in Knowledge Graphs of Research Concepts
Borrego, A., Dessì, D., Hernández, I., Osborne, F., Recupero, D. R., Ruiz, D., Buscaldi, D., & Motta, E. (2022) · IEEE Access · doi:10.1109/access.2022.3220241 · Cited by 29
Self-citationWalker, V. R., Schmitt, C. P., Wolfe, M. S., Nowak, A. J., Kulesza, K., Williams, A. R., Shin, R., Cohen, J., Burch, D., Stout, M. D., et al. (2022) · Environment International · doi:10.1016/j.envint.2021.107025 · Cited by 28 · cited 4×
“Likewise, several tools continue to employ BoW methods for text representation, although some of the most recent ones [42, 45, 71] have shifted towards adopting word and sentence embedding techniques, such as GloVe [95], ELMo [103], SciBERT [90], and Sentence-BERT [92].”
InfluentialChatGPT-like AIs are coming to major science search engines
Van Noorden, R. (2023) · Nature · doi:10.1038/d41586-023-02470-3 · Cited by 28 · cited 1×
“For instance, Scopus and Dimensions 16 are working on their own chatbot engine and are planning to release them throughout the 2024 [160, 161].”
Pride, D., Cancellieri, M., & Knoth, P. (2023) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-43849-3_13 · Cited by 28
Vision for SLR tooling infrastructure: Prioritizing value-added requirements
Al-Zubidy, A., Carver, J. C., Hale, D. P., & Hassler, E. E. (2017) · Information and Software Technology · doi:10.1016/j.infsof.2017.06.007 · Cited by 27 · cited 1×
“This suggests that usability and accessibility should be prioritised in the design process to encourage wider adoption of these tools [153–155].”
AI science search engines are exploding in number — are they any good?
Sanderson, K. (2023) · Nature · doi:10.1038/d41586-023-01273-w · Cited by 27 · cited 1×
“This development is largely influenced by the advancements in Large Language Models [159].”
Hannousse, A. (2021) · IET Software · doi:10.1049/sfw2.12011 · Cited by 27 · cited 1×
“This search was carried out on Semantic Scholar 4 , selected for its extensive coverage in Computer Science , particularly for snowballing searches [58].”
Pride, D., Cancellieri, M., & Knoth, P. (2023) · International Conference on Theory and Practice of Digital Libraries · doi:10.48550/arxiv.2307.04683 · Cited by 27
AGATHA: Automatic Graph Mining And Transformer based Hypothesis Generation Approach
Sybrandt, J., Tyagin, I., Shtutman, M., & Safro, I. (2020) · Proceedings of the 29th ACM International Conference on Information & Knowledge Management · doi:10.1145/3340531.3412684 · Cited by 26
Brody, S. (2021) · Journal of the Medical Library Association · doi:10.5195/jmla.2021.1331 · Cited by 24
Kusa, W., Lipani, A., Knoth, P., & Hanbury, A. (2023) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2023.200193 · Cited by 21
Tool Features to Support Systematic Reviews in Software Engineering - A Cross Domain Study
Marshall, C., Kitchenham, B. A., & Brereton, P. (2018) · Keele Research Repository (Keele University) · doi:10.5277/e-inf180104 · Cited by 20
Semi-automated Tools for Systematic Searches
Adam, G. P., Wallace, B. C., & Trikalinos, T. A. (2022) · Methods in Molecular Biology · doi:10.1007/978-1-0716-1566-9_2 · Cited by 19 · cited 1×
“Traditionally, the search phase had not been significantly supported by artificial intelligence techniques [34].”
A checklist for choosing between R packages in ecology and evolution
Lortie, C. J., Braun, J., Filazzola, A., & Miguel, M. F. (2020) · Ecology and Evolution · doi:10.1002/ece3.5970 · Cited by 17 · cited 1×
“Historically, this stage did not benefit from the use of artificial intelligence techniques [53, 54].”
A Survey on Open Information Extraction from Rule-based Model to Large Language ModelPreprint
Liu, P., Gao, W., Dong, W., Ai, L., Gong, Z., Huang, S., Li, Z., Hoque, M. E., Hirschberg, J., & Zhang, Y. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2208.08690 · Cited by 16
Self-Tuned Descriptive Document Clustering Using a Predictive Network
Brockmeier, A. J., Mu, T., Ananiadou, S., & Goulermas, J. Y. (2018) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2017.2781721 · Cited by 15 · cited 1×
“It also incorporates a statistical selection process for identifying the key terms characterising each cluster [98].”
Wijkstra, M., Lek, T., Kuhn, T., Welbers, K., & Steijaert, M. (2021) · Proceedings of the 11th Knowledge Capture Conference · doi:10.1145/3460210.3493567 · Cited by 13 · cited 2×
“In recent years, we saw the emergence of several knowledge graphs that offer machine-readable, semantically rich, interlinked descriptions of the content of research publications [126–128].”
Chen, Z., He, L., Liu, S., Liu, H., Yu, J., & Li, Y. (2022) · Journal of Tissue Viability · doi:10.1016/j.jtv.2022.06.005 · Cited by 13 · cited 1×
“This analysis provides evidence of the overall strength and the level of trustworthiness presented in the review [50, 51].”
A commitment to act on data sharing
Abbasi, K. A. (2023) · BMJ · doi:10.1136/bmj.p1609 · Cited by 12
Technology for Research Synthesis: An Application of Sociotechnical Systems Theory
Dell, N. A., Maynard, B. R., Murphy, A. M., & Stewart, M. (2021) · Journal of the Society for Social Work and Research · doi:10.1086/713525 · Cited by 11 · cited 1×
“…compatibility with other software systems [148, 149], iii) distrust , as there is uncertainty about the reliability and the mechanisms of these tools [150, 151], and iv) financial obstacles , predominantly arising from licensing expenses, along with feature restrictions in trial versions [152].”
Bornmann, L., & Mutz, R. (2014) · arXiv · doi:10.48550/arxiv.1402.4578 · Cited by 11
Fontaine, G., Maheu‐Cadotte, M., Lavallée, A., Mailhot, T., Lavoie, P., Rouleau, G., Vinette, B., Ramirez-Garcìa, P., & Bourbonnais, A. (2022) · Worldviews on Evidence-Based Nursing · doi:10.1111/wvn.12609 · Cited by 8 · cited 1×
“A detailed protocol is also developed during this stage, outlining the appropriate methodologies that will be adopted to carry out the review [27].”
2. Research, Policy and Practice: Why Developing Countries are Different
Young, J. (2006) · Practical Action Publishing eBooks · doi:10.3362/9781780445632.002 · Cited by 8
Napoleão, B. M., Petrillo, F. D. S., & Hallé, S. (2021) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa53835.2021.00015 · Cited by 7 · cited 1×
“While it exists an extensive body of literature examining earlier iterations of SLR tools, it appears that these studies have largely overlooked the role of AI embedded within such tools [17–23].”
Nunzio, G. M. D., Kanoulas, E., & Majumder, P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99739-7_69 · Cited by 7
AGATHA: Automatic Graph-mining And Transformer based Hypothesis generation ApproachPreprint
Sybrandt, J., Tyagin, I., Shtutman, M. S., & Safro, I. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.05635 · Cited by 6
Automatic classification of sentences for evidence based medicine
Kim, S. N., Martínez, D., & Cavedon, L. (2010) · Data and Text Mining in Bioinformatics · doi:10.1145/1871871.1871876 · Cited by 6
scite: The next generation of citations
Rife, S. C., Rosati, D., & Nicholson, J. M. (2021) · Learned Publishing · doi:10.1002/leap.1379 · Cited by 5
Research works printed but not linked: 12
- Iansiti M, Lakhani KR (2020) Competing in the age of AI: strategy and leadership when algorithms and networks run the world. Harvard Business Press, Boston
- Kusa W, Knoth P, Hanbury A (2022) Evaluation of automated citation screening in systematic literature reviews with work saved over sampling: an analysis. In: 1st workshop on augmented intelligence for technology-assisted reviews systems: evaluation metrics and protocols for ediscovery and systematic review systems, pp 1–7
- Le Q, Mikolov T (2014) Distributed representations of sentences and documents. In: International Conference on Machine Learning. PMLR, pp 1188–1196
- Lewis P, Perez E, Piktus A, Petroni F, Karpukhin V, Goyal N, Küttler H, Lewis M, Yih W-T, Rocktäschel T et al (2020) Retrieval-augmented generation for knowledge-intensive NLP tasks. Adv Neural Inf Process Syst 33:9459–9474
- Li Q, Li J, Sheng J, Cui S, Wu J, Hei Y, Peng H, Guo S, Wang L, Beheshti A et al (2022) A survey on deep learning event extraction: approaches and applications. IEEE Trans Neural Netw Learn Syst
- Lipscomb CE (2000) Medical subject headings (mesh). Bull Med Libr Assoc 88(3):265
- Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. Adv Neural Inf Process Syst 26
- Minion JT, Egunsola O, Mastikhina L, Farkas B, Hofmeister M, Flanagan J, Salmon C, Clement F (2021) Pico portal. J Can Health Libraries Assoc 42(3):181
- Ng A, Jordan M, Weiss Y (2001) On spectral clustering: analysis and an algorithm. Adv Neural Inf Process Syst 14
- National Academies of Sciences, Engineering, and Medicine; Policy and Global Affairs; Board on Research Data and Information; Committee on Toward an Open Science Enterprise (2018) Open science by design: realizing a vision for 21st century research. National Academies Press, Washington
- Quesenbery W (2014) The five dimensions of usability. Content and complexity. Routledge, New York, pp 93–114
- Wang LL, DeYoung J, Wallace B (2022) Overview of mslr2022: a shared task on multi-document summarization for literature reviews. In: Proceedings of the third workshop on scholarly document processing
Cited by
Resnik, D. B., & Hosseini, M. (2025) · AI and Ethics · doi:10.1007/s43681-024-00493-8 · Cited by 228
“• Searching and reviewing the scientific literature [30, 72].”
Bandi, A., Kongari, B., Naguru, R., Pasnoor, S., & Vilipala, S. V. (2025) · Future Internet · doi:10.3390/fi17090404 · Cited by 207
Suchikova, Y., Tsybuliak, N., Teixeira da Silva, J. A., & Nazarovets, S. (2026) · Accountability in Research · doi:10.1080/08989621.2025.2544331 · Cited by 70
Automating Research Synthesis with Domain-Specific Large Language Model Fine-Tuning
Sušnjak, T., Hwang, P., Reyes, N. H., Barczak, A. L. C., McIntosh, T. R., & Ranathunga, S. (2025) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3715964 · Cited by 143
“In their most recent review, Bolanos et al. [28] found that LLM usage for SLRs is hampered by limitations such as reduced efficacy in domain-specific and narrower subject areas, their propensity for hallucinating and generating misleading information, alongside their opaque decision processes which…”
BackgroundEconomic impacts of AI-augmented R&D
Besiroglu, T., Emery-Xu, N., & Thompson, N. (2024) · Research Policy · doi:10.1016/j.respol.2024.105037 · Cited by 118
Metadata
- Authors
- Bolaños, F., Salatino, A., Osborne, F., & Motta, E.
- Affiliations
- The Open University (GB)
- Corresponding author
- Bolaños, Francisco
- Venue
- Artificial Intelligence Review
- Published
- 2024-08-17
- Volume
- 57
- Issue
- 10
- ISSN
- 0269-2821 (ISSN-L)
- 1573-7462
- OpenAlex
- W4401667275
- Cited by
- 246
- References
- 170
- Citations by year
- 2026: 124 · 2025: 103 · 2024: 19
- FWCI
- 69.13
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Artificial Intelligence Review (publishedVersion, cc-by)
- Open Research Online (The Open University) (acceptedVersion, cc-by)
- BOA (University of Milano-Bicocca) (submittedVersion, cc-by)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- artificial intelligence, systematic reviews, large language models, literature screening, data extraction, knowledge graphs, academic writing support, usability, transparency
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-28, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-28 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.