Using artificial intelligence for systematic review: the example of elicit
Open accessAbstract
BACKGROUND: Artificial intelligence (AI) tools are increasingly being used to assist researchers with various research tasks, particularly in the systematic review process. Elicit is one such tool that can generate a summary of the question asked, setting it apart from other AI tools. The aim of this study is to determine whether AI-assisted research using Elicit adds value to the systematic review process compared to traditional screening methods. METHODS: We compare the results from an umbrella review conducted independently of AI with the results of the AI-based searching using the same criteria. Elicit contribution was assessed based on three criteria: repeatability, reliability and accuracy. For repeatability the search process was repeated three times on Elicit (trial 1, trial 2, trial 3). For accuracy, articles obtained with Elicit were reviewed using the same inclusion criteria as the umbrella review. Reliability was assessed by comparing the number of publications with those without AI-based searches. RESULTS: The repeatability test found 246,169 results and 172 results for the trials 1, 2, and 3 respectively. Concerning accuracy, 6 articles were included at the conclusion of the selection process. Regarding, revealed 3 common articles, 3 exclusively identified by Elicit and 17 exclusively identified by the AI-independent umbrella review search. CONCLUSION: Our findings suggest that AI research assistants, like Elicit, can serve as valuable complementary tools for researchers when designing or writing systematic reviews. However, AI tools have several limitations and should be used with caution. When using AI tools, certain principles must be followed to maintain methodological rigour and integrity. Improving the performance of AI tools such as Elicit and contributing to the development of guidelines for their use during the systematic review process will enhance their effectiveness.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Bernard et al. found that Elicit's sensitivity was poor, identifying only 3 out of 17 studies (17.6%) included in an umbrella review.
“However, just like our study, Bernard et al. found that the sensitivity of Elicit is poor. As it was only able to find 3 out of 17 studies (17.6%) included in the umbrella review.”
From Comparison of Elicit AI and Traditional Literature Searching in Evidence Syntheses Using Four Case Studies (2025) · cites this paper as Bernard et al. [109]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Are systematic reviews up-to-date at the time of publication?
Beller, E. M., Chen, J. K.-H., Wang, U. L.-H., & Glasziou, P. P. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-36 · Cited by 196 · see citing works
PRISMA AI reporting guidelines for systematic reviews and meta-analyses on AI in healthcare
Cacciamani, G. E., Chu, T. N., Sanford, D. I., Abreu, A., Duddalwar, V., Oberai, A., Kuo, C.-C. J., Liu, X., Denniston, A. K., Vasey, B., et al. (2023) · Nature Medicine · doi:10.1038/s41591-022-02139-w · Cited by 143 · cited 1× · see citing works
“Finally, the implementation of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) AI reporting guidelines which are currently underway, will be very useful for providing a framework for the use of AI [22].”
Flagged first, then by citations
Research works printed but not linked: 18
- [1]Kebede MM, Le Cornet C, Fortner RT. In-depth evaluation of machine learning methods for semi-automating Article screening in a systematic review of mechanistic literature. Res Synth Methods Mars. 2023;14(2):156–72. 35798691 · DOI · Paper page
- [2]Blaizot A, Veettil SK, Saidoung P, Moreno-Garcia CF, Wiratunga N, Aceves-Martins M, et al. Using artificial intelligence methods for systematic review in health sciences: A systematic review. Res Synth Methods Mai. 2022;13(3):353–62. 35174972 · DOI · Paper page
- [3]Zhang Y, Liang S, Feng Y, Wang Q, Sun F, Chen S, et al. Automation of literature screening using machine learning in medical evidence synthesis: a diagnostic test accuracy systematic review protocol. Syst Rev 15 Janv. 2022;11(1):11. PMC8760775 35031074 · DOI
- [5]Tannou T, Lihoreau T, Couture M, Giroux S, Wang RH, Spalla G, et al. Is research on « smart living environments » based on unobtrusive technologies for older adults going in circles? Evidence from an umbrella review. Ageing Res Rev Févr. 2023;84:101830. 36565962 · DOI
- [6]Kitchenham B, Brereton P, Li Z, Budgen D, Burn A. Repeatability of systematic literature reviews. In: 15th Annual Conference on Evaluation & Assessment in Software Engineering (EASE 2011). 2011 [cité 21 déc 2023]. pp. 46–55. Disponible sur: https://ieeexplore.ieee.org/abstract/document/6083161
- [7]Khosravi P, Ghapanchi AH. Investigating the effectiveness of technologies applied to assist seniors: A systematic literature review. Int J Med Inf Janv. 2016;85(1):17–26. 26216463 · DOI
- [8]Lussier M, Lavoie M, Giroux S, Consel C, Guay M, Macoir J, et al. Early detection of mild cognitive impairment with In-Home monitoring sensor technologies using functional measures: A systematic review. IEEE J Biomed Health Inf Mars. 2019;23(2):838–47. 29994013 · DOI
- [9]Reeder B, Meyer E, Lazar A, Chaudhuri S, Thompson HJ, Demiris G. Framing the evidence for health smart homes and home-based consumer health technologies as a public health intervention for independent aging: a systematic review. Int J Med Inf Juill. 2013;82(7):565–79. PMC3740158 23639263 · DOI
- [10]Lenouvel E, Novak L, Nef T, Klöppel S. Advances in sensor monitoring effectiveness and applicability: A systematic review and update. Gerontologist 15 Mai. 2020;60(4):e299–308. 31102436
- [11]Richardson MX, Ehn M, Stridsberg SL, Redekop K, Wamala-Andersson S. Nocturnal digital surveillance in aged populations and its effects on health, welfare and social care provision: a systematic review. BMC Health Serv Res 30 Juin. 2021;21(1):622. PMC8241531 34187472 · DOI
- [12]Morris M, Adair B, Miller K, Ozanne E, Hampson R, Pearce A, et al. Smart-Home technologies to assist older people to live well at home. J Aging Sci 1 Janv. 2013;1:101.
- [13]Aromataris E, Pearson A. The systematic review: an overview. AJN Am J Nurs Mars. 2014;114(3):53. 24572533 · DOI · Paper page
- [14]Gusenbauer M. Google scholar to overshadow them all? Comparing the sizes of 12 academic search engines and bibliographic databases. Scientometrics 1 Janv. 2019;118(1):177–214. · DOI · Paper page
- [16]Wohlin C, Runeson P, da Mota Silveira Neto PA, Engström E, do, Carmo Machado I, de Almeida ES. oct. On the reliability of mapping studies in software engineering. J Syst Softw. 1 2013;86(10):2594–610.
- [17]MacDonell S, Shepperd M, Kitchenham B, Mendes E. How reliable are systematic reviews in empirical software engineering?? IEEE Trans Softw Eng Sept. 2010;36(5):676–87. · DOI · Paper page
- [18]Fabiano N, Gupta A, Bhambra N, Luu B, Wong S, Maaz M, et al. How to optimize the systematic review process using AI tools. JCPP Adv 23 Avr. 2024;4(2):e12234. PMC11143948 38827982 · DOI · Paper page
- [19]van Dijk SHB, Brusse-Keizer MGJ, Bucsán CC, van der Palen J, Doggen CJM, Lenferink A. Artificial intelligence in systematic reviews: promising when appropriately used. BMJ Open 7 Juill. 2023;13(7):e072254. PMC10335470 37419641 · DOI · Paper page
- [21]Beller E, Clark J, Tsafnat G, Adams C, Diehl H, Lund H et al. Making progress with the automation of systematic reviews: principles of the International Collaboration for the Automation of Systematic Reviews (ICASR). Syst Rev. 19 mai. 2018;7(1):77. PMC5960503 29778096
Cited by
Sorted by citations
Lau, O., & Golder, S. (2025) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70050 · Cited by 11 · see citing works
- Authors
- Bernard, N., Sagawa Jr, Y., Bier, N., Lihoreau, T., Pazart, L., & Tannou, T.
- Affiliations
- Inserm (FR)
- Institut Universitaire de Gériatrie de Montréal (CA)
- Corresponding author
- Bernard, Nathan
- Venue
- BMC Medical Research Methodology
- Published
- 2025-03-18
- Volume
- 25
- Issue
- 1
- Pages
- 75-75
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC11921719
- OpenAlex
- W4408546444
- Cited by
- 93
- References
- 2
- Citations by year
- 2026: 61 · 2025: 31 · 2023: 1
- FWCI
- 17.57
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Keywords
- artificial intelligence, Elicit, umbrella review, literature searching, AI research assistant, search accuracy, repeatability, reliability statistics, research integrity
- MeSH terms
- Systematic Reviews as Topic, Artificial Intelligence, Humans, Research Design, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
