A full systematic review was completed in 2 weeks using automation tools: a case study
Open accessAbstract
Background and objectives Systematic reviews (SRs) are time and resource intensive, requiring approximately 1 year from protocol registration to submission for publication. Our aim was to describe the process, facilitators, and barriers to completing the first 2-week full SR.Study design and setting We systematically reviewed evidence of the impact of increased fluid intake, on urinary tract infection (UTI) recurrence, in individuals at risk for UTIs. The review was conducted by experienced systematic reviewers with complementary skills (two researcher clinicians, an information specialist, and an epidemiologist), using Systematic Review Automation tools, and blocked off time for the duration of the project. The outcomes were time to complete the SR, time to complete individual SR tasks, facilitators and barriers to progress, and peer reviewer feedback on the SR manuscript. Times to completion were analyzed quantitatively (minutes and calendar days); facilitators and barriers were mapped onto the Theoretical Domains Framework; and peer reviewer feedback was analyzed quantitatively and narratively.Results The SR was completed in 61 person-hours (9 workdays; 12 calendar days); accepted version of the manuscript required 71 person-hours. Individual SR tasks ranged from 16 person-minutes (deduplication of search results) to 461 person-minutes (data extraction). The least time-consuming SR tasks were obtaining full-texts, searches, citation analysis, data synthesis, and deduplication. The most time-consuming tasks were data extraction, write-up, abstract screening, full-text screening, and risk of bias. Facilitators and barriers mapped onto the following domains: knowledge; skills; memory, attention, and decision process; environmental context and resources; and technology and infrastructure. Two sets of peer reviewer feedback were received on the manuscript: the first included 34 comments requesting changes, 17 changes were made, requiring 173 person-minutes; the second requested 13 changes, and eight were made, requiring 121 person-minutes.Conclusion A small and experienced systematic reviewer team using Systematic Review Automation tools who have protected time to focus solely on the SR can complete a moderately sized SR in 2 weeks.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous AI tools were incapable of reviewing full-texts.
“the incapability to either review full-texts (Clark et al., 2020; Nye et al., 2018)”
From Can large language models replace humans in systematic reviews? Evaluating <scp>GPT</scp> ‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languages (2024) · cites this paper as Clark et al., 2020
The ‘Modifications to standard SR processes’ table (page unavailable) lists SRA Helper as a tool for screening full-text articles. The Discussion says most tools cannot completely replace a user, but that is different from saying they are incapable of assisting with full-text review. (LLM, conf=0.82)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredForward and/or backward citation quality checks can be automated through tools such as SR-Accelerator's Spidercite and Citation Chaser.
“This can be automated, for example, through the use of SR-Accelerator’s Spidercite [14], Citation Chaser [19].”
From The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses (2024) · cites this paper as [14]
The 'Facilitators and Barriers' section proposes a citation collator that would automatically identify cited and citing articles, supporting the general possibility of automating forward and backward citation searches. But the paper presents this as a tool for future development; its 'Modifications to standard SR processes' table lists no tool for citation analysis. Our copy does not identify Spidercite or Citation Chaser as tools for this task. (LLM, conf=0.82)
Experimental — accuracy not yet measuredSystematic reviews and meta-analyses require a median of five researchers and around 40 weeks of work to reach submission.
“requiring a median of five researchers and around 40 weeks of work to reach submission [ 3 , 4 ]”
From Reducing systematic review burden using Deduklick: a novel, automated, reliable, and explainable deduplication algorithm to foster medical research (2022) · cites this paper as [3]
Machine learning tools assist the reviewer by suggesting, classifying, or selecting records, and can either help or even replace the reviewer during certain parts of the process.
“these tools assist the reviewer by suggesting, classifying, or selecting records, and can either help or even replace the reviewer during certain parts of the process [28, 52, 71]”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [28]
Projects such as SR-Accelerator integrate several tools in a suite to assist in more than one step of the systematic review procedure.
“Other projects as SR-Accelerator [28], integrate several tools in a suite to assist in more than one step of the SR procedure”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [28]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823 · see citing works
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859 · see citing works
Making psychological theory useful for implementing evidence based practice: a consensus approach
Michie, S. (2005) · Quality and Safety in Health Care · doi:10.1136/qshc.2004.011155 · Cited by 3,440 · see citing works
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737 · see citing works
Systematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472 · see citing works
Self-citationMarshall, I. J., Noel‐Storr, A., Kuiper, J., Thomas, J., & Wallace, B. C. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1287 · Cited by 435 · see citing works
RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv044 · Cited by 317 · see citing works
Rathbone, J., Carter, M., Hoffmann, T., & Glasziou, P. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-6 · Cited by 206 · see citing works
Self-citationOn behalf of the founding members of the ICASR group, Beller, E., Clark, J., Tsafnat, G., Adams, C., Diehl, H., Lund, H., Ouzzani, M., Thayer, K., Thomas, J., et al. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0740-7 · Cited by 191 · see citing works
Self-citationUsage of automation tools in systematic reviews
van Altena, A. J., Spijker, R., & Olabarriaga, S. D. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1335 · Cited by 79 · see citing works
O’Connor, A. M., Tsafnat, G., Gilbert, S. B., Thayer, K. A., & Wolfe, M. S. (2018) · Systematic Reviews · doi:10.1186/s13643-017-0667-4 · Cited by 69 · see citing works
Increased fluid intake to prevent urinary tract infections: systematic review and meta-analysis
Scott, A. M., Clark, J. M., Mar, C. D., & Glasziou, P. (2020) · British Journal of General Practice · doi:10.3399/bjgp20x708125 · Cited by 49 · see citing works
Self-citationImproving the conduct of systematic reviews: a process mining perspective
Pham, B., Ebrahim Bagheri, A., Rios, P. D. L., Pourmasoumi, A., Robson, R. C., Hwee, J., Isaranuwatchai, W., Darvesh, N., Page, M. J., & Tricco, A. C. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.06.011 · Cited by 40 · see citing works
Flagged first, then by citations
Research works printed but not linked: 2
- [6]6. Higgins JPT, Green S, editors. Cochrane handbook for systematic reviews of interventions Chichester, West Sussex ; Hoboken NJ: Wiley-Blackwell; 2011.
- [14]14. Clark J, Sanders S, Carter M, Honeyman D, Cleo G, Auld Y, et al. (Under review) Improving the translation of search strategies using the Polyglot Search Translator: a randomised controlled trial. Journal of the American Library Association. 2019.
Cited by
Sorted by citations
Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 11,883 · see citing works
Automation of systematic literature reviews: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106589 · Cited by 432 · see citing works
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2023) · arXiv · doi:10.48550/arxiv.2310.17526 · Cited by 268 · see citing works
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1715 · Cited by 269 · see citing works
Tools to support the automation of systematic reviews: a scoping review.
Khalil, H., Ameen, D., & Zarnegar, A. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.12.005 · Cited by 190 · see citing works
Jukic, I., Castilla, A. P., Ramos, A. G., Van Hooren, B., McGuigan, M. R., & Helms, E. R. (2023) · Sports Medicine · doi:10.1007/s40279-022-01754-4 · Cited by 138 · see citing works
Boetje, J., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02502-7 · Cited by 146 · see citing works
“This can be automated, for example, through the use of SR-Accelerator’s Spi-dercite [14], Citation Chaser [19].”
MethodHow to optimize the systematic review process using AI tools
Fabiano, N., Gupta, A., Bhambra, N., Luu, B., Wong, S., Maaz, M., Fiedorowicz, J. G., Smith, A. L., & Solmi, M. (2024) · JCPP Advances · doi:10.1002/jcv2.12234 · Cited by 130 · see citing works
Borissov, N., Haas, Q., Minder, B., Kopp-Heim, D., von Gernler, M., Janka, H., Teodoro, D., & Amini, P. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02045-9 · Cited by 116 · see citing works
“Such analyses are time and resource intensive, requiring a median of five researchers and around 40 weeks of work to reach submission [3, 4].”
BackgroundMethodKaragiannis, T., Malandris, K., Avgerinos, I., Stamati, A., Kakotrichi, P., Liakos, A., Vasilakou, D., Kakaletsis, N., Tsapas, A., & Bekiari, E. (2024) · Diabetologia · doi:10.1007/s00125-024-06144-1 · Cited by 102 · see citing works
Resource use during systematic review production varies widely: a scoping review
Nussbaumer-Streit, B., Ellen, M., Klerings, I., Sfetcu, R., Riva, N., Mahmić-Kaknjo, M., Poulentzas, G., Martinez, P., Baladia, E., Ziganshina, L. E., et al. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.05.019 · Cited by 98 · see citing works
Cierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01805-4 · Cited by 85 · see citing works
Scott, A. M., Forbes, C., Clark, J., Carter, M., Glasziou, P., & Munn, Z. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.06.030 · Cited by 81 · see citing works
Self-citationYates, N., Gough, S., & Brazil, V. (2022) · Medical Teacher · doi:10.1080/0142159x.2022.2093704 · Cited by 67 · see citing works
Hamel, C., Michaud, A., Thuku, M., Affengruber, L., Skidmore, B., Nussbaumer-Streit, B., Stevens, A., & Garritty, C. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.06.027 · Cited by 36 · see citing works
Xu, C., Fan, S., Tian, Y., Liu, F., Furuya-Kanamori, L., Clark, J., Zhang, C., Li, S., Lin, L., Chu, H., et al. (2025) · BMJ · doi:10.1136/bmj-2024-082068 · Cited by 39 · see citing works
Self-citationDecoding semi-automated title-abstract screening: findings from a convenience sample of reviews
Gates, A., Gates, M., DaRosa, D., Elliott, S. A., Pillay, J., Rahman, S., Vandermeer, B., & Hartling, L. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01528-x · Cited by 31 · see citing works
Walker, V. R., Schmitt, C. P., Wolfe, M. S., Nowak, A. J., Kulesza, K., Williams, A. R., Shin, R., Cohen, J., Burch, D., Stout, M. D., et al. (2022) · Environment International · doi:10.1016/j.envint.2021.107025 · Cited by 28 · see citing works
Codepori: Large-Scale System for Autonomous Software Development Using Multi-Agent Technology
Rasheed, Z., Malik, S., Kemell, K.-K., Muhammad, W., Saari, M., Systä, K., & Abrahamsson, P. (2024) · doi:10.2139/ssrn.4979510 · Cited by 27 · see citing works
Human Performance in Deepfake Detection: A Systematic Review
Somoray, K., Miller, D. J., & Holmes, M. (2025) · Human Behavior and Emerging Technologies · doi:10.1155/hbe2/1833228 · Cited by 19 · see citing works
Role of Nasal Surgery in Adult Obstructive Sleep Apnea: A Systematic Review
Correa, E. J., Conti, D. M., Moreno-Luna, R., Sánchez-Gómez, S., & O'Connor Reina, C. (2024) · Sleep Science · doi:10.1055/s-0044-1782527 · Cited by 15 · see citing works
The Impact of Query Refinement on Systematic Review Literature Search: A Query Log Analysis
Scells, H., Forbes, C., Clark, J., Koopman, B., & Zuccon, G. (2022) · Proceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval · doi:10.1145/3539813.3545143 · Cited by 12 · see citing works
Self-citationWeighall, A., & Kellar, I. (2023) · Emerging Topics in Life Sciences · doi:10.1042/etls20230110 · Cited by 13 · see citing works
Do it faster with PICOS: Generative AI-Assisted systematic review screening
Vallamchetla, S. K., Abdelkader, O., Elnaggar, A., Ramadan, D., Islam Shourav, M. M., Riaz, I. B., & Lin, M. P. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104860 · Cited by 13 · see citing works
Novel AI applications in systematic review: gPT-4 assisted data extraction, analysis, review of bias
Kim, J. K., Chua, M. E., Li, T. G., Rickard, M., & Lorenzo, A. J. (2025) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2024-113066 · Cited by 11 · see citing works
Robleto, E., & Shehadeh, L. A. (2025) · American Journal of Physiology. Heart and Circulatory Physiology · doi:10.1152/ajpheart.00374.2025 · Cited by 2 · see citing works
- Authors
- Clark, J., Glasziou, P., Del Mar, C., Bannach-Brown, A., Stehlik, P., & Scott, A. M.
- Affiliations
- Bond University (AU)
- Corresponding author
- Clark, Justin
- Venue
- Journal of Clinical Epidemiology
- Publisher
- Elsevier BV
- Published
- 2020-05
- Volume
- 121
- Pages
- 81-90
- ISSN
- 0895-4356 (ISSN-L)
- OpenAlex
- W3003735349
- Cited by
- 687
- References
- 13
- Citations by year
- 2026: 110 · 2025: 166 · 2024: 160 · 2023: 116 · 2022: 50 · 2021: 52 · 2020: 32
- FWCI
- 12.87
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of Clinical Epidemiology (publishedVersion)
- PubMed (publishedVersion)
- Bond University Research Portal (Bond University) (submittedVersion, cc-by-nc-nd)
- Griffith Research Online (submittedVersion, cc-by-nc-nd)
- Funders
- National Health and Medical Research Council
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- fluid intake, recurrent urinary tract infection, Theoretical Domains Framework, data extraction, risk of bias assessment
- MeSH terms
- Abstracting and Indexing, Data Collection, Fluid Therapy, Humans, Publishing, Recurrence, Time Factors, Urinary Tract Infections, Writing, Randomized Controlled Trials as Topic, Peer Review, Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
