CoChat
Sign Up
Cited by 43Meta-analysis and systematic reviewsscientometrics and bibliometrics researchHealth Sciences Research and Education

Automation tools to support undertaking scoping reviews

Open access
Khalil, Hanan, Pollock, Danielle, McInerney, Patricia, Evans, Catrin, Moraes, Erica B., Godfrey, Christina M., Alexander, Lyndsay, Tricco, Andrea +6 more·2024·Research Synthesis Methods
DOI 10.1002/jrsm.1731
Open this paper in CoChat — free accountOpen PDF

Abstract

Objective: This paper describes several automation tools and software that can be considered during evidence synthesis projects and provides guidance for their integration in the conduct of scoping reviews. Study Design and Setting: The guidance presented in this work is adapted from the results of a scoping review and consultations with the JBI Scoping Review Methodology group. Results: This paper describes several reliable, validated automation tools and software that can be used to enhance the conduct of scoping reviews. Developments in the automation of systematic reviews, and more recently scoping reviews, are continuously evolving. We detail several helpful tools in order of the key steps recommended by the JBI's methodological guidance for undertaking scoping reviews including team establishment, protocol development, searching, de‐duplication, screening titles and abstracts, data extraction, data charting, and report writing. While we include several reliable tools and software that can be used for the automation of scoping reviews, there are some limitations to the tools mentioned. For example, some are available in English only and their lack of integration with other tools results in limited interoperability. Conclusion: This paper highlighted several useful automation tools and software programs to use in undertaking each step of a scoping review. This guidance has the potential to inform collaborative efforts aiming at the development of evidence informed, integrated automation tools and software packages for enhancing the conduct of high‐quality scoping reviews.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1002/jrsm.1731

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Grey literature was searched.

    “Grey literature was also searched as it has been found to be increasingly recognised as valuable sources of literature when conducting scoping reviews in recent years (27).”

    From How is dysfunctional breathing characterised in the clinical literature? A scoping review (2026) · cites this paper as [27]

    Section 4.3 (Step 3: Searching; page unavailable) recommends searching grey literature sources when conducting scoping reviews, supporting the citing sentence’s rationale for including them. It does not say that grey literature has become increasingly recognised as valuable in recent years. The statement that grey literature was searched describes the citing authors’ own method, not a finding attributed to this source. (LLM, conf=0.84)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 70 · Cited by 43

Research works printed but not linked: 48

  1. Müller SM, Mueller GF, Navarini AA, Brandt O. National pub- lication productivity during the COVID-19 pandemic-a pre- liminary exploratory analysis of the 30 countries most affected. Biology. 2020;9(9):271.
  2. Aromataris E, Stern C, Lockwood C, et al. JBI series paper 2: tailored evidence synthesis approaches are required to answer diverse questions: a pragmatic evidence synthesis toolkit from JBI. J Clin Epidemiol. 2022;150:196-202.
  3. Khalil H, Tamara L, Rada G, Akl EA. Challenges of evidence synthesis during the 2020 COVID pandemic: a scoping review. J Clin Epidemiol. 2022;142:10-18.
  4. Khalil H, Shahid M, Roughead L. Medication safety programs in primary care: a scoping review. JBI Evid Synth. 2017;15(10): 2512-2526.
  5. Khalil H, Downie A, Ristevski E. Mapping palliative and end of care research in Australia (2000-2018). Palliat Support Care. 2020;18(6):713-721.
  6. Wallace BC, Small K, Brodley CE, Lau J, Trikalinos TA. Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. Proceedings of the 2nd ACM SIGHIT international health informatics symposium. 2012.
  7. O'Connor AM, Tsafnat G, Gilbert SB, Thayer KA, Wolfe MS. Moving toward the automation of the systematic review pro- cess: a summary of discussions at the second meeting of inter- national collaboration for the automation of systematic reviews (ICASR). Syst Rev. 2018;7(1):1-5.
  8. Munn Z, Brandt L, Kuijpers T, Whittington C, Wiles L, Karge T. Are systematic review and guideline development tools useful? A guidelines international network survey of user preferences. JBI Evid Implement. 2020;18(3):345-352.
  9. Van Altena A, Spijker R, Olabarriaga S. Usage of automation tools in systematic reviews. Res Synth Methods. 2019;10(1): 72-82.
  10. Aguinis H, Ramani RS, Alabduljader N. Best-practice recom- mendations for producers, evaluators, and users of methodo- logical literature reviews. Organ Res Methods. 2023;26(1):46-76.
  11. Gates A, Gates M, Sebastianski M, Guitard S, Elliott SA, Hartling L. The semi-automation of title and abstract screen- ing: a retrospective exploration of ways to leverage Abstrackr's relevance predictions in systematic and rapid reviews. BMC Med Res Methodol. 2020;20(1):139.
  12. Munn Z, Pollock D, Khalil H, et al. What are scoping reviews? Providing a formal definition of scoping reviews as a type of evidence synthesis. JBI evidence. Synthesis. 2022;20:950-952.
  13. Jordan Z, Lockwood C, Aromataris E, et al. JBI series paper 1: introducing JBI and the JBI model of EHBC. J Clin Epidemiol. 2022;150:191-195.
  14. Nyanchoka L, Tudur-Smith C, Thu VN, Iversen V, Tricco AC, Porcher R. A scoping review describes methods used to iden- tify, prioritize and display gaps in health research. J Clin Epide- miol. 2019;109:99-110.
  15. Kellermeyer L, Harnke B, Knight S. Covidence and rayyan. J Med Library Assoc. 2018;106(4):580.
  16. Munn Z, Aromataris E, Tufanaru C, et al. The development of software to support multiple systematic review types: the Joanna Briggs institute system for the unified management, assessment and review of information (JBI SUMARI). Int J Evid Based Healthc. 2019;17(1):36-43.
  17. Mortensen ML, Adam GP, Trikalinos TA, Kraska T, Wallace BC. An exploration of crowdsourcing citation screening for systematic reviews. Res Synth Methods. 2017;8(3):366-386.
  18. Nama N, Sampson M, Barrowman N, et al. Crowdsourcing the citation screening process for systematic reviews: validation study. J Med Internet Res. 2019;21(4):e12953.
  19. Nama N, Iliriani K, Xia MY, et al. A pilot validation study of crowdsourcing systematic reviews: update of a searchable data- base of pediatric clinical trials of high-dose vitamin D. Transl Pediatr. 2017;6(1):18-26.
  20. Johnson HA. Trello. J Med Library Assoc. 2017;105(2):209.
  21. Peters MD, Godfrey C, McInerney P, et al. Best practice guid- ance and reporting items for the development of scoping review protocols. JBI Evid Synth. 2022;20(4):953-968.
  22. Clark J, Glasziou P, Del Mar C, Bannach-Brown A, Stehlik P, Scott AM. A full systematic review was completed in 2 weeks using automation tools: a case study. J Clin Epidemiol. 2020; 121:81-90.
  23. Clark JM, Sanders S, Carter M, et al. Improving the translation of search strategies using the polyglot search translator: a ran- domized controlled trial. J Med Libr Assoc. 2020;108(2):195-207.
  24. Choong MK, Galgani F, Dunn AG, Tsafnat G. Automatic evi- dence retrieval for systematic reviews. J Med Internet Res. 2014; 16(10):e223.
  25. Beller E, Clark J, Tsafnat G, et al. Making progress with the automation of systematic reviews: principles of the international collaboration for the automation of system- atic reviews (ICASR). Syst Rev. 2018;7(1):1-7.
  26. Balk EM, Chung M, Chen ML, Chang LKW, Trikalinos TA. Data extraction from machine-translated versus original lan- guage randomized trial reports: a comparative study. Syst Rev. 2013;2(1):1-6.
  27. Yulianto A, Supriatnaningsih R. Google translate vs. DeepL: a quantitative evaluation of close-language pair translation (French to English). Asian J Engl Lang Pedagogy. 2021;9(2): 109-127.
  28. Idri N. Zotero software: A means of bibliographic research and data organisation; teaching bibliographic research. Arab World English Journal (AWEJ) Special Issue on CALL. 2015.
  29. Basak SK. Comparison of Researcher’ s reference manage- ment software: Refworks, Mendeley, and EndNote. J Econ Behav Stud. 2014;6(7):561-568.
  30. Kwon Y, Lemieux M, McTavish J, Wathen N. Identifying and removing duplicate records from systematic review searches. J Med Libr Assoc. 2015;103(4):184-188.
  31. Orgeolet L, Foulquier N, Misery L, et al. Can artificial intelli- gence replace manual search for systematic literature? Review on cutaneous manifestations in primary Sjögren's syndrome. Rheumatology. 2020;59(4):811-819.
  32. Ferdinands G. AI-assisted systematic reviewing: selecting stud- ies to compare Bayesian versus frequentist SEM for small sam- ple sizes. Multivar Behav Res. 2021;56(1):153-154.
  33. Shemilt I, Noel-Storr A, Thomas J, Featherstone R, Mavergames C. Machine learning reduced workload for the Cochrane COVID-19 study register: development and evalua- tion of the Cochrane COVID-19 study classifier. Syst Rev. 2022; 11(1):15.
  34. Tsubota T, Bollegala D, Zhao Y, Jin Y, Kozu T. Improvement of intervention information detection for automated clinical lit- erature screening during systematic review. J Biomed Inform. 2022;104185:104185.
  35. Escaldelai FMD, Escaldelai L, Bergamaschi DP. Systematic review support software system: web-based solution for manag- ing duplicates and screening eligible studies. Rev Bras Epide- miol. 2022;25:25.
  36. Marshall IJ, Johnson BT, Wang Z, Rajasekaran S, Wallace BC. Semi-automated evidence synthesis in health psychology: current methods and future prospects. Health Psychol Rev. 2020;14(1):145-158.
  37. Marshall IJ, Wallace BC. Toward systematic review automa- tion: a practical guide to using machine learning tools in research synthesis. Syst Rev. 2019;8(1):1-10.
  38. Squires JE, Graham ID, Grinspun D, et al. Inappropriateness of health care in Canada: a systematic review protocol. Syst Rev. 2019;8(1):1-8.
  39. Van Grootven B, Jepma P, Rijpkema C, et al. Prediction models for hospital readmissions in patients with heart disease: a sys- tematic review and meta-analysis. BMJ Open. 2021;11(8): e047576.
  40. Halamoda-Kenzaoui B, Rolland E, Piovesan J, Puertas Gallardo A, Bremer-Hoffmann S. Toxic effects of nanomaterials for health applications: how automation can support a system- atic review of the literature? J Appl Toxicol. 2022;42(1):41-51.
  41. Mohan K, Alam MA, Patnaik R, Mohan K. A review on use of automation in systematic reviews for scientific evidence gener- ation. Eur J Mol Clin Med. 2020;8(2):2021.
  42. Park SE, Thomas J. Evidence synthesis software. BMJ Evide Med. 2018;23:140-141.
  43. Dhakal K. NVivo. J Med Libr Assoc. 2022;110(2):270-272.
  44. Jena B. An approach for forecast prediction in data analytics field by tableau software. Int J Inform Eng Electr Bus. 2019; 11(1):19-26.
  45. Cierco Jimenez R, Lee T, Rosillo N, et al. Machine learning computational tools to assist the performance of systematic reviews: a mapping review. BMC Med Res Methodol. 2022; 22(1):322.
  46. Qureshi R, Shaughnessy D, Gill KA, Robinson KA, Li T, Agai E. Are ChatGPT and large language models "the answer" to bringing us closer to systematic review automation? Syst Rev. 2023;12(1):72.
  47. Sun C, Emonet V, Dumontier M. A comprehensive comparison of automated FAIRness Evaluation Tools. 13th International Conference on Semantic Web Applications and Tools for Health Care and Life Sciences. 2022.
  48. Teijema JJ, Seuren S, Anadria D, Bagheri A, van de Schoot R. Simulation-based Active Learning for Systematic Reviews: A Systematic Review of the Literature. Retrieved from psyarxiv com/67zmt. 2023.

Authors
Khalil, H., Pollock, D., McInerney, P., Evans, C., Moraes, E. B., Godfrey, C. M., Alexander, L., Tricco, A., Peters, M. D. J., Pieper, D., Saran, A., Ameen, D., Taneri, P. E., & Munn, Z.
Affiliations
  • La Trobe University (AU)
  • The University of Adelaide (AU)
  • University of the Witwatersrand (ZA)
  • University of Nottingham (GB)
  • Nursing School, Department of Nursing Fundamentals and Administration Federal Fluminense University Rio de Janeiro Brazil
  • Queen's University (CA)
  • Robert Gordon University (GB)
  • St. Michael's Hospital (CA)
  • University of South Australia (AU)
  • Medizinische Hochschule Brandenburg Theodor Fontane (DE)
  • Campbell South Asia India
  • Monash University (AU)
  • HRB‐Trials Methodology Research Network, College of Medicine, Nursing and Health Sciences University of Galway Galway Ireland
Corresponding author
Khalil, Hanan
ORCID
Venue
Research Synthesis Methods
Publisher
Wiley
Published
2024-06-17
Volume
15
Issue
6
Pages
839-850
ISSN
  • 1759-2879 (ISSN-L)
  • 1759-2887
DOI
10.1002/jrsm.1731
OpenAlex
W4399722064
Cited by
43
References
70
Citations by year
2026: 20 · 2025: 20 · 2024: 3
FWCI
12.00
Type
journal article
Language
en
Open access
Licence
Locations
  • Research Synthesis Methods (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Repository@Nottingham (University of Nottingham) (publishedVersion, other-oa)
  • Repository@Nottingham (University of Nottingham) (submittedVersion, cc-by)
  • Open Access Institutional Repository at Robert Gordon University (Robert Gordon University) (submittedVersion, cc-by)
  • UniSA Research Outputs Repository (University of South Australia) (submittedVersion, other-oa)
  • Figshare (submittedVersion, cc-by)
Funders
  • La Trobe University
  • National Health and Medical Research Council (1195676)
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
  • Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
Keywords
scoping review, automation tools, computer software, evidence synthesis, systematic review automation, literature searching, data deduplication, title and abstract screening, data extraction, report writing, interoperability
MeSH terms
Scoping Reviews As Topic, Scoping Reviews as Topic, Automation, Humans, Research Design, Review Literature as Topic, Software, Reproducibility of Results, Information Storage and Retrieval

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review