Usability and acceptability of four systematic review automation software packages: a mixed method design
Open accessThe DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
AIM: New software packages help to improve the efficiency of conducting a systematic review through automation of key steps in the systematic review. The aim of this study was to gather qualitative data on the usability and acceptability of four systematic review automation software packages (Covidence, SRA-Helper for EndNote, Rayyan and RobotAnalyst) for the citation screening step of a systematic review. METHODS: We recruited three volunteer systematic reviewers and asked them to use allocated software packages during citation screening. They then completed a 12-item online questionnaire which was tailored to capture data for the software packages used. FINDINGS: All four software packages were reported to be easy or very easy to learn and use. SRA-Helper for EndNote was most favoured by participants for screening citations and Covidence for resolving conflicts. Overall, participants reported that SRA-Helper for EndNote would be their software package of choice, primarily due to its efficiency. CONCLUSION: This study identified a number of considerations which systematic reviewers can use as a basis of their decision which software to use when performing the citation screening and dispute resolution steps of a systematic review.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the ML tools identified by this review have been created with different aims by a great variety of developers, ranging from individuals and small research groups to large organisations dedicated to evidence-based medicine and systematic reviewing at a large scale [31, 37, 44, 56, 57]”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [37]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“There has not yet been significant primary research into barriers and facilitators to uptake of automation in health evidence synthesis contexts [7, 8].”
From The views of health guideline developers on the use of automation in health evidence synthesis (2021) · cites this paper as [8]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Automated screening of research studies for systematic reviews using study characteristics
Tsafnat, G., Glasziou, P., Karystianis, G., & Coiera, E. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0724-7 · Cited by 44
Cited by
Items 101–117 of 117
Female Representation Among Head and Neck Surgical Oncology Publications
Trent, M. S., Frank, M., Meller, L. L. T., Chan, D., Tsang, C., Tokhi, W., Tjoa, T., & Haidar, Y. M. (2025) · Head & Neck · doi:10.1002/hed.28066
Dialectical nursing care: a concept analysis
Bitencourt, J. V. D. O. V., de Léo, M. M. F., Luzardo, A. R., Biffi, P., Farias, D. C. S., & Araújo, J. S. (2025) · Investigación y Educación en Enfermería · doi:10.17533/udea.iee.v43n2e04
Jiménez, R. C., Lee, T., Rosillo, N., Córdova, R., Cree, I. A., González, Á., & Indave Ruiz, B. I. (2023) · Figshare · doi:10.6084/m9.figshare.22609363
Mukumbang, F. C., Guajardo, M. G. U., Eastwood, J. G., Katz, I., & Schneider, C. H. (2026) · Systematic Reviews · doi:10.1186/s13643-026-03135-8
“Following the completion of the search, citations will be uploaded directly into the Covidence Systematic Review Software—an online software tool designed to streamline the process of conducting a systematic review [26].”
Ripa, M., Schipa, C., Aceto, P., Prasad, S., & Shah, N. A. (2026) · Vision · doi:10.3390/vision10020020
“Covidence systematic review software© (Veritas Health Innovation, Melbourne, Australia) was employed to record and evaluate the study data between 8 May 2024 and 8 June 2025 [19].”
Müller-Eberstein, R., & Barnes, B. (2026) · Research Square · doi:10.21203/rs.3.rs-9789751/v1
Jacob, R., Ortolano, L. C., Dicks, T., Kemmerer, P., Reichel, J. L., Eggert, V., Rigotti, T., & Dietz, P. (2026) · Applied Psychology Health and Well-Being · doi:10.1111/aphw.70152
“Dupli-cates were removed using both the Deduplicator Tool (Institute for Evidence-Based Healthcare, Bond University, 2024) and the automatic deduplication feature of the Covidence software (Cleo et al., 2019).”
InfluentialMeta-research on key metrics of preregistered scoping reviews
Nordmann, K., Kuhn, P., & Fischer, F. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2026.10078
Roga, E. Y., Moti, G., Chaka, E. E., Gabriel, L., Creedy, D. K., & Hastie, C. R. (2026) · Women s Health · doi:10.1177/17455057261450176 · Cited by 1
“33 Two independent reviewers carefully evaluated the complete text of chosen citations against the inclusion criteria.”
Pereira, B. A., McMurray, A., Manoharan, A., Rajesh, J. I., & Jang, R. (2026) · Health Care Management Review · doi:10.1097/hmr.0000000000000484 · Cited by 1
Beyond the shoreline: making sense of coexistence for the marine governance realm
Crowley-Aksamit, C., Knol-Kauffman, M., Mannherz, F., & Nielsen, K. N. (2026) · Ecology and Society · doi:10.5751/es-17555-310301
Barragán, A. M., Bonett, S. E. O., Rodríguez-Grande, E.-I., Orjuela-Cañón, Á. D., Perdómo, O., & Sánchez, G. (2026) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70098
“A user's background can also affect assessment; what is considered an advantage or disadvantage will vary among reviewers depending on their context and objectives [184], thereby limiting feature assessment methodology's generalizability.”
Shahid, F., Zare, E., Tikhonova, S., Jessani, A., & Santos, M. J. (2026) · Journal of Dental Education · doi:10.1002/jdd.70360
Local government climate change strategies: A systematic literature review
Musa, S., Mansi, M., Pandey, R., & Crawford, J. (2026) · Cities · doi:10.1016/j.cities.2026.107567
Heidari, K., De-Heer, M., Neef, A., Datta, R. K., Kwok, S. M., Drolet, J. L., & Salami, B. O. (2026) · World Development Sustainability · doi:10.1016/j.wds.2026.100370
Lu, S., Phang, S., Judge, S. K., Liu, Y., & Fu, K. (2026) · Discover Education · doi:10.1007/s44217-026-02125-1 · Cited by 1
“Database search results were first exported to EndNote for duplicate removal and initial extraction [41].”
Cheng, X., Luo, J., Ge, B., Wang, D., Liu, R., & Liao, X. (2026) · Frontiers in Immunology · doi:10.3389/fimmu.2026.1918701
Metadata
- Authors
- Cleo, G., Scott, A. M., Islam, F., Julien, B., & Beller, E.
- Affiliations
- Bond University (AU)
- Corresponding author
- Cleo, Gina
- Venue
- Systematic Reviews
- Published
- 2019-06-20
- Volume
- 8
- Issue
- 1
- Pages
- 145-145
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC6587262
- OpenAlex
- W2952098834
- Cited by
- 112
- References
- 4
- Citations by year
- 2026: 22 · 2025: 13 · 2024: 21 · 2023: 21 · 2022: 21 · 2021: 8 · 2020: 6
- FWCI
- 1.77
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Bond University Research Portal (Bond University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Health and Medical Research Council (APP1044904)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- citation screening, software usability, Covidence, Rayyan, conflict resolution, software selection
- MeSH terms
- Systematic Reviews as Topic, Automation, Humans, Software, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.