Usability and acceptability of four systematic review automation software packages: a mixed method design
Open accessThe DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
AIM: New software packages help to improve the efficiency of conducting a systematic review through automation of key steps in the systematic review. The aim of this study was to gather qualitative data on the usability and acceptability of four systematic review automation software packages (Covidence, SRA-Helper for EndNote, Rayyan and RobotAnalyst) for the citation screening step of a systematic review. METHODS: We recruited three volunteer systematic reviewers and asked them to use allocated software packages during citation screening. They then completed a 12-item online questionnaire which was tailored to capture data for the software packages used. FINDINGS: All four software packages were reported to be easy or very easy to learn and use. SRA-Helper for EndNote was most favoured by participants for screening citations and Covidence for resolving conflicts. Overall, participants reported that SRA-Helper for EndNote would be their software package of choice, primarily due to its efficiency. CONCLUSION: This study identified a number of considerations which systematic reviewers can use as a basis of their decision which software to use when performing the citation screening and dispute resolution steps of a systematic review.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the ML tools identified by this review have been created with different aims by a great variety of developers, ranging from individuals and small research groups to large organisations dedicated to evidence-based medicine and systematic reviewing at a large scale [31, 37, 44, 56, 57]”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [37]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“There has not yet been significant primary research into barriers and facilitators to uptake of automation in health evidence synthesis contexts [7, 8].”
From The views of health guideline developers on the use of automation in health evidence synthesis (2021) · cites this paper as [8]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Automated screening of research studies for systematic reviews using study characteristics
Tsafnat, G., Glasziou, P., Karystianis, G., & Coiera, E. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0724-7 · Cited by 44
Cited by
The digital and sustainable transition of the agri-food sector
Abbate, S., Centobelli, P., & Cerchione, R. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.122222 · Cited by 200
Tools to support the automation of systematic reviews: a scoping review.
Khalil, H., Ameen, D., & Zarnegar, A. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.12.005 · Cited by 190
Use of Epic Electronic Health Record System for Health Care Research: Scoping Review
Chishtie, J., Sapiro, N., Wiebe, N., Rabatach, L., Lorenzetti, D., Leung, A. A. C., Rabi, D. M., Quan, H., & Eastwood, C. A. (2023) · Journal of Medical Internet Research · doi:10.2196/51003 · Cited by 155
“The Covidence (Level 10) web platform was used to import citations and facilitate the screening process [33].”
Giri, S., Grimshaw, A. A., Bal, S., Godby, K. N., Kharel, P., Djulbegović, B., Dimopoulos, M. A., Facon, T. T. F., Usmani, S. Z., Mateos, M., et al. (2020) · JAMA Oncology · doi:10.1001/jamaoncol.2020.4338 · Cited by 107
Cierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01805-4 · Cited by 85
Metadata
- Authors
- Cleo, G., Scott, A. M., Islam, F., Julien, B., & Beller, E.
- Affiliations
- Bond University (AU)
- Corresponding author
- Cleo, Gina
- Venue
- Systematic Reviews
- Published
- 2019-06-20
- Volume
- 8
- Issue
- 1
- Pages
- 145-145
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC6587262
- OpenAlex
- W2952098834
- Cited by
- 112
- References
- 4
- Citations by year
- 2026: 22 · 2025: 13 · 2024: 21 · 2023: 21 · 2022: 21 · 2021: 8 · 2020: 6
- FWCI
- 1.77
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Bond University Research Portal (Bond University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Health and Medical Research Council (APP1044904)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- citation screening, software usability, Covidence, Rayyan, conflict resolution, software selection
- MeSH terms
- Systematic Reviews as Topic, Automation, Humans, Software, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Usability and acceptability of four systematic review automation software packages: a mixed method design. https://app.cochat.ai/papers/10.1186/s13643-019-1069-6
- The paper
- Cleo, G., Scott, A. M., Islam, F., Julien, B., & Beller, E. (2019). Usability and acceptability of four systematic review automation software packages: a mixed method design. Systematic Reviews. https://doi.org/10.1186/s13643-019-1069-6
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.