Usability and acceptability of four systematic review automation software packages: a mixed method design
Open accessThe DOI matches its Crossref record; 4 of the 4 references it cites were found in a registry or an index.
Abstract
AIM: New software packages help to improve the efficiency of conducting a systematic review through automation of key steps in the systematic review. The aim of this study was to gather qualitative data on the usability and acceptability of four systematic review automation software packages (Covidence, SRA-Helper for EndNote, Rayyan and RobotAnalyst) for the citation screening step of a systematic review. METHODS: We recruited three volunteer systematic reviewers and asked them to use allocated software packages during citation screening. They then completed a 12-item online questionnaire which was tailored to capture data for the software packages used. FINDINGS: All four software packages were reported to be easy or very easy to learn and use. SRA-Helper for EndNote was most favoured by participants for screening citations and Covidence for resolving conflicts. Overall, participants reported that SRA-Helper for EndNote would be their software package of choice, primarily due to its efficiency. CONCLUSION: This study identified a number of considerations which systematic reviewers can use as a basis of their decision which software to use when performing the citation screening and dispute resolution steps of a systematic review.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“the ML tools identified by this review have been created with different aims by a great variety of developers, ranging from individuals and small research groups to large organisations dedicated to evidence-based medicine and systematic reviewing at a large scale [31, 37, 44, 56, 57]”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [37]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“There has not yet been significant primary research into barriers and facilitators to uptake of automation in health evidence synthesis contexts [7, 8].”
From The views of health guideline developers on the use of automation in health evidence synthesis (2021) · cites this paper as [8]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Automated screening of research studies for systematic reviews using study characteristics
Tsafnat, G., Glasziou, P., Karystianis, G., & Coiera, E. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0724-7 · Cited by 44
Cited by
The digital and sustainable transition of the agri-food sector
Abbate, S., Centobelli, P., & Cerchione, R. (2023) · Technological Forecasting and Social Change · doi:10.1016/j.techfore.2022.122222 · Cited by 200
Tools to support the automation of systematic reviews: a scoping review.
Khalil, H., Ameen, D., & Zarnegar, A. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.12.005 · Cited by 190
Use of Epic Electronic Health Record System for Health Care Research: Scoping Review
Chishtie, J., Sapiro, N., Wiebe, N., Rabatach, L., Lorenzetti, D., Leung, A. A. C., Rabi, D. M., Quan, H., & Eastwood, C. A. (2023) · Journal of Medical Internet Research · doi:10.2196/51003 · Cited by 155
“The Covidence (Level 10) web platform was used to import citations and facilitate the screening process [33].”
Giri, S., Grimshaw, A. A., Bal, S., Godby, K. N., Kharel, P., Djulbegović, B., Dimopoulos, M. A., Facon, T. T. F., Usmani, S. Z., Mateos, M., et al. (2020) · JAMA Oncology · doi:10.1001/jamaoncol.2020.4338 · Cited by 107
Cierco Jimenez, R., Lee, T., Rosillo, N., Cordova, R., Cree, I. A., Gonzalez, A., & Indave Ruiz, B. I. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01805-4 · Cited by 85
Meherali, S. M., Rahim, K. A., Campbell, S., & Lassi, Z. S. (2021) · Frontiers in Public Health · doi:10.3389/fpubh.2021.761394 · Cited by 108
Affengruber, L., van der Maten, M. M., Spiero, I., Nussbaumer-Streit, B., Mahmić-Kaknjo, M., Ellen, M. E., Goossen, K., Kantorova, L., Hooft, L., Riva, N., et al. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02320-4 · Cited by 52
“…37, 39, 60, 74, 95, 97, 120, 128] • RCT classifier [118] • Research screener [33] • RobotAnalyst [36, 37, 48, 106, 109] • SRA-helper for EndNote [36] • SWIFT-active screener [37, 64] • SWIFT-Review [74] Full-text retrieval - - Full-text selection • Crowdsourcing using different (automation-)…”
BackgroundMethodInfluentialClark, J., McFarlane, C., Cleo, G., Ishikawa Ramos, C., & Marshall, S. (2021) · JMIR Medical Education · doi:10.2196/24418 · Cited by 54
Self-citationTóth, B., Berek, L., Gulácsi, L., Péntek, M., & Zrubka, Z. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02592-3 · Cited by 48
“…freeware screening automation software were Abstrackr [90, 94, 97, 98, 105, 113, 132], EPPI Reviewer [80, 128, 132, 136], RobotAnalyst [91, 94, 113, 131], Distiller SR [94, 99, 126], Rayyan [103, 131], Systematic Review Accelerator [18, 135], RCT Tagger [77, 78], SWIFT Review [125, 126], SyRF [92,…”
BackgroundMethodCoiera, E., & Liu, S. (2022) · Cell Reports Medicine · doi:10.1016/j.xcrm.2022.100860 · Cited by 48
Access to Cardiovascular Care for Indigenous Peoples in Canada: A Rapid Review
Vervoort, D., Kimmaliardjuk, D. M., Ross, H. J., Fremes, S. E., Ouzounian, M., & Mashford‐Pringle, A. (2022) · CJC Open · doi:10.1016/j.cjco.2022.05.010 · Cited by 56
“Articles were screened for relevance based on (i) their title-abstract and (ii) being full texts identified using Covidence screening software.(22) Articles presenting original research, conference abstracts, or articles presenting a review of the literature were included.”
MethodChishtie, J., Marchand, J.-S., Turcotte, L., Bielska, I. A., Babineau, J., Cepoiu‐Martin, M., Irvine, M. A., Munce, S., Abudiab, S., Bjelica, M., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/17892 · Cited by 52
“The review management software Covidence was used to manage the search results, including the importing of references, screening of citations, and conflict resolution [27].”
Automation tools to support undertaking scoping reviews
Khalil, H., Pollock, D., McInerney, P., Evans, C., Moraes, E. B., Godfrey, C. M., Alexander, L., Tricco, A., Peters, M. D. J., Pieper, D., et al. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1731 · Cited by 43
“12,13,20,21 However, to date, there is no guidance for conducting scoping reviews using automation tools or existing software to facilitate the process of scoping reviews.”
InfluentialGerdes, A., & Frandsen, T. F. (2023) · Ethics and Information Technology · doi:10.1007/s10676-023-09700-2 · Cited by 53
“After removing duplicates, all retrieved references were added to a reference management database using Covidence, which is developed for screening of literature for systematic reviews (Cleo et al., 2019).”
BackgroundMethodLagorio, S., Blettner, M., Baaken, D., Feychting, M., Karipidis, K., Loney, T., Orsini, N., Röösli, M., Paulo, M. S., & Elwood, M. (2021) · Environment International · doi:10.1016/j.envint.2021.106828 · Cited by 36
The views of health guideline developers on the use of automation in health evidence synthesis
Arno, A., Elliott, J., Wallace, B., Turner, T., & Thomas, J. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01569-2 · Cited by 40
Moreno-Garcia, C. F., Jayne, C., Elyan, E., & Aceves-Martins, M. (2023) · Decision Analytics Journal · doi:10.1016/j.dajour.2023.100162 · Cited by 35
Koszalinski, R. S., & Olmos, B. (2022) · Perspectives In Psychiatric Care · doi:10.1111/ppc.13115 · Cited by 47
Lanke, V., Trimm, K., Habib, B., & Tamblyn, R. M. (2025) · Journal of Medical Internet Research · doi:10.2196/60822 · Cited by 44
“All studies identified in the 3 databases were compiled in Covidence, a systematic review management web-based platform [23].”
Chacon, L., Mitchell, G., & Golder, S. (2024) · PLOS Global Public Health · doi:10.1371/journal.pgph.0002736 · Cited by 33
Abdul Wahab, A., Kusuma, Y. S., Juandi, D., Turmudi, T., Buhaerah, B., & Syaiful, S. (2024) · Jurnal Pendidikan Progresif · doi:10.23960/jpp.v14.i3.2024118 · Cited by 28
SCLERAL FIXATION OF CARLEVALE INTRAOCULAR LENS
Ripa, M., Angunawela, R. I., & Motta, L. (2023) · Retina · doi:10.1097/iae.0000000000003873 · Cited by 27
Catheter and Surgical Ablation for Atrial Fibrillation: A Systematic Review and Meta-analysis
Montané, B., Zhang, S., Wolfe, J. D., Prime, S., Luo, C., Cooper, D. H., Doering, M. M., Blomström‐Lundqvist, C., Nashef, S. A. M., Osmancik, P. P., et al. (2025) · Annals of Internal Medicine · doi:10.7326/annals-25-00253 · Cited by 27
Reyes‐Ferrada, W., Chirosa-Rios, L., García, D. M., Perea, Á. R., & Jerez‐Mayorga, D. (2022) · Journal of Back and Musculoskeletal Rehabilitation · doi:10.3233/bmr-210261 · Cited by 36
“interface [38], and has been previously used in system138”
MethodRehmani, A. I., Au, A., Montgomery, C., & Papathanassoglou, E. D. E. (2024) · Nursing in Critical Care · doi:10.1111/nicc.13076 · Cited by 31
Giri, S., Aryal, M. R., Yu, H., Grimshaw, A. A., Pathak, R., Huntington, S. P., & Dhakal, B. (2020) · Journal of Geriatric Oncology · doi:10.1016/j.jgo.2020.05.013 · Cited by 20
“After removing, remaining set of articles was ingested into CovidenceTM, a screening and data extraction tool(5).”
MethodLukman, K. M., Hastuti, L. P., Oktavia, D., Pratiwi, D., Uchiyama, Y., & Harding, D. (2025) · Journal of Environmental Management · doi:10.1016/j.jenvman.2025.126132 · Cited by 28
Information seeking for coping with cancer: a systematic review
Petersen, E., Jensen, J. G., & Frandsen, T. F. (2021) · Aslib Journal of Information Management · doi:10.1108/ajim-01-2021-0004 · Cited by 18
Detecting Metabolic Thresholds from Nonlinear Analysis of Heart Rate Time Series: A Review
Zimatore, G., Gallotta, M. C., Campanella, M., Skarzynski, P. H., Maulucci, G., Serantoni, C., Spirito, M. D., Curzi, D., Guidetti, L., Baldari, C., et al. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph191912719 · Cited by 22
Cai, J., Yang, M., Zhang, N., Chen, Y., Wei, J., Wang, J., Liu, Q., Li, W., Shi, W., & Liu, W. (2024) · Building and Environment · doi:10.1016/j.buildenv.2024.111226 · Cited by 22
Leech, J. B., Owen, W. E., Young, J. L., & Rhon, D. I. (2022) · Journal of Manual & Manipulative Therapy · doi:10.1080/10669817.2022.2113295 · Cited by 17
Machine learning for accelerating screening in evidence reviews
Chappell, M., Edwards, M., Watkins, D., Marshall, C., & Graziadio, S. (2023) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.12021 · Cited by 19
“The function and usability of machine learning tools has been assessed in a number of reviews [6, 9 – 12] and some of the tools have been identified as having good usability and acceptability or being rated highly by users.”
Rajkumar, V., McCausland, K., & Lobo, R. C. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19105947 · Cited by 16
Moran, G. W., Kurtzman, J. T., & Carpenter, C. P. (2022) · Journal of Pediatric Urology · doi:10.1016/j.jpurol.2022.08.015 · Cited by 19
Silva, N., & Wickramaarachchi, D. (2025) · International Conference on Advanced Research in Computing (ICARC) · doi:10.1109/icarc64760.2025.10963273 · Cited by 21
“These challenges include labor-intensive processes for identifying literature, biases introduced by human reviewers, inconsistencies in data interpretation and time inefficiencies that can delay research outcomes [1].”
Penfornis, K. M., van Vliet, M. H. M., Meijer, E., & Gebhardt, W. A. (2022) · BMJ Open · doi:10.1136/bmjopen-2021-058405 · Cited by 15
ASReview: Open Source Software for Efficient and Transparent Active Learning for Systematic Reviews.Preprint
van de Schoot, R., Bruin, J. D., Schram, R. D., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B. M. R., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2020) · arXiv (Cornell University) · Cited by 21
Fütterer, T., Campos, D., Gfrörer, T., Lavelle-Hill, R. E., Murayama, K., & Scherer, R. (2025) · Learning and Individual Differences · doi:10.1016/j.lindif.2025.102849 · Cited by 18
Scott, A. M., Glasziou, P., & Clark, J. M. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.03.007 · Cited by 15
Self-citationMak, V., Brand, G., & Morphet, J. (2022) · Journal of Clinical Nursing · doi:10.1111/jocn.16426 · Cited by 11
Ripa, M., Kopsacheilis, N., Kanellopoulou, K., Nomikarios, M., & Motta, L. (2022) · Diagnostics · doi:10.3390/diagnostics12092100 · Cited by 11
Shen, L., Mo, J., Yang, C., Jiang, Y., Ke, L., Hou, D., Yan, J., Zhang, T., & Fan, W. (2023) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1010830 · Cited by 10
Ripa, M., Panos, G. D., Ręjdak, R., Empeslidis, T., Toro, M. D., Costagliola, C., Ferrara, A., Gotzaridis, S., Frisina, R., & Motta, L. (2023) · Journal of Clinical Medicine · doi:10.3390/jcm12062316 · Cited by 13
“org [22], (accessed on 11 November 2022) to record and evaluate the study data between 11 October 2012 and 11 November 2022.”
BackgroundMasterson, S., Heffernan, E., Keegan, D., Clarke, B., Deasy, C., O'Donnell, C., Crowley, P., Breen, R., Kelly, M., & Murphy, A. W. (2020) · HRB Open Research · doi:10.12688/hrbopenres.13149.1 · Cited by 10
Vyas, N., Hendren, S., Sehgal, D. M. T., Monga, C., Ranjan, R., Chaturvedi, H., Subramanian, A., & Vashistha, V. (2022) · Indian Journal of Hematology and Blood Transfusion · doi:10.1007/s12288-022-01543-z · Cited by 9
“Abstracts captured by the search strategy were extracted from all five databases and uploaded to the online Covidence platform [9].”
MethodChristopoulou, S. C. (2022) · BioMedInformatics · doi:10.3390/biomedinformatics2030032 · Cited by 11
“, 2019) [32] Covidence, SRA-Helper for EndNote, Rayyan and RobotAnalyst”
BackgroundLiu, J., Xiao, J., Ou, S. I., Chen, J., & Tang, S. (2025) · Asia-Pacific Journal of Oncology Nursing · doi:10.1016/j.apjon.2025.100734 · Cited by 10
“Vision Loss” and COVID-19 Infection: A Systematic Review and Meta-Analysis
Ripa, M., Motta, L., Schipa, C., Rizzo, S., Sollazzi, L., & Aceto, P. (2022) · Vision · doi:10.3390/vision6040060 · Cited by 8
“org (accessed on 9 June 2022) [37], to record and evaluate the study data between 22 May 2022 and 09 June 2022.”
BackgroundMachine Learning Methods for Systematic Reviews:: A Rapid Scoping Review.
Roth, S. C., & Wermer‐Colan, A. (2023) · PubMed Central · doi:10.32481/djph.2023.11.008 · Cited by 39
Rossi, A., Heyman, N. B., Rossi, M. O., Wolf, S., & White, T. (2023) · HERD Health Environments Research & Design Journal · doi:10.1177/19375867231151687 · Cited by 9
“The Covidence citation program was used to organize and evaluate articles for inclusion (Cleo et al., 2019).”
MethodRakhshanda, S., Briggs, N. E., Rhee, J., Rye, K.-A., Liaw, S.-T., & Jonnagaddala, J. (2026) · European Journal of Preventive Cardiology · doi:10.1093/eurjpc/zwag175 · Cited by 6
Ayaaba, E., Adusei‐Asante, K., Nunfam, V. F., Rumchev, K. B., & Amponsah, S. K. (2024) · Critical Public Health · doi:10.1080/09581596.2024.2438654 · Cited by 9
“Rayyan software (Cleo et al., 2019; Couban, 2016; Haritsa et al., 2021; H. Harrison et al., 2020; R. L. Harrison et al., 2020; Johnson & Phillips, 2018) was used to screen the remaining 4,771 titles and abstracts and excluded 4,734 that were unrelated to the topic.”
Family implications during deprivation of liberty and Attachment Theory: a qualitative metasynthesis
Gato, C. M., Nascimento, L. C., de Oliveira, R. A. A., Massaroli, A., da Silva, S. É. D., & Araújo, J. S. (2022) · Revista Brasileira de Enfermagem · doi:10.1590/0034-7167-2021-0722pt · Cited by 7
Du, J. T., Ke, Q., Chu, C. M., Partridge, H., & Ma, D. (2024) · Journal of Documentation · doi:10.1108/jd-10-2023-0222 · Cited by 6
Amofa-Adade, E., & Koltai, J. P. (2024) · Journal of Infrastructure Policy and Development · doi:10.24294/jipd.v8i11.8400 · Cited by 6
“The Covidence platform provides the opportunity to streamline literature by avoiding bias in the synthesis of research material (Cleo et al., 2019; Kellermeyer et al., 2018).”
BackgroundTowards the Utilization of AI-Powered Assistance for Systematic Literature Review
Đukić, M., Škembarević, M., Jejić, O., & Luković, I. (2024) · Communications in computer and information science · doi:10.1007/978-3-031-70421-5_16 · Cited by 7
Zahra, G. T., & Sudihartinih, E. (2025) · Universal Education Journal of Teaching and Learning · doi:10.63081/uejtl.v2i2.54 · Cited by 6
Masterson, S., Heffernan, E., Keegan, D., Clarke, B., Deasy, C., O'Donnell, C., Crowley, P., Breen, R., Kelly, M., & Murphy, A. W. (2021) · HRB Open Research · doi:10.12688/hrbopenres.13149.2 · Cited by 6
“Literature review will be carried out in collaboration with a HSE expert librarian and using the Covidence systematic review management tool through the HSE National Health Library’s Covidence account 21,22 .”
A Mixed‐Method Usability Study on User Experience with Systematic Review Software
Wang, M., Sharmin, S., Wang, M., & Yu, F. (2021) · Proceedings of the Association for Information Science and Technology · doi:10.1002/pra2.462 · Cited by 10
“In addition to Abstrackr, Ouzzani (2016) introduced Rayyan as a semi-automation tool for abstract screening.”
BackgroundInfluentialHow do search systems impact systematic searching? A qualitative study
Hickner, A. (2023) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2023.1647 · Cited by 7
When to stop reviewing: validation of stop criteria in ASReview
Kempny, C., Annac, K., Wahidie, D., Yilmaz-Aslan, Y., & Brzoska, P. (2026) · BMC Medical Research Methodology · doi:10.1186/s12874-026-02866-5 · Cited by 5
“Systematic Reviews play a key role in evidence-based research [1].”
Askin, N., & Okoli, G. N. (2024) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-024-00335-4 · Cited by 4
“While participants generally found these software easy to learn, they had concerns about response time and software “glitches” (Cleo et al. 2019).”
Nunu, W. N., Ndirangu, J., & Tsoka-Gwegweni, J. M. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-076140 · Cited by 4
“28 It has been reported that Web of Science, Scopus and Dimensions, the combination of the three databases, would give an accuracy rate of over 96% in accessing required literature, thus making the scoping review using these databases comprehensive.”
BackgroundTu, J., Nacke, L. E., & Rogers, K. (2025) · Interacting with Computers · doi:10.1093/iwc/iwaf001 · Cited by 4
Tanlim, K., Nita, S., & Badry, A. I. (2024) · Pena Justisia Media Komunikasi dan Kajian Hukum · doi:10.31941/pj.v23i1.4203 · Cited by 5
Women’s reported experience of respectful maternity care measures worldwide: a systematic review
Roga, E. Y., Moti, G., Chaka, E. E., Gabriel, L., Creedy, D. K., & Hastie, C. R. (2026) · BMC Public Health · doi:10.1186/s12889-026-27157-y · Cited by 3
“Articles were imported to Endnote and then uploaded into the data screening software tool, Covidence [38], where duplicates and irrelevant articles were eliminated.”
Ripa, M., Schipa, C., Aceto, P., Kanikaram, G., & Shah, N. A. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13216572 · Cited by 3
What Do We Know About Informal Caregiving in the Field of Addiction?: A Scoping Review
Côté, M., Morency, E.-E., Roussel, N., Ferland, F., Tremblay, J., & Blanchette-Martin, N. (2024) · International Journal of Mental Health and Addiction · doi:10.1007/s11469-024-01301-4 · Cited by 2
“Covidence software, which is specialized in systematic review data management, was used to facilitate the article management process (Cleo et al., 2019; Kellermeyer et al., 2018).”
Rungstrøm, K., Frisk, C. H., Hulgaard, D. R., & Lichtenstein, M. B. (2024) · Scandinavian Journal of Sport and Exercise Psychology · doi:10.7146/sjsep.v6i.140705 · Cited by 2
Farias, A. N., Pucinelli, R. H., Ramos, R. C., Fernández‐Río, J., & Impolcetto, F. M. (2024) · Caderno Pedagógico · doi:10.54033/cadpedv21n8-267 · Cited by 2
Cardenas, D. A., & Preston, N. (2025) · Journal of Bodywork and Movement Therapies · doi:10.1016/j.jbmt.2025.08.023 · Cited by 2
Ohta, R., Fujimori, T., & Ichinose, K. (2026) · Cureus · doi:10.7759/cureus.102091 · Cited by 2
“Study screening, selection, and data management were conducted using Covidence systematic review software (Melbourne, Australia: Veritas Health Innovation) [10].”
Amankwaa, I., Odoom, A., Kasim, A., Kobiah, E., Diebieri, M., Boateng, E. A., Gyamfi, S., & Hales, C. (2026) · Nurse Education Today · doi:10.1016/j.nedt.2026.107154 · Cited by 3
O que é uma revisão de literatura?
Brandão, A. S., Araújo, M. E. L., Garcia, T. M. P., & de Oliveira, R. D. C. S. (2023) · Editora Pascal LTDA eBooks · doi:10.29327/5293250.1-1 · Cited by 2
Usabilidad del sistema computacional “Apoyo a la Revisión Sistemática”: un estudio metodológico
Escaldelai, F. M. D., Escaldelai, L., & Bergamaschi, D. P. (2024) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.7081.4373 · Cited by 3
Rachmi, C. N., Jupri, A., Juandi, D., Putri, N., & Bulubaan, S. (2025) · Jurnal Pendidikan MIPA · doi:10.23960/jpmipa.v26i1.pp227-243 · Cited by 2
“By leveraging Covidence, researchers could conduct their systematic literature review more efficiently, maintaining high standards of academic rigor and methodological precision (Cleo et al., 2019).”
Krommes, K. K., Nielsen, M. F., Krohn, L. M., Grønfeldt, B. M., Thorborg, K., Hölmich, P., & Ishøi, L. (2020) · medRxiv · doi:10.1101/2020.04.01.20048686 · Cited by 2
Quality of neonatal intensive care in Africa: a systematic review and meta-analysis
Biadgo, A., Mekonnen, A., Getahun, K. A., Yitayal, M., & Fekadu, G. A. (2026) · BMC Health Services Research · doi:10.1186/s12913-026-14355-y · Cited by 1
“Appropriate papers selection was conducted using Covi-dence (a web-based tool to streamline the systematic review process [16]) by two independent reviewers (ABK and KAG).”
Ripa, M., Shah, N. A., Schipa, C., Aceto, P., & Donati, T. (2024) · Journal of Clinical Medicine · doi:10.3390/jcm13092486 · Cited by 1
“The Covidence systematic review software© (Veritas Health Innovation, Melbourne, Australia), available at www.covidence.org [31] (accessed on 1 March 2024), was employed to record and evaluate the study data until 1 March 2024.”
MethodUsabilidade do sistema computacional “Apoio à Revisão Sistemática”: um estudo metodológico
Escaldelai, F. M. D., Escaldelai, L., & Bergamaschi, D. P. (2024) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.7081.4375 · Cited by 1
Heaney, C., Lally, S., & Dowling, M. (2025) · British Journal of Nursing · doi:10.12968/bjon.2024.0393 · Cited by 1
Spatial and spatio-temporal analysis for malaria hotspot identification: a scoping review protocol
Monteiro, G. M., Sonounameto, R. C., Djogbénou, L., & Sedda, L. (2025) · BMJ Open · doi:10.1136/bmjopen-2025-101375 · Cited by 2
Tuncer, M., & Tuncer, G. Z. (2025) · Journal of Nursing Scholarship · doi:10.1111/jnu.70048 · Cited by 2
Henriksen, S. B., Hornemann, C., & Hulgaard, D. R. (2026) · Journal of Human Nutrition and Dietetics · doi:10.1111/jhn.70226 · Cited by 1
Palliative Care Needs of Older Adults with Stroke: A Scoping Review
Inocian, E. P., Milone, G. A., Jiang, N., Inocian, E., & Felicilda‐Reynaldo, R. F. D. (2026) · Journal of Holistic Nursing · doi:10.1177/08980101261431419 · Cited by 1
Ge, B., Luo, J., Cheng, X., Wang, D., Liu, R., & Liao, X. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1916495 · Cited by 1
“All retrieved records were managed using EndNote, Covidence, and Microsoft Excel throughout the review process (23).”
Ge, B., Luo, J., Cheng, X., Wang, D., Liu, R., & Liao, X. (2026) · Frontiers in Public Health · doi:10.3389/fpubh.2026.1918722 · Cited by 2
“Throughout the review process, we used Covidence, Endnote, and Excel applications to manage records (29).”
Bawazir, N., Johnson, J. M., Abdulhadi, K., & Forgrave, D. (2022) · International Journal of Healthcare · doi:10.5430/ijh.v9n1p1 · Cited by 3
Ripa, M., Schipa, C., Aceto, P., Tshomba, Y., Rizzo, S., Baldascino, A., & Donati, T. (2024) · European Journal of Ophthalmology · doi:10.1177/11206721241277247 · Cited by 1
Leenaars, C. H. C., Stafleu, F. R., & Bleich, A. (2024) · ALTEX · doi:10.14573/altex.2409251 · Cited by 1
“Concerning qualitative studies addressing user experiences with SR tools, Cleo et al. investigated volunteers using Covidence, SRA-Helper for EndNote, Rayyan or RobotAnalyst for title abstract screening of >200 references (Cleo et al., 2019).”
RISK FACTORS AFFECTING THE OCCURRENCE OF BRAIN TUMORS
Safarova, S. A. K., Zp, K., & Makimbetov, E. К. (2022) · Научное обозрение Медицинские науки (Scientific Review Medical Sciences) · doi:10.17513/srms.1239 · Cited by 1
Sam, D. M., Bukirwa, C., & Ekongefeyin, S. N. N. (2023) · medRxiv · doi:10.1101/2023.09.19.23295754 · Cited by 1
Gerenciando suas referências bibliográficas
Shinkai, A., Rosa, I. R., Correa, R. M., & de Oliveira, R. D. C. S. (2023) · Editora Pascal LTDA eBooks · doi:10.29327/5293250.1-8
Inserindo sua pesquisa em bases de registro de protocolos de revisão sistemática
Rosa, I. R., Gonçalves, M. J., Alvarez, V. M. C., & de Oliveira, R. D. C. S. (2023) · Editora Pascal LTDA eBooks · doi:10.29327/5293250.1-3
Como realizar uma busca em base de dados?
Rosa, I. R., Ferreira, L. G. S., Mesquita, L. N., Sampaio, S. S., & de Oliveira, R. D. C. S. (2023) · Editora Pascal LTDA eBooks · doi:10.29327/5293250.1-4
Como analisar a qualidade de uma revisão sistemática?
Aranha, M. F. D. A. C., Lima, J. D. C., dos Santos, F. B. G., Pinheiro, M. M., & de Oliveira, R. D. C. S. (2023) · Editora Pascal LTDA eBooks · doi:10.29327/5293250.1-10
Como estruturar a escrita da sua revisão sistemática?
Aranha, M. F. D. A. C., Garcia, T. M. P., Pinheiro, M. M., & de Oliveira, R. D. C. S. (2023) · Editora Pascal LTDA eBooks · doi:10.29327/5293250.1-7
Goldfarb, N., Tal, N., Cohen, I.-C., Elyashar, A., Ellen, M., Puzis, R., & Cohen, O. (2024) · doi:10.2196/preprints.59322
Dodd, S., & Sydes, M. L. (2024) · European Journal on Criminal Policy and Research · doi:10.1007/s10610-024-09596-8 · Cited by 2
“…a web-based reference management software, facilitated the review’s information management, allowing for the importation and screening of citations and full-text articles, identification of duplicate entries, resolution of conflicts, and data extraction using customisable forms (Cleo et al., 2019).”
MethodUsability of the “Systematic Review Support” computer system: a methodological study
Escaldelai, F. M. D., Escaldelai, L., & Bergamaschi, D. P. (2024) · Revista Latino-Americana de Enfermagem · doi:10.1590/1518-8345.7081.4374 · Cited by 1
Metadata
- Authors
- Cleo, G., Scott, A. M., Islam, F., Julien, B., & Beller, E.
- Affiliations
- Bond University (AU)
- Corresponding author
- Cleo, Gina
- Venue
- Systematic Reviews
- Published
- 2019-06-20
- Volume
- 8
- Issue
- 1
- Pages
- 145-145
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC6587262
- OpenAlex
- W2952098834
- Cited by
- 112
- References
- 4
- Citations by year
- 2026: 22 · 2025: 13 · 2024: 21 · 2023: 21 · 2022: 21 · 2021: 8 · 2020: 6
- FWCI
- 1.77
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Bond University Research Portal (Bond University) (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- National Health and Medical Research Council (APP1044904)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- citation screening, software usability, Covidence, Rayyan, conflict resolution, software selection
- MeSH terms
- Systematic Reviews as Topic, Automation, Humans, Software, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.