Evaluating software tools to conduct systematic reviews: a feature analysis and user survey
Open accessThe DOI matches its Crossref record; 21 of the 22 references it cites were found in a registry or an index.
Abstract
Systematic reviews are research synthesis methods increasingly used in educational research to support evidence-based decision making. The conduction of a systematic review is a complex process with several phases usually supported by software tools. These tools used at the international level are not currently common in the Italian educational research. This work describes four software tools used in the international educational research and evaluates their general functionality and specific features’ usability to conduct the systematic review phases of study screening and selection. For this purpose, this study uses two methods: a feature analysis (Kitchenham et al., 1997) and an expert survey (Harrison et al., 2020). The results of both investigation methods agree to consider Covidence and Rayyan the most functional software tools in conducting SR. Among the four tools, ASReview has the greatest potential for making the process of a SR more efficient. Una valutazione dei software per condurre revisioni sistematiche: analisi delle caratteristiche e sondaggio a esperti Le revisioni sistematiche sono metodi di sintesi di ricerca sempre più utilizzati in campo educativo al fine di supportare il processo decisionale basato su evidenze. La conduzione di una revisione sistematica, poiché complessa e su più fasi, è sovente supportata in ambito internazionale da software specifici attualmente poco diffusi nella ricerca educativa italiana. Il contributo presenta quattro software utilizzati nella ricerca educativa internazionale e valuta le funzionalità generali e l’usabilità di caratteristiche specifiche per condurre le fasi di screening e di selezione degli studi. A questo scopo lo studio impiega i metodi dell’analisi delle caratteristiche (Kitchenham et al., 1997) e del sondaggio ad esperti in revisioni sistematiche (Harrison et al., 2020) in campo educativo. I risultati di entrambi i metodi di indagine concordano nel ritenere Covidence e Rayyan gli strumenti più funzionali per condurre revisioni sistematiche, mentre ASReview risulta il software con maggiore potenzialità per rendere il processo più efficiente.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, automation can be used to accelerate several steps in the process, particularly the screening phase [1, 13, 16, 23, 26, 31, 33, 34, 36, 43, 45, 48, 51].”
From The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses (2024) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092 · cited 2×
“I ricercatori possono ordinare gli studi sulla base di tale valutazione, ricalcolando il computing rating man mano che prosegue lo screening, riuscendo così a risparmiare oltre il 50% del tempo (Ouzzani et al., 2016).”
BackgroundA typology of reviews: an analysis of 14 review types and associated methodologies
Grant, M. J., & Booth, A. (2009) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2009.00848.x · Cited by 10,817 · cited 1×
“Altri ancora permettono di accorciare, per quanto possibile (Grant & Booth, 2009), le lunghe e dispendiose fasi di screening di titoli e abstract, come il software ASReview, oppure lo screening dei full-text, come per esempio JBI SUMARI.”
BackgroundThe Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · cited 1×
“In seguito, nella fase di full-text review gli articoli che hanno superato la fase di screening sono letti interamente da uno o più ricercatori e inclusi nella SR sulla base dei criteri di inclusione stabiliti a priori nel protocollo di sintesi (Cooper et al., 2019; Pellegrini & Vivanet, 2021).”
BackgroundKohl, C., McIntosh, E. J., Unger, S., Haddaway, N. R., Kecke, S., Schiemann, J., & Wilhelm, R. (2018) · Environmental Evidence · doi:10.1186/s13750-018-0115-5 · Cited by 471 · cited 3×
“Covidence Covidence è un software creato da un’organizzazione australiana non-profit che, in partnership con la Cochrane Collaboration, offre un servizio per migliorare la progettazione e la conduzione di più fasi di una SR (Kohl et al., 2018).”
BackgroundHarrison, H., Griffin, S. J., Kuhn, I., & Usher-Smith, J. A. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-0897-3 · Cited by 449
Kellermeyer, L., Harnke, B., & Knight, S. (2018) · Journal of the Medical Library Association · doi:10.5195/jmla.2018.513 · Cited by 380 · cited 3×
“Seppur nati in seno alla tradizione dell’evidence-based medicine, approccio longevo e consolidato (Cottini & Morganti, 2016), i software si profilano come sostegno e guida interdisciplinare, adatti a qualsiasi campo di ricerca (Kellermeyer et al., 2018; Kohl et al., 2018).”
BackgroundHunt, H., Pollock, A., Campbell, P., Estcourt, L., & Brunton, G. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0695-8 · Cited by 314
Johnson, N., & Phillips, M. (2018) · Journal of Electronic Resources Librarianship · doi:10.1080/1941126x.2018.1444339 · Cited by 252 · cited 1×
“Rayyan Rayyan è un software progettato e realizzato dalla fondazione Quatar Computing Research Institute e si configura come strumento completamente gratuito (Johnson & Phillips, 2018).”
MethodDESMET: a methodology for evaluating software engineering methods and tools
Kitchenham, B., Linkman, S., & Law, D. (1997) · Computing & Control Engineering Journal · doi:10.1049/cce:19970304 · Cited by 247 · cited 3×
“A questo scopo lo studio impiega i metodi dell’analisi delle caratteristiche (Kitchenham et al., 1997) e del sondaggio ad esperti in revisioni sistematiche (Harrison et al., 2020) in campo educativo.”
BackgroundMethodInfluentialHow evidence-based reform will transform research and practice in education
Slavin, R. E. (2020) · Educational Psychologist · doi:10.1080/00461520.2019.1611432 · Cited by 220
Research works printed but not linked: 2
- Calvani, A. (2013). Evidence Based (Informed?) Education: neopositivismo ingenuo o opportunità epistemologica? Form@re -Open Journal per la formazione in rete, 13(2), 91-101.
- Marshall, C., & Brereton, P. (2015). Systematic review catalogue of tools to support systematic reviews. Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, 1-16.
Cited by
Boetje, J., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02502-7 · Cited by 146
“However, automation can be used to accelerate several steps in the process, particularly the screening phase [1, 13, 16, 23, 26, 31, 33, 34, 36, 43, 45, 48, 51].”
Machine learning to optimize literature screening in medical guideline development
Harmsen, W., de Groot, J., Harkema, A., van Dusseldorp, I., de Bruin, J., van den Brand, S., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02590-5 · Cited by 21
Gianvincenzi, M., Mosconi, E. M., Marconi, M., & Tola, F. (2024) · Recycling · doi:10.3390/recycling9010013 · Cited by 25
Systematic Review of Battery Life Cycle Management: A Framework for European Regulation Compliance
Gianvincenzi, M., Marconi, M., Mosconi, E. M., Favi, C., & Tola, F. (2024) · Sustainability · doi:10.3390/su162210026 · Cited by 26
Simulation-based Active Learning for Systematic Reviews: A Scoping Review of Literature
Teijema, J. J., Ribeiro, G., Seuren, S., Anadria, D., Bagheri, A., & van de Schoot, R. (2023) · doi:10.31234/osf.io/67zmt · Cited by 10
“Active learning is already incorporated into numerous software tools[7, 5, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18] 1 , and there’s an extensive field where the performance of active learning is assessed using simulation studies.”
Large-scale simulation study of active learning models for systematic reviews
Teijema, J. J., de Bruin, J., Bagheri, A., & van de Schoot, R. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00777-0 · Cited by 13
Systematic Literature Review on Open Educational Resources Recommender Systems
Rima, S., Hnida, M., Daoudi, N., & Ajhoun, R. (2022) · International Journal of Interactive Mobile Technologies (iJIM) · doi:10.3991/ijim.v16i18.32197 · Cited by 18
Boetje, J., & van de Schoot, R. (2023) · doi:10.31234/osf.io/c93gq · Cited by 10
Redefining design features for mitigating performance gaps in higher education buildings
Nazeer, F. S., Kamardeen, I., & Hasan, A. (2025) · Smart and Sustainable Built Environment · doi:10.1108/sasbe-02-2025-0059 · Cited by 9
Large-Scale Simulation Study of Active Learning Models for Systematic Reviews
Teijema, J. J., de Bruin, J., Bagheri, A., & van de Schoot, R. (2023) · doi:10.31234/osf.io/2w3rm · Cited by 8
Bellacicco, R., Capone, F., Sorrentino, C., & Martino, V. D. (2025) · Education Sciences · doi:10.3390/educsci15010047 · Cited by 27
“After downloading from databases and search engines, citations were uploaded into Covidence, a web-based collaboration software platform for conducting systematic reviews as a team (Pellegrini & Marsili, 2021).”
Kao, H.-E. (2024) · Annual International Conference on Digital Government Research · doi:10.1145/3657054.3657121 · Cited by 2
“Same result also shows on the research by Pellegrini & Marsili (2021) [26] using feature analysis and expert survey to compare four different literature review software: while the Rayyan and Covidence have more features, ASReview has the greatest potential for making the process of a SR more…”
MethodSimulation-based active learning for systematic reviews: A scoping review of literature
Teijema, J. J., Ribeiro, G., Seuren, S., Anadria, D., Bagheri, A., & van de Schoot, R. (2025) · Journal of Information Science · doi:10.1177/01655515251379058 · Cited by 1
李, 静. (2025) · E-Commerce Letters · doi:10.12677/ecl.2025.1462066
Femminini, A., Salerni, A., & Stanzione, I. (2025) · EDUCATION SCIENCES AND SOCIETY · doi:10.3280/ess1-2025oa19732
Boetje, J., & van de Schoot, R. (2024) · Figshare · doi:10.6084/m9.figshare.26687522
Zappa, C. (2026) · Communications in computer and information science · doi:10.1007/978-3-032-31850-3_2
Metadata
- Authors
- Pellegrini, M., & Marsili, F.
- Affiliations
- University of Florence (IT)
- University of Perugia (IT)
- Venue
- Form@re - Open Journal per la formazione in rete
- Publisher
- Firenze University Press
- Published
- 2021-07-31
- Volume
- 21
- Issue
- 2
- Pages
- 124-140
- ISSN
- 1825-7321 (ISSN-L)
- OpenAlex
- W3193973034
- Cited by
- 17
- References
- 22
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 6 · 2023: 3 · 2022: 1
- FWCI
- 0.10
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Form re - Open Journal per la formazione in rete (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Florence Research (University of Florence) (submittedVersion, cc-by-nc-nd)
- UNICA IRIS Institutional Research Information System (University of Cagliari) (submittedVersion, other-oa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Educational Research and Methods
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- systematic review software, software evaluation, feature analysis, usability, study screening, study selection, Covidence, Rayyan, ASReview, educational research, evidence-based decision making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.