Evaluating software tools to conduct systematic reviews: a feature analysis and user survey
Open accessThe DOI matches its Crossref record; 21 of the 22 references it cites were found in a registry or an index.
Abstract
Systematic reviews are research synthesis methods increasingly used in educational research to support evidence-based decision making. The conduction of a systematic review is a complex process with several phases usually supported by software tools. These tools used at the international level are not currently common in the Italian educational research. This work describes four software tools used in the international educational research and evaluates their general functionality and specific features’ usability to conduct the systematic review phases of study screening and selection. For this purpose, this study uses two methods: a feature analysis (Kitchenham et al., 1997) and an expert survey (Harrison et al., 2020). The results of both investigation methods agree to consider Covidence and Rayyan the most functional software tools in conducting SR. Among the four tools, ASReview has the greatest potential for making the process of a SR more efficient. Una valutazione dei software per condurre revisioni sistematiche: analisi delle caratteristiche e sondaggio a esperti Le revisioni sistematiche sono metodi di sintesi di ricerca sempre più utilizzati in campo educativo al fine di supportare il processo decisionale basato su evidenze. La conduzione di una revisione sistematica, poiché complessa e su più fasi, è sovente supportata in ambito internazionale da software specifici attualmente poco diffusi nella ricerca educativa italiana. Il contributo presenta quattro software utilizzati nella ricerca educativa internazionale e valuta le funzionalità generali e l’usabilità di caratteristiche specifiche per condurre le fasi di screening e di selezione degli studi. A questo scopo lo studio impiega i metodi dell’analisi delle caratteristiche (Kitchenham et al., 1997) e del sondaggio ad esperti in revisioni sistematiche (Harrison et al., 2020) in campo educativo. I risultati di entrambi i metodi di indagine concordano nel ritenere Covidence e Rayyan gli strumenti più funzionali per condurre revisioni sistematiche, mentre ASReview risulta il software con maggiore potenzialità per rendere il processo più efficiente.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“However, automation can be used to accelerate several steps in the process, particularly the screening phase [1, 13, 16, 23, 26, 31, 33, 34, 36, 43, 45, 48, 51].”
From The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses (2024) · cites this paper as [31]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092 · cited 2×
“I ricercatori possono ordinare gli studi sulla base di tale valutazione, ricalcolando il computing rating man mano che prosegue lo screening, riuscendo così a risparmiare oltre il 50% del tempo (Ouzzani et al., 2016).”
BackgroundA typology of reviews: an analysis of 14 review types and associated methodologies
Grant, M. J., & Booth, A. (2009) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2009.00848.x · Cited by 10,817 · cited 1×
“Altri ancora permettono di accorciare, per quanto possibile (Grant & Booth, 2009), le lunghe e dispendiose fasi di screening di titoli e abstract, come il software ASReview, oppure lo screening dei full-text, come per esempio JBI SUMARI.”
BackgroundThe Handbook of Research Synthesis and Meta-Analysis
(2019) · doi:10.7758/9781610448864 · Cited by 4,553 · cited 1×
“In seguito, nella fase di full-text review gli articoli che hanno superato la fase di screening sono letti interamente da uno o più ricercatori e inclusi nella SR sulla base dei criteri di inclusione stabiliti a priori nel protocollo di sintesi (Cooper et al., 2019; Pellegrini & Vivanet, 2021).”
BackgroundKohl, C., McIntosh, E. J., Unger, S., Haddaway, N. R., Kecke, S., Schiemann, J., & Wilhelm, R. (2018) · Environmental Evidence · doi:10.1186/s13750-018-0115-5 · Cited by 471 · cited 3×
“Covidence Covidence è un software creato da un’organizzazione australiana non-profit che, in partnership con la Cochrane Collaboration, offre un servizio per migliorare la progettazione e la conduzione di più fasi di una SR (Kohl et al., 2018).”
BackgroundHarrison, H., Griffin, S. J., Kuhn, I., & Usher-Smith, J. A. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-0897-3 · Cited by 449
Kellermeyer, L., Harnke, B., & Knight, S. (2018) · Journal of the Medical Library Association · doi:10.5195/jmla.2018.513 · Cited by 380 · cited 3×
“Seppur nati in seno alla tradizione dell’evidence-based medicine, approccio longevo e consolidato (Cottini & Morganti, 2016), i software si profilano come sostegno e guida interdisciplinare, adatti a qualsiasi campo di ricerca (Kellermeyer et al., 2018; Kohl et al., 2018).”
BackgroundHunt, H., Pollock, A., Campbell, P., Estcourt, L., & Brunton, G. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0695-8 · Cited by 314
Johnson, N., & Phillips, M. (2018) · Journal of Electronic Resources Librarianship · doi:10.1080/1941126x.2018.1444339 · Cited by 252 · cited 1×
“Rayyan Rayyan è un software progettato e realizzato dalla fondazione Quatar Computing Research Institute e si configura come strumento completamente gratuito (Johnson & Phillips, 2018).”
MethodDESMET: a methodology for evaluating software engineering methods and tools
Kitchenham, B., Linkman, S., & Law, D. (1997) · Computing & Control Engineering Journal · doi:10.1049/cce:19970304 · Cited by 247 · cited 3×
“A questo scopo lo studio impiega i metodi dell’analisi delle caratteristiche (Kitchenham et al., 1997) e del sondaggio ad esperti in revisioni sistematiche (Harrison et al., 2020) in campo educativo.”
BackgroundMethodInfluentialHow evidence-based reform will transform research and practice in education
Slavin, R. E. (2020) · Educational Psychologist · doi:10.1080/00461520.2019.1611432 · Cited by 220
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., Bruin, J. D., Schram, R. D., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B. M. R., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Utrecht University Repository (Utrecht University) · Cited by 106
Lorenzetti, D. L., & Ghali, W. A. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-141 · Cited by 102
Software Tools for Literature Screening in Systematic Reviews in Biomedical Research
van der Mierden, S. (2019) · ALTEX · doi:10.14573/altex.1902131 · Cited by 99 · cited 2×
“Questi risultati sono inoltre in linea con le opinioni espresse da ricercatori in ambito medico (Harrison et al., 2020; Van der Mierden et al., 2019).”
BackgroundEvidence Based Education e modelli di valutazione formativa per le scuole
Calvani, A., & Vivanet, G. (2014) · Journal of Educational Cultural and Psychological Studies (ECPS Journal) · doi:10.7358/ecps-2014-009-calv · Cited by 85 · cited 1×
“Questi metodi hanno iniziato a diffondersi nella ricerca educativa italiana grazie al pionieristico volume di Di Nuovo del 1995 e al lavoro di altri ricercatori in diversi ambiti delle scienze sociali (Calvani & Vivanet, 2014; Crocetti, 2015).”
BackgroundBedenlier, S., Bond, M., Buntins, K., Zawacki‐Richter, O., & Kerres, M. (2019) · Springer eBooks · doi:10.1007/978-3-658-27602-7_7 · Cited by 51
Active learning for screening prioritization in systematic reviews - A simulation study
Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., & van de Schoot, R. (2020) · doi:10.31219/osf.io/w6qbg · Cited by 51
Evidence-based education e Pedagogia speciale. Principi e modelli per l'inclusione.
Cottini, L., & Morganti, A. (2015) · Cited by 16
Sintesi di ricerca in educazione. Basi teoriche e metodologiche.
Pellegrini, M., & Vivanet, G. (2018) · Florence Research (University of Florence) · Cited by 9
Self-citationNuovi orizzonti di ricerca in educazione speciale: le sintesi di sintesi
Marsili, F., Morganti, A., & Vivanet, G. (2020) · UNICA IRIS Institutional Research Information System (University of Cagliari) · doi:10.7346/sipes-01-2020-15 · Cited by 7
Self-citationRassegne sistematiche, sintesi della ricerca e meta-analisi
Crocetti, E. (2015) · Archivio istituzionale della ricerca (Alma Mater Studiorum Università di Bologna) · Cited by 5
Ferdinands, G. (2020) · OSF · doi:10.17605/osf.io/7mr2g · Cited by 2
Research works printed but not linked: 2
- Calvani, A. (2013). Evidence Based (Informed?) Education: neopositivismo ingenuo o opportunità epistemologica? Form@re -Open Journal per la formazione in rete, 13(2), 91-101.
- Marshall, C., & Brereton, P. (2015). Systematic review catalogue of tools to support systematic reviews. Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering, 1-16.
Cited by
Boetje, J., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02502-7 · Cited by 146
“However, automation can be used to accelerate several steps in the process, particularly the screening phase [1, 13, 16, 23, 26, 31, 33, 34, 36, 43, 45, 48, 51].”
Machine learning to optimize literature screening in medical guideline development
Harmsen, W., de Groot, J., Harkema, A., van Dusseldorp, I., de Bruin, J., van den Brand, S., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02590-5 · Cited by 21
Gianvincenzi, M., Mosconi, E. M., Marconi, M., & Tola, F. (2024) · Recycling · doi:10.3390/recycling9010013 · Cited by 25
Systematic Review of Battery Life Cycle Management: A Framework for European Regulation Compliance
Gianvincenzi, M., Marconi, M., Mosconi, E. M., Favi, C., & Tola, F. (2024) · Sustainability · doi:10.3390/su162210026 · Cited by 26
Simulation-based Active Learning for Systematic Reviews: A Scoping Review of Literature
Teijema, J. J., Ribeiro, G., Seuren, S., Anadria, D., Bagheri, A., & van de Schoot, R. (2023) · doi:10.31234/osf.io/67zmt · Cited by 10
“Active learning is already incorporated into numerous software tools[7, 5, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18] 1 , and there’s an extensive field where the performance of active learning is assessed using simulation studies.”
Metadata
- Authors
- Pellegrini, M., & Marsili, F.
- Affiliations
- University of Florence (IT)
- University of Perugia (IT)
- Venue
- Form@re - Open Journal per la formazione in rete
- Publisher
- Firenze University Press
- Published
- 2021-07-31
- Volume
- 21
- Issue
- 2
- Pages
- 124-140
- ISSN
- 1825-7321 (ISSN-L)
- OpenAlex
- W3193973034
- Cited by
- 17
- References
- 22
- Citations by year
- 2026: 1 · 2025: 6 · 2024: 6 · 2023: 3 · 2022: 1
- FWCI
- 0.10
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Form re - Open Journal per la formazione in rete (publishedVersion, cc-by-nc-nd)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Florence Research (University of Florence) (submittedVersion, cc-by-nc-nd)
- UNICA IRIS Institutional Research Information System (University of Cagliari) (submittedVersion, other-oa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Educational Research and Methods
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- systematic review software, software evaluation, feature analysis, usability, study screening, study selection, Covidence, Rayyan, ASReview, educational research, evidence-based decision making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.