Choosing the Right Tool for the Job: Screening Tools for Systematic Reviews in Education
Abstract
In recent years, the rapid development of artificial intelligence has enabled the launch of many new screening tools. This review aims to facilitate screening tool selection through a systematic narrative review and feature analysis. The current adoption rate of transparent tool reporting is low: by screening 191 studies published in the Review of Educational Research since 2015, we found that only eight studies reported screening tools. More research is needed to understand the reasons behind this phenomenon. After consulting various sources, 26 available screening tools in the market were found. Among them, we identified and evaluated 12 screening tools for educational reviewers and ranked them in descending order of feature score: Covidence (1), DistillerSR (2, tied), EPPI-Reviewer (2, tied), CADIMA (4), Swift-Active (5), Rayyan (6, tied), SysRev (6, tied), Abstrackr (8, tied), ReLiS (8, tied), RevMan (8, tied), ASReview (11), and Excel (12). In the discussion, we provide insights into the promise and bias in tools’ machine learning algorithms. Our results encourage researchers to report their tool usage in publications and select tools based on suitability instead of convenience.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Machine learning could reveal objective skills indicators, and AI-based assessments demonstrated high levels of accuracy.
“Zhang et al. (2023) found that machine learning could reveal objective skills indicators and Kirubarajan et al. (2022) found that AI-based assessments demonstrated high levels of accuracy.”
From A meta systematic review of artificial intelligence in higher education: a call for increased ethics, collaboration, and rigour (2024) · cites this paper as Zhang et al., 2023
Cites
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092 · see citing works
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157 · see citing works
Harrison, H., Griffin, S. J., Kuhn, I., & Usher-Smith, J. A. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-0897-3 · Cited by 449 · see citing works
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · see citing works
Surr, C. A., Gates, C., Irving, D., Oyebode, J., Smith, S. J., Parveen, S., Drury, M., & Dennison, A. (2017) · Review of Educational Research · doi:10.3102/0034654317723305 · Cited by 327 · see citing works
McKeown, S., & Mir, Z. M. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01583-y · Cited by 251 · see citing works
Error rates of human reviewers during abstract screening in systematic reviews
Wang, Z., Nayfeh, T., Tetzlaff, J., O’Blenis, P., & Murad, M. H. (2020) · PLOS ONE · doi:10.1371/journal.pone.0227742 · Cited by 217 · see citing works
Rathbone, J., Hoffmann, T., & Glasziou, P. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0067-6 · Cited by 184 · cited 1× · see citing works
“The additional functions enabled by these semi-automated software tools hold promise to save a tremendous amount of time and labor in evidence-based synthesis (Ouzzani et al., 2016; Rathbone et al., 2015) if selected and used appropriately.”
Rowan, L., Bourke, T., L’Estrange, L., Lunn Brownlee, J., Ryan, M., Walker, S., & Churchward, P. (2021) · Review of Educational Research · doi:10.3102/0034654320979171 · Cited by 176 · see citing works
Gates, A., Johnson, C., & Hartling, L. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0707-8 · Cited by 148 · cited 1× · see citing works
“Another method is to replace one of the two human screeners with the machine screener to increase reliability since both human and machine screeners are imperfect (Gates et al., 2018).”
Gates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1222-2 · Cited by 133 · see citing works
Sysrev: A FAIR Platform for Data Curation and Systematic Evidence Review
Bozada, T., Borden, J., Workman, J., Del Cid, M., Malinowski, J., & Luechtefeld, T. (2021) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2021.685298 · Cited by 118 · see citing works
Gaining Insight Into Human Nature: A Review of Literature Classroom Intervention Studies
Schrijvers, M., Janssen, T., Fialho, O., & Rijlaarsdam, G. (2019) · Review of Educational Research · doi:10.3102/0034654318812914 · Cited by 76 · see citing works
Braunack-Mayer, A. J., Street, J. M., Tooher, R., Feng, X., & Scharling-Gamba, K. (2020) · Review of Educational Research · doi:10.3102/0034654320960213 · Cited by 41 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Bond, M., Khosravi, H., De Laat, M., Bergdahl, N., Negrea, V., Oxley, E., Pham, P., Chong, S. W., & Siemens, G. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00436-z · Cited by 852 · see citing works
“Zhang et al. (2023) found that machine learning could reveal objective skills indicators and Kirubarajan et al. (2022) found that AI-based assessments demonstrated high levels of accuracy.”
BackgroundMethodInfluentialZhang, Q., Zhang, R., Xiong, Y., Sui, Y., Tong, C., & Lin, F.-H. (2025) · Journal of Medical Internet Research · doi:10.2196/78238 · Cited by 56 · see citing works
“In conducting the screening process, we used Covidence software owing to its robust full-text review features and the provision of complimentary licenses available via our affiliated institutions [32].”
Self-citation
- Authors
- Zhang, Q., & Neitzel, A.
- Affiliations
- Johns Hopkins University (US)
- Corresponding author
- Zhang, Qiyang
- Venue
- Journal of Research on Educational Effectiveness
- Publisher
- Informa UK Limited
- Published
- 2023-05-11
- Volume
- 17
- Issue
- 3
- Pages
- 513-539
- ISSN
- 1934-5739 (ISSN-L)
- 1934-5747
- OpenAlex
- W4376129636
- Cited by
- 28
- References
- 14
- Citations by year
- 2026: 4 · 2025: 17 · 2024: 6 · 2023: 1
- FWCI
- 1.96
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Journal of Research on Educational Effectiveness (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Education › Educational Research and Methods
- Social Sciences › Psychology › Experimental and Cognitive Psychology › Psychology Research and Bibliometrics
- Keywords
- screening tool, tool selection, machine learning bias, Covidence, Rayyan, ASReview, feature analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
