Research Screener: a machine learning tool to semi-automate abstract screening for systematic reviews
Open accessThe DOI matches its Crossref record; 56 of the 57 references it cites were found in a registry or an index.
Abstract
Background: Systematic reviews and meta-analyses provide the highest level of evidence to help inform policy and practice, yet their rigorous nature is associated with significant time and economic demands. The screening of titles and abstracts is the most time consuming part of the review process with analysts required review thousands of articles manually, taking on average 33 days. New technologies aimed at streamlining the screening process have provided initial promising findings, yet there are limitations with current approaches and barriers to the widespread use of these tools. In this paper, we introduce and report initial evidence on the utility of Research Screener, a semi-automated machine learning tool to facilitate abstract screening. Methods: Three sets of analyses (simulation, interactive and sensitivity) were conducted to provide evidence of the utility of the tool through both simulated and real-world examples. Results: Research Screener delivered a workload saving of between 60 and 96% across nine systematic reviews and two scoping reviews. Findings from the real-world interactive analysis demonstrated a time saving of 12.53 days compared to the manual screening, which equates to a financial saving of USD 2444. Conservatively, our results suggest that analysts who scan 50% of the total pool of articles identified via a systematic search are highly likely to have identified 100% of eligible papers. Conclusions: In light of these findings, Research Screener is able to reduce the burden for researchers wishing to conduct a comprehensive systematic review without reducing the scientific rigour for which they strive to achieve.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, these tools, sometimes referred to as discriminative artificial intelligence (AI) technologies, generally require human researchers providing large amount of sufficiently diverse data for training and active learning [1, 3, 5, 6].”
From Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis (2025) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In summary, ML-assisted systematic reviews can significantly reduce workload and have proven accurate across various fields and review topics.”
From Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics (2025) · cites this paper as Chai et al. (2021)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Traditional systematic reviews, despite their high-quality evidence, are labor-intensive and error-prone, especially during the abstract screening phase.”
From Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics (2025) · cites this paper as Chai et al., 2021
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Screening titles and abstracts is a critical component of the review process (Chai et al., 2021).”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Chai et al., 2021
The cited source says what the claim attributes to it.
In the citing paper
“However, these tools are still evolving, and their performance may vary across domains and even within the same domain (Burgard & Bittermann, 2023; Chai et al., 2021; van de Schoot et al., 2021).”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Chai et al., 2021
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
Advances in neural information processing systems 7
Cios, K., & Shields, M. (1997) · Neurocomputing · doi:10.1016/s0925-2312(97)90018-7 · Cited by 22,059
Distributed Representations of Sentences and Documents
Le, Q. V., & Mikolov, T. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1405.4053 · Cited by 4,934
De-duplication of database search results for systematic reviews in EndNote
Bramer, W. M., Giustini, D., de Jonge, G. B., Holland, L., & Bekhuis, T. (2016) · Journal of the Medical Library Association : JMLA · doi:10.3163/1536-5050.104.3.014 · Cited by 2,103 · cited 1×
“Using a traditional manual approach [6], the de-duplication took approximately 7 h to complete, though with Research Screener this process was completed in a matter of seconds.”
MethodSeventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157
Munn, Z., Aromataris, E., Tufanaru, C., Stern, C., Porritt, K., Farrow, J., Lockwood, C., Stephenson, M., Moola, S., Lizarondo, L., et al. (2019) · International Journal of Evidence-Based Healthcare · doi:10.1097/xeb.0000000000000152 · Cited by 977
Ambati, B. R., Deoskar, T., Johnson, M., & Steedman, M. J. (2015) · The Association for Computational Linguistics · Cited by 860
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 7×
“Text mining is defined as ‘ the process of discovering knowledge and structure from unstructured data (i.e. text) ’ ([30]; p. 2).”
BackgroundMethodInfluentialMarshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638
Deploying an interactive machine learning system in an evidence-based practice center: abstrackr
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012) · Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium · doi:10.1145/2110363.2110464 · Cited by 532
Registration of systematic reviews in PROSPERO: 30,000 records and counting
Page, M. J., Shamseer, L., & Tricco, A. C. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0699-4 · Cited by 509
Systematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472 · cited 1×
“A review of automation technologies identified 14 tasks within the review process which have the potential to be automated (e.g. de-duplication and screening titles and abstracts [39]).”
BackgroundKohl, C., McIntosh, E. J., Unger, S., Haddaway, N. R., Kecke, S., Schiemann, J., & Wilhelm, R. (2018) · Environmental Evidence · doi:10.1186/s13750-018-0115-5 · Cited by 471 · cited 1×
“For example, a recent review of tools developed to support researchers through the review process identified 20 packages (of 22) that included help for the article screening stage [17].”
MethodHarrison, H., Griffin, S. J., Kuhn, I., & Usher-Smith, J. A. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-0897-3 · Cited by 449
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 1×
“Performance was assessed using work saved over random sampling (WSS [11]), which provides an indication of the time saved over manual screening in the form a percentage reduction in the number of papers needed to be screened.”
MethodBest practice guidelines for abstract screening large‐evidence systematic reviews and meta‐analyses
Polanin, J. R., Pigott, T. D., Espelage, D. L., & Grotpeter, J. K. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1354 · Cited by 401
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Semi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · cited 2×
“In a simulation study over two data sets, Abstrackr demonstrated a similar saving with WSS@100, whereby a workload saving of approaching 50% was found [43].”
MethodReviewing retrieved references for inclusion in systematic reviews using EndNote
Bramer, W. M., Milic, J., & Mast, F. (2017) · Journal of the Medical Library Association · doi:10.5195/jmla.2017.111 · Cited by 264 · cited 1×
“Past work suggests that it takes be-tween 30 [7] and 60 s [38] to screen each abstract.”
BackgroundPreoperative exercise training for patients with non-small cell lung cancer
Cavalheri, V., & Granger, C. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd012020.pub2 · Cited by 260
Reducing systematic review workload through certainty-based screening
Miwa, M., Thomas, J., O’Mara-Eves, A., & Ananiadou, S. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.06.005 · Cited by 190
Bannach-Brown, A., Przybyła, P., Thomas, J., Rice, A. S. C., Ananiadou, S., Liao, J., & Macleod, M. R. (2019) · Systematic Reviews · doi:10.1186/s13643-019-0942-7 · Cited by 185 · cited 3×
“Machine learning approaches to the facilitation of title and abstract screening have been examined within many scientific disciplines, such as public health [24], animal studies [3] and genetics [42].”
BackgroundRathbone, J., Hoffmann, T., & Glasziou, P. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0067-6 · Cited by 184 · cited 1×
“In two retrospective, independent evaluations of Abstrackr ’ s performance, the workload savings varied between 9 and 57% [36], and 9.5 and 88.4% [14].”
BackgroundInfluentialShemilt, I., Simon, A., Hollands, G. J., Marteau, T. M., Ogilvie, D., O'Mara‐Eves, A., Kelly, M. P., & Thomas, J. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1093 · Cited by 170
Shemilt, I., Khan, N., Park, S., & Thomas, J. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0315-4 · Cited by 168
O’Connor, A. M., Tsafnat, G., Thomas, J., Glasziou, P., Gilbert, S. B., & Hutton, B. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1062-0 · Cited by 167 · cited 1×
“For such a tool to be considered seriously by researchers, clinicians and policy-makers, they require evidence to support the effectiveness of the semi-automation tool and the potential time savings associated with its adoption [28].”
BackgroundEstimating Time to Conduct a Meta-analysis From Number of Citations Retrieved
Allen, I. E. (1999) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.282.7.634 · Cited by 163 · cited 1×
“Towards the end of the twentieth century, meta-analyses took on average around 1139 h to complete [1].”
BackgroundKite, C., Lahart, I. M., Afzal, I., Broom, D. R., Randeva, H., Kyrou, I., & Brown, J. E. (2019) · Systematic Reviews · doi:10.1186/s13643-019-0962-3 · Cited by 155
Michelson, M., & Reuter, K. (2019) · Contemporary Clinical Trials Communications · doi:10.1016/j.conctc.2019.100443 · Cited by 152
Gates, A., Johnson, C., & Hartling, L. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0707-8 · Cited by 148 · cited 5×
“This stage of a systematic review involves reviewers screening the titles and abstracts of papers identified from the initial search — oftentimes in the tens of thousands [30] — to make an assessment on its relevance for inclusion, which typically takes around 30 s per paper [14].”
BackgroundMethodInfluentialPrioritising references for systematic reviews with RobotAnalyst: A user study
Przybyła, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M., McNaught, J., von Elm, E., Nolan, K., & Ananiadou, S. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1311 · Cited by 148 · cited 5×
“For example, ASReviewer provides a mean saving of 83% over four reviews [49], Rayyan demonstrates a mean workload saving of 49% over 15 reviews [31], and RobotAnalyst provides a mean workload saving of 43% [35].”
BackgroundResultInfluentialCavalheri, V., Burtin, C., Formico, V. R., Nonoyama, M. L., JENKINS, S. C., Spruit, M. A., & Hill, K. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009955.pub3 · Cited by 144
Lorenzetti, D., & Ghali, W. A. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-141 · Cited by 142
Cavalheri, V., Tahirah, F., Nonoyama, M. L., Jenkins, S., & Hill, K. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009955.pub2 · Cited by 134
Gates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1222-2 · Cited by 133 · cited 2×
“There is a growing interest in tools to facilitate and speed up the screening process (e.g. [3, 13, 21, 28]).”
BackgroundMethodPredicting the time needed for environmental systematic reviews and systematic maps
Haddaway, N. R., & Westgate, M. J. (2019) · Conservation Biology · doi:10.1111/cobi.13231 · Cited by 120
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., Bruin, J. D., Schram, R. D., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B. M. R., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Utrecht University Repository (Utrecht University) · Cited by 106
Can abstract screening workload be reduced using text mining? User experiences of the tool Rayyan
Olofsson, H., Brolund, A., Hellberg, C., Silverstein, R., Stenström, K., Österberg, M., & Dagerhamn, J. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1237 · Cited by 102
Saraceni, N., Kent, P. M., Ng, L., Campbell, A., Straker, L. M., & O’Sullivan, P. (2019) · Journal of Orthopaedic and Sports Physical Therapy · doi:10.2519/jospt.2020.9218 · Cited by 97
Self-citationO’Connor, A. M., Tsafnat, G., Gilbert, S. B., Thayer, K. A., & Wolfe, M. S. (2018) · Systematic Reviews · doi:10.1186/s13643-017-0667-4 · Cited by 69
Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., Schmid, C. H., Bertram, L., Lill, C. M., Cohen, J. T., & Trikalinos, T. A. (2012) · Genetics in Medicine · doi:10.1038/gim.2012.7 · Cited by 62
Sport, Exercise, and Performance Psychology.
Martin, J. J. (2015) · Sport Exercise and Performance Psychology · doi:10.1037/spy0000035 · Cited by 57
Systematic review toolbox: a catalogue of tools to support systematic reviews
Marshall, C., & Brereton, P. (2015) · Proceedings of the 19th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2745802.2745824 · Cited by 44
Automated screening of research studies for systematic reviews using study characteristics
Tsafnat, G., Glasziou, P., Karystianis, G., & Coiera, E. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0724-7 · Cited by 44 · cited 2×
“Machine learning utilises the mined text for training algorithms to create a classifier model based upon reviewers ’ inclusion and exclusion decisions [40].”
BackgroundMethodGray, C., Crawford, G., Maycock, B. R., & Lobo, R. C. (2021) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph18052469 · Cited by 33
Editorial: Systematic review automation thematic series
Lau, J. (2019) · Systematic Reviews · doi:10.1186/s13643-019-0974-z · Cited by 27
Burton, E., Farrier, K., Lewin, G. F., Petrich, M., Boyle, E., & Hill, K. (2018) · Disability and Rehabilitation · doi:10.1080/09638288.2018.1508509 · Cited by 24
Assessing the landscape of US postdoctoral salaries
Athanasiadou, R., Bankston, A., Carlisle, M., Niziolek, C. A., & McDowell, G. S. (2018) · Studies in Graduate and Postdoctoral Education · doi:10.1108/sgpe-d-17-00048 · Cited by 21
Lines, R. L. J., Pietsch, S., Crane, M. F., Ntoumanis, N., Temby, P., Graham, S., & Gucciardi, D. F. (2021) · Sport Exercise and Performance Psychology · doi:10.1037/spy0000251 · Cited by 19
Self-citationZabatiero, J. G., Ng, L., Clayton, R., Middlemiss, S., Kang, K., Harrold, M., & Cavalheri, V. (2018) · Obesity Research & Clinical Practice · doi:10.1016/j.orcp.2018.10.004 · Cited by 13
Self-citation
Cited by
The role of digitalization in business and management: a systematic literature review
Calderon-Monge, E., & Ribeiro-Soriano, D. (2024) · Review of Managerial Science · doi:10.1007/s11846-023-00647-8 · Cited by 233
Artificial intelligence for literature reviews: opportunities and challenges
Bolaños, F., Salatino, A., Osborne, F., & Motta, E. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10902-3 · Cited by 246
Automated Paper Screening for Clinical Reviews Using Large Language Models
Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C. (2023) · arXiv · doi:10.48550/arxiv.2305.00844 · Cited by 230
Automated paper screening for clinical reviews using large language models: Data analysis study
Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C. (2024) · Journal of Medical Internet Research · doi:10.2196/48996 · Cited by 205
“This is especially problematic for small research groups, review projects with broad inclusion criteria (such as network meta-analyses), or time-constrained review projects (such as reviews relating to COVID-19 during the early stages of the pandemic) [7].”
Tools to support the automation of systematic reviews: a scoping review.
Khalil, H., Ameen, D., & Zarnegar, A. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.12.005 · Cited by 190
Metadata
- Authors
- Chai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L.
- Affiliations
- Curtin University (AU)
- Corresponding author
- Ng, Leo
- Venue
- Systematic Reviews
- Published
- 2021-04-01
- Volume
- 10
- Issue
- 1
- Pages
- 93-93
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC8017894
- OpenAlex
- W3143985904
- Cited by
- 190
- References
- 57
- Citations by year
- 2026: 38 · 2025: 50 · 2024: 43 · 2023: 40 · 2022: 13 · 2021: 6
- FWCI
- 1.06
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- abstract screening, machine learning, semi-automated screening, scoping review, workload reduction, eligibility screening
- MeSH terms
- Machine Learning, Systematic Reviews as Topic, Humans, Mass Screening, Research, Workload
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

