Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index; 8 of the 8 sources its claims cite were checked for support.
Abstract
BACKGROUND: Improving the speed of systematic review (SR) development is key to supporting evidence-based medicine. Machine learning tools which semi-automate citation screening might improve efficiency. Few studies have assessed use of screening prioritization functionality or compared two tools head to head. In this project, we compared performance of two machine-learning tools for potential use in citation screening. METHODS: Using 9 evidence reports previously completed by the ECRI Institute Evidence-based Practice Center team, we compared performance of Abstrackr and EPPI-Reviewer, two off-the-shelf citations screening tools, for identifying relevant citations. Screening prioritization functionality was tested for 3 large reports and 6 small reports on a range of clinical topics. Large report topics were imaging for pancreatic cancer, indoor allergen reduction, and inguinal hernia repair. We trained Abstrackr and EPPI-Reviewer and screened all citations in 10% increments. In Task 1, we inputted whether an abstract was ordered for full-text screening; in Task 2, we inputted whether an abstract was included in the final report. For both tasks, screening continued until all studies ordered and included for the actual reports were identified. We assessed potential reductions in hypothetical screening burden (proportion of citations screened to identify all included studies) offered by each tool for all 9 reports. RESULTS: For the 3 large reports, both EPPI-Reviewer and Abstrackr performed well with potential reductions in screening burden of 4 to 49% (Abstrackr) and 9 to 60% (EPPI-Reviewer). Both tools had markedly poorer performance for 1 large report (inguinal hernia), possibly due to its heterogeneous key questions. Based on McNemar's test for paired proportions in the 3 large reports, EPPI-Reviewer outperformed Abstrackr for identifying articles ordered for full-text review, but Abstrackr performed better in 2 of 3 reports for identifying articles included in the final report. For small reports, both tools provided benefits but EPPI-Reviewer generally outperformed Abstrackr in both tasks, although these results were often not statistically significant. CONCLUSIONS: Abstrackr and EPPI-Reviewer performed well, but prioritization accuracy varied greatly across reports. Our work suggests screening prioritization functionality is a promising modality offering efficiency gains without giving up human involvement in the screening process.
Overview
CoChat Summary
## TL;DR
This study compared Abstrackr and EPPI-Reviewer for screening prioritization in 9 completed evidence reports and found that both tools could substantially reduce screening burden, especially in larger reviews. Performance varied widely by report, with EPPI-Reviewer generally doing better for early identification in Task 1 and mixed results for final-inclusion predictions in Task 2.
## Key findings
- For the 3 large reports, potential reductions in screening burden were 4 to 49% with Abstrackr and 9 to 60% with EPPI-Reviewer.
- In one large report, indoor allergen reduction, nearly all relevant citations were identified by the time 50% of citations had been screened; screening burden was 39.9% for EPPI-Reviewer and 51.5% for Abstrackr.
- In the pancreatic cancer imaging report, EPPI-Reviewer identified all abstracts ordered for full text 868 citations earlier than Abstrackr, with a potential screening-burden reduction of nearly 10%.
- In 4 of 6 small reports, EPPI-Reviewer identified all included citations earlier than Abstrackr, offering reductions in screening burden of 30 to 70%.
## Methods
- Comparative study using citations from 9 previously completed evidence reports: 3 large and 6 small reports across clinical topics; the large reports were pancreatic cancer imaging, indoor allergen reduction, and inguinal hernia repair.
- The tools were trained on the same citation sets and evaluated in 10% increments; Task 1 used decisions about whether an abstract was ordered for full-text screening, and Task 2 used whether an abstract was included in the final report.
- Main analyses used hypothetical screening burden, percentile rank/sensitivity, and McNemar’s test for paired proportions; the dataset included 19,149 citations, minus 505 duplicates and 363 citations excluded for publication year, leaving 18,281 articles.
## Limitations
- The paper states that performance may vary depending on the composition of the initial training set; different random training samples could have produced different results.
- The authors did not use some tool features that might improve performance in real-world use, such as keyword identification and automatic classification.
- Evident from the text: the study measured potential reductions in screening burden under retrospective conditions, and the paper notes these may not translate to actual reductions in real-time use because reviewers cannot know in advance whether performance will be good or bad.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures











Verification
Claims the paper makes
4 Valid · 1 Not valid · 0 Not checked · 48 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Many studies evaluating these tools have assessed automatic classification functionality, in which screening continued until the tool predicts whether studies should be included or not, and remaining citations are not screened [19, 21].”
§ Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer > Discussion > Encouraging wider adoption
What the sources say
- Supportedcites [19] Gates A, Johnson C, Hartling L. Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool. Syst Rev. 2018;7 [cited 2019 Oct 10] Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5848519/. source
In Methods > Data collection (page unavailable), the authors say they screened records until Abstrackr produced relevance predictions, then discontinued screening. The Discussion also identifies earlier evaluations of Abstrackr’s prediction-based screening, supporting the sentence’s characterization of this evaluation approach. (LLM, conf=0.89)
- Partly supportedcites [21] Rathbone J, Hoffmann T, Glasziou P. Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers. Syst Rev. 2015;4:80. sourceExperimental — accuracy not yet measured
The Results (Atypical haemolytic uremic syndrome dataset) show that Abstrackr made predictions after an initial set of citations was screened, so this paper is an example of evaluating classification-based screening. But the Methods state that reviewers subsequently assessed citations predicted relevant; only those predicted irrelevant counted as screening workload saved. Thus, the claim that the remaining citations were not screened is too broad. (LLM, conf=0.84)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“Despite numerous tools for automation, adoption of these tools has lagged [8, 22].”
§ Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer > Discussion > Encouraging wider adoption
What the sources say
- Supportedcites [8] Lau J. Editorial: systematic review automation thematic series. Syst Rev. 2019;8(1):70. source
The editorial says researchers have developed automation tools for systematic reviews over the past 15 years, but that a lack of user-friendly software and commercial products has hindered their widespread adoption. This directly supports the attributed claim in the context of systematic reviews. (LLM, conf=0.92)
- Supportedcites [22] O’Connor AM, Tsafnat G, Thomas J, Glasziou P, Gilbert SB, Hutton B. A question of trust: can we build an evidence base to gain trust in systematic review automation technologies? Syst Rev. 2019;8(1):143. source
The 'Compatibility with current practice—set-up challenges' section says many automation tools exist, citing 159 indexed tools. The 'Barriers to adoption of automation' section describes an absence of adoption, and the Abstract discussion explicitly calls adoption slow. Page numbers were not supplied. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“After 70% of citations had been screened, both citation tools had not yet identified 4 (of 223 total) studies that would be included in the final report.”
§ Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“but Abstrackr performed better in 2 of 3 reports for identifying articles included in the final report”
§ Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer > Abstract > Results
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, for the pancreatic imaging report, EPPI-Reviewer identified all citations ordered for full-text review after only 62% of citations had been screened.”
§ Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer > Discussion > Limitations
plainstates a causeSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption not shown here.
28 more claims
Create a free account to see all 53 claims and every source verdict.
- Probably not a substantial claim. 7
- It may state two separate claims as one. 15
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Screening burden to identify all title/abstract includes for the de novo review was 85% or more for seven of the nine reviews for both Abstrackr (median: 93.8%, range: 71.1 to 99.0%) and EPPI-Reviewer (median: 91.3%, range: 39.9 to 97.9%).”
From An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes (2020) · cites this paper as [15]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The most advanced technology in this respect is the use of machine learning (ML) to prioritize studies for screening, which has been found to be accurate and efficient in a number of methodological studies [12e14], and is available at the time of writing in several systematic review authoring tools [15,16].”
From In a pilot study, automated real-time systematic review updates were feasible, accurate, and work-saving (2023) · cites this paper as [16]
Citations
Cites
Evidence-Based Medicine: How to Practice and Teach EBM
Sackett, D. L. (2000) · Cited by 5,950
Clinical Practice Guidelines We Can Trust
Committee on Standards for Developing Trustworthy Clinical Practice Guidelines, Board on Health Care Services, & Institute of Medicine (2011) · National Academies Press eBooks · doi:10.17226/13058 · Cited by 3,081
Statistical Methods in Medical Research
Lachenbruch, P. A. (1972) · Technometrics · doi:10.1080/00401706.1972.10488903 · Cited by 1,703
Finding What Works in Health Care: Standards for Systematic Reviews
Eden, J., Levit, L. A., Berg, A. O., & Morton, S. C. (2013) · Cited by 943
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
Deploying an interactive machine learning system in an evidence-based practice center: abstrackr
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012) · Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium · doi:10.1145/2110363.2110464 · Cited by 532 · cited 3×
“Both tools are web-based and facilitate citation screening through text-mining tools and machine learning techniques; they employ “ active learning, ” in which ongoing feedback from reviewers is used dynamically to improve predictive accuracy [14, 17].”
BackgroundMethodHarrison, H., Griffin, S. J., Kuhn, I., & Usher-Smith, J. A. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-0897-3 · Cited by 449
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Semi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351
Cited by
Bond, M., Khosravi, H., De Laat, M., Bergdahl, N., Negrea, V., Oxley, E., Pham, P., Chong, S. W., & Siemens, G. (2024) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00436-z · Cited by 852
“Given the affordances in efficiency that machine learning can bring to evidence synthesis (e.g., Stansfield et al., 2022; Tsou et al., 2020), as well as the enhanced transparency through visualisation tools such as EPPI Visualiser, it is surprising that the AIHEd community has not made more use of…”
MethodArtificial intelligence to automate the systematic review of scientific literature
de la Torre-López, J., Ramírez, A., & Romero, J. R. (2023) · Computing · doi:10.1007/s00607-023-01181-x · Cited by 170
“Five new methods have been proposed in the last four years [26, 27, 34, 35, 40], and supporting tools are subject of evaluations [11, 49, 50].”
BackgroundMethodValizadeh, A., Moassefi, M., Nakhostin-Ansari, A., Hosseini Asl, S. H., Saghab Torbati, M., Aghajani, R., Maleki Ghorbani, Z., & Faghani, S. (2022) · BMC Medical Research Methodology · doi:10.1186/s12874-022-01631-8 · Cited by 159
“Chai et al. in 2021 [21] introduced a new tool named “Research Screener” which utilizes deep learning algorithms for the semi-automated screening process.”
MethodResultInfluentialScherbakov, D., Hubig, N., Jansari, V., Bakumenko, A., & Lenert, L. A. (2024) · arXiv · doi:10.48550/arxiv.2409.04600 · Cited by 156
Scherbakov, D., Hubig, N., Jansari, V., Bakumenko, A., & Lenert, L. A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf063 · Cited by 154
Metadata
- Authors
- Tsou, A. Y., Treadwell, J. R., Erinoff, E., & Schoelles, K.
- Affiliations
- ECRI Institute (US)
- Corresponding author
- Tsou, Amy Y.
- Venue
- Systematic Reviews
- Published
- 2020-04-02
- Volume
- 9
- Issue
- 1
- Pages
- 73-73
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC7118839
- OpenAlex
- W3014512586
- Semantic Scholar
- 935489033eb450aaf007cc89d974c431b27cdd74
- Cited by
- 122
- References
- 22
- Citations by year
- 2026: 26 · 2025: 24 · 2024: 20 · 2023: 16 · 2022: 9 · 2021: 20 · 2020: 7
- FWCI
- 1.94
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Agency for Healthcare Research and Quality (R03 HS25859)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- citation screening, machine learning, abstract screening, full-text screening, tool comparison, evidence reports
- MeSH terms
- Machine Learning, Systematic Reviews as Topic, Humans, Mass Screening, Research, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-11, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-11 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer. https://app.cochat.ai/papers/10.1186/s13643-020-01324-7
- The paper
- Tsou, A. Y., Treadwell, J. R., Erinoff, E., & Schoelles, K. (2020). Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer. Systematic Reviews. https://doi.org/10.1186/s13643-020-01324-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.