The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index; 11 of the 20 sources its claims cite were checked for support.
Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool
Open accessAbstract
BACKGROUND: Machine learning tools can expedite systematic review (SR) processes by semi-automating citation screening. Abstrackr semi-automates citation screening by predicting relevant records. We evaluated its performance for four screening projects. METHODS: We used a convenience sample of screening projects completed at the Alberta Research Centre for Health Evidence, Edmonton, Canada... RESULTS: Abstrackr's sensitivity was > 0.75 for all projects and the mean specificity ranged from 0.69 to 0.90 with the exception of Child Health SRs, for which it was 0.19...
Overview
CoChat Summary
## TL;DR
This retrospective evaluation of Abstrackr across four heterogeneous screening projects found that it generally achieved high sensitivity but variable specificity, precision, and workload savings. The time saved came with some risk of missing relevant records, so the authors concluded that more testing is needed before routine use.
## Key findings
- Across the four projects, Abstrackr’s sensitivity was >0.75 for all projects; mean sensitivity was 0.96 for Child Health SRs, 0.92 for Bronchiolitis, 0.82 for Diabetes, and 0.79 for Antipsychotics.
- Mean specificity ranged from 0.69 to 0.90 for three projects, but was 0.19 for Child Health SRs; precision ranged from 14.8% to 64.7%.
- Median workload savings was 67.2% overall, with project-level workload savings of 88.4% for Diabetes, 70.0% for Bronchiolitis, 64.5% for Antipsychotics, and 9.5% for Child Health SRs.
- The proportion missed was 0.1% for all SRs and 6.4% for the descriptive analysis; this corresponded to 4.2% of the records in the final reports.
## Methods
- Retrospective evaluation of Abstrackr using a convenience sample of four screening projects completed or ongoing at the Alberta Research Centre for Health Evidence: three systematic reviews and one descriptive analysis.
- Projects varied in size and topic: Antipsychotics, Bronchiolitis, Child Health SRs, and Diabetes; search yields ranged from 5,243 to 47,385 records.
- Records were uploaded to Abstrackr and screened in single-screen mode until predictions were produced; across three trials per project, predictions were compared with human reviewer decisions and standard performance metrics were calculated.
## Limitations
- The paper states the sample was heterogeneous and convenience-based, so the findings “should not be generalized.”
- It notes possible variation in predictions between trials of the same screening task, and that the first record could not be controlled, affecting prioritization and predictions.
- Evident from the text: the study was retrospective and did not test Abstrackr prospectively as a second reviewer in a real screening workflow, which the authors themselves say should be studied further.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
9 Valid · 1 Not valid · 6 Not checked · 40 Own results and interpretations
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the mean specificity ranged from 0.69 to 0.90 with the exception of Child Health SRs, for which it was 0.19”
§ Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool > Abstract > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Abstrackr’s sensitivity was > 0.75 for all projects”
§ Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool > Abstract > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Across the three trials, Abstrackr missed 16, 20, and 25 (7.8, 9.8, and 12.2%) of the studies included in the final SR for Diabetes”
§ Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool > Results > Records missed and time savings
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For the SRs, Abstrackr reduced the number of records to screen by 65 to 88%”
§ Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool > Discussion
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“24, 31, and 35 (1.5, 1.9, and 2.2%) of the studies for Child Health SRs”
§ Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool > Results > Records missed and time savings
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
17 more claims
Create a free account to see all 56 claims and every source verdict.
- It may state two separate claims as one. 19
- Probably not a substantial claim. 17
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Each study is typically reviewed within 30 s (Gates et al., 2018)”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Gates et al., 2018
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The application of active learning models in systematic reviews has been extensively studied in simulation studies [11-21].”
From Performance of active learning models for screening prioritization in systematic reviews: a simulation study into the Average Time to Discover relevant records (2023) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“However, developing automatic screening tools based on combined ML techniques seems feasible; plenty of research and developments have been done in this field in recent years [26, 39, 41, 46, 52].”
From Machine learning computational tools to assist the performance of systematic reviews: A mapping review. (2022) · cites this paper as [39]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In summary, ML-assisted systematic reviews can significantly reduce workload and have proven accurate across various fields and review topics.”
From Machine Learning-Assisted Systematic Review: A Case Study in Learning Analytics (2025) · cites this paper as Gates et al. (2018)
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Many studies evaluating these tools have assessed automatic classification functionality, in which screening continued until the tool predicts whether studies should be included or not, and remaining citations are not screened [19, 21].”
From Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer (2020) · cites this paper as [19]
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042 · cited 1×
“Specialized guidance documents [2, 3] aim to ensure that reviewers produce evidence syntheses that are methodologically rigorous and transparently reported.”
BackgroundPreferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
The Levels of Evidence and Their Role in Evidence-Based Medicine
Burns, P. B., Rohrich, R. J., & Chung, K. C. (2011) · Plastic and Reconstructive Surgery · doi:10.1097/prs.0b013e318219c171 · Cited by 2,155 · cited 1×
“Systematic reviews (SRs) provide the highest level of evidence to inform clinical and policy decisions [1].”
BackgroundStatistics Notes: Diagnostic tests 1: sensitivity and specificity
Altman, D. G., & Bland, J. M. (1994) · BMJ · doi:10.1136/bmj.308.6943.1552 · Cited by 2,061
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188 · cited 1×
“The disconnect between the work required to complete a SR and the rate of publication of new evidence from trials [6] means that many SRs are out of date before they are published [7].”
BackgroundPage, M. J., Shamseer, L., Altman, D. G., Tetzlaff, J., Sampson, M., Tricco, A. C., Catalá-López, F., Li, L., Reid, E. K., Sarkis-Onofre, R., et al. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1002028 · Cited by 768 · cited 1×
“As the standards for the conduct and reporting of SRs have become more stringent [4], they have become more labor-intensive to produce.”
BackgroundBorah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737 · cited 1×
“On average, after registering a protocol, it takes author teams more than 1 year of work before their SR is published [5].”
Method
Research works printed but not linked: 1
- Higgins JPT, Green S, Cochrane handbook for systematic reviews of interventions. Version 5.1.0. The Cochrane collaboration; 2011. www.handbook.cochrane.org .
Cited by
Automated Paper Screening for Clinical Reviews Using Large Language Models
Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C. (2023) · arXiv · doi:10.48550/arxiv.2305.00844 · Cited by 230
Using artificial intelligence methods for systematic review in health sciences: A systematic review
Blaizot, A., Veettil, S. K., Saidoung, P., Moreno‐Garcia, C. F., Wiratunga, N., Aceves‐Martins, M., Lai, N. M., & Chaiyakunapruk, N. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1553 · Cited by 223
Kazda, L., Bell, K., Thomas, R., McGeechan, K., Sims, R., & Barratt, A. (2021) · JAMA Network Open · doi:10.1001/jamanetworkopen.2021.5335 · Cited by 219
Diet and Anxiety: A Scoping Review
Aucoin, M., LaChance, L., Naidoo, U., Remy, D., Shekdar, T., Sayar, N., Cardozo, V., Rawana, T., Chan, I., & Cooley, K. (2021) · Nutrients · doi:10.3390/nu13124418 · Cited by 212
“The study authors manually screened 13,286 results while Abstrackr’s artificial intelligence screened the remaining results.”
MethodInfluentialAutomated paper screening for clinical reviews using large language models: Data analysis study
Guo, E., Gupta, M., Deng, J., Park, Y.-J., Paget, M., & Naugler, C. (2024) · Journal of Medical Internet Research · doi:10.2196/48996 · Cited by 205
“The comparison of our proposed machine learning method to other tools, such as Abstrackr [18], DistillerSR [17], and RobotAnalyst [36], provides evidence of its efficacy and reliability in the context of systematic review processes.”
Metadata
- Authors
- Gates, A., Johnson, C., & Hartling, L.
- Affiliations
- University of Alberta (CA)
- Corresponding author
- Hartling, Lisa
- Venue
- Systematic Reviews
- Published
- 2018-03-12
- Volume
- 7
- Issue
- 1
- Pages
- 45-45
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC5848519
- OpenAlex
- W2807522649
- Cited by
- 148
- References
- 30
- Citations by year
- 2026: 13 · 2025: 22 · 2024: 23 · 2023: 17 · 2022: 15 · 2021: 22 · 2020: 24 · 2019: 8 · 2018: 4
- FWCI
- 2.89
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Canadian Institutes of Health Research
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- machine learning, citation screening, title and abstract screening, relevance prediction, false negative rate, sensitivity and specificity, human-machine comparison, research automation, pediatric research, antipsychotic drugs, bronchiolitis, diabetes mellitus, reproducibility
- MeSH terms
- Machine Learning, Systematic Reviews as Topic, Canada, Humans, Mass Screening, Records, Retrospective Studies, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool. https://app.cochat.ai/papers/10.1186/s13643-018-0707-8
- The paper
- Gates, A., Johnson, C., & Hartling, L. (2018). Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool. Systematic Reviews. https://doi.org/10.1186/s13643-018-0707-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
