Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index; 11 of the 20 sources its claims cite were checked for support.

Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool

Open access
Gates, Allison, Johnson, Cydney, Hartling, Lisa · 2018 · Systematic Reviews

Abstract

BACKGROUND: Machine learning tools can expedite systematic review (SR) processes by semi-automating citation screening. Abstrackr semi-automates citation screening by predicting relevant records. We evaluated its performance for four screening projects. METHODS: We used a convenience sample of screening projects completed at the Alberta Research Centre for Health Evidence, Edmonton, Canada... RESULTS: Abstrackr's sensitivity was > 0.75 for all projects and the mean specificity ranged from 0.69 to 0.90 with the exception of Child Health SRs, for which it was 0.19...

Overview

CoChat Summary

## TL;DR

This retrospective evaluation of Abstrackr across four heterogeneous screening projects found that it generally achieved high sensitivity but variable specificity, precision, and workload savings. The time saved came with some risk of missing relevant records, so the authors concluded that more testing is needed before routine use.

## Key findings

  • Across the four projects, Abstrackr’s sensitivity was >0.75 for all projects; mean sensitivity was 0.96 for Child Health SRs, 0.92 for Bronchiolitis, 0.82 for Diabetes, and 0.79 for Antipsychotics.
  • Mean specificity ranged from 0.69 to 0.90 for three projects, but was 0.19 for Child Health SRs; precision ranged from 14.8% to 64.7%.
  • Median workload savings was 67.2% overall, with project-level workload savings of 88.4% for Diabetes, 70.0% for Bronchiolitis, 64.5% for Antipsychotics, and 9.5% for Child Health SRs.
  • The proportion missed was 0.1% for all SRs and 6.4% for the descriptive analysis; this corresponded to 4.2% of the records in the final reports.

## Methods

  • Retrospective evaluation of Abstrackr using a convenience sample of four screening projects completed or ongoing at the Alberta Research Centre for Health Evidence: three systematic reviews and one descriptive analysis.
  • Projects varied in size and topic: Antipsychotics, Bronchiolitis, Child Health SRs, and Diabetes; search yields ranged from 5,243 to 47,385 records.
  • Records were uploaded to Abstrackr and screened in single-screen mode until predictions were produced; across three trials per project, predictions were compared with human reviewer decisions and standard performance metrics were calculated.

## Limitations

  • The paper states the sample was heterogeneous and convenience-based, so the findings “should not be generalized.”
  • It notes possible variation in predictions between trials of the same screening task, and that the first record could not be controlled, affecting prioritization and predictions.
  • Evident from the text: the study was retrospective and did not test Abstrackr prospectively as a second reviewer in a real screening workflow, which the authors themselves say should be studied further.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13643-018-0707-8

Cites

30 of 30 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Abstrackr’s mean sensitivity and specificity across three trials for each project
Fig. 1 Abstrackr’s mean sensitivity and specificity across three trials for each projectCC BY · PubMed Central

Verification

Claims the paper makes

9 Valid · 1 Not valid · 6 Not checked · 40 Own results and interpretations

17 more claims

Create a free account to see all 56 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 19
  • Probably not a substantial claim. 17

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

1 more claim about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Show all 30 references

Research works printed but not linked: 1

  1. Higgins JPT, Green S, Cochrane handbook for systematic reviews of interventions. Version 5.1.0. The Cochrane collaboration; 2011. www.handbook.cochrane.org .

Cited by

See all 148

Metadata

Authors
Gates, A., Johnson, C., & Hartling, L.
Affiliations
University of Alberta (CA)
Corresponding author
Hartling, Lisa
Venue
Systematic Reviews
Published
2018-03-12
Volume
7
Issue
1
Pages
45-45
ISSN
2046-4053 (ISSN-L)
OpenAlex
W2807522649
Cited by
148
References
30
Citations by year
2026: 13 · 2025: 22 · 2024: 23 · 2023: 17 · 2022: 15 · 2021: 22 · 2020: 24 · 2019: 8 · 2018: 4
FWCI
2.89
Type
journal article
Language
en
Open access
Licence
  • cc-by
  • creativecommons.org/licenses/by/4.0/ (tdm)
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
Funders
Canadian Institutes of Health Research
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
Keywords
machine learning, citation screening, title and abstract screening, relevance prediction, false negative rate, sensitivity and specificity, human-machine comparison, research automation, pediatric research, antipsychotic drugs, bronchiolitis, diabetes mellitus, reproducibility
MeSH terms
Machine Learning, Systematic Reviews as Topic, Canada, Humans, Mass Screening, Records, Retrospective Studies, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-10, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-10 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool. https://app.cochat.ai/papers/10.1186/s13643-018-0707-8
The paper
Gates, A., Johnson, C., & Hartling, L. (2018). Technology-assisted title and abstract screening for systematic reviews: a retrospective evaluation of the Abstrackr machine learning tool. Systematic Reviews. https://doi.org/10.1186/s13643-018-0707-8

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.