Student or educator? Sign up with your school email and get 6 months of Pro free.

The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index; 10 of the 11 sources its claims cite were checked for support.

Statistical stopping criteria for automated screening in systematic reviews

Open access
Callaghan, Max W, Müller-Hansen, Finn · 2020 · Systematic Reviews

Published version of a preprint

Abstract

Active learning for systematic review screening promises to reduce the human effort required to identify relevant documents for a systematic review. Machines and humans work together, with humans providing training data, and the machine optimising the documents that the humans screen. This enables the identification of all relevant documents after viewing only a fraction of the total documents. However, current approaches lack robust stopping criteria, so that reviewers do not know when they have seen all or a certain proportion of relevant documents. This means that such systems are hard to implement in live reviews. This paper introduces a workflow with flexible statistical stopping criteria, which offer real work reductions on the basis of rejecting a hypothesis of having missed a given recall target with a given level of confidence. The stopping criteria are shown on test datasets to achieve a reliable level of recall, while still providing work reductions of on average 17%. Other methods proposed previously are shown to provide inconsistent recall and work reductions across datasets.

Overview

CoChat Summary

TL;DR

The paper introduces statistical stopping criteria for active-learning screening in systematic reviews, using a hypergeometric test to reject the hypothesis that recall is below a target with a chosen confidence. On test datasets, the proposed criteria achieve reliable recall while saving on average 17% of screening work, whereas existing sampling and heuristic criteria are inconsistent.

Key findings

  • The ranked quasi-sampling criterion achieved the 95% recall target in more than 95% of cases, saved a mean of 17% of work, and missed the target in only 0.95% of cases; the random sampling criterion saved 15% and missed the target in 3.29% of cases.
  • The baseline sampling criterion missed the 95% recall target in 39.67% of cases, with the most common work saving being 0%.
  • The heuristic criterion stopping after 50 consecutive irrelevant results had a mean work saving of 41% but missed the target in 39% of cases; the 200-irrelevant-result heuristic also performed unreliably.
  • The proposed criteria are independent of model performance: if the model performs badly, work saved is low, but recall is not compromised.

Methods

  • The study simulates active-learning screening on real systematic review datasets, including the Cohen et al. 2006 collection, Proton Beam, COPD, and FASTREAD computer science datasets, running 100 iterations per dataset.
  • A basic SVM with 1–2 word n-grams from abstracts is used; screening starts with random samples of 200 documents, then batches of the 20 highest predicted relevance scores are revealed and the model retrained after each batch. Target recall is 95% and confidence level is 95%.
  • The statistical test uses the hypergeometric distribution to test the null hypothesis that recall is below target; screening stops when the maximum probability of observed results under the null is below 1−α. Ranked quasi-sampling treats previously screened batches as quasi-samples to decide when to begin random sampling and as an independent stopping criterion.

Limitations

  • The paper states that it did not formally address multiple testing; the authors call such a derivation mathematically challenging and beyond the scope, though they say performance suggests limited practical concern.
  • The paper states that ranked quasi-sampling is unlikely to be accurate because documents seen during active learning are ranked, not random; the estimator is conservative if the assumption holds that machine-learning-selected documents are not less likely to be relevant than random ones.
  • Evident from the text: evaluation uses a basic SVM and specific datasets, and the authors note that work savings, even maximum work savings, are below the state of the art recorded for those datasets.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13643-020-01521-4

Cites

28 of 28 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Distribution of under- or overestimation errors using the BIR sampling method in a dataset of 20,000 documents of which 500 are relevant. a The probability distribution of the estimated number of relevant documents after a sample of 1000 documents. b The probability of each type of error according to the sample size
Fig. 1 Distribution of under- or overestimation errors using the BIR sampling method in a dataset of 20,000 documents of which 500 are relevant. a The probability distribution of the estimated number of relevant documents after a sample of 1000 documents. b The probability of each type of error according to the sample sizeCC BY · PubMed Central
Fig. 2 Similar low proportions of relevant documents in unseen documents with different consequences for recall. The top bar shows a random distribution of relevant documents (green) and irrelevant documents (red) at a given proportion of relevance. The bottom bar shows distributions of relevant and irrelevant documents in hypothetical sets of seen (right) and unseen (left—transparent) documents
Fig. 2 Similar low proportions of relevant documents in unseen documents with different consequences for recall. The top bar shows a random distribution of relevant documents (green) and irrelevant documents (red) at a given proportion of relevance. The bottom bar shows distributions of relevant and irrelevant documents in hypothetical sets of seen (right) and unseen (left—transparent) documentsCC BY · PubMed Central
Fig. 3 A workflow for active learning in screening with a statistical stopping criterion
Fig. 3 A workflow for active learning in screening with a statistical stopping criterionCC BY · PubMed Central
Fig. 4 The distribution of achieved recall values given our random sampling stopping criterion for 6 scenarios with different recall values at the start of sampling
Fig. 4 The distribution of achieved recall values given our random sampling stopping criterion for 6 scenarios with different recall values at the start of samplingCC BY · PubMed Central
Fig. 5 Distribution of recall and work saved after each stopping criteria. Green dots show results for datasets with less than 1000 documents, orange dots show datasets with 1000–2000 documents, and blue dots show datasets with more than 2000 documents
Fig. 5 Distribution of recall and work saved after each stopping criteria. Green dots show results for datasets with less than 1000 documents, orange dots show datasets with 1000–2000 documents, and blue dots show datasets with more than 2000 documentsCC BY · PubMed Central
Fig. 6 Distribution of recall and additional burden after each stopping criterion. Additional burden is the work saved when the criterion was triggered minus the work saved when the target was reached. Colouring of data points as in Fig. 5
Fig. 6 Distribution of recall and additional burden after each stopping criterion. Additional burden is the work saved when the criterion was triggered minus the work saved when the target was reached. Colouring of data points as in Fig. 5CC BY · PubMed Central
Fig. 7 Work saved for the ranked quasi-sampling method in each dataset. Labels show the number of relevant documents and the total number of documents. The datasets are presented in order of the number of documents. The whiskers represent the 5th and 95th percentiles. The grey line shows work savings of 5%
Fig. 7 Work saved for the ranked quasi-sampling method in each dataset. Labels show the number of relevant documents and the total number of documents. The datasets are presented in order of the number of documents. The whiskers represent the 5th and 95th percentiles. The grey line shows work savings of 5%CC BY · PubMed Central
Fig. 8 The path of recall (yellow) and the p value of H0 for four different datasets
Fig. 8 The path of recall (yellow) and the p value of H0 for four different datasetsCC BY · PubMed Central

Verification

Claims the paper makes

5 Valid · 1 Not valid · 0 Not checked · 32 Own results and interpretations

16 more claims

Create a free account to see all 38 claims and every source verdict.

See all claims — free account
  • It may state two separate claims as one. 14
  • Probably not a substantial claim. 4

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

4 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 26 references

Cited by

See all 121

Metadata

Authors
Callaghan, M. W., & Müller-Hansen, F.
Affiliations
  • University of Leeds (GB)
  • Leibniz Association (DE)
Corresponding author
Callaghan, Max W
Venue
Systematic Reviews
Published
2020-11-28
Volume
9
Issue
1
Pages
273-273
ISSN
2046-4053 (ISSN-L)
OpenAlex
W3110318438
Cited by
121
References
26
Citations by year
2026: 28 · 2025: 28 · 2024: 26 · 2023: 23 · 2022: 5 · 2021: 10 · 2020: 1
FWCI
2.02
Type
journal article
Language
en
Open access
Licence
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • Research Square (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
  • Publication Database PIK (Potsdam Institute for Climate Impact Research (PIK)) (submittedVersion, other-oa)
Funders
  • Bundesministerium für Bildung und Forschung (03EK3046B)
  • Heinrich Böll Stiftung
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
Keywords
active learning (machine learning), systematic review screening, stopping rules, memory retrieval, human-machine collaboration, hypothesis testing
MeSH terms
Machine Learning, Systematic Reviews as Topic, Humans, Research

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Statistical stopping criteria for automated screening in systematic reviews. https://app.cochat.ai/papers/10.1186/s13643-020-01521-4
The paper
Callaghan, M. W., & Müller-Hansen, F. (2020). Statistical stopping criteria for automated screening in systematic reviews. Systematic Reviews. https://doi.org/10.1186/s13643-020-01521-4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.