The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index; 10 of the 11 sources its claims cite were checked for support.
Statistical stopping criteria for automated screening in systematic reviews
Open accessPublished version of a preprint
Abstract
Active learning for systematic review screening promises to reduce the human effort required to identify relevant documents for a systematic review. Machines and humans work together, with humans providing training data, and the machine optimising the documents that the humans screen. This enables the identification of all relevant documents after viewing only a fraction of the total documents. However, current approaches lack robust stopping criteria, so that reviewers do not know when they have seen all or a certain proportion of relevant documents. This means that such systems are hard to implement in live reviews. This paper introduces a workflow with flexible statistical stopping criteria, which offer real work reductions on the basis of rejecting a hypothesis of having missed a given recall target with a given level of confidence. The stopping criteria are shown on test datasets to achieve a reliable level of recall, while still providing work reductions of on average 17%. Other methods proposed previously are shown to provide inconsistent recall and work reductions across datasets.
Overview
CoChat Summary
TL;DR
The paper introduces statistical stopping criteria for active-learning screening in systematic reviews, using a hypergeometric test to reject the hypothesis that recall is below a target with a chosen confidence. On test datasets, the proposed criteria achieve reliable recall while saving on average 17% of screening work, whereas existing sampling and heuristic criteria are inconsistent.
Key findings
- The ranked quasi-sampling criterion achieved the 95% recall target in more than 95% of cases, saved a mean of 17% of work, and missed the target in only 0.95% of cases; the random sampling criterion saved 15% and missed the target in 3.29% of cases.
- The baseline sampling criterion missed the 95% recall target in 39.67% of cases, with the most common work saving being 0%.
- The heuristic criterion stopping after 50 consecutive irrelevant results had a mean work saving of 41% but missed the target in 39% of cases; the 200-irrelevant-result heuristic also performed unreliably.
- The proposed criteria are independent of model performance: if the model performs badly, work saved is low, but recall is not compromised.
Methods
- The study simulates active-learning screening on real systematic review datasets, including the Cohen et al. 2006 collection, Proton Beam, COPD, and FASTREAD computer science datasets, running 100 iterations per dataset.
- A basic SVM with 1–2 word n-grams from abstracts is used; screening starts with random samples of 200 documents, then batches of the 20 highest predicted relevance scores are revealed and the model retrained after each batch. Target recall is 95% and confidence level is 95%.
- The statistical test uses the hypergeometric distribution to test the null hypothesis that recall is below target; screening stops when the maximum probability of observed results under the null is below 1−α. Ranked quasi-sampling treats previously screened batches as quasi-samples to decide when to begin random sampling and as an independent stopping criterion.
Limitations
- The paper states that it did not formally address multiple testing; the authors call such a derivation mathematically challenging and beyond the scope, though they say performance suggests limited practical concern.
- The paper states that ranked quasi-sampling is unlikely to be accurate because documents seen during active learning are ranked, not random; the estimator is conservative if the assumption holds that machine-learning-selected documents are not less likely to be relevant than random ones.
- Evident from the text: evaluation uses a basic SVM and specific datasets, and the authors note that work savings, even maximum work savings, are below the state of the art recorded for those datasets.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures








Verification
Claims the paper makes
5 Valid · 1 Not valid · 0 Not checked · 32 Own results and interpretations
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“Active learning for systematic review screening promises to reduce the human effort required to identify relevant documents for a systematic review.”
- Also stated in: § Background · plain
“Several papers have developed active learning (AL) approaches [3–7] to reduce the time required to screen documents.”
§ Statistical stopping criteria for automated screening in systematic reviews > Abstract
What the sources say
- Supportedcites [3] Miwa M, Thomas J, O’Mara-Eves A, Ananiadou S. Reducing systematic review workload through certainty-based screening. J Biomed Inform. 2014; 51:242–53. https://doi.org/10.1016/j.jbi.2014.06.005. source
The Introduction says active learning can make manual screening more efficient by suggesting items for review. The Conclusions report reduced screening burden and faster identification of relevant studies; the paper also notes a risk of missing some studies. (LLM, conf=0.94)
- Supportedcites [4] Wallace B, Trikalinos T, Lau J, Brodley C, Schmid C. Semi-automated screening of biomedical citations for systematic reviews. BMC Bioinforma. 2010; 11. https://doi.org/10.1186/1471-2105-11-55. source
The Results state that the paper’s active-learning strategy significantly reduced reviewers’ screening burden without excluding relevant citations. The Abstract Results further report fewer citations requiring manual screening across three systematic-review datasets. (LLM, conf=0.94)
- Not checked — no full text of the sourcecites [5] Wallace BC, Small K, Brodley CE, Trikalinos TA. Active learning for biomedical citation screening. In: Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD ’10). New York: Association for Computing Machinery: 2010. p. 173–82. https://doi.org/10.1145/1835804.1835829. source
- Supportedcites [6] Jonnalagadda S, Petitti D. A new iterative method to reduce workload in systematic review process. Int J Comput Biol Drug Des. 2013; 6((1/2)):5. https://doi.org/10.1504/ijcbdd.2013.052198. source
The Background describes active learning as an approach to reducing the effort of manually screening abstracts for systematic reviews. The claim says it promises to reduce effort, not that this paper demonstrated a workload reduction specifically for active learning. (LLM, conf=0.90)
- Supportedcites [7] Przybyła P, Brockmeier A, Kontonatsios G, Le Pogam M, McNaught J, von Elm E, Nolan K, Ananiadou S. Prioritising references for systematic reviews with RobotAnalyst: a user study. Res Synth Methods. 2018; 9(3):470–88. https://doi.org/10.1002/jrsm.1311. source
The Introduction (page not supplied) says machine learning can prioritise references based on screening decisions, potentially allowing reviewers to screen only part of a collection while finding the vast majority of relevant references. The Results (page not supplied) report faster identification of relevant references with active-learning prioritisation than with random sampling, supporting the citing sentence’s qualified claim that it promises to reduce human effort. (LLM, conf=0.93)
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Background · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The stopping criteria are shown on test datasets to achieve a reliable level of recall, while still providing work reductions of on average 17%.”
- Also stated in: § Results · plain
“Both the random sampling and the ranked sampling criteria achieve the target threshold of 95% in more than 95% of cases.”
§ Statistical stopping criteria for automated screening in systematic reviews > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In only 23% of cases can work savings be achieved while still achieving a recall of at least 95%.”
§ Statistical stopping criteria for automated screening in systematic reviews > Methods review > Sampling-based stopping criteria
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In each case, 95% recall is achieved with close to 20% of documents remaining.”
§ Statistical stopping criteria for automated screening in systematic reviews > Results
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We find this a promising intuition, but argue that (1) it ignores uncertainty, as discussed in relation to the previous method”
§ Statistical stopping criteria for automated screening in systematic reviews > Methods review > Sampling-based stopping criteria
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
16 more claims
Create a free account to see all 38 claims and every source verdict.
- It may state two separate claims as one. 14
- Probably not a substantial claim. 4
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“However, maintaining a high recall rate (i.e., identifying most relevant studies) when the proportion of relevant studies is larger often results in minimal time savings, as a substantial portion of literature still requires manual review 22 .”
From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [22]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In contrast, Callaghan and Müller-Hansen (2020) found that data-driven approaches could not find 95% of all relevant records in 39% of the 20 systematic reviews analyzed.”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Callaghan and Müller-Hansen (2020)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“as their development depends on knowledge of the true relevant records (Callaghan & Müller-Hansen, 2020; Hamel et al., 2021).”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Callaghan & Müller-Hansen, 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Lastly, researchers have an ongoing uncertainty regarding the optimal number of articles to screen using these tools, and the existing simulation studies have yielded limited and conflicting findings (Callaghan & MüllerHansen, 2020; Wallace et al., 2010).”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Callaghan & Müller-Hansen, 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Overall, the performance of heuristic stopping rules should not be generalized to databases where the number of relevant records is unknown, as their development depends on knowledge of the true relevant records (Callaghan & Müller-Hansen, 2020; Hamel et al., 2021).”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as Callaghan & Müller-Hansen, 2020
Experimental — accuracy not yet measured
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ Statistical stopping criteria for automated screening in systematic reviews > Results
“In Fig. 6, we rescale the x axis, calling it additional burden, which is simply the work saved when the criterion is triggered minus the work saved when the recall target was actually achieved.”
§ Statistical stopping criteria for automated screening in systematic reviews > Methods > Evaluation
“Additional Burden: The work saved when the criterion was triggered subtracted from the work saved when the recall target was actually achieved.”
Citations
Cites
Scikit-learn: Machine Learning in Python
Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Müller, A., Nothman, J., Louppe, G., et al. (2012) · arXiv · doi:10.48550/arxiv.1201.0490 · Cited by 71,016
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Cortes, C., & Vapnik, V. (1995) · Machine-mediated learning · doi:10.1023/a:1022627411411 · Cited by 48,374
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
Searching for and selecting studies
Lefebvre, C., Glanville, J., Briscoe, S., Littlewood, A., Marshall, C., Metzendorf, M., Noel‐Storr, A., Rader, T., Shokraneh, F., Thomas, J., et al. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch4 · Cited by 647 · cited 1×
“This is particularly problematic in systematic reviews because low recall increases the risk of bias [9].”
BackgroundMarshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638 · cited 1×
“The lack of appropriate stopping criteria has therefore been identified as a research gap [10, 11], although some approaches have been suggested.”
BackgroundReducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Cited by
Khanna, T. M., Baiocchi, G., Callaghan, M., Creutzig, F., Guias, H., Haddaway, N. R., Hirth, L., Javaid, A., Koch, N., Laukemper, S., et al. (2021) · Nature Energy · doi:10.1038/s41560-021-00866-x · Cited by 186
Self-citationSystematic review and meta-analysis of ex-post evaluations on the effectiveness of carbon pricing
Döbbeling-Hildebrandt, N., Miersch, K., Khanna, T. M., Bachelet, M., Bruns, S. B., Callaghan, M., Edenhofer, O., Flachsland, C., Forster, P. M., Kalkuhl, M., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-48512-w · Cited by 164
“A systematic review and meta-analysis of the ex-post literature””
BackgroundMethodSelf-citationHamel, C., Hersi, M., Kelly, S. E., Tricco, A. C., Straus, S., Wells, G., Pham, B., & Hutton, B. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01451-2 · Cited by 104
“There have also been some more complex evaluations to implement a stopping decision [44].”
BackgroundA systematic review highlights that there are multiple benefits of urban agriculture besides food
Pradhan, P., Callaghan, M., Hu, Y., Dahal, K., Hunecke, C., Reußwig, F., Lotze‐Campen, H., & Kropp, J. P. (2023) · Global Food Security · doi:10.1016/j.gfs.2023.100700 · Cited by 107
Self-citationCampos, D. G., Fütterer, T., Gfrörer, T., Lavelle-Hill, R., Murayama, K., König, L., Hecht, M., Zitzmann, S., & Scherer, R. (2024) · Educational Psychology Review · doi:10.1007/s10648-024-09862-5 · Cited by 75
“…that the results on the performance of heuristic stopping criteria are highly dependent on dataset characteristics, it is advisable to extend the study to evaluate the effectiveness and generalizability of statistical stopping criteria, such as those proposed by Callaghan and Müller-Hansen (2020).”
Influential
Metadata
- Authors
- Callaghan, M. W., & Müller-Hansen, F.
- Affiliations
- University of Leeds (GB)
- Leibniz Association (DE)
- Corresponding author
- Callaghan, Max W
- Venue
- Systematic Reviews
- Published
- 2020-11-28
- Volume
- 9
- Issue
- 1
- Pages
- 273-273
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC7700715
- OpenAlex
- W3110318438
- Cited by
- 121
- References
- 26
- Citations by year
- 2026: 28 · 2025: 28 · 2024: 26 · 2023: 23 · 2022: 5 · 2021: 10 · 2020: 1
- FWCI
- 2.02
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0/ (tdm)
- creativecommons.org/licenses/by/4.0/ (vor)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Publication Database PIK (Potsdam Institute for Climate Impact Research (PIK)) (submittedVersion, other-oa)
- Funders
- Bundesministerium für Bildung und Forschung (03EK3046B)
- Heinrich Böll Stiftung
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- active learning (machine learning), systematic review screening, stopping rules, memory retrieval, human-machine collaboration, hypothesis testing
- MeSH terms
- Machine Learning, Systematic Reviews as Topic, Humans, Research
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-09-29, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-09-29 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Statistical stopping criteria for automated screening in systematic reviews. https://app.cochat.ai/papers/10.1186/s13643-020-01521-4
- The paper
- Callaghan, M. W., & Müller-Hansen, F. (2020). Statistical stopping criteria for automated screening in systematic reviews. Systematic Reviews. https://doi.org/10.1186/s13643-020-01521-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.