The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index; none of the 21 sources its claims cite is checked for support yet.
When to stop reviewing: validation of stop criteria in ASReview
Open accessAbstract
Background: Systematic reviews are essential for evidence-based research, but they often require a great deal of time and effort. Although title and abstract (T&A) screening is just one part of the review process, it can be very time-consuming when search strategies retrieve large numbers of records. Given the exponential growth of scientific publications in recent decades, tools such as ASReview, which use machine learning (ML) for active learning-based screening, aim to reduce the workload. However, since ASReview helps users prioritise potentially relevant studies rather than supporting them in screening all records, a key question arises: at what point can the screening process safely stop without risking the omission of relevant studies when the entire dataset is not being reviewed? Methods: This simulation study tested three proposed stop criteria for terminating screening in ASReview without loss of relevant data: (1) stopping after a calculated number of relevant studies based on an initial sample; (2) stopping after a fixed number of consecutively studies deemed irrelevant; (3) stopping after a predefined percentage of the dataset has been screened. A total of 35,000 automated title and abstract screenings were conducted using five datasets from the SYNERGY repository. Key outcomes included the percentage of studies screened until the last relevant study was found and the number of relevant studies missed under each stop criterion. Results: The proportion of the dataset that needed to be screened to identify all relevant studies (as pre-classified in the SYNERGY dataset) varied greatly across datasets, ranging from 2.9% to 76.9% on average. None of the tested stop criteria could consistently identify all relevant studies across all datasets. Stop criterion 1 was reliable in only 2% of simulations. Stop criterion 2 showed high variability, with thresholds ranging from 2% to 61%, depending on the dataset. Stop criterion 3 failed to define a universal percentage applicable across datasets. Conclusions: ASReview can reduce screening workload by prioritizing potentially relevant studies through ML-based ranking, thereby allowing researchers to identify relevant studies earlier in the screening process. However, no stop criterion reliably ensures that all relevant studies are identified. Early stopping may result in missed studies, depending on dataset characteristics. Current stop criteria should be applied cautiously and potentially combined with quality assurance measures. Further research is needed to develop more robust and generalizable stopping rules. Trial registration: Not applicable – this is a simulation study, not a registered systematic review.
Overview
CoChat Summary
TL;DR
This simulation study tested three proposed stop criteria for ASReview-assisted title and abstract screening across five SYNERGY datasets. None reliably identified all relevant studies, so early stopping risked missing relevant records and universal stopping rules were not supported.
Key findings
- The proportion of each dataset that needed to be screened to identify all relevant studies varied widely, with means ranging from 2.9% to 76.9% across datasets; in the Donners (2022) dataset, up to 94.5% still required screening in the least favorable run.
- Stop criterion 1 was reliable in only 2% of simulations; in about 40% of simulations more relevant studies were assumed than existed, in 22% one study was erroneously excluded, in 36% more than one was excluded, and in one simulation 84 relevant studies were missed.
- Stop criterion 2 showed high variability, with functional thresholds ranging from 2% to 61% depending on the dataset.
- Stop criterion 3 failed to define a universal percentage applicable across datasets.
Methods
- A simulation study conducted 35,000 automated title and abstract screenings (7,000 per dataset) using ASReview version 1.6.1 and a Python script on five datasets from the SYNERGY repository.
- Initial training subsets were varied at 1%, 5%, 10%, 15%, 20%, 25%, and 30% of each dataset; for each dataset and subset size, 1,000 independent simulations were run.
- The three stop criteria were evaluated retrospectively using known relevance labels: estimated versus actual relevant studies for criterion 1, largest gap between consecutive relevant studies for criterion 2, and position of the last relevant study as a percentage of dataset size for criterion 3.
Limitations
- The paper states that publicly available datasets already used in other ASReview projects were used, so results could differ with new datasets; only the default and most established ASReview model was tested; and it was not checked whether further studies would have been found retrospectively by screening bibliographies.
- Evident from the text: the retrospective simulation uses known relevance labels and cannot be directly applied to prospective systematic reviews where the number of relevant studies is unknown.
- Evident from the text: only five datasets were tested, and results varied greatly by dataset, limiting generalizability of the findings.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
21 Own result · 17 Own interpretation · 7 Not checked yet · 3 No full text of the sources · 1 Check pending · 1 Own method
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, it can be seen that relevant studies are found with less effort on average across all datasets and all simulations than in manual screening processes without ASReview.”
§ When to stop reviewing: validation of stop criteria in ASReview > Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In any case, all relevant studies could be identified earlier compared to a manual screening of the entire dataset.”
§ When to stop reviewing: validation of stop criteria in ASReview > Conclusion
hedgedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Stop criterion 3 failed to define a universal percentage applicable across datasets.”
- Also stated in: § Results · plain
“Therefore, no overall value can be given for stop criterion 3, in which all relevant studies were always included in all datasets.”
§ When to stop reviewing: validation of stop criteria in ASReview > Abstract > Results
plainn = 7,000- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Results · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“None of the tested stop criteria could consistently identify all relevant studies across all datasets.”
- Also stated in: § Conclusion · plain
“Furthermore, no overall value could be found for any of the stop criteria tested that proved useful in all the data records tested.”
§ When to stop reviewing: validation of stop criteria in ASReview > Abstract > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Conclusion · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In 22% of the simulations, 1 study was erroneously excluded and in 36% of the simulations, more than 1 study was excluded.”
§ When to stop reviewing: validation of stop criteria in ASReview > Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
24 more claims
Create a free account to see all 50 claims and to verify the 11 references still open.
- It may state two separate claims as one. 19
- Probably not a substantial claim. 6
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ When to stop reviewing: validation of stop criteria in ASReview > Results
“1 = Appenzeller-Herzog( 2019) [16]”
§ When to stop reviewing: validation of stop criteria in ASReview > Methods
“Appenzeller-Herzog (2019) [17]”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ When to stop reviewing: validation of stop criteria in ASReview > Background
“100 consecutive irrelevant papers, or about 0.3% of the total dataset.”
§ When to stop reviewing: validation of stop criteria in ASReview > Conclusion
“The use of stop criterion 2 in the various studies at 0.03% [13] or 5% [14] seems highly questionable”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“| Total | 66.7 | 46.3 | 7.3 | 29.2 | 4.5 | 21.6 | 4.0 | 17.4 | 3.7 | 14.9 | 3.2 | 13.4 | 3.6 | 12.3 |”
§ When to stop reviewing: validation of stop criteria in ASReview > Results
Reported 7.3 · recomputed 4.08
Citations
Cites
Cochrane Handbook for Systematic Reviews of Interventions
(2019) · doi:10.1002/9781119536604 · Cited by 14,828
Bornmann, L., & Mutz, R. (2015) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.23329 · Cited by 1,576
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157
A Systematic Literature Review on Fault Prediction Performance in Software Engineering
Hall, T., Beecham, S., Bowes, D., Gray, D., & Counsell, S. (2012) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2011.103 · Cited by 1,154
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638
Harrison, H., Griffin, S. J., Kuhn, I., & Usher-Smith, J. A. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-0897-3 · Cited by 449
Kellermeyer, L., Harnke, B., & Knight, S. L. (2018) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2018.513 · Cited by 404
Bos, D., Wolters, F. J., Darweesh, S. K. L., Vernooij, M. W., de Wolf, F., Ikram, M. A., & Hofman, A. (2018) · Alzheimer's & Dementia · doi:10.1016/j.jalz.2018.04.007 · Cited by 192
Functional magnetic resonance imaging, deep learning, and Alzheimer's disease: A systematic review
Warren, S. L., & Moustafa, A. A. (2023) · Journal of Neuroimaging · doi:10.1111/jon.13063 · Cited by 161
Statistical stopping criteria for automated screening in systematic reviews
Callaghan, M. W., & Müller-Hansen, F. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01521-4 · Cited by 121
Cleo, G., Scott, A. M., Islam, F., Julien, B., & Beller, E. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1069-6 · Cited by 112
SYMBALS: A Systematic Review Methodology Blending Active Learning and Snowballing
van Haastrecht, M., Sarhan, I., Yigit Ozkan, B., Brinkhuis, M., & Spruit, M. (2021) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2021.685591 · Cited by 80
Appenzeller‐Herzog, C., Mathes, T., Heeres, M. L. S., Weiss, K. H., Houwen, R. H. J., & Ewald, H. (2019) · Liver International · doi:10.1111/liv.14179 · Cited by 51
Pharmacokinetics and Associated Efficacy of Emicizumab in Humans: A Systematic Review
Donners, A. A. M. T., Rademaker, C. M. A., Bevers, L. A. H., Huitema, A. D. R., Schutgens, R. E. G., Egberts, T. C. G., & Fischer, K. (2021) · Clinical Pharmacokinetics · doi:10.1007/s40262-021-01042-w · Cited by 40
Callaghan, M., Müller-Hansen, F., Bond, M., Hamel, C., Devane, D., Kusa, W., O’Mara-Eves, A., Spijker, R., Stevenson, M., Stansfield, C., et al. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02699-7 · Cited by 27
Assessing the article screening efficiency of artificial intelligence for Systematic Reviews
Chan, Y.-T., Abad, J. E., Dibart, S., & Kernitsky, J. R. (2024) · Journal of Dentistry · doi:10.1016/j.jdent.2024.105259 · Cited by 23
Franzoi, D., Bockting, C. L., Bennett, K. F., Odom, A., Lucassen, P. J., Pathania, A., Lee, A., Brouwer, M. E., van de Schoot, R., Wiers, R. W., et al. (2024) · SSM - Population Health · doi:10.1016/j.ssmph.2023.101575 · Cited by 21
Adamse, I., Eichelsheim, V., Blokland, A., & Schoonmade, L. (2024) · European Journal of Criminology · doi:10.1177/14773708241250278 · Cited by 20
SYNERGY - Open machine learning dataset on study selection in systematic reviews
De Bruin, J., Ma, Y., Ferdinands, G., Teijema, J., & Van de Schoot, R. (2023) · DataverseNL · doi:10.34894/he6naq · Cited by 20
A meta-analysis on the role older adults with cancer favour in treatment decision making
van der Waal, M. S., Seghers, N., Welsing, P. M. J., van Huis, L. H., Emmelot-Vonk, M. H., & Hamaker, M. E. (2023) · Journal of Geriatric Oncology · doi:10.1016/j.jgo.2022.09.012 · Cited by 7
Cited by
Xu, H., Liu, J., & Cheng, Q. (2026) · Applied Sciences · doi:10.3390/app16125827
Steijger, D. J. M., Thissen, S., Aarts, S., Verbeek, H., de Vugt, M. E., & Christie, H. L. (2026) · JMIR Formative Research · doi:10.2196/84989
PASHA, M., & Shaikᵇ, M. A. (2026) · Research Square · doi:10.21203/rs.3.rs-10938971/v1
Wijaya, D., Murdani, A. L., Balafas, S. E., Rollema, C., Jansman, F. G. A., & Taxis, K. (2026) · British Journal of Clinical Pharmacology · doi:10.1002/bcp.70883
Romaniuk, P., Kaczmarek, K., Alcer, K., Zadworna‐Strzeszyna, N., Kaner, G., Depboylu, G. Y., van Duin, E. W., Sonneveld, N., O’Connor, N., Djojosoeparto, S. K., et al. (2026) · Obesity Reviews · doi:10.1111/obr.70252
Metadata
- Authors
- Kempny, C., Annac, K., Wahidie, D., Yilmaz-Aslan, Y., & Brzoska, P.
- Affiliations
- Witten/Herdecke University (DE)
- Corresponding author
- Kempny, C.
- Venue
- BMC Medical Research Methodology
- Published
- 2026-05-09
- Volume
- 26
- Issue
- 1
- ISSN
- 1471-2288 (ISSN-L)
- PMC
- PMC13156873
- OpenAlex
- W7160704877
- Semantic Scholar
- 1aa54fc96e93d26a878fe61c08ee316562b20db5
- Cited by
- 5
- References
- 22
- Citations by year
- 2026: 5
- FWCI
- 11.02
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- BMC Medical Research Methodology (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- Private Universität Witten/Herdecke gGmbH
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- ASReview, title and abstract screening, active learning (machine learning), machine learning screening
- MeSH terms
- Machine Learning, Computer Simulation, Humans, Review Literature as Topic, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-11-12, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-11-12 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
