The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.
Computer-assisted screening in systematic evidence synthesis requires robust and well-evaluated stopping criteria
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These findings are consistent with critiques raised by Callaghan and Müller-Hansen (2020) [11], who demonstrated statistical limitations of fixed stopping criteria, and Callaghan et al. (2024) [22], who argued that universal stopping criteria are unlikely to be sufficient without considering additional parameters such as dataset characteristics and topic complexity.”
From When to stop reviewing: validation of stop criteria in ASReview (2026) · cites this paper as [22]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
Self-citationLLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
Self-citationReducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411
Howard, B. E., Phillips, J., Tandon, A., Maharana, A., Elmore, R., Mav, D., Sedykh, A., Thayer, K., Merrick, B. A., Walker, V., et al. (2020) · Environment International · doi:10.1016/j.envint.2020.105623 · Cited by 157
Four principles to make evidence synthesis more useful for policy
Donnelly, C. A., Boyd, I., Campbell, P., Craig, C., Vallance, P., Walport, M., Whitty, C. J. M., Woods, E., & Wormald, C. (2018) · Nature · doi:10.1038/d41586-018-05414-4 · Cited by 157
Michelson, M., & Reuter, K. (2019) · Contemporary Clinical Trials Communications · doi:10.1016/j.conctc.2019.100443 · Cited by 152
Prioritising references for systematic reviews with RobotAnalyst: A user study
Przybyła, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M., McNaught, J., von Elm, E., Nolan, K., & Ananiadou, S. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1311 · Cited by 148
Boetje, J., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02502-7 · Cited by 146
Research works printed but not linked: 21
- Saldanha, I. J., Adam, G. P., Schmid, C. H., Trikalinos, T. A., & Konnyu, K. J. (2023). Modernizing evidence synthesis for evidence-based medicine. In Clinical Decision Support and beyond: Progress and Opportunities in Knowledge-Enhanced Health and Healthcare (pp. 257–278). Elsevier. https://doi.org/10.1016/B978-0-323-91200-6.00006-1 · DOI
- Surkovic E, Vigar D. Scientific advice for policymakers on climate change: the role of evidence synthesis. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 2022;380(2221):20210147. https://doi.org/10.1098/rsta.2021.0147. · DOI
- Lefebvre, C., Glanville, J., Briscoe, S., Featherstone, R., Metzendorf, M.-I., Noel-Storr, A., Paynter, R., Rader, T., Thomas, J., & Wieland, L. (2023). Chapter 4: Searching for and selecting studies. In J. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M. Page, & V. Welch, Cochrane Handbook for Systematic Reviews of Interventions (Version 6.4 (updated October 2023)). https://training.cochrane.org/handbook/current/chapter-04
- Higgins, J., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M., & Welch, V. (Eds.). (2019). Cochrane Handbook for Systematic Reviews of Interventions (2nd ed.). John Wiley & Sons. · DOI · Paper page
- Bornmann L, Mutz R. Growth rates of modern science: a bibliometric analysis based on the number of publications and cited references. J Am Soc Inf Sci. 2015;66(11):2215–22. https://doi.org/10.1002/asi.23329. · DOI · Paper page
- Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M. T., & Zhang, Y. (2023). Sparks of artificial general intelligence: early experiments with GPT-4 (arXiv:2303.12712). arXiv. https://doi.org/10.48550/arXiv.2303.12712 · DOI · Paper page
- Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., … Rush, A. (2020). Transformers: state-of-the-art natural language processing. In Q. Liu & D. Schlangen (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations (pp. 38–45). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-demos.6 · DOI · Paper page
- Chappell M, Edwards M, Watkins D, Marshall C, Graziadio S. Machine learning for accelerating screening in evidence reviews. Cochrane Evidence Synthesis and Methods. 2023;1(5): e12021. https://doi.org/10.1002/cesm.12021. · DOI
- Cormack, G. V., & Grossman, M. R. (2014). Evaluation of machine-learning protocols for technology-assisted review in electronic discovery. Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, 153–162. https://doi.org/10.1145/2600428.2609601 · DOI · Paper page
- Lewis, D. D., Gray, L., & Noel, M. (2023). Confidence sequences for evaluating one-phase technology-assisted review. Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, 131–140. https://doi.org/10.1145/3594536.3595167 · DOI
- Callaghan, M., & Müller-Hansen, F. (2020). Statistical stopping criteria for automated screening in systematic reviews. Systematic Reviews. https://doi.org/10.21203/rs.2.18218/v2 · DOI
- Lefebvre, C., Glanville, J., Briscoe, S., A Littlewood, Marshall, C., Metzendorf, M.-I., Noel-Storr, A., Rader, T., Shokraneh, F., Thomas, J., & Wieland, L. (2019). Chapter 4: Searching for and selecting studies. In J. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M. Page, & V. Welch, Cochrane Handbook for Systematic Reviews of Interventions (Version 6 (updated October 2019)). https://training.cochrane.org/handbook/current/chapter-04 · DOI · Paper page
- MacDonald H, Comer C, Foster M, Labelle PR, Marsalis S, Nyhan K, Premji Z, Rogers M, Splenda R, Stansfield C, Young S. Searching for studies: a guide to information retrieval for Campbell systematic reviews. Campbell Syst Rev. 2024;20(3): e1433. https://doi.org/10.1002/cl2.1433. · DOI
- Molinari A, Esuli A. SALτ: efficiently stopping TAR by improving priors estimates. Data Min Knowl Disc. 2024;38(2):535–68. https://doi.org/10.1007/s10618-023-00961-5. · DOI
- Sneyd, A., & Stevenson, M. (2019). Modelling stopping criteria for search results using poisson processes. In K. Inui, J. Jiang, V. Ng, & X. W. 0001 (Eds.), Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, EMNLP-IJCNLP 2019, hong kong, china, november 3–7, 2019 (pp. 3482–3487). Association for Computational Linguistics. https://doi.org/10.18653/v1/D19-1351 · DOI
- Stevenson M, Bin-Hezam R. Stopping methods for technology assisted reviews based on point processes. ACM Transactions on Information Systems. 2024;42(3):1–37. https://doi.org/10.1145/3631990. · DOI
- The Systematic Review Toolbox. (n.d.). Retrieved 12 October 2023, from http://systematicreviewtools.com/software.php
- Jonnalagadda S, Petitti D. A new iterative method to reduce workload in systematic review process. Int J Comput Biol Drug Des. 2013;6(1–2):5–17. https://doi.org/10.1504/IJCBDD.2013.052198. · DOI · Paper page
- Kusa, W., Zuccon, G., Knoth, P., & Hanbury, A. (2023). Outcome-based evaluation of systematic review automation. Proceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval, 125–133. https://doi.org/10.1145/3578337.3605135 · DOI
- Scholer, F., Kelly, D., Wu, W.-C., Lee, H. S., & Webber, W. (2013). The effect of threshold priming and need for cognition on relevance calibration and assessment. Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval, 623–632. https://doi.org/10.1145/2484028.2484090 · DOI
- Stansfield C, Stokes G, Thomas J. Applying machine classifiers to update searches: analysis from two case studies. Research Synthesis Methods. 2022;13(1):121–33. https://doi.org/10.1002/jrsm.1537. · DOI
Cited by
When to stop reviewing: validation of stop criteria in ASReview
Kempny, C., Annac, K., Wahidie, D., Yilmaz-Aslan, Y., & Brzoska, P. (2026) · BMC Medical Research Methodology · doi:10.1186/s12874-026-02866-5 · Cited by 5
Metadata
- Authors
- Callaghan, M., Müller-Hansen, F., Bond, M., Hamel, C., Devane, D., Kusa, W., O’Mara-Eves, A., Spijker, R., Stevenson, M., Stansfield, C., Thomas, J., & Minx, J. C.
- Venue
- Systematic Reviews
- Published
- 2024-11-22
- Volume
- 13
- Issue
- 1
- ISSN
- 2046-4053
- PMC
- PMC11583672
- Cited by
- 27
- References
- 12
- Language
- en
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Computer-assisted screening in systematic evidence synthesis requires robust and well-evaluated stopping criteria. https://app.cochat.ai/papers/10.1186/s13643-024-02699-7
- The paper
- Callaghan, M., Müller-Hansen, F., Bond, M., Hamel, C., Devane, D., Kusa, W., O’Mara-Eves, A., Spijker, R., Stevenson, M., Stansfield, C., Thomas, J., & Minx, J. C. (2024). Computer-assisted screening in systematic evidence synthesis requires robust and well-evaluated stopping criteria. Systematic Reviews. https://doi.org/10.1186/s13643-024-02699-7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
