An Evaluation of the Performance of Stopping Rules in AI-aided Screening for Psychological Meta-Analytical Research
Open accessAbstract
Several AI-aided screening tools have emerged to tackle the ever-expanding body of literature. These tools employ active learning, where algorithms sort abstracts based on human feedback. However, researchers using these tools face a crucial dilemma: When should they stop screening without knowing the proportion of relevant studies? Although numerous stopping rules have been proposed to guide users in this decision, they have yet to undergo comprehensive evaluation. In this study, we evaluated the performance of three stopping rules: the knee method, a data-driven heuristic, and a prevalence estimation technique. We measured performance via sensitivity, specificity, and screening cost and explored the influence of the prevalence of relevant studies and the choice of the learning algorithm. We curated a dataset of abstract collections from meta-analyses across five psychological research domains. Our findings revealed performance differences between stopping rules regarding all performance measures and variations in the performance of stopping rules across different prevalence ratios. Moreover, despite the relatively minor impact of the learning algorithm, we found that specific combinations of stopping rules and learning algorithms were most effective for certain prevalence ratios of relevant abstracts. Based on these results, we derived practical recommendations for users of AI-aided screening tools. Furthermore, we discuss possible implications and offer suggestions for future research.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The effectiveness of heuristic strategies depends on various factors, including the database size, the percentage of relevant abstracts, the topic complexity, and the learning algorithm used.
“Consistent with prior studies, our exploratory analysis suggests that the effectiveness of heuristic strategies depends on various factors, including the database size, the percentage of relevant abstracts (König et al., 2023), the topic complexity, and the learning algorithm used (Howard et al., 2020).”
From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as König et al., 2023
The Performance Results section reports that the heuristic stopping rule performed best in collections with many relevant abstracts, supporting the point about the percentage of relevant abstracts. It also reports variation across collections, but says the study could not distinguish an effect of collection size from an effect of the number of relevant abstracts. The citation to König et al. is attached to size and percentage; the sentence cites Howard et al. separately for topic complexity and learning algorithm. (LLM, conf=0.87)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
No citation lists for this paper
No citation lists for this paper yet — checked again daily.
References 99 · Cited by 33
Research works printed but not linked: 61
- Aggarwal, C. C. (2018). Neural networks and deep learning: A textbook. Springer International Publishing. https://doi.org/10.1007/978-3-319-94463-0 · DOI
- Alden, L. E., Matthews, L. R., Wagner, S., Fyfe, T., Randall, C., Regehr, C., White, M., Buys, N., Carey, M. G., Corneil, W., White, N., Fraess-Phillips, A., & Krutop, E. (2021). Systematic literature review of psychological interventions for first responders. Work & Stress, 35(2), 193-215. https://doi.org/10.1080/02678373.2020.1758833 · DOI
- Boetje, J., & van de Schoot, R. van. (2023). The SAFE procedure: A practical stopping heuristic for active learning-Based screening in systematic reviews and meta-analyses [Preprint]. Research Square. https://doi.org/10.21203/rs.3.rs-2856011/v1 · DOI
- Bottema-Beutel, K., Crowley, S., Sandbank, M., & Woynaroski, T. G. (2021). Research Review: Conflicts of Interest (COIs) in autism early intervention research -a meta- analysis of COI influences on intervention effects. Journal of Child Psychology and Psychiatry, 62(1), 5-15. https://doi.org/10.1111/jcpp.13249 · DOI · Paper page
- Bourke, M., Haddara, A., Loh, A., Carson, V., Breau, B., & Tucker, P. (2023). Adherence to the World Health Organization's physical activity recommendation in preschool-aged children: A systematic review and meta-analysis of accelerometer studies. International Journal of Behavioral Nutrition and Physical Activity, 20(1), 52. https://doi.org/10.1186/s12966-023-01450-0 · DOI
- Bourke, M., Patten, R. K., Dash, S., Pascoe, M., Craike, M., Firth, J., Bailey, A., Jacka, F., & Parker, A. G. (2022). The Effect of Interventions That Target Multiple Modifiable Health Behaviors on Symptoms of Anxiety and Depression in Young People: A Meta- Analysis of Randomized Controlled Trials. Journal of Adolescent Health, 70(2), 208- 219. https://doi.org/10.1016/j.jadohealth.2021.08.005 · DOI
- Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5-32. https://doi.org/10.1023/A:1010933404324 · DOI · Paper page
- Burgard, T., & Bittermann, A. (2023). Reducing literature screening workload with machine learning: A systematic review of tools and their performance. Zeitschrift Für Psychologie, 231(1), 3-15. https://doi.org/10.1027/2151-2604/a000509 · DOI · Paper page
- Campos, D., Fütterer, T., Gfrörer, T., Lavelle-Hill, R. E., Murayama, K., König, L., Hecht, M., Zitzmann, S., & Scherer, R. (2023). Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/fpwc2 · DOI
- Castro-Alonso, J. C., Wong, R. M., Adesope, O. O., & Paas, F. (2021). Effectiveness of Multimedia Pedagogical Agents Predicted by Diverse Theories: A Meta-Analysis. Educational Psychology Review, 33(3), 989-1015. https://doi.org/10.1007/s10648- Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006). Reducing workload in systematic review preparation using automated citation classification. Journal of the American Medical Informatics Association, 13(2), 206-219. https://doi.org/10.1197/jamia.M1929 · DOI
- Cohen, J. (2013). Statistical power analysis for the behavioral sciences (0 ed.). Routledge. https://doi.org/10.4324/9780203771587 · DOI · Paper page
- Cooper, H. M., Hedges, L. V., & Valentine, J. C. (Eds.). (2009). The handbook of research synthesis and meta-analysis (2nd ed). Russell Sage Foundation.
- Cormack, G. V., & Grossman, M. R. (2016). Engineering quality and reliability in technology-assisted review. Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, 75-84. https://doi.org/10.1145/2911451.2911510 · DOI · Paper page
- Dailey, S., & Bergelson, E. (2022). Language input to infants of different socioeconomic statuses: A quantitative meta-analysis. Developmental Science, 25(3), e13192. https://doi.org/10.1111/desc.13192 · DOI
- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. https://doi.org/10.48550/ARXIV.1810.04805 · DOI · Paper page
- Elliott, J. H., Synnot, A., Turner, T., Simmonds, M., Akl, E. A., McDonald, S., Salanti, G., Meerpohl, J., MacLehose, H., Hilton, J., Tovey, D., Shemilt, I., Thomas, J., Agoritsas, T., Hilton, J., Perron, C., Akl, E., Hodder, R., Pestridge, C., … Pearson, L. (2017). Living systematic review: 1. Introduction -the why, what, when, and how. Journal of Clinical Epidemiology, 91, 23-30. https://doi.org/10.1016/j.jclinepi.2017.08.010 · DOI · Paper page
- Endendijk, J. J., van Baar, A. L., & Deković, M. (2020). He is a Stud, She is a Slut! A Meta- Analysis on the Continued Existence of Sexual Double Standards. Personality and Social Psychology Review, 24(2), 163-190. https://doi.org/10.1177/1088868319891310 · DOI · Paper page
- Estevez Cores, S., Sayed, A. A., Tracy, D. K., & Kempton, M. J. (2021). Individual-focused occupational health interventions: A meta-analysis of randomized controlled trials. Journal of Occupational Health Psychology, 26(3), 189-203. https://doi.org/10.1037/ocp0000249 · DOI
- Ferdinands, G. (2021). AI-assisted systematic reviewing: Selecting studies to compare bayesian versus frequentist SEM for small sample sizes. Multivariate Behavioral Research, 56(1), 153-154. https://doi.org/10.1080/00273171.2020.1853501 · DOI
- Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., & van de Schoot, R. (2020). Active learning for screening prioritization in systematic reviews- A simulation study [Preprint]. Open Science Framework. https://doi.org/10.31219/osf.io/w6qbg · DOI · Paper page
- Gomes, S. R., Saroar, S. G., Mosfaiul, M., Telot, A., Khan, B. N., Chakrabarty, A., & Mostakim, M. (2017). A comparative approach to email classification using Naive Bayes classifier and hidden Markov model. 2017 4th International Conference on Advances in Electrical Engineering (ICAEE), 482-487. https://doi.org/10.1109/ICAEE.2017.8255404 · DOI
- Greenhouse, S. W., & Geisser, S. (1959). On methods in the analysis of profile data. Psychometrika, 24(2), 95-112. https://doi.org/10.1007/BF02289823 · DOI
- Guan, X., Feng, X., & Islam, A. Y. M. A. (2023). The dilemma and countermeasures of educational data ethics in the age of intelligence. Humanities and Social Sciences Communications, 10(1), 138. https://doi.org/10.1057/s41599-023-01633-x · DOI · Paper page
- Hall, C., Dahl-Leonard, K., Cho, E., Solari, E. J., Capin, P., Conner, C. L., Henry, A. R., Cook, L., Hayes, L., Vargas, I., Richmond, C. L., & Kehoe, K. F. (2023). Forty Years of Reading Intervention Research for Elementary Students with or at Risk for Dyslexia: A Systematic Review and Meta-Analysis. Reading Research Quarterly, 58(2), 285-312. https://doi.org/10.1002/rrq.477 · DOI
- Harmsen, W., de Groot, J., Harkema, A., van Dusseldorp, I., De Bruin, J., Van den Brand, S., & Van de Schoot, R. (2021). Artificial intelligence supports literature screening in medical guideline development: Towards up-to-date medical guidelines. [Preprint]. https://doi.org/10.5281/ZENODO.5031907 · DOI
- Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. (2013). Applied logistic regression (Third edition). Wiley.
- Hsieh, W., Faulkner, N., & Wickes, R. (2022). What reduces prejudice in the real world? A meta-analysis of prejudice reduction field experiments. British Journal of Social Psychology, 61(3), 689-710. https://doi.org/10.1111/bjso.12509 · DOI
- Jackson, P., & Moulinier, I. (2007). Natural language processing for online applications: Text retrieval, extraction and categorization. Second revised edition (2nd ed., Vol. 5). John Benjamins Publishing Company. https://doi.org/10.1075/nlp. · DOI
- Kastner, M., Straus, S. E., McKibbon, K. A., & Goldsmith, C. H. (2009). The capture-mark- recapture technique can be used as a stopping rule when searching in systematic reviews. Journal of Clinical Epidemiology, 62(2), 149-157. https://doi.org/10.1016/j.jclinepi.2008.06.001 · DOI · Paper page
- Khazanov, G. K., Morris, P. E., Beed, A., Jager-Hyman, S., Myhre, K., McKay, J. R., Feinn, R. S., Boland, E. M., & Thase, M. E. (2022). Do financial incentives increase mental health treatment engagement? A meta-analysis. Journal of Consulting and Clinical Psychology, 90(6), 528-544. https://doi.org/10.1037/ccp0000737 · DOI
- Le, Q. V., & Mikolov, T. (2014). Distributed representations of sentences and documents. https://doi.org/10.48550/ARXIV.1405.4053 · DOI
- Leijten, P., Weisz, J. R., & Gardner, F. (2021). Research Strategies to Discern Active Psychological Therapy Components: A Scoping Review. Clinical Psychological Science, 9(3), 307-322. https://doi.org/10.1177/2167702620978615 · DOI · Paper page
- Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval (1st ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511809071 · DOI · Paper page
- Marshall, I. J., & Wallace, B. C. (2019). Toward systematic review automation: A practical guide to using machine learning tools in research synthesis. Systematic Reviews, 8(1), 163, s13643-019-1074-1079. https://doi.org/10.1186/s13643-019-1074-9 · DOI · Paper page
- Marsili, F., & Pellegrini, M. (2022). The relation between nominations and traditional measures in the gifted identification process: A meta-analysis. School Psychology International, 43(4), 321-338. https://doi.org/10.1177/01430343221105398 · DOI
- Ober, T. M., Brooks, P. J., Homer, B. D., & Rindskopf, D. (2020). Executive Functions and Decoding in Children and Adolescents: A Meta-analytic Investigation. Educational Psychology Review, 32(3), 735-763. https://doi.org/10.1007/s10648-020-09526-0 · DOI
- Pham, B., Bagheri, E., Rios, P., Pourmasoumi, A., Robson, R. C., Hwee, J., Isaranuwatchai, W., Darvesh, N., Page, M. J., & Tricco, A. C. (2018). Improving the conduct of systematic reviews: A process mining perspective. Journal of Clinical Epidemiology, 103, 101-111. https://doi.org/10.1016/j.jclinepi.2018.06.011 · DOI
- Platt, J. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers (Vol. 3, pp. 61-74).
- Reimer, N. K., & Sengupta, N. K. (2023). Meta-analysis of the "ironic" effects of intergroup contact. Journal of Personality and Social Psychology, 124(2), 362-380. https://doi.org/10.1037/pspi0000404 · DOI
- Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3980-3990. https://doi.org/10.18653/v1/D19-1410 · DOI · Paper page
- Ros, R., Bjarnason, E., & Runeson, P. (2017). A machine learning approach for semi- automated search and selection in literature studies. Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, 118-127. https://doi.org/10.1145/3084226.3084243 · DOI · Paper page
- Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing & Management, 24(5), 513-523. https://doi.org/10.1016/0306-4573(88)90021-0 · DOI
- Sammons, M., Christodoulopoulos, C., Kordjamshidi, P., Kordjamshidi, D., Srikumar, V., Vijayakumar, P., Bokhari, M., Wu, X., & Roth, D. (2016). Edison: Feature extraction for NLP, simplified. In Proceedings of the 10th International Conference on Language Resources and Evaluation, LREC 2016 (pp. 4085-4092). European Language Resources Association (ELRA).
- Satopaa, V., Albrecht, J., Irwin, D., & Raghavan, B. (2011). Finding a "Kneedle" in a haystack: Detecting knee points in system behavior. 2011 31st International Conference on Distributed Computing Systems Workshops, 166-171. https://doi.org/10.1109/ICDCSW.2011. · DOI
- Schielzeth, H., Dingemanse, N. J., Nakagawa, S., Westneat, D. F., Allegue, H., Teplitsky, C., Réale, D., Dochtermann, N. A., Garamszegi, L. Z., & Araya-Ajoy, Y. G. (2020). Robustness of linear mixed-effects models to violations of distributional assumptions. Methods in Ecology and Evolution, 11(9), 1141-1152. https://doi.org/10.1111/2041- 210X.13434 · DOI · Paper page
- Shemilt, I., Khan, N., Park, S., & Thomas, J. (2016). Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Systematic Reviews, 5(1), 140. https://doi.org/10.1186/s13643-016-0315-4 · DOI · Paper page
- Simonsmeier, B. A., Flaig, M., Deiglmayr, A., Schalk, L., & Schneider, M. (2022). Domain- specific prior knowledge and learning: A meta-analysis. Educational Psychologist, 57(1), 31-54. https://doi.org/10.1080/00461520.2021.1939700 · DOI
- Tang, X., Renninger, K. A., Hidi, S. E., Murayama, K., Lavonen, J., & Salmela-Aro, K. (2022). The differences and similarities between curiosity and interest: Meta-analysis and network analyses. Learning and Instruction, 80, 101628. https://doi.org/10.1016/j.learninstruc.2022.101628 · DOI
- Teijema, J. J., Hofstee, L., Brouwer, M., de Bruin, J., Ferdinands, G., de Boer, J., Vizan, P., van den Brand, S., Bockting, C., van de Schoot, R., & Bagheri, A. (2023). Active learning-based systematic reviewing using switching classification models: The case of the onset, maintenance, and relapse of depressive disorders. Frontiers in Research Metrics and Analytics, 8, 1178181. https://doi.org/10.3389/frma.2023.1178181 · DOI · Paper page
- Tian, X., Xia, Z., Xie, J., Zhang, C., Liu, Y., & Xu, M. (2022). A meta-analytical review of intervention experiments to reduce food waste. Environmental Research Letters, 17(6), 064041. https://doi.org/10.1088/1748-9326/ac72b6 · DOI
- Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014). Systematic review automation technologies. Systematic Reviews, 3(1), 74. https://doi.org/10.1186/2046-4053-3-74 · DOI · Paper page
- van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., Harkema, A., Willemsen, J., Ma, Y., Fang, Q., Hindriks, S., Tummers, L., & Oberski, D. L. (2021b). An open source machine learning framework for efficient and transparent systematic reviews. Nature Machine Intelligence, 3(2), 125-133. https://doi.org/10.1038/s42256-020-00287-7 · DOI · Paper page
- van Haastrecht, M., Sarhan, I., Yigit Ozkan, B., Brinkhuis, M., & Spruit, M. (2021). SYMBALS: A Systematic Review Methodology Blending Active Learning and Snowballing. Frontiers in Research Metrics and Analytics, 6, 685591. https://doi.org/10.3389/frma.2021.685591 · DOI · Paper page
- Vapnik, V. N. (1995). The nature of statistical learning theory. Springer New York. https://doi.org/10.1007/978-1-4757-2440-0 · DOI · Paper page
- Vermillet, A., Tølbøll, K., Litsis Mizan, S., C. Skewes, J., & Parsons, C. E. (2022). Crying in the first 12 months of life: A systematic review and meta-analysis of cross-country parent-reported data and modeling of the "cry curve." Child Development, 93(4), 1201-1222. https://doi.org/10.1111/cdev.13760 · DOI
- Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010). Semi- automated screening of biomedical citations for systematic reviews. BMC Bioinformatics, 11(1), 55. https://doi.org/10.1186/1471-2105-11-55 · DOI · Paper page
- Wang, Z., Nayfeh, T., Tetzlaff, J., O'Blenis, P., & Murad, M. H. (2020). Error rates of human reviewers during abstract screening in systematic reviews. PLOS ONE, 15(1), e0227742. https://doi.org/10.1371/journal.pone.0227742 · DOI · Paper page
- Westgate, M. J. (2019). revtools: An R package to support article screening for evidence synthesis. Research Synthesis Methods, 10(4), 606-614. https://doi.org/10.1002/jrsm.1374 · DOI · Paper page
- Woods, S., Dunne, S., Gallagher, P., & McNicholl, A. (2022). A systematic review of the factors associated with athlete burnout in team sports. International Review of Sport and Exercise Psychology, 1-41. https://doi.org/10.1080/1750984X.2022.2148225 · DOI
- Yu, Z., & Menzies, T. (2019). FAST2: An intelligent assistant for finding relevant papers. Expert Systems with Applications, 120, 57-71. https://doi.org/10.1016/j.eswa.2018.11.021 · DOI · Paper page
- Zaneva, M., Guzman-Holst, C., Reeves, A., & Bowes, L. (2022). The Impact of Monetary Poverty Alleviation Programs on Children's and Adolescents' Mental Health: A Systematic Review and Meta-Analysis Across Low-, Middle-, and High-Income Countries. Journal of Adolescent Health, 71(2), 147-156. https://doi.org/10.1016/j.jadohealth.2022.02.011 2444; 218 9.79 2010; 10 2121; 21 2205; 105 2277; 207 · DOI
- Authors
- König, L., Zitzmann, S., Fütterer, T., Campos, D., Scherer, R., & Hecht, M.
- Affiliations
- Helmut Schmidt University (DE)
- Universität Hamburg (DE)
- University of Tübingen (DE)
- University of Oslo (NO)
- Corresponding author
- König, Lars
- Venue
- Research Synthesis Methods
- Publisher
- Center for Open Science
- Published
- 2023-11-30
- Volume
- 15
- Issue
- 6
- Pages
- 1120-1146
- ISSN
- 1759-2879 (ISSN-L)
- OpenAlex
- W4403465916
- Cited by
- 33
- References
- 99
- Citations by year
- 2026: 18 · 2025: 12 · 2024: 2 · 2023: 1
- FWCI
- 8.87
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Locations
- Research Synthesis Methods (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
- Funders
- European Commission (101132474)
- Norges Forskningsråd (331640)
- Universität Hamburg
- HORIZON EUROPE Framework Programme (101132474)
- NextGenerationEU
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › General Social Sciences › Computational and Text Analysis Methods
- Keywords
- stopping rules, active learning (machine learning), prevalence estimation, abstract screening, meta-analysis, psychology research, screening costs, sensitivity and specificity, learning algorithm comparison
- MeSH terms
- Algorithms, Artificial Intelligence, Humans, Research Design, Sensitivity and Specificity, Meta-Analysis as Topic, Reproducibility of Results
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review