PreprintCited by 33Meta-analysis and systematic reviewsInformation Retrieval and Search BehaviorComputational and Text Analysis Methods

An Evaluation of the Performance of Stopping Rules in AI-aided Screening for Psychological Meta-Analytical Research

Open access
König, Lars, Zitzmann, Steffen, Fütterer, Tim, Campos, Diego, Scherer, Ronny, Hecht, Martin·2023·Research Synthesis Methods

Abstract

Several AI-aided screening tools have emerged to tackle the ever-expanding body of literature. These tools employ active learning, where algorithms sort abstracts based on human feedback. However, researchers using these tools face a crucial dilemma: When should they stop screening without knowing the proportion of relevant studies? Although numerous stopping rules have been proposed to guide users in this decision, they have yet to undergo comprehensive evaluation. In this study, we evaluated the performance of three stopping rules: the knee method, a data-driven heuristic, and a prevalence estimation technique. We measured performance via sensitivity, specificity, and screening cost and explored the influence of the prevalence of relevant studies and the choice of the learning algorithm. We curated a dataset of abstract collections from meta-analyses across five psychological research domains. Our findings revealed performance differences between stopping rules regarding all performance measures and variations in the performance of stopping rules across different prevalence ratios. Moreover, despite the relatively minor impact of the learning algorithm, we found that specific combinations of stopping rules and learning algorithms were most effective for certain prevalence ratios of relevant abstracts. Based on these results, we derived practical recommendations for users of AI-aided screening tools. Furthermore, we discuss possible implications and offer suggestions for future research.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.31234/osf.io/ybu3w

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • The effectiveness of heuristic strategies depends on various factors, including the database size, the percentage of relevant abstracts, the topic complexity, and the learning algorithm used.

    “Consistent with prior studies, our exploratory analysis suggests that the effectiveness of heuristic strategies depends on various factors, including the database size, the percentage of relevant abstracts (König et al., 2023), the topic complexity, and the learning algorithm used (Howard et al., 2020).”

    From Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research (2024) · cites this paper as König et al., 2023

    The Performance Results section reports that the heuristic stopping rule performed best in collections with many relevant abstracts, supporting the point about the percentage of relevant abstracts. It also reports variation across collections, but says the study could not distinguish an effect of collection size from an effect of the number of relevant abstracts. The citation to König et al. is attached to size and percentage; the sentence cites Howard et al. separately for topic complexity and learning algorithm. (LLM, conf=0.87)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

No citation lists for this paper

No citation lists for this paper yet — checked again daily.

References 99 · Cited by 33

Research works printed but not linked: 61

  1. Aggarwal, C. C. (2018). Neural networks and deep learning: A textbook. Springer International Publishing. https://doi.org/10.1007/978-3-319-94463-0 · DOI
  2. Alden, L. E., Matthews, L. R., Wagner, S., Fyfe, T., Randall, C., Regehr, C., White, M., Buys, N., Carey, M. G., Corneil, W., White, N., Fraess-Phillips, A., & Krutop, E. (2021). Systematic literature review of psychological interventions for first responders. Work & Stress, 35(2), 193-215. https://doi.org/10.1080/02678373.2020.1758833 · DOI
  3. Boetje, J., & van de Schoot, R. van. (2023). The SAFE procedure: A practical stopping heuristic for active learning-Based screening in systematic reviews and meta-analyses [Preprint]. Research Square. https://doi.org/10.21203/rs.3.rs-2856011/v1 · DOI
  4. Bottema-Beutel, K., Crowley, S., Sandbank, M., & Woynaroski, T. G. (2021). Research Review: Conflicts of Interest (COIs) in autism early intervention research -a meta- analysis of COI influences on intervention effects. Journal of Child Psychology and Psychiatry, 62(1), 5-15. https://doi.org/10.1111/jcpp.13249 · DOI · Paper page
  5. Bourke, M., Haddara, A., Loh, A., Carson, V., Breau, B., & Tucker, P. (2023). Adherence to the World Health Organization's physical activity recommendation in preschool-aged children: A systematic review and meta-analysis of accelerometer studies. International Journal of Behavioral Nutrition and Physical Activity, 20(1), 52. https://doi.org/10.1186/s12966-023-01450-0 · DOI
  6. Bourke, M., Patten, R. K., Dash, S., Pascoe, M., Craike, M., Firth, J., Bailey, A., Jacka, F., & Parker, A. G. (2022). The Effect of Interventions That Target Multiple Modifiable Health Behaviors on Symptoms of Anxiety and Depression in Young People: A Meta- Analysis of Randomized Controlled Trials. Journal of Adolescent Health, 70(2), 208- 219. https://doi.org/10.1016/j.jadohealth.2021.08.005 · DOI
  7. Breiman, L. (2001). Random Forests. Machine Learning, 45(1), 5-32. https://doi.org/10.1023/A:1010933404324 · DOI · Paper page
  8. Burgard, T., & Bittermann, A. (2023). Reducing literature screening workload with machine learning: A systematic review of tools and their performance. Zeitschrift Für Psychologie, 231(1), 3-15. https://doi.org/10.1027/2151-2604/a000509 · DOI · Paper page
  9. Campos, D., Fütterer, T., Gfrörer, T., Lavelle-Hill, R. E., Murayama, K., König, L., Hecht, M., Zitzmann, S., & Scherer, R. (2023). Screening Smarter, Not Harder: A Comparative Analysis of Machine Learning Screening Algorithms and Heuristic Stopping Criteria for Systematic Reviews in Educational Research [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/fpwc2 · DOI
  10. Castro-Alonso, J. C., Wong, R. M., Adesope, O. O., & Paas, F. (2021). Effectiveness of Multimedia Pedagogical Agents Predicted by Diverse Theories: A Meta-Analysis. Educational Psychology Review, 33(3), 989-1015. https://doi.org/10.1007/s10648- Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006). Reducing workload in systematic review preparation using automated citation classification. Journal of the American Medical Informatics Association, 13(2), 206-219. https://doi.org/10.1197/jamia.M1929 · DOI
  11. Cohen, J. (2013). Statistical power analysis for the behavioral sciences (0 ed.). Routledge. https://doi.org/10.4324/9780203771587 · DOI · Paper page
  12. Cooper, H. M., Hedges, L. V., & Valentine, J. C. (Eds.). (2009). The handbook of research synthesis and meta-analysis (2nd ed). Russell Sage Foundation.
  13. Cormack, G. V., & Grossman, M. R. (2016). Engineering quality and reliability in technology-assisted review. Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, 75-84. https://doi.org/10.1145/2911451.2911510 · DOI · Paper page
  14. Dailey, S., & Bergelson, E. (2022). Language input to infants of different socioeconomic statuses: A quantitative meta-analysis. Developmental Science, 25(3), e13192. https://doi.org/10.1111/desc.13192 · DOI
  15. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. https://doi.org/10.48550/ARXIV.1810.04805 · DOI · Paper page
  16. Elliott, J. H., Synnot, A., Turner, T., Simmonds, M., Akl, E. A., McDonald, S., Salanti, G., Meerpohl, J., MacLehose, H., Hilton, J., Tovey, D., Shemilt, I., Thomas, J., Agoritsas, T., Hilton, J., Perron, C., Akl, E., Hodder, R., Pestridge, C., … Pearson, L. (2017). Living systematic review: 1. Introduction -the why, what, when, and how. Journal of Clinical Epidemiology, 91, 23-30. https://doi.org/10.1016/j.jclinepi.2017.08.010 · DOI · Paper page
  17. Endendijk, J. J., van Baar, A. L., & Deković, M. (2020). He is a Stud, She is a Slut! A Meta- Analysis on the Continued Existence of Sexual Double Standards. Personality and Social Psychology Review, 24(2), 163-190. https://doi.org/10.1177/1088868319891310 · DOI · Paper page
  18. Estevez Cores, S., Sayed, A. A., Tracy, D. K., & Kempton, M. J. (2021). Individual-focused occupational health interventions: A meta-analysis of randomized controlled trials. Journal of Occupational Health Psychology, 26(3), 189-203. https://doi.org/10.1037/ocp0000249 · DOI
  19. Ferdinands, G. (2021). AI-assisted systematic reviewing: Selecting studies to compare bayesian versus frequentist SEM for small sample sizes. Multivariate Behavioral Research, 56(1), 153-154. https://doi.org/10.1080/00273171.2020.1853501 · DOI
  20. Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., & van de Schoot, R. (2020). Active learning for screening prioritization in systematic reviews- A simulation study [Preprint]. Open Science Framework. https://doi.org/10.31219/osf.io/w6qbg · DOI · Paper page
  21. Gomes, S. R., Saroar, S. G., Mosfaiul, M., Telot, A., Khan, B. N., Chakrabarty, A., & Mostakim, M. (2017). A comparative approach to email classification using Naive Bayes classifier and hidden Markov model. 2017 4th International Conference on Advances in Electrical Engineering (ICAEE), 482-487. https://doi.org/10.1109/ICAEE.2017.8255404 · DOI
  22. Greenhouse, S. W., & Geisser, S. (1959). On methods in the analysis of profile data. Psychometrika, 24(2), 95-112. https://doi.org/10.1007/BF02289823 · DOI
  23. Guan, X., Feng, X., & Islam, A. Y. M. A. (2023). The dilemma and countermeasures of educational data ethics in the age of intelligence. Humanities and Social Sciences Communications, 10(1), 138. https://doi.org/10.1057/s41599-023-01633-x · DOI · Paper page
  24. Hall, C., Dahl-Leonard, K., Cho, E., Solari, E. J., Capin, P., Conner, C. L., Henry, A. R., Cook, L., Hayes, L., Vargas, I., Richmond, C. L., & Kehoe, K. F. (2023). Forty Years of Reading Intervention Research for Elementary Students with or at Risk for Dyslexia: A Systematic Review and Meta-Analysis. Reading Research Quarterly, 58(2), 285-312. https://doi.org/10.1002/rrq.477 · DOI
  25. Harmsen, W., de Groot, J., Harkema, A., van Dusseldorp, I., De Bruin, J., Van den Brand, S., & Van de Schoot, R. (2021). Artificial intelligence supports literature screening in medical guideline development: Towards up-to-date medical guidelines. [Preprint]. https://doi.org/10.5281/ZENODO.5031907 · DOI
  26. Hosmer, D. W., Lemeshow, S., & Sturdivant, R. X. (2013). Applied logistic regression (Third edition). Wiley.
  27. Hsieh, W., Faulkner, N., & Wickes, R. (2022). What reduces prejudice in the real world? A meta-analysis of prejudice reduction field experiments. British Journal of Social Psychology, 61(3), 689-710. https://doi.org/10.1111/bjso.12509 · DOI
  28. Jackson, P., & Moulinier, I. (2007). Natural language processing for online applications: Text retrieval, extraction and categorization. Second revised edition (2nd ed., Vol. 5). John Benjamins Publishing Company. https://doi.org/10.1075/nlp. · DOI
  29. Kastner, M., Straus, S. E., McKibbon, K. A., & Goldsmith, C. H. (2009). The capture-mark- recapture technique can be used as a stopping rule when searching in systematic reviews. Journal of Clinical Epidemiology, 62(2), 149-157. https://doi.org/10.1016/j.jclinepi.2008.06.001 · DOI · Paper page
  30. Khazanov, G. K., Morris, P. E., Beed, A., Jager-Hyman, S., Myhre, K., McKay, J. R., Feinn, R. S., Boland, E. M., & Thase, M. E. (2022). Do financial incentives increase mental health treatment engagement? A meta-analysis. Journal of Consulting and Clinical Psychology, 90(6), 528-544. https://doi.org/10.1037/ccp0000737 · DOI
  31. Le, Q. V., & Mikolov, T. (2014). Distributed representations of sentences and documents. https://doi.org/10.48550/ARXIV.1405.4053 · DOI
  32. Leijten, P., Weisz, J. R., & Gardner, F. (2021). Research Strategies to Discern Active Psychological Therapy Components: A Scoping Review. Clinical Psychological Science, 9(3), 307-322. https://doi.org/10.1177/2167702620978615 · DOI · Paper page
  33. Manning, C. D., Raghavan, P., & Schütze, H. (2008). Introduction to information retrieval (1st ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511809071 · DOI · Paper page
  34. Marshall, I. J., & Wallace, B. C. (2019). Toward systematic review automation: A practical guide to using machine learning tools in research synthesis. Systematic Reviews, 8(1), 163, s13643-019-1074-1079. https://doi.org/10.1186/s13643-019-1074-9 · DOI · Paper page
  35. Marsili, F., & Pellegrini, M. (2022). The relation between nominations and traditional measures in the gifted identification process: A meta-analysis. School Psychology International, 43(4), 321-338. https://doi.org/10.1177/01430343221105398 · DOI
  36. Ober, T. M., Brooks, P. J., Homer, B. D., & Rindskopf, D. (2020). Executive Functions and Decoding in Children and Adolescents: A Meta-analytic Investigation. Educational Psychology Review, 32(3), 735-763. https://doi.org/10.1007/s10648-020-09526-0 · DOI
  37. Pham, B., Bagheri, E., Rios, P., Pourmasoumi, A., Robson, R. C., Hwee, J., Isaranuwatchai, W., Darvesh, N., Page, M. J., & Tricco, A. C. (2018). Improving the conduct of systematic reviews: A process mining perspective. Journal of Clinical Epidemiology, 103, 101-111. https://doi.org/10.1016/j.jclinepi.2018.06.011 · DOI
  38. Platt, J. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers (Vol. 3, pp. 61-74).
  39. Reimer, N. K., & Sengupta, N. K. (2023). Meta-analysis of the "ironic" effects of intergroup contact. Journal of Personality and Social Psychology, 124(2), 362-380. https://doi.org/10.1037/pspi0000404 · DOI
  40. Reimers, N., & Gurevych, I. (2019). Sentence-BERT: Sentence embeddings using siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3980-3990. https://doi.org/10.18653/v1/D19-1410 · DOI · Paper page
  41. Ros, R., Bjarnason, E., & Runeson, P. (2017). A machine learning approach for semi- automated search and selection in literature studies. Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering, 118-127. https://doi.org/10.1145/3084226.3084243 · DOI · Paper page
  42. Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing & Management, 24(5), 513-523. https://doi.org/10.1016/0306-4573(88)90021-0 · DOI
  43. Sammons, M., Christodoulopoulos, C., Kordjamshidi, P., Kordjamshidi, D., Srikumar, V., Vijayakumar, P., Bokhari, M., Wu, X., & Roth, D. (2016). Edison: Feature extraction for NLP, simplified. In Proceedings of the 10th International Conference on Language Resources and Evaluation, LREC 2016 (pp. 4085-4092). European Language Resources Association (ELRA).
  44. Satopaa, V., Albrecht, J., Irwin, D., & Raghavan, B. (2011). Finding a "Kneedle" in a haystack: Detecting knee points in system behavior. 2011 31st International Conference on Distributed Computing Systems Workshops, 166-171. https://doi.org/10.1109/ICDCSW.2011. · DOI
  45. Schielzeth, H., Dingemanse, N. J., Nakagawa, S., Westneat, D. F., Allegue, H., Teplitsky, C., Réale, D., Dochtermann, N. A., Garamszegi, L. Z., & Araya-Ajoy, Y. G. (2020). Robustness of linear mixed-effects models to violations of distributional assumptions. Methods in Ecology and Evolution, 11(9), 1141-1152. https://doi.org/10.1111/2041- 210X.13434 · DOI · Paper page
  46. Shemilt, I., Khan, N., Park, S., & Thomas, J. (2016). Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Systematic Reviews, 5(1), 140. https://doi.org/10.1186/s13643-016-0315-4 · DOI · Paper page
  47. Simonsmeier, B. A., Flaig, M., Deiglmayr, A., Schalk, L., & Schneider, M. (2022). Domain- specific prior knowledge and learning: A meta-analysis. Educational Psychologist, 57(1), 31-54. https://doi.org/10.1080/00461520.2021.1939700 · DOI
  48. Tang, X., Renninger, K. A., Hidi, S. E., Murayama, K., Lavonen, J., & Salmela-Aro, K. (2022). The differences and similarities between curiosity and interest: Meta-analysis and network analyses. Learning and Instruction, 80, 101628. https://doi.org/10.1016/j.learninstruc.2022.101628 · DOI
  49. Teijema, J. J., Hofstee, L., Brouwer, M., de Bruin, J., Ferdinands, G., de Boer, J., Vizan, P., van den Brand, S., Bockting, C., van de Schoot, R., & Bagheri, A. (2023). Active learning-based systematic reviewing using switching classification models: The case of the onset, maintenance, and relapse of depressive disorders. Frontiers in Research Metrics and Analytics, 8, 1178181. https://doi.org/10.3389/frma.2023.1178181 · DOI · Paper page
  50. Tian, X., Xia, Z., Xie, J., Zhang, C., Liu, Y., & Xu, M. (2022). A meta-analytical review of intervention experiments to reduce food waste. Environmental Research Letters, 17(6), 064041. https://doi.org/10.1088/1748-9326/ac72b6 · DOI
  51. Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014). Systematic review automation technologies. Systematic Reviews, 3(1), 74. https://doi.org/10.1186/2046-4053-3-74 · DOI · Paper page
  52. van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., Harkema, A., Willemsen, J., Ma, Y., Fang, Q., Hindriks, S., Tummers, L., & Oberski, D. L. (2021b). An open source machine learning framework for efficient and transparent systematic reviews. Nature Machine Intelligence, 3(2), 125-133. https://doi.org/10.1038/s42256-020-00287-7 · DOI · Paper page
  53. van Haastrecht, M., Sarhan, I., Yigit Ozkan, B., Brinkhuis, M., & Spruit, M. (2021). SYMBALS: A Systematic Review Methodology Blending Active Learning and Snowballing. Frontiers in Research Metrics and Analytics, 6, 685591. https://doi.org/10.3389/frma.2021.685591 · DOI · Paper page
  54. Vapnik, V. N. (1995). The nature of statistical learning theory. Springer New York. https://doi.org/10.1007/978-1-4757-2440-0 · DOI · Paper page
  55. Vermillet, A., Tølbøll, K., Litsis Mizan, S., C. Skewes, J., & Parsons, C. E. (2022). Crying in the first 12 months of life: A systematic review and meta-analysis of cross-country parent-reported data and modeling of the "cry curve." Child Development, 93(4), 1201-1222. https://doi.org/10.1111/cdev.13760 · DOI
  56. Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010). Semi- automated screening of biomedical citations for systematic reviews. BMC Bioinformatics, 11(1), 55. https://doi.org/10.1186/1471-2105-11-55 · DOI · Paper page
  57. Wang, Z., Nayfeh, T., Tetzlaff, J., O'Blenis, P., & Murad, M. H. (2020). Error rates of human reviewers during abstract screening in systematic reviews. PLOS ONE, 15(1), e0227742. https://doi.org/10.1371/journal.pone.0227742 · DOI · Paper page
  58. Westgate, M. J. (2019). revtools: An R package to support article screening for evidence synthesis. Research Synthesis Methods, 10(4), 606-614. https://doi.org/10.1002/jrsm.1374 · DOI · Paper page
  59. Woods, S., Dunne, S., Gallagher, P., & McNicholl, A. (2022). A systematic review of the factors associated with athlete burnout in team sports. International Review of Sport and Exercise Psychology, 1-41. https://doi.org/10.1080/1750984X.2022.2148225 · DOI
  60. Yu, Z., & Menzies, T. (2019). FAST2: An intelligent assistant for finding relevant papers. Expert Systems with Applications, 120, 57-71. https://doi.org/10.1016/j.eswa.2018.11.021 · DOI · Paper page
  61. Zaneva, M., Guzman-Holst, C., Reeves, A., & Bowes, L. (2022). The Impact of Monetary Poverty Alleviation Programs on Children's and Adolescents' Mental Health: A Systematic Review and Meta-Analysis Across Low-, Middle-, and High-Income Countries. Journal of Adolescent Health, 71(2), 147-156. https://doi.org/10.1016/j.jadohealth.2022.02.011 2444; 218 9.79 2010; 10 2121; 21 2205; 105 2277; 207 · DOI

Authors
König, L., Zitzmann, S., Fütterer, T., Campos, D., Scherer, R., & Hecht, M.
Affiliations
  • Helmut Schmidt University (DE)
  • Universität Hamburg (DE)
  • University of Tübingen (DE)
  • University of Oslo (NO)
Corresponding author
König, Lars
Venue
Research Synthesis Methods
Published
2023-11-30
Volume
15
Issue
6
Pages
1120-1146
ISSN
1759-2879 (ISSN-L)
OpenAlex
W4403465916
Cited by
33
References
99
Citations by year
2026: 18 · 2025: 12 · 2024: 2 · 2023: 1
FWCI
8.87
Type
journal article
Language
en
Open access
Locations
  • Research Synthesis Methods (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • PsyArXiv (OSF Preprints) (submittedVersion, cc-by)
Funders
  • European Commission (101132474)
  • Norges Forskningsråd (331640)
  • Universität Hamburg
  • HORIZON EUROPE Framework Programme (101132474)
  • NextGenerationEU
Topics
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Social Sciences › General Social Sciences › Computational and Text Analysis Methods
Keywords
stopping rules, active learning (machine learning), prevalence estimation, abstract screening, meta-analysis, psychology research, screening costs, sensitivity and specificity, learning algorithm comparison
MeSH terms
Algorithms, Artificial Intelligence, Humans, Research Design, Sensitivity and Specificity, Meta-Analysis as Topic, Reproducibility of Results

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review