Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat
Sign Up
Cited by 7

The DOI matches its Crossref record; 12 of the 12 references it cites were found in a registry or an index.

Using Chao’s Estimator as a Stopping Criterion for Technology-Assisted Review

Bron, Michiel, van der Heijden, Peter G. M., Feelders, Ad, Siebes, Arno·2025·ACM Transactions on Information Systems
DOI 10.1145/3724116
Open this paper in CoChat — free account

Abstract

Technology-Assisted Review aims to reduce the human effort required for screening processes such as abstract screening for Systematic Literature Reviews. Human reviewers label documents as relevant or irrelevant during this process, while the system incrementally updates a prediction model based on the reviewers’ previous decisions. After each model update, the system proposes new documents it deems relevant, to prioritize relevant documents over irrelevant ones. A stopping criterion is necessary to guide users in stopping the review process to minimize the number of missed relevant documents and the number of read irrelevant documents. In this article, we propose and evaluate a new ensemble-based Active Learning strategy and a stopping criterion based on Chao’s Population Size Estimator that estimates the prevalence of relevant documents in the dataset. Our simulation study demonstrates that this criterion performs well on several datasets and is compared to other methods presented in the literature.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3724116

Cites

12 of 12 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1 This figure shows an architectural overview of our method. The Active Learning module consists of several committee members {C 1 , . . . , C n } , with each its own labeled and unlabeled state. Each of the members can have a Machine Learning Model (for illustrative purposes represented as an Artificial Neural Network). The rankings of each of the members are combined by going through each member in a round-robin or random fashion and selecting the top of the stack. The estimation module can query the labeled states of each of the member to construct a contingency table and fit a PSE model.
Figure 1 This figure shows an architectural overview of our method. The Active Learning module consists of several committee members {C 1 , . . . , C n } , with each its own labeled and unlabeled state. Each of the members can have a Machine Learning Model (for illustrative purposes represented as an Artificial Neural Network). The rankings of each of the members are combined by going through each member in a round-robin or random fashion and selecting the top of the stack. The estimation module can query the labeled states of each of the member to construct a contingency table and fit a PSE model.CC-BY · From the PDF
Figure 2 An example run for 500 iterations on a dataset. The Ensemble curve shows the number of documents that have been found by the overall system. The other curves display the number of relevant documents that have been found by the individual members within C . The reader may notice that curves start slightly after 0 documents and end slightly after 500 documents. This is caused by the fact that our method requires five relevant and five irrelevant documents at the start of the process (see Section 3.5.4), which results in this shift.
Figure 2 An example run for 500 iterations on a dataset. The Ensemble curve shows the number of documents that have been found by the overall system. The other curves display the number of relevant documents that have been found by the individual members within C . The reader may notice that curves start slightly after 0 documents and end slightly after 500 documents. This is caused by the fact that our method requires five relevant and five irrelevant documents at the start of the process (see Section 3.5.4), which results in this shift.CC-BY · From the PDF
Figure 3 Calculating the 95 % confidence interval using the profile likelihood method for the frequency statistics in Table 1. In this figure, the log-likelihood for u ⋆ is subtracted from the likelihood in aid of the visualization. Additionally, we inverted the y -axis for this purpose. By finding the values u that intersect with the line y = k α =0 . 05 = 3 . 84 , we can find the lower bound and upper bound of the interval.
Figure 3 Calculating the 95 % confidence interval using the profile likelihood method for the frequency statistics in Table 1. In this figure, the log-likelihood for u ⋆ is subtracted from the likelihood in aid of the visualization. Additionally, we inverted the y -axis for this purpose. By finding the values u that intersect with the line y = k α =0 . 05 = 3 . 84 , we can find the lower bound and upper bound of the interval.CC-BY · From the PDF
Image 4
CC-BY · From the PDF
Figure 5 These figures display the results of all runs on all datasets in terms of Work Saved over Sampling and Recall for the Conservative Stopping criteria with a 95 % recall target. The colors are indicative of the dataset size. The outer graphs show the overall distribution of WSS and Recall. For each method, the mean WSS and the 5th percentile of the recall are shown.
Figure 5 These figures display the results of all runs on all datasets in terms of Work Saved over Sampling and Recall for the Conservative Stopping criteria with a 95 % recall target. The colors are indicative of the dataset size. The outer graphs show the overall distribution of WSS and Recall. For each method, the mean WSS and the 5th percentile of the recall are shown.CC-BY · From the PDF
Figure 6 These figures display the results of all runs on all datasets in terms of Work Saved over Sampling and Recall for the Conservative Stopping criteria with a 100 % recall target. The colors are indicative of the dataset size. The outer graphs show the overall distribution of WSS and Recall. For each method, the mean WSS and the 5th percentile of the recall are shown.
Figure 6 These figures display the results of all runs on all datasets in terms of Work Saved over Sampling and Recall for the Conservative Stopping criteria with a 100 % recall target. The colors are indicative of the dataset size. The outer graphs show the overall distribution of WSS and Recall. For each method, the mean WSS and the 5th percentile of the recall are shown.CC-BY · From the PDF
Figure 7 These figures display the results of all runs on all datasets in terms of Work Saved over Sampling and Recall for the Optimistic Stopping criteria with a 95 % recall target. The colors are indicative of the dataset size. The outer graphs show the overall distribution of WSS and Recall. For each method, the mean WSS and the 5th percentile of the recall are shown.
Figure 7 These figures display the results of all runs on all datasets in terms of Work Saved over Sampling and Recall for the Optimistic Stopping criteria with a 95 % recall target. The colors are indicative of the dataset size. The outer graphs show the overall distribution of WSS and Recall. For each method, the mean WSS and the 5th percentile of the recall are shown.CC-BY · From the PDF
Figure 8 These figures display the results of all runs on all datasets in terms of Work Saved over Sampling and Recall for the Optimistic Stopping criteria with a 100 % recall target. The colors are indicative of the dataset size. The outer graphs show the overall distribution of WSS and Recall. For each method, the mean WSS and the 5th percentile of the recall are shown.
Figure 8 These figures display the results of all runs on all datasets in terms of Work Saved over Sampling and Recall for the Optimistic Stopping criteria with a 100 % recall target. The colors are indicative of the dataset size. The outer graphs show the overall distribution of WSS and Recall. For each method, the mean WSS and the 5th percentile of the recall are shown.CC-BY · From the PDF
Figure 9 These figures display the results of all runs on all datasets in terms of Work Saved over Sampling and Recall for the CMH criteria.
Figure 9 These figures display the results of all runs on all datasets in terms of Work Saved over Sampling and Recall for the CMH criteria.CC-BY · From the PDF
Figure 10 These figures display the results of all runs on all datasets in terms of Work Saved over Sampling and Recall for the hybrid stopping criteria.
Figure 10 These figures display the results of all runs on all datasets in terms of Work Saved over Sampling and Recall for the hybrid stopping criteria.CC-BY · From the PDF
Figure 11 These figures display the results of all runs on all datasets in terms of Work Saved over Sampling and Recall for the standoff heuristic stopping criteria.
Figure 11 These figures display the results of all runs on all datasets in terms of Work Saved over Sampling and Recall for the standoff heuristic stopping criteria.CC-BY · From the PDF

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 38

  1. 10.5555/69372 · DOI
  2. 10.1007/BF01002566 · DOI
  3. 10.18637/jss.v019.i05 · DOI
  4. 10.1007/s10651-020-00440-w · DOI
  5. 10.4324/9781315151939 · DOI
  6. 10.1093/biomet/65.3.625 · DOI
  7. 10.2307/1936861 · DOI
  8. 10.2307/2531532 · DOI
  9. 10.1002/0471667196.ess5051 · DOI
  10. 10.2436/20.8080.02.49 · DOI
  11. Carl V. L. Charlier. 1905. Die zweite form des fehlergesetzes. Meddelanden Fran Lunds Astronomiska Observatorium Serie I 26 (1905), 1–8.
  12. 10.1016/j.ejor.2005.06.023 · DOI
  13. Gordon V. Cormack and Maura R. Grossman. 2015. Autonomy and reliability of continuous active learning for technology-assisted review. arXiv:1504.06868. Retrieved from https://arxiv.org/abs/1504.06868 · arXiv
  14. Gordon V. Cormack and Maura R. Grossman. 2017. Technology-assisted review in empirical medicine: Waterloo participation in CLEF eHealth 2017. In Working Notes of CLEF 2017; Conference and Labs of the Evaluation Forum (CEUR Workshop Proceedings, Vol. 1866), Linda Cappellato, Nicola Ferro, Lorraine Goeuriot, and Thomas Mandl (Eds.), CEUR-WS.org.
  15. 10.2307/2532310 · DOI
  16. A. W. F. Edwards. 1972. Likelihood: An Account of the Statistical Concept of Likelihood and Its Application to Scientific Inference. University Press, Cambridge [Eng.].
  17. 10.2307/2987516 · DOI
  18. 10.1109/ICDAR.1995.598994 · DOI
  19. 10.1002/0471715816 · DOI
  20. Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2017. CLEF 2017 technologically assisted reviews in empirical medicine overview. CEUR Workshop Proceedings 1866 (2017), 1–29.
  21. Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2018. CLEF 2018 technologically assisted reviews in empirical medicine overview: 19th working notes of CLEF conference and labs of the evaluation forum, CLEF 2018. CEUR Workshop Proceedings 2125 (2018), 1–34.
  22. Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker. 2019. CLEF 2019 technology assisted reviews in empirical medicine overview. CEUR Workshop Proceedings 2380 (2019), 9–12.
  23. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: A highly efficient gradient boosting decision tree. In Advances in Neural Information Processing Systems. I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc, 3149–3157.
  24. 10.1126/science.abl7655 · DOI
  25. 10.1186/1471-2288-14-58 · DOI
  26. 10.1145/3459637.3482415 · DOI
  27. 10.1145/3411755 · DOI
  28. 10.1016/j.cegh.2023.101485 · DOI
  29. 10.1201/b17222 · DOI
  30. 10.1111/j.1541-0420.2007.00779.x · DOI
  31. 10.1016/j.jclinepi.2011.03.008 · DOI
  32. 10.1145/130385.130417 · DOI
  33. 10.1111/1467-9574.00232 · DOI
  34. Byron C. Wallace, Issa J. Dahabreh, Kelly H. Moran, Carla E. Brodley, and Thomas A. Trikalinos. 2013. Active literature discovery for scoping evidence reviews. In KDD Workshop on Data Mining for Healthcare (KDDDMH’13), 14–19.
  35. 10.1080/00031305.2013.783881 · DOI · Paper page
  36. 10.1145/3477495.3531663 · DOI
  37. 10.1145/3469096.3469873 · DOI
  38. 10.1007/978-3-030-99736-6_34 · DOI

Cited by

See all 7

Metadata

Authors
Bron, M., van der Heijden, P. G. M., Feelders, A., & Siebes, A.
Venue
ACM Transactions on Information Systems
Publisher
Association for Computing Machinery (ACM)
Published
2025-05-09
Volume
43
Issue
3
Pages
1-51
ISSN
  • 1046-8188
  • 1558-2868
DOI
10.1145/3724116
Cited by
7
References
12
Language
en
Licence
creativecommons.org/licenses/by/4.0/ (vor)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.