An intrinsic stopping criterion for committee-based active learning
Open accessAbstract
As supervised machine learning methods are increasingly used in language technology, the need for high-quality annotated language data becomes imminent.Active learning (AL) is a means to alleviate the burden of annotation.This paper addresses the problem of knowing when to stop the AL process without having the human annotator make an explicit decision on the matter.We propose and evaluate an intrinsic criterion for committee-based AL of named entity recognizers.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Heuristics have been proposed as a practical and effective way to define stopping rules.
“Alternatively, heuristics have been proposed as a practical and effective way to define stopping rules (e.g., [7, 27, 35, 47]).”
From The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses (2024) · cites this paper as [27]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,953 · see citing works
Witten, I. H. (2008) · Cited by 9,115 · see citing works
A maximum entropy approach to natural language processing
Berger, A. L., Pietra, V. J. D., & Pietra, S. A. D. (1996) · doi:10.5555/234285.234289 · Cited by 3,083 · see citing works
Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition
Tjong Kim Sang, E. F., & De Meulder, F. (2003) · Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - · doi:10.3115/1119176.1119195 · Cited by 2,788 · see citing works
Text Chunking using Transformation-Based Learning
Ramshaw, L., & Marcus, M. P. (1995) · VLC@ACL · doi:10.1007/978-94-017-2390-9_10 · Cited by 2,031 · see citing works
A sequential algorithm for training text classifiersPreprint
Lewis, D., & Gale, W. A. (1994) · arXiv (Cornell University) · doi:10.5555/188490.188495 · Cited by 1,838 · see citing works
Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity RecognitionPreprint
Sang, E. T. K. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0209010 · Cited by 1,677 · see citing works
A Sequential Algorithm for Training Text Classifiers
Lewis, D. D., & Gale, W. A. (1994) · SIGIR ’94 · doi:10.1007/978-1-4471-2099-5_1 · Cited by 1,208 · see citing works
Less is More: Active Learning with Support Vector Machines
Schohn, G., & Cohn, D. A. (2000) · Cited by 771 · see citing works
MultiBoosting: A Technique for Combining Boosting and Wagging
Webb, G. I. (2000) · Machine Learning · doi:10.1023/a:1007659514849 · Cited by 698 · see citing works
Penn, T. (1999) · Medical Entomology and Zoology · Cited by 652 · see citing works
Text Chunking using Transformation-Based LearningPreprint
Ramshaw, L., & Marcus, M. P. (1995) · arXiv (Cornell University) · doi:10.48550/arxiv.cmp-lg/9505040 · Cited by 475 · see citing works
A non-projective dependency parser
Tapanainen, P., & Järvinen, T. (1997) · Conference on Applied Natural Language Processing · doi:10.3115/974557.974568 · Cited by 371 · see citing works
Active Learning for Natural Language Parsing and Information Extraction
Thompson, C. A., Califf, M. E., & Mooney, R. J. (1999) · Cited by 288 · see citing works
Large-scale text categorization by batch mode active learning
Hoi, S. C. H., Jin, R., & Lyu, M. R.-T. (2006) · International World Wide Web Conference (WWW) · doi:10.1145/1135777.1135870 · Cited by 257 · see citing works
Multi-criteria-based active learning for named entity recognition
Shen, D., Zhang, J., Su, J., et al. (2004) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/1218955.1219030 · Cited by 234 · see citing works
Zhu, J., & Hovy, E. H. (2007) · Cited by 156 · see citing works
A stopping criterion for active learning
Vlachos, A. (2008) · Computer Speech & Language · doi:10.1016/j.csl.2007.12.001 · Cited by 153 · see citing works
Tomanek, K., Wermter, J., & Hahn, U. (2007) · Cited by 97 · see citing works
Self-citationMinimizing manual annotation cost in supervised training from corpora
Engelson, S. P., & Dagan, I. (1996) · Annual Meeting of the Association for Computational Linguistics (ACL) · doi:10.3115/981863.981905 · Cited by 83 · see citing works
Stopping criteria for active learning of named entity recognition
Laws, F., & Schätze, H. (2008) · International Conference on Computational Linguistics · doi:10.3115/1599081.1599140 · Cited by 78 · see citing works
Active learning for part-of-speech tagging
Ringger, E. K., McClanahan, P., Haertel, R., et al. (2007) · doi:10.3115/1642059.1642075 · Cited by 69 · see citing works
Performance thresholding in practical text classification
Schütze, H., Velipasaoglu, E., & Pedersen, J. (2006) · doi:10.1145/1183614.1183709 · Cited by 54 · see citing works
Multi-criteria-based strategy to stop active learning for data annotation
Zhu, J., Wang, H., & Hovy, E. H. (2008) · International Conference on Computational Linguistics · doi:10.3115/1599081.1599223 · Cited by 47 · see citing works
Zhu, J., Wang, H., & Hovy, E. H. (2008) · Cited by 40 · see citing works
Olsson, F. K. (2008) · KTH Publication Database DiVA (KTH Royal Institute of Technology) · Cited by 37 · see citing works
Self-citationA two-stage method for active learning of statistical grammars
Becker, M. C., & Osborne, M. (2005) · Cited by 34 · see citing works
Approximating Learning Curves for Active-Learning-Driven Annotation
Tomanek, K., & Hahn, U. (2008) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/4smeu6r7yan5 · Cited by 13 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- Florian Laws and Hinrich Schütze. 2008. Stopping Cri- teria for Active Learning of Named Entity Recogni- tion. In Proc 22nd COLING, Manchester, England.
- Ian H. Witten and Eibe Frank. 2005. Data Mining: Prac- tical Machine Learning Tools with Java Implementa- tions. 2nd Edition. Morgan Kaufmann, San Fransisco.
Cited by
Sorted by citations
Settles, B. (2012) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.2200/s00429ed1v01y201207aim018 · Cited by 605 · see citing works
Boetje, J., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02502-7 · Cited by 146 · see citing works
“Alternatively, heuristics have been proposed as a practical and effective way to define stopping rules (e.g., [7, 27, 35, 47]).”
BackgroundUtilizing ASReview in screening primary studies for meta-research in SLA: A step-by-step tutorial
Quan, Y., Tytko, T., & Hui, B. (2024) · Research Methods in Applied Linguistics · doi:10.1016/j.rmal.2024.100101 · Cited by 64 · see citing works
- Authors
- Olsson, F., & Tomanek, K.
- Affiliations
- Friedrich Schiller University Jena (DE)
- ORCID
- Tomanek, Katrin
- Venue
- Proceedings of the Thirteenth Conference on Computational Natural Language Learning - CoNLL '09
- Published
- 2009
- Pages
- 138
- OpenAlex
- W2021894617
- Cited by
- 37
- References
- 28
- Citations by year
- 2025: 2 · 2024: 6 · 2023: 4 · 2022: 2 · 2021: 2 · 2020: 1 · 2019: 1 · 2017: 1 · 2016: 1 · 2015: 2 · 2014: 3 · 2013: 3 · 2012: 6
- FWCI
- 2.11
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Conference on Computational Natural Language Learning (publishedVersion)
- Diva portal (Dalarna University Library) (submittedVersion, other-oa)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- active learning (machine learning), stopping rules, named entity recognition, intrinsic evaluation, annotation cost reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
