Reducing Labeling Effort for Structured Prediction Tasks
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
A common obstacle preventing the rapid deployment of supervised machine learning algorithms is the lack of labeled training data. This is particularly expensive to obtain for structured prediction tasks, where each training instance may have multiple, interacting labels, all of which must be correctly annotated for the instance to be of use to the learner. Traditional active learning addresses this problem by optimizing the order in which the examples are labeled to increase learning efficiency. However, this approach does not consider the difficulty of labeling each example, which can vary widely in structured prediction tasks. For example, the labeling predicted by a partially trained system may be easier to correct for some instances than for others. We propose a new active learning paradigm which reduces not only how many instances the annotator must label, but also how difficult each instance is to annotate. The system leverages information from partially correct predictions to efficiently solicit annotations from the user. We validate this active learning framework in an interactive information extraction system, reducing the total number of annotation actions by 22%.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
A tutorial on hidden Markov models and selected applications in speech recognition
Rabiner, L. R. (1989) · Proceedings of the IEEE · doi:10.1109/5.18626 · Cited by 22,743 · cited 1×
“For linear-chain CRFs, inference is performed using an analog of the Viterbi algorithm, a dynamic program well-known for its use in inference in hidden Markov models (Rabiner 1989).”
MethodConditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Self-citationImproving generalization with active learning
Cohn, D., Atlas, L., & Ladner, R. (1994) · Machine Learning · doi:10.1007/bf00993277 · Cited by 1,591
Active Learning with Statistical Models
Cohn, D. A., Ghahramani, Z., & Jordan, M. I. (1996) · Journal of Artificial Intelligence Research · doi:10.1613/jair.295 · Cited by 1,317
Heterogeneous Uncertainty Sampling for Supervised Learning
Lewis, D. D., & Catlett, J. (1994) · Machine Learning Proceedings 1994 · doi:10.1016/b978-1-55860-335-6.50026-x · Cited by 1,150
Active Hidden Markov Models for Information Extraction
Scheffer, T., Decomain, C., & Wrobel, S. (2001) · Lecture Notes in Computer Science · doi:10.1007/3-540-44816-0_31 · Cited by 534
Active Learning for Natural Language Parsing and Information Extraction
Thompson, C. A., Califf, M. E., & Mooney, R. J. (1999) · International Conference on Machine Learning · Cited by 282
Schwartz, R. M., & Chow, Y.-L. (2002) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp.1990.115542 · Cited by 192
Active Learning with Statistical ModelsPreprint
Cohn, D. A., Ghahramani, Z., & Jordan, M. I. (1996) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/9603104 · Cited by 155
Interactive information extraction with constrained conditional random fields
Kristjansson, T. T., Culotta, A., Viola, P. A., & McCallum, A. (2004) · Scholarworks (University of Massachusetts Amherst) · Cited by 142
Self-citation
Cited by
(2009) · Minds at UW (University of Wisconsin) · Cited by 4,791
An analysis of active learning strategies for sequence labeling tasks
Settles, B., & Craven, M. (2008) · Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08 · doi:10.3115/1613715.1613855 · Cited by 972
Settles, B. (2012) · Synthesis Lectures on Artificial Intelligence and Machine Learning · doi:10.2200/s00429ed1v01y201207aim018 · Cited by 605
Deep Active Learning for Named Entity Recognition
Shen, Y., Yun, H., Lipton, Z., Kronrod, Y., & Anandkumar, A. (2017) · Proceedings of the 2nd Workshop on Representation Learning for NLP · doi:10.18653/v1/w17-2630 · Cited by 370
A survey on instance selection for active learning
Fu, Y., Zhu, X., & Li, B. (2013) · Knowledge and Information Systems · doi:10.1007/s10115-012-0507-8 · Cited by 364
Metadata
- Authors
- Culotta, A., & McCallum, A.
- Publisher
- Defense Technical Information Center
- Published
- 2005-01-01
- OpenAlex
- W2117763124
- Cited by
- 356
- References
- 10
- Type
- report
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.