Contrastive estimation: training log-linear models on unlabeled data
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
Conditional random fields (Lafferty et al., 2001) are quite effective at sequence labeling tasks like shallow parsing (Sha and Pereira, 2003) and named-entity extraction (McCallum and Li, 2003). CRFs are log-linear, allowing the incorporation of arbitrary features into the model. To train on unlabeled data, we require unsupervised estimation methods for log-linear models; few exist. We describe a novel approach, contrastive estimation. We show that the new technique can be intuitively understood as exploiting implicit negative evidence and is computationally efficient. Applied to a sequence labeling problem---POS tagging given a tagging dictionary and unlabeled text---contrastive estimation outperforms EM (with the same feature set), is more robust to degradations of the dictionary, and can largely recover by modeling additional features.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
On the limited memory BFGS method for large scale optimization
Liu, D. C., & Nocedal, J. (1989) · Mathematical Programming · doi:10.1007/bf01589116 · Cited by 8,560
Training Products of Experts by Minimizing Contrastive Divergence
Hinton, G. E. (2002) · Neural Computation · doi:10.1162/089976602760128018 · Cited by 4,879 · cited 1×
“Not to be confused withcontrastive divergence minimization (Hinton, 2003), a technique for training products of experts.”
BackgroundMaximum likelihood estimation from incomplete data via the EM algorithm
Dempster, A. P. (1977) · Journal of the Royal Statistical Society Series A (Statistics in Society) · Cited by 4,543
Unsupervised word sense disambiguation rivaling supervised methods
Yarowsky, D. (1995) · Proceedings of the 33rd annual meeting on Association for Computational Linguistics - · doi:10.3115/981658.981684 · Cited by 2,385
On the algorithmic implementation of multiclass kernel-based vector machines
Crammer, K., & Singer, Y. (2002) · Journal of Machine Learning Research · Cited by 2,311
Shallow parsing with conditional random fields
Sha, F., & Pereira, F. (2003) · Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - NAACL '03 · doi:10.3115/1073445.1073473 · Cited by 1,246
McCallum, A., & Li, W. (2003) · Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - · doi:10.3115/1119176.1119206 · Cited by 1,160
On the algorithmic implementation of multiclass kernel-based vector machines
CrammerKoby, & SingerYoram (2002) · Journal of Machine Learning Research · doi:10.5555/944790.944813 · Cited by 874
Eugene Charniak, D. (1994) · Medical Entomology and Zoology · Cited by 811
Discriminative learning for minimum error classification (pattern recognition)
Juang, B.-H., & Katagiri, S. (1992) · IEEE Transactions on Signal Processing · doi:10.1109/78.175747 · Cited by 714
Discriminative Reranking for Natural Language Parsing
Collins, M., & Koo, T. (2005) · Computational Linguistics · doi:10.1162/0891201053630273 · Cited by 710
Tagging English text with a probabilistic model
Mérialdo, B. (1994) · Cited by 474
Statistical Language Learning.
SW, & Charniak, E. (1994) · Journal of the American Statistical Association · doi:10.2307/2291039 · Cited by 365
Adaptive Statistical Language Modeling: A Maximum Entropy Approach
Rosenfeld, R. (2018) · doi:10.1184/r1/6603032 · Cited by 269
A Unified Approach to Path Problems
Tarjan, R. E. (1981) · Journal of the ACM · doi:10.1145/322261.322272 · Cited by 253
Parameter estimation for probabilistic finite-state transducers
Eisner, J. M. (2001) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1073083.1073085 · Cited by 197
Self-citationMMIE training of large vocabulary recognition systems
Valtchev, V, Odell, J. J., Woodland, P. C., & Young, S. J. (1997) · Speech Communication · doi:10.1016/s0167-6393(97)00029-0 · Cited by 175
Maximum Conditional Likelihood via Bound Maximization and the CEM Algorithm
Jebara, T., & Pentland, A. (1998) · Cited by 65
Riezler, S., Kuhn, J., Prescher, D., & Johnson, M. (2000) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1075218.1075279 · Cited by 61
Annealing techniques for unsupervised statistical language learning
Smith, N. A., & Eisner, J. M. (2004) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1218955.1219017 · Cited by 56
Self-citationProbabilistic Constraint Logic ProgrammingPreprint
Riezler, S. (1997) · arXiv (Cornell University) · doi:10.48550/arxiv.cmp-lg/9711001 · Cited by 55
Joint and conditional estimation of tagging and parsing models
Johnson, M. (2001) · Proceedings of the 39th Annual Meeting on Association for Computational Linguistics - ACL '01 · doi:10.3115/1073012.1073054 · Cited by 54
Towards semi-supervised classification with Markov random fields
Zhu, X., Ghahramani, Z., & Lafferty, J. (2002) · Cambridge University Engineering Department Publications Database · Cited by 41
Cited by
Natural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Clark, K., Luong, M.-T., Le, Q. V., & Manning, C. D. (2020) · arXiv · doi:10.48550/arxiv.2003.10555 · Cited by 1,844
Modeling Relations and Their Mentions without Labeled Text
Riedel, S., Yao, L., & McCallum, A. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-15939-8_10 · Cited by 1,270
A Universal Part-of-Speech Tagset
Petrov, S., Das, D., & McDonald, R. (2012) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3pjyez8kmkhz · Cited by 722
Improved Part-of-Speech Tagging for Online Conversational Text with Word Clusters
Owoputi, O., O'Connor, B., Dyer, C., Gimpel, K., Schneider, N., & Smith, N. A. (2018) · Figshare · doi:10.1184/r1/6473408 · Cited by 662
Self-citation
Metadata
- Authors
- Smith, N. A., & Eisner, J.
- Venue
- Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics - ACL '05
- Published
- 2005
- Pages
- 354-362
- OpenAlex
- W2137807925
- Cited by
- 305
- References
- 24
- FWCI
- 10.02
- Type
- conference paper
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.