The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Semi-supervised sequence tagging with bidirectional language models
Open accessAbstract
Pre-trained word embeddings learned from unlabeled text have become a standard component of neural network architectures for NLP tasks.However, in most cases, the recurrent network that operates on word-level representations to produce context sensitive representations is trained on relatively little labeled data.In this paper, we demonstrate a general semi-supervised approach for adding pretrained context embeddings from bidirectional language models to NLP systems and apply it to sequence labeling tasks.We evaluate our model on two standard datasets for named entity recognition (NER) and chunking, and in both cases achieve state of the art results, surpassing previous systems that use other forms of transfer or joint learning with additional labeled data and task specific gazetteers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Prompt-based methods have shown great power in few-shot setting [199, 200, 70, 72], zero-shot setting [129, 201], and even fully-supervised setting [74, 75].”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [200]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“LSTM LMs have been shown to discover high-level semantic and contextual representations that are useful for many natural language processing tasks (18; 13; 19).”
From Incorporating Context into Language Encoding Models for fMRI (2018) · cites this paper as [18]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The key difference between our system and the previous state of the art from Peters et al. (2017) is that we allowed the task model to learn a weighted average of all biLM layers, whereas Peters et al. (2017) only use the top biLM layer.”
From Deep Contextualized Word Representations (2018) · cites this paper as Peters et al., 2017
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Neural Architectures for Named Entity Recognition
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1030 · Cited by 4,401
Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition
Tjong Kim Sang, E. F., & De Meulder, F. (2003) · Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - · doi:10.3115/1119176.1119195 · Cited by 2,788
Text classification from labeled and unlabeled documents using EM
Nigam, K., Mccallum, A. K., Thrun, S., & Mitchell, T. (2000) · Machine Learning · doi:10.1023/a:1007692713085 · Cited by 2,736
Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., & Fidler, S. (2015) · arXiv · doi:10.48550/arxiv.1506.06726 · Cited by 1,293
One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling
Chelba, C., Mikolov, T., Schuster, M., Ge, Q., Brants, T., Koehn, P., & Robinson, T. (2013) · arXiv · doi:10.48550/arxiv.1312.3005 · Cited by 1,051
Visualizing and Understanding Recurrent Networks
Karpathy, A., Johnson, J., & Fei-Fei, L. (2015) · arXiv · doi:10.48550/arxiv.1506.02078 · Cited by 845
Learning Distributed Representations of Sentences from Unlabelled Data
Hill, F., Cho, K., & Korhonen, A. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1162 · Cited by 566
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks
Hashimoto, K., xiong, C., Tsuruoka, Y., & Socher, R. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1206 · Cited by 535
Research works printed but not linked: 28
- Waleed Ammar, Matthew E. Peters, Chandra Bhaga- vatula, and Russell Power. 2017. The AI2 sys- tem at SemEval-2017 Task 10 (ScienceIE): semi- supervised end-to-end entity and relation extraction. In ACL workshop (SemEval).
- Rie Kubota Ando and Tong Zhang. 2005. A high- performance semi-supervised learning method for text chunking. In ACL.
- Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic lan- guage model. In JMLR.
- Avrim Blum and Tom Mitchell. 1998. Combining la- beled and unlabeled data with co-training. In COLT.
- Jason Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. In TACL.
- Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bah- danau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder ap- proaches. In SSST@EMNLP.
- Michael Collins. 2002. Discriminative training meth- ods for hidden markov models: Theory and experi- ments with perceptron algorithms. In EMNLP.
- Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. Natural language processing (almost) from scratch. In JMLR.
- Andrew M. Dai and Quoc V. Le. 2015. Semi- supervised sequence learning. In NIPS.
- Volkmar Frinken, Alicia Fornés, Josep Lladós, and Jean-Marc Ogier. 2012. Bidirectional language model for handwriting recognition. In Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syn- tactic Pattern Recognition (SSPR).
- Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the lim- its of language modeling. CoRR abs/1602.02410. · arXiv
- Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP.
- Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
- Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press.
- John D. Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Prob- abilistic models for segmenting and labeling se- quence data. In ICML.
- Quoc V. Le and Tomas Mikolov. 2014. Distributed rep- resentations of sentences and documents. In ICML.
- Wei Li and Andrew McCallum. 2005. Semi-supervised sequence modeling with syntactic topic models. In AAAI.
- Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. In TACL.
- Gang Luo, Xiaojiang Huang, Chin-Yew Lin, and Za- iqing Nie. 2015. Joint entity recognition and disam- biguation. In EMNLP.
- Xuezhe Ma and Eduard H. Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs- CRF. In ACL.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositional- ity. In NIPS.
- Behrang Mohit and Rebecca Hwa. 2005. Syntax-based semi-supervised named entity tagging. In ACL.
- Jeffrey Pennington, Richard Socher, and Christo- pher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
- David Pierce and Claire Cardie. 2001. Limitations of co-training for natural language learning from large datasets. In EMNLP.
- Hasim Sak, Andrew W. Senior, and Franoise Beaufays. 2014. Long short-term memory recurrent neural network architectures for large scale acoustic mod- eling. In INTERSPEECH.
- Jun Suzuki, Akinori Fujino, and Hideki Isozaki. 2007. Semi-supervised structured output learning based on a hybrid generative and discriminative approach. In EMNLP-CoNLL.
- Jun Suzuki and Hideki Isozaki. 2008. Semi-supervised sequential labeling and segmentation using giga- word scale unlabeled data. In ACL.
- Zhilin Yang, Ruslan Salakhutdinov, and William W. Cohen. 2017. Transfer learning for sequence tag- ging with hierarchical recurrent networks. In ICLR.
Cited by
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Self-citationKirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953
Universal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100
Metadata
- Authors
- Peters, M., Ammar, W., Bhagavatula, C., & Power, R.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- Venue
- Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Published
- 2017
- Pages
- 1756-1765
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2610748790
- Cited by
- 655
- References
- 13
- Citations by year
- 2026: 5 · 2025: 30 · 2024: 38 · 2023: 68 · 2022: 66 · 2021: 102 · 2020: 116 · 2019: 149 · 2018: 75 · 2017: 6
- FWCI
- 42.23
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Domain Adaptation and Few-Shot Learning
- Keywords
- semi-supervised learning, sequence labeling, bidirectional language models, pre-trained word embeddings, context embedding, named entity recognition, chunking, transfer learning, unlabeled text
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Semi-supervised sequence tagging with bidirectional language models. https://app.cochat.ai/papers/10.18653/v1/p17-1161
- The paper
- Peters, M., Ammar, W., Bhagavatula, C., & Power, R. (2017). Semi-supervised sequence tagging with bidirectional language models. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). https://doi.org/10.18653/v1/p17-1161
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.