The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
Fast and Accurate Entity Recognition with Iterated Dilated Convolutions
Open accessAbstract
Today when many practitioners run basic NLP on the entire web and large-volume traffic, faster methods are paramount to saving time and energy costs.Recent advances in GPU hardware have led to the emergence of bi-directional LSTMs as a standard method for obtaining pertoken vector representations serving as input to labeling tasks such as NER (often followed by prediction in a linear-chain CRF).Though expressive and accurate, these models fail to fully exploit GPU parallelism, limiting their computational efficiency.This paper proposes a faster alternative to Bi-LSTMs for NER: Iterated Dilated Convolutional Neural Networks (ID-CNNs), which have better capacity than traditional CNNs for large context and structured prediction.Unlike LSTMs whose sequential processing on sentences of length N requires O(N ) time even in the face of parallelism, ID-CNNs permit fixed-depth convolutions to run in parallel across entire documents.We describe a distinct combination of network structure, parameter sharing and training procedures that enable dramatic 14-20x testtime speedups while retaining accuracy comparable to the Bi-LSTM-CRF.Moreover, ID-CNNs trained to aggregate context from the entire document are even more accurate while maintaining 8x faster test time speeds.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
(2014) · Cited by 33,419
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Self-citationMulti-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738 · cited 7×
“We used the modified version of identity initialization (Le et al., 2015) re-ported by Yu and Koltun (2016) to initialize our dilated layers, which we found to perform the best in initial experiments compared to orthogonal and Xavier initialization (Glorot and Bengio, 2010).”
BackgroundMethodInfluentialNatural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911
Neural Architectures for Named Entity Recognition
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1030 · Cited by 4,401
Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2014) · arXiv · doi:10.48550/arxiv.1412.7062 · Cited by 4,213
A Convolutional Neural Network for Modelling Sentences
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1062 · Cited by 3,601
Incorporating non-local information into information extraction systems by Gibbs sampling
Finkel, J. R., Grenager, T., & Manning, C. (2005) · Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics - ACL '05 · doi:10.3115/1219840.1219885 · Cited by 3,008
The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions
Hochreiter, S. (1998) · International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems · doi:10.1142/s0218488598000094 · Cited by 2,939
FitNets: Hints for Thin Deep Nets
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1412.6550 · Cited by 2,859
Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition
Tjong Kim Sang, E. F., & De Meulder, F. (2003) · Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - · doi:10.3115/1119176.1119195 · Cited by 2,788
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
Ma, X., & Hovy, E. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1101 · Cited by 2,567 · cited 3×
“…models for sequence labeling include an inference step that searches the space of possible output sequences of a chain-structured graphical model, or approximates this search with a beam (Collobert et al., 2011; Weiss et al., 2015; Lample et al., 2016; Ma and Hovy, 2016; Chiu and Nichols, 2016).”
BackgroundNamed Entity Recognition with Bidirectional LSTM-CNNs
Chiu, J. P. C., & Nichols, E. (2016) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00104 · Cited by 1,909
(2010) · Cited by 1,906
Design challenges and misconceptions in named entity recognition
Ratinov, L., & Roth, D. (2009) · Proceedings of the Thirteenth Conference on Computational Natural Language Learning - CoNLL '09 · doi:10.3115/1596374.1596399 · Cited by 1,448
Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., & Weischedel, R. (2006) · Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers on XX - NAACL '06 · doi:10.3115/1614049.1614064 · Cited by 806 · cited 1×
“We evaluate using labeled data from the CoNLL-2003 shared task (Tjong Kim Sang and De Meul-der, 2003) and OntoNotes 5.0 (Hovy et al., 2006; Pradhan et al., 2006).”
MethodInfluentialRepresenting Text for Joint Embedding of Text and Knowledge Bases
Toutanova, K., Chen, D., Pantel, P., Poon, H., Choudhury, P., & Gamon, M. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1174 · Cited by 750
(2012) · Cited by 618
A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
Le, Q. V., Jaitly, N., & Hinton, G. E. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1504.00941 · Cited by 513
Search-based structured prediction
Daumé, H., Langford, J., & Marcu, D. (2009) · Machine Learning · doi:10.1007/s10994-009-5106-x · Cited by 465
Neural Machine Translation in Linear Time
Kalchbrenner, N., Espeholt, L., Simonyan, K., Oord, A. V. D., Graves, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1610.10099 · Cited by 414 · cited 1×
“Additionally, we present a novel loss and parameter sharing scheme to facilitate training models on much smaller datasets than those used by Kalchbrenner et al. (2016).”
MethodA Convolutional Neural Network for Modelling Sentences
Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1404.2188 · Cited by 407
Named entity recognition with long short-term memory
Hammerton, J. (2003) · Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - · doi:10.3115/1119176.1119202 · Cited by 264
A Joint Model for Entity Analysis: Coreference, Typing, and Linking
Durrett, G., & Klein, D. (2014) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00197 · Cited by 234
Multi-Task Cross-Lingual Sequence Tagging from Scratch
Yang, Z., Salakhutdinov, R., & Cohen, W. W. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.06270 · Cited by 178
Not All Contexts Are Created Equal: Better Word Representations with Variable Attention
Wang, L., Tsvetkov, Y., Amir, S., Fermandez, R., Dyer, C., Black, A. W., Trancoso, I. M., & Lin, C. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1161 · Cited by 147
Molding CNNs for text: non-linear, non-consecutive convolutions
Lei, T., Barzilay, R., & Jaakkola, T. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1180 · Cited by 146
Collective Segmentation and Labeling of Distant Entities in Information Extraction
Sutton, C. A., & McCallum, A. (2004) · Edinburgh Research Explorer (University of Edinburgh) · Cited by 141
Self-citationKnowledge Matters: Importance of Prior Information for OptimizationPreprint
Gülçehre, Ç., & Bengio, Y. (2013) · arXiv (Cornell University) · doi:10.48550/arxiv.1301.4083 · Cited by 140
Collective information extraction with relational Markov networks
Bunescu, R., & Mooney, R. J. (2004) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/1218955.1219011 · Cited by 136
Lexicon Infused Phrase Embeddings for Named Entity Resolution
Passos, A., Kumar, V., & McCallum, A. (2014) · Conference on Computational Natural Language Learning · doi:10.3115/v1/w14-1609 · Cited by 130
Self-citationStructure compilation: trading structure for features
Liang, P., Daumé, H., & Klein, D. (2008) · International Conference on Machine Learning · doi:10.1145/1390156.1390231 · Cited by 75
Structured Training for Neural Network Transition-Based ParsingPreprint
Weiss, D., Alberti, C., Collins, M., & Petrov, S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.06158 · Cited by 75
Dropout with Expectation-linear RegularizationPreprint
Ma, X., Gao, Y., Hu, Z., Yu, Y., Deng, Y., & Hovy, E. H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1609.08017 · Cited by 44
Stability and generalization in structured prediction
London, B., Huang, B., & Getoor, L. C. (2016) · Journal of Machine Learning Research · Cited by 31
(2015) · doi:10.4230/oasics.slate.2021.8 · Cited by 1
Cited by
A Survey on Deep Learning for Named Entity Recognition
Li, J., Sun, A., Han, J., & Li, C. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.2981314 · Cited by 1,567
Recent Trends in Deep Learning Based Natural Language Processing
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.02709 · Cited by 1,192
A Unified MRC Framework for Named Entity Recognition
Li, X., Feng, J., Meng, Y., Han, Q., Wu, F., & Li, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.519 · Cited by 697
Opportunities and Challenges for Machine Learning in Materials Science
Morgan, D., & Jacobs, R. (2020) · Annual Review of Materials Research · doi:10.1146/annurev-matsci-070218-010015 · Cited by 549
Named Entity Recognition as Dependency Parsing
Yu, J., Bohnet, B., & Poesio, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.577 · Cited by 458
Metadata
- Authors
- Strubell, E., Verga, P., Belanger, D., & McCallum, A.
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 2670-2680
- OpenAlex
- W2740462959
- Cited by
- 474
- References
- 40
- FWCI
- 23.63
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.