Fast and Accurate Entity Recognition with Iterated Dilated Convolutions
Open accessThe DOI matches its Crossref record; 42 of the 42 references it cites were found in a registry or an index.
Abstract
Today when many practitioners run basic NLP on the entire web and large-volume traffic, faster methods are paramount to saving time and energy costs.Recent advances in GPU hardware have led to the emergence of bi-directional LSTMs as a standard method for obtaining pertoken vector representations serving as input to labeling tasks such as NER (often followed by prediction in a linear-chain CRF).Though expressive and accurate, these models fail to fully exploit GPU parallelism, limiting their computational efficiency.This paper proposes a faster alternative to Bi-LSTMs for NER: Iterated Dilated Convolutional Neural Networks (ID-CNNs), which have better capacity than traditional CNNs for large context and structured prediction.Unlike LSTMs whose sequential processing on sentences of length N requires O(N ) time even in the face of parallelism, ID-CNNs permit fixed-depth convolutions to run in parallel across entire documents.We describe a distinct combination of network structure, parameter sharing and training procedures that enable dramatic 14-20x testtime speedups while retaining accuracy comparable to the Bi-LSTM-CRF.Moreover, ID-CNNs trained to aggregate context from the entire document are even more accurate while maintaining 8x faster test time speeds.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · cited 1×
“LSTMs (Hochreiter and Schmidhuber, 1997) were used for NER as early as the CoNLL shared task in 2003 (Hammerton, 2003; Tjong Kim Sang and De Meulder, 2003).”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
Going deeper with convolutions
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015) · 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2015.7298594 · Cited by 48,562
(2014) · Cited by 33,419
Convolutional Neural Networks for Sentence Classification
Kim, Y. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1181 · Cited by 14,831
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Self-citationMulti-scale context aggregation by dilated convolutions
Yu, F., & Koltun, V. (2015) · arXiv · doi:10.48550/arxiv.1511.07122 · Cited by 7,738 · cited 7×
“We used the modified version of identity initialization (Le et al., 2015) re-ported by Yu and Koltun (2016) to initialize our dilated layers, which we found to perform the best in initial experiments compared to orthogonal and Xavier initialization (Glorot and Bengio, 2010).”
BackgroundMethodInfluentialNatural Language Processing (almost) from Scratch
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 4,962
WaveNet: a generative model for raw audio
Oord, A. V. D., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., & Kavukcuoglu, K. (2016) · arXiv · doi:10.48550/arxiv.1609.03499 · Cited by 4,911
Neural Architectures for Named Entity Recognition
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1030 · Cited by 4,401 · cited 5×
“The massively parallel computation facilitated by GPU hardware has led to a surge of successful neural network architectures for sequence labeling (Ling et al., 2015; Ma and Hovy, 2016; Chiu and Nichols, 2016; Lample et al., 2016).”
BackgroundMethodInfluential
Cited by
A Survey on Deep Learning for Named Entity Recognition
Li, J., Sun, A., Han, J., & Li, C. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2020.2981314 · Cited by 1,567
Recent Trends in Deep Learning Based Natural Language Processing
Young, T., Hazarika, D., Poria, S., & Cambria, E. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1708.02709 · Cited by 1,192
A Unified MRC Framework for Named Entity Recognition
Li, X., Feng, J., Meng, Y., Han, Q., Wu, F., & Li, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.519 · Cited by 697
Opportunities and Challenges for Machine Learning in Materials Science
Morgan, D., & Jacobs, R. (2020) · Annual Review of Materials Research · doi:10.1146/annurev-matsci-070218-010015 · Cited by 549
Named Entity Recognition as Dependency Parsing
Yu, J., Bohnet, B., & Poesio, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.577 · Cited by 458
Double Embeddings and CNN-based Sequence Labeling for Aspect Extraction
Xu, H., Liu, B., Shu, L., & Yu, P. S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2094 · Cited by 414
Unified Named Entity Recognition as Word-Word Relation Classification
Li, J., Fei, H., Liu, J., Wu, S., Zhang, M., Teng, C., Ji, D., & Li, F. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21344 · Cited by 394
A Comprehensive Survey on Automatic Knowledge Graph Construction
Zhong, L., Wu, J., Li, Q., Peng, H., & Wu, X. (2024) · ACM Computing Surveys · doi:10.1145/3618295 · Cited by 382
Template-Based Named Entity Recognition Using BART
Cui, L., Wu, Y., Liu, J., Yang, S., & Zhang, Y. (2021) · arXiv · doi:10.48550/arxiv.2106.01760 · Cited by 342
“…BiLSTM (Hochreiter and Schmidhuber, 1997) and BERT (Devlin et al., 2019) is used to represent the input text, and softmax (Chiu and Nichols, 2016; Strubell et al., 2017; Cui and Zhang, 2019) or CRF (Lample et al., 2016; Ma and Hovy, 2016; Luo et al., 2020) output layers to assign named entity…”
MethodInfluentialTemplate-Based Named Entity Recognition Using BART
Cui, L., Wu, Y., Liu, J., Yang, S., & Zhang, Y. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.161 · Cited by 314
A Neural Layered Model for Nested Named Entity Recognition
Ju, M., Miwa, M., & Ananiadou, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1131 · Cited by 309
CNN-Based Chinese NER with Lexicon Rethinking
Gui, T., Ma, R., Zhang, Q., Zhao, L., Jiang, Y.-G., & Huang, X. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/692 · Cited by 265
Deep Exhaustive Model for Nested Named Entity Recognition
Sohrab, M. G., & Miwa, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1309 · Cited by 235
“Recent successes in neural networks have shown impressive performance gains on flat named entity recognition in several domains (Lample et al., 2016; Ma and Hovy, 2016; Gridach, 2017; Strubell et al., 2017).”
BackgroundTENER: Adapting Transformer Encoder for Named Entity Recognition
Yan, H. R., Deng, B., Li, X., & Qiu, X. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.04474 · Cited by 227
“Therefore, CNN has been proposed by (Strubell et al., 2017; Gui et al., 2019a) to encode words concurrently.”
MethodInfluentialA Hierarchical Multi-Task Approach for Learning Embeddings from Semantic Tasks
Sanh, V., Wolf, T., & Ruder, S. (2019) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v33i01.33016949 · Cited by 203
On the Evolution of Knowledge Graphs: A Survey and Perspective
Jiang, X., Xu, C., Shen, Y., Sun, X., Tang, L., Wang, S., Chen, Z., Wang, Y., & Guo, J. (2023) · arXiv · doi:10.48550/arxiv.2310.04835 · Cited by 42
“Moreover, Emma et al. [37] propose ID-CNNs, employing fixed-depth convolutions for parallelism to conduct NER tasks.”
Metadata
- Authors
- Strubell, E., Verga, P., Belanger, D., & McCallum, A.
- Venue
- Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
- Published
- 2017
- Pages
- 2670-2680
- OpenAlex
- W2740462959
- Cited by
- 474
- References
- 42
- FWCI
- 23.63
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.