The DOI matches its Crossref record; 54 of the 57 references it cites were found in a registry or an index.
Dissecting Contextual Word Embeddings: Architecture and Representation
Open accessAbstract
Contextual word representations derived from pre-trained bidirectional language models (biLMs) have recently been shown to provide significant improvements to the state of the art for a wide range of NLP tasks.However, many questions remain as to how and why these models are so effective.In this paper, we present a detailed empirical study of how the choice of neural architecture (e.g.LSTM, CNN, or self attention) influences both end task accuracy and qualitative properties of the representations that are learned.We show there is a tradeoff between speed and accuracy, but all architectures learn high quality contextual representations that outperform word embeddings for four challenging NLP tasks.Additionally, all architectures learn representations that vary with network depth, from exclusively morphological based at the word embedding layer through local syntax based in the lower contextual layers to longer range semantics such coreference at the upper layers.Together, these results suggest that unsupervised biLMs, independent of architecture, are learning much more about the structure of language than previously appreciated.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Following Peters et al. (2018b); Liu et al. (2019a), we observe that the final 2-3 BERT layers provide embeddings that excel at predictive language modeling; as such, Figure 4 uses embeddings from layer 9 of S2ORC-SCIBERT.”
From S2ORC: The Semantic Scholar Open Research Corpus (2020) · cites this paper as Peters et al. (2018b)
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The study by Peters et al. (2018) suggests that the context vectors of bi-directional language models undergo abrupt changes at syntactic boundaries, allowing them to be utilized for constructing representations of spans or consecutive token sequences.”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Peters et al., 2018
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Deep Residual Learning for Image Recognition
He, K., Zhang, X., Ren, S., & Sun, J. (2016) · 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2016.90 · Cited by 235,474
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
(2008) · Journal of Machine Learning Research · Cited by 33,051
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Visualizing and understanding convolutional networks
Zeiler, M. D., & Fergus, R. (2014) · Lecture Notes in Computer Science · doi:10.1007/978-3-319-10590-1_53 · Cited by 15,839
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Self-citationBuilding a Large Annotated Corpus of English: The Penn Treebank
Marcus, M. (1993) · doi:10.21236/ada273556 · Cited by 7,405
ADADELTA: An Adaptive Learning Rate Method
Zeiler, M. D. (2012) · arXiv · doi:10.48550/arxiv.1212.5701 · Cited by 5,322
Flagged first, then by citations
3 of the 57 references couldn’t be matched to a known paper.
Research works printed but not linked: 4
- Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Has- san Sajjad, and James R. Glass. 2017. What do neu- ral machine translation models learn about morphol- ogy? In ACL.
- Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Open- NMT: Open-source toolkit for neural machine trans- lation. In ACL.
- Liyuan Liu, Xiang Ren, Jingbo Shang, Jian Peng, and Jiawei Han. 2018. Efficient contextualized represen- tation: Language model pruning for sequence label- ing. In EMNLP.
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL.
Cited by
Sorted by citations
Items 101–200 of 436
Towards better representation learning using hybrid deep learning model for fake news detection
Kausar, N., AliKhan, A., & Sattar, M. (2022) · Social Network Analysis and Mining · doi:10.1007/s13278-022-00986-6 · Cited by 38
Introduction to Neural Transfer Learning With Transformers for Social Science Text Analysis
Wankmüller, S. (2022) · Sociological Methods & Research · doi:10.1177/00491241221134527 · Cited by 45
Contextual Word Representations: A Contextual IntroductionPreprint
Smith, N. A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.06006 · Cited by 41
Choenni, R., & Shutova, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.12862 · Cited by 40
Probing BERT in Hyperbolic SpacesPreprint
Chen, B., Fu, Y., Xu, G., Xie, P., Tan, C., Chen, M., & Jing, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.03869 · Cited by 46
Roles and Utilization of Attention Heads in Transformer-based Neural Language Models
Jo, J.-Y., & Myaeng, S.-H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.311 · Cited by 33
BERTnesia: Investigating the capture and forgetting of knowledge in BERT
Wallat, J., Singh, J., & Anand, A. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.17 · Cited by 40
Asking without Telling: Exploring Latent Ontologies in Contextual Representations
Michael, J., Botha, J. A., & Tenney, I. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.552 · Cited by 38
Frequency, acceptability, and selection: A case study of clause-embedding
White, A. S., & Rawlins, K. (2020) · Glossa a journal of general linguistics · doi:10.5334/gjgl.1001 · Cited by 32
Multi-Head Multi-Layer Attention to Deep Language Representations for Grammatical Error Detection
Kaneko, M., & Komachi, M. (2019) · Computación y Sistemas · doi:10.13053/cys-23-3-3271 · Cited by 35
Probing for Referential Information in Language Models
Sorodoc, I.-T., Gulordava, K., & Boleda, G. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.384 · Cited by 37
Hybrid Text Representation for Explainable Suicide Risk Identification on Social Media
Naseem, U., Khushi, M., Kim, J., & Dunn, A. G. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3184984 · Cited by 31
Bagchi, S., Aggarwal, V., Chaterji, S., Douglis, F., Gamal, A. E., Han, J., Henz, B. J., Hoffmann, H., Jana, S., Kulkarni, M., et al. (2020) · IEEE Open Journal of the Computer Society · doi:10.1109/ojcs.2020.3006807 · Cited by 28
A Relation-Oriented Clustering Method for Open Relation Extraction
Zhao, J., Gui, T., Zhang, Q., & Zhou, Y. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.765 · Cited by 28
Conditional probing: measuring usable information beyond a baseline
Hewitt, J., Ethayarajh, K., Liang, P., & Manning, C. D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.122 · Cited by 40
A deep learning approach to text-based personality prediction using multiple data sources mapping
Johnson, S. J., & Malla, R. M. (2023) · Neural Computing and Applications · doi:10.1007/s00521-023-08846-w · Cited by 29
Evaluation Benchmarks and Learning Criteria for Discourse-Aware Sentence Representations
Chen, M., Chu, Z., & Gimpel, K. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1060 · Cited by 29
Mixed-Precision Embedding Using a CachePreprint
Yang, J., Huang, J., Park, J., Tang, P. T. P., & Tulloch, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.11305 · Cited by 35
LMMS reloaded: Transformer-based sense embeddings for disambiguation and beyond
Loureiro, D., Jorge, A., & Camacho-Collados, J. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2022.103661 · Cited by 28
A Review of State of the Art Deep Learning Models for Ontology Construction
Zengeya, T., & Fonou-Dombeu, J. V. (2024) · IEEE Access · doi:10.1109/access.2024.3406426 · Cited by 29
Franzon, F., & Zanini, C. (2022) · Journal of Quantitative Linguistics · doi:10.1080/09296174.2022.2063501 · Cited by 25
Mining arguments in scientific abstracts with discourse-level embeddings
Accuosto, P., & Saggion, H. (2020) · Data & Knowledge Engineering · doi:10.1016/j.datak.2020.101840 · Cited by 27
E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoning
Chen, J., Xu, R., Fu, Z., Shi, W., Li, Z., Zhang, X., Sun, C., Li, L., Xiao, Y., & Zhou, H. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.311 · Cited by 30
Investigating Learning Dynamics of BERT Fine-Tuning
Hao, Y., Dong, L., Wei, F., & Xu, K. (2020) · Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing (AACL-IJCNLP) · doi:10.18653/v1/2020.aacl-main.11 · Cited by 25
Discovering the Compositional Structure of Vector Representations with Role Learning Networks
Soulos, P., McCoy, T., Linzen, T., & Smolensky, P. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.23 · Cited by 38
Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019)
Augenstein, I., Gella, S., Ruder, S., Kann, K., Can, B., Conneau, A., Welbl, J., Ren, X., & Rei, M. (2019) · doi:10.18653/v1/w19-43 · Cited by 24
Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling.
Bowman, S. R., Pavlick, E., Grave, É., Durme, B. V., Wang, A., Hůla, J., Xia, P., Pappagari, R. R., McCoy, R. T., Patel, R., et al. (2018) · Cited by 23
On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Zhang, T., & Hashimoto, T. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.404 · Cited by 27
Newman-Griffis, D., & Fosler-Lussier, E. (2021) · Frontiers in Digital Health · doi:10.3389/fdgth.2021.620828 · Cited by 24
ATOM-1: A Foundation Model for RNA Structure and Function Built on Chemical Mapping DataPreprint
Boyd, N. R., Anderson, B., Townshend, B., Chow, R., Stephens, C. J., Rangan, R., Kaplan, M., Corley, M., Tambe, A., Ido, Y., et al. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.12.13.571579 · Cited by 23
SemGloVe: Semantic Co-Occurrences for GloVe From BERT
Gan, L., Teng, Z., Zhang, Y., Zhu, L., Wu, F., & Yang, Y. (2022) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2022.3197316 · Cited by 22
Contextualized embeddings for semantic change detection: Lessons learned
Kutuzov, A., Velldal, E., & Øvrelid, L. (2022) · Northern European Journal of Language Technology · doi:10.3384/nejlt.2000-1533.2022.3478 · Cited by 30
Multilingual Text Summarization for German Texts Using Transformer Models
Alcantara, T. H. M., Krütli, D., Ravada, R., & Hanne, T. (2023) · Information · doi:10.3390/info14060303 · Cited by 24
Embeddings in Natural Language Processing
Camacho-Collados, J., & Pilehvar, M. T. (2020) · doi:10.18653/v1/2020.coling-tutorials.2 · Cited by 22
Binder, M., Heinrich, B., Hopf, M., & Schiller, A. (2022) · Electronic Markets · doi:10.1007/s12525-022-00612-5 · Cited by 27
Towards Interpreting BERT for Reading Comprehension Based QA
Ramnath, S., Nema, P., Sahni, D., & Khapra, M. M. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.261 · Cited by 28
My Body is a Cage: the Role of Morphology in Graph-Based Incompatible ControlPreprint
Kurin, V., Igl, M., Rocktäschel, T., Böhmer, W., & Whiteson, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.01856 · Cited by 21
Evaluating Document Coherence Modeling
Shen, A., Mistica, M., Salehi, B., Li, H., Baldwin, T., & Qi, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00388 · Cited by 22
Comparative Analysis of BERT-base Transformers and Deep Learning Sentiment Prediction Models
Chinnalagu, A., & Durairaj, A. K. (2022) · International Conference on System Modeling & Advancement in Research Trends (SMART) · doi:10.1109/smart55829.2022.10047651 · Cited by 20
Probing for Constituency Structure in Neural Language Models
Arps, D., Samih, Y., Kallmeyer, L., & Sajjad, H. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.502 · Cited by 28
Appellate Court Modifications Extraction for Portuguese
Fernandes, W. P. D., Schirmer, L., Frajhof, I. Z., Almeida, G., Konder, C. N., Nasser, R., de Carvalho, G. R., Barbosa, S. D. J., & Lopes, H. (2019) · Artificial Intelligence and Law · doi:10.1007/s10506-019-09256-x · Cited by 20
ELMo and BERT in Semantic Change Detection for Russian
Rodina, J., Trofimova, Y., Kutuzov, A., & Artemova, E. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72610-2_13 · Cited by 19
Performance Comparison of Transformer-Based Models on Twitter Health Mention Classification
Khan, P. I., Razzak, I., Dengel, A. R., & Ahmed, S. (2022) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2022.3143768 · Cited by 19
Tabaie, A., Sengupta, S., Pruitt, Z. M., & Fong, A. (2023) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2022-100731 · Cited by 18
Using Multi-Sense Vector Embeddings for Reverse Dictionaries
Hedderich, M. A., Yates, A., Klakow, D., & de Melo, G. (2019) · International Conference on Computational Semantics (IWCS) · doi:10.18653/v1/w19-0421 · Cited by 19
Distributional Semantics: Meaning Through Culture and Interaction
Contreras Kallens, P., & Christiansen, M. H. (2025) · Topics in Cognitive Science · doi:10.1111/tops.12771 · Cited by 17
Improving Named Entity Recognition for Social Media with Data Augmentation
Liu, W., & Cui, X. (2023) · Applied Sciences · doi:10.3390/app13095360 · Cited by 17
Grishina, A., Hort, M., & Moonen, L. (2023) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3611643.3616304 · Cited by 21
Ramadhani, D. P., Alamsyah, A., Febrianta, M. Y., Fajriananda, M. N., Nada, M. S., & Hasanah, F. (2025) · Computers · doi:10.3390/computers14010027 · Cited by 17
A Linguistic Study on Relevance Modeling in Information Retrieval
Fan, Y., Guo, J., Ma, X., Zhang, R., Lan, Y., & Cheng, X. (2021) · International World Wide Web Conference (WWW) · doi:10.1145/3442381.3450009 · Cited by 21
Exploring the Role of BERT Token Representations to Explain Sentence Probing Results
Mohebbi, H., Modarressi, A., & Pilehvar, M. T. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.61 · Cited by 28
Inducing Causal Structure for Interpretable Neural NetworksPreprint
Geiger, A., Wu, Z., Lu, H., Rozner, J., Kreiss, E., Icard, T., Goodman, N. D., & Potts, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2112.00826 · Cited by 32
Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)Preprint
Oota, S. R., Chen, Z., Gupta, M., Bapi, R. S., Jobard, G., Alexandre, F., & Hinaut, X. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.10246 · Cited by 34
Neural DatabasesPreprint
Thorne, J., Yazdani, M., Saeidi, M., Silvestri, F., Riedel, S., & Halevy, A. Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.06973 · Cited by 20
Is Anisotropy Truly Harmful? A Case Study on Text Clustering
Ait-Saada, M., & Nadif, M. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.103 · Cited by 15
Zhao, Y. D., Gong, L., Huang, Z., Wang, Y., Wei, M., & Wu, F. Q. (2024) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/3691620.3695539 · Cited by 18
Evaluating Pronominal Anaphora in Machine Translation: An Evaluation Measure and a Test Suite
Jwalapuram, P., Joty, S., Temnikova, I., & Nakov, P. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1294 · Cited by 17
Improving Biomedical Information Extraction with Word Embeddings Trained on Closed-Domain Corpora
Silvestri, S., Gargiulo, F., & Ciampi, M. (2019) · IEEE Symposium on Computers and Communications (ISCC) · doi:10.1109/iscc47284.2019.8969769 · Cited by 15
van Staden, L., & Kamper, H. (2021) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt48900.2021.9383625 · Cited by 17
Chowdhury, S. A., Durrani, N., & Ali, A. M. A. (2023) · Computer Speech & Language · doi:10.1016/j.csl.2023.101539 · Cited by 16
Pre- Trained Language Models for Mental Health: An Empirical Study on Arabic Q&A Classification
Alhuzali, H., & Alasmari, A. (2025) · Healthcare · doi:10.3390/healthcare13090985 · Cited by 14
Zero-Shot Clinical Acronym Expansion via Latent Meaning Cells
Adams, G., Ketenci, M., Bhave, S., Perotte, A., & Elhadad, N. (2020) · PubMed · doi:10.48550/arxiv.2010.02010 · Cited by 14
Personality Identification from Social Media Using Ensemble Bert and Roberta
Tsani, E. F., & Suhartono, D. (2023) · Informatica · doi:10.31449/inf.v47i4.4771 · Cited by 14
A Survey of the Model Transfer Approaches to Cross-Lingual Dependency Parsing
Das, A., & Sarkar, S. (2020) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3383772 · Cited by 14
Li, J., Cotterell, R., & Sachan, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.84 · Cited by 17
Improved Transformer With Multi-Head Dense Collaboration
Wang, H., Shen, X., Tu, M., Zhuang, Y., & Liu, Z. (2022) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2022.3199648 · Cited by 15
Empowering the Fact-checkers! Automatic Identification of Claim Spans on Twitter
Sundriyal, M., Kulkarni, A., Pulastya, V., Akhtar, M. S., & Chakraborty, T. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.525 · Cited by 17
Multilingual Probing of Deep Pre-Trained Contextual Encoders
Ravishankar, V., Gökırmak, M., Øvrelid, L., & Velldal, E. (2019) · DSpace repository (University of Tartu) · Cited by 14
Multi-view self-attention networks
Xu, M., Yang, B., Wong, D. F., & Chao, L. S. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.108268 · Cited by 14
Towards Robust Named Entity Recognition for Historic German
Schweter, S., & Baiter, J. (2019) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/w19-4312 · Cited by 20
Learning Syntax Without Planting Trees: Understanding Hierarchical Generalization in Transformers
Ahuja, K., Balachandran, V., Panwar, M., He, T., Smith, N. A., Goyal, N., & Tsvetkov, Y. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00733 · Cited by 21
Deep Contextualized Word Embeddings for Universal Dependency Parsing
Liu, Y., Che, W., Wang, Y., Zheng, B., Qin, B., & Liu, T. (2019) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3326497 · Cited by 13
Zhang, Z., Zhu, L., & Yu, P. (2020) · JMIR Medical Informatics · doi:10.2196/17637 · Cited by 13
Hou, Y. T., Wang, H., & Wang, H. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103033 · Cited by 13
On the Interplay Between Fine-tuning and Composition in Transformers
Yu, L., & Ettinger, A. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.201 · Cited by 13
Revisiting the Importance of Encoding Logic Rules in Sentiment Classification
Krishna, K., Jyothi, P., & Iyyer, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1505 · Cited by 13
Understanding Transformers for Bot Detection in TwitterPreprint
García-Silva, A., Berrío, C., & Gómez-Pérez, J. M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.06182 · Cited by 16
Naseem, U., Khushi, M., Kim, J., & Dunn, A. G. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533454 · Cited by 13
MCRSpell: A metric learning of correct representation for Chinese spelling correction
Li, C., Zhang, M., Zhang, X., & Yan, Y. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121513 · Cited by 13
Yang, H., Wang, E., Song, Y., Chen, D., Wang, X., Wang, D., & Li, J. (2024) · Physics of Fluids · doi:10.1063/5.0219409 · Cited by 12
A Comparison of Context-sensitive Models for Lexical Substitution
Soler, A. G., Cocos, A., Apidianaki, M., & Callison-Burch, C. (2019) · International Conference on Computational Semantics (IWCS) · doi:10.18653/v1/w19-0423 · Cited by 12
EntEval: A Holistic Evaluation Benchmark for Entity Representations
Chen, M., Chu, Z., Chen, Y., Stratos, K., & Gimpel, K. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1040 · Cited by 12
Enhancing Claims Handling Processes with Insurance Based Language Models
Dimri, A., Yerramilli, S., Lee, P., Afra, S., & Jakubowski, A. (2019) · International Conference on Machine Learning and Applications · doi:10.1109/icmla.2019.00284 · Cited by 12
Quantifying the Contextualization of Word Representations with Semantic Class Probing
Zhao, M., Dufter, P., Yaghoobzadeh, Y., & Schütze, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.109 · Cited by 14
Mediators in Determining what Processing BERT Performs First
Slobodkin, A., Choshen, L., & Abend, O. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.8 · Cited by 13
Structural generalization is hard for sequence-to-sequence models
Yao, Y., & Koller, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.337 · Cited by 21
Saberi, I., Fard, F. H., & Chen, F. (2024) · Empirical Software Engineering · doi:10.1007/s10664-024-10457-5 · Cited by 12
Knowledge-Aware Conversational Semantic Parsing over Web Tables
Sun, Y., Tang, D., Xu, J., Duan, N., Feng, X., Qin, B., Liu, T., & Zhou, M. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32233-5_64 · Cited by 12
Dependency-Guided LSTM-CRF for Named Entity RecognitionPreprint
Jie, Z., & Lu, W. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.10148 · Cited by 18
Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability
Geiger, A., Ibeling, D., Zur, A., Chaudhary, M., Chauhan, S., Huang, J., Arora, A., Wu, Z., Goodman, N., Potts, C., et al. (2023) · arXiv · doi:10.48550/arxiv.2301.04709 · Cited by 11
ELion: An Intelligent Chinese Composition Tutoring System Based on Large Language Models
Zheng, C., Guo, S., & Xia, W. (2023) · Chinese/English Journal of Educational Measurement and Evaluation · doi:10.59863/mpjo6480 · Cited by 12
A multi-facet analysis of BERT-based entity matching models
Paganelli, M., Tiano, D., & Guerra, F. (2023) · The VLDB Journal · doi:10.1007/s00778-023-00824-x · Cited by 15
Zou, X., Li, W., Wu, W., Hunt, A. S. P., & Lu, H. (2025) · Systems · doi:10.3390/systems13060494 · Cited by 11
Molecular Topological Deep Learning for Polymer Property Prediction
Shen, C., Zhang, Y., Er, T. K. G., Han, F., Goto, A., & Xia, K. (2026) · ACS Nano · doi:10.1021/acsnano.5c11744 · Cited by 13
Unsupervised Labeled Parsing with Deep Inside-Outside Recursive Autoencoders
Drozdov, A., Verga, P. W., Chen, Y.-P., Iyyer, M., & McCallum, A. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1161 · Cited by 11
Differentiable Window for Dynamic Local Attention
Nguyen, T.-T., Nguyen, X.-P., Joty, S., & Li, X. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.589 · Cited by 11
Does BERT Understand Idioms? A Probing-Based Empirical Study of BERT Encodings of Idioms
Tan, M., & Jiang, J. (2021) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-072-4_156 · Cited by 11
Metadata
- Authors
- Peters, M., Neumann, M., Zettlemoyer, L., & Yih, W.-T.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- University of Washington (US)
- Venue
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Published
- 2018
- Pages
- 1499-1509
- OpenAlex
- W2888329843
- Cited by
- 434
- References
- 57
- Citations by year
- 2026: 23 · 2025: 27 · 2024: 37 · 2023: 53 · 2022: 55 · 2021: 75 · 2020: 83 · 2019: 77 · 2018: 4
- FWCI
- 24.08
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- contextual word embeddings, bidirectional language models, LSTM, convolutional neural networks, self-attention, representation learning, network depth, morphological representation, syntactic representation, coreference, word embeddings, speed-accuracy trade-off
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
