The DOI matches its Crossref record; 26 of the 27 references it cites were found in a registry or an index.
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
Open accessAbstract
Replacing static word embeddings with contextualized word representations has yielded significant improvements on many NLP tasks.However, just how contextual are the contextualized representations produced by models such as ELMo and BERT?Are there infinitely many context-specific representations for each word, or are words essentially assigned one of a finite number of word-sense representations?For one, we find that the contextualized representations of all words are not isotropic in any layer of the contextualizing model.While representations of the same word in different contexts still have a greater cosine similarity than those of two different words, this self-similarity is much lower in upper layers.This suggests that upper layers of contextualizing models produce more context-specific representations, much like how upper layers of LSTMs produce more task-specific representations.In all layers of ELMo, BERT, and GPT-2, on average, less than 5% of the variance in a word's contextualized representations can be explained by a static embedding for that word, providing some justification for the success of contextualized representations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Precedent exists for using PCA to extract network-level patterns from high-dimensional neural data, including fMRI (Smith et al., 2015), EEG (Blankertz et al., 2011), and even LLM embeddings (Ethayarajh, 2019).”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Ethayarajh, 2019
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Ethayarajh (2019) notes that unidirectional models produce embeddings that vary smoothly across tokens, implying that the difference operation can still capture meaningful transitions within a span.”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Ethayarajh, 2019
Experimental — accuracy not yet measuredIn the citing paper
“Previous literature showed the BERT embeddings54 capture rich semantic information and are widely recognized as effective for sentence similarity tasks55–57.”
From Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation (2025) · cites this paper as [57]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061
Linguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · Cited by 2,913
Neural Word Embedding as Implicit Matrix Factorization
Levy, O., & Goldberg, Y. (2014) · neural information processing systems · Cited by 1,581
Flagged first, then by citations
1 of the 27 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez- Agirre, and Weiwei Guo. 2013. Sem 2013 shared task: Semantic textual similarity, including a pilot on typed-similarity. In SEM 2013: The Second Joint Conference on Lexical and Computational Seman- tics. Association for Computational Linguistics.
- John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word represen- tations. In North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies. Association for Computational Linguistics.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013a. Distributed representa- tions of words and phrases and their compositional- ity. In Advances in Neural Information Processing Systems. pages 3111-3119.
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long Papers). pages 2227-2237.
- Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipan- jan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In Inter- national Conference on Learning Representations. https://openreview.net/forum?id=SJzSgnRcKX.
Cited by
Sorted by citations
Items 501–600 of 898
Rethinking Word Similarity: Semantic Similarity through Classification Confusion
Zhou, K., Gao, H., Chen, S. L., Edelstein, D., Jurafsky, D., & Shani, C. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.299 · Cited by 3
Answering Fill-in-the-Blank Questions in Portuguese with Transformer Language Models
Oliveira, H. G. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86230-5_58 · Cited by 3
Weighted Sampling for Masked Language Modeling
Zhang, L., Chen, Q., Wang, W., Deng, C., Cao, X., Hao, K., Jiang, Y., & Wang, W. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10096946 · Cited by 3
The Task of Question Answering in NLP: A Comprehensive Review
Sarkar, S., Singh, P., Kumari, N., & Kashtriya, P. (2023) · Lecture notes in electrical engineering · doi:10.1007/978-981-99-0601-7_46 · Cited by 3
Xu, B., Wei, S., Cheng, L., Huang, S., Song, H., Du, M., & Wang, H. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191335 · Cited by 3
Garden Path Traversal in GPT-2
Jurayj, W., Rudman, W., & Eickhoff, C. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.25 · Cited by 5
ConGen: Unsupervised control and generalization distillation for sentence representation
Limkonchotiwat, P., Ponwitayarat, W., Lowphansirikul, L., Udomcharoenchaikit, C., Chuangsuwanich, E., & Nutanong, S. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.483 · Cited by 3
Structure-Aware Dialogue Modeling Methods for Conversational Semantic Role Labeling
Wu, H., Xu, K., & Song, L. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3331576 · Cited by 3
Hierarchical Topic-Aware Contextualized Transformers
Lu, R., Chen, B., Guo, D., Wang, D., & Zhou, M. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3339344 · Cited by 3
Shi, Z., Wang, G., Bai, K., Li, J., Li, X., Cui, Q., Zeng, B., Chilimbi, T. M., & Zhu, X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.448 · Cited by 5
Improving word mover’s distance by leveraging self-attention matrix
Yamagiwa, H., Yokoi, S., & Shimodaira, H. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.746 · Cited by 3
Research on Automatic Generation of Social Short Text Based on Backtracking Pattern
Ma, R., Gao, Y., Li, X., & Yang, L. (2023) · Asia-Pacific Conference on Image Processing, Electronics and Computers (IPEC) · doi:10.1109/ipec57296.2023.00066 · Cited by 3
Wei, C., Zheng, D., Fu, S., & Cui, J. (2024) · Scientometrics · doi:10.1007/s11192-024-05080-6 · Cited by 3
Effects of Architecture and Training on Embedding Geometry and Feature Discriminability in BERT
Podkorytov, M., Biś, D., Cai, J., Amirizirtol, K., & Liu, X. (2020) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn48605.2020.9206645 · Cited by 5
Improving Social Meaning Detection with Pragmatic Masking and Surrogate Fine-Tuning
Zhang, C., & Abdul-Mageed, M. (2022) · Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA) · doi:10.18653/v1/2022.wassa-1.14 · Cited by 3
Comparing Text Representations: A Theory-Driven Approach
Yauney, G., & Mimno, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.449 · Cited by 4
Unsupervised Contextualized Document Representation
Gupta, A., & Gupta, V. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.17 · Cited by 3
Ensemble Multi-Channel Neural Networks for Scientific Language Editing Evaluation
Lee, L., Wang, Y.-S., Chen, C.-Y., & Yu, L.-C. (2021) · IEEE Access · doi:10.1109/access.2021.3130042 · Cited by 3
CNERVis: a visual diagnosis tool for Chinese named entity recognition
Lo, P.-S., Wu, J.-L., Deng, S.-T., & Wang, K.-C. (2021) · Journal of Visualization · doi:10.1007/s12650-021-00799-3 · Cited by 3
Evaluation of Vector Transformations for Russian Static and Contextualized Embeddings
Korogodina, O., Koulichenko, V., Karpik, O. V., & Klyshinsky, E. S. (2021) · International Conference on Computer Graphics and Vision (GraphiCon) · doi:10.20948/graphicon-2021-3027-349-357 · Cited by 3
Bridging Pre-trained Language Models and Hand-crafted Features for Unsupervised POS Tagging
Zhou, H., Li, Y., Li, Z., & Zhang, M. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.259 · Cited by 3
Regularized Contrastive Learning of Semantic Search
Tan, M., Rolland, A., & Tian, A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-17120-8_10 · Cited by 3
ACF: Aligned Contrastive Finetuning For Language and Vision Tasks
Zhu, W., Wang, P., Wang, X., Ni, Y., & Xie, G. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10094859 · Cited by 3
FCGCL: Fine- and Coarse-Granularity Contrastive Learning for Speech Translation
Zhang, H., Si, N., Chen, Y., Li, Z., Niu, T., Yang, X., & Qu, D. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.222 · Cited by 3
Yang, H., Zhao, J., Haffari, G., & Shareghi, E. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.141 · Cited by 3
A self-explanatory contrastive logical knowledge learning method for sentiment analysis
Chen, Y., Yuan, B., Liao, B., & Gabbay, D. M. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110863 · Cited by 3
Rust, P., & Søgaard, A. (2023) · Research at the University of Copenhagen (University of Copenhagen) · doi:10.48550/arxiv.2308.08774 · Cited by 3
Osuna, J. J. O., Jorge‐Botana, G., Martínez‐Huertas, J. Á., Olmos, R., & Martínez-Mingo, A. (2023) · Political Research Exchange · doi:10.1080/2474736x.2023.2287036 · Cited by 3
Pre-Training a Graph Recurrent Network for Text Understanding
Wang, Y., Yang, L., Teng, Z., Zhou, M., & Zhang, Y. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3538221 · Cited by 4
Basic Methodologies Used in NLP Area
Wang, Y. (2020) · IEEE International Conference on Automation, Electronics and Electrical Engineering (AUTEEE) · doi:10.1109/auteee50969.2020.9315550 · Cited by 3
On the Importance of Word Embedding in Automated Harmful Information Detection
Mohtaj, S., & Möller, S. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-16270-1_21 · Cited by 3
JETAM: A Joint Embedding Method for Research Paper Mutil-Label Classification
Yan, W. Q., He, Y., & Yang, C. (2022) · International Conference on Image Processing, Computer Vision and Machine Learning (ICICML) · doi:10.1109/icicml57342.2022.10009721 · Cited by 3
Knowledge-aware response selection with semantics underlying multi-turn open-domain conversations
Nakatsuji, M., Ozeki, Y., Tateishi, S., Kano, Y., & Zhang, Q. (2023) · World Wide Web · doi:10.1007/s11280-023-01164-0 · Cited by 3
Paraphrase Generation via Adversarial Penalizations
Vizcarra, G., & Ochoa-Luna, J. E. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.32 · Cited by 3
LAWDR: Language-Agnostic Weighted Document Representations from Pre-trained ModelsPreprint
Gong, H., Chaudhary, V., Tang, Y., & Guzmán, F. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.03379 · Cited by 3
Good Arguments Against the People Pleasers: How Reasoning Mitigates (Yet Masks) LLM Sycophancy
Feng, Z., Chen, Z., Ma, J., Po, Y. T., Chersoni, E., & Li, B. (2026) · Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2026.acl-long.1126 · Cited by 2
Li, J. (2025) · Natural Language Processing Journal · doi:10.1016/j.nlp.2025.100163 · Cited by 3
Pande, M., Budhraja, A., Nema, P., Kumar, P., & Khapra, M. M. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17605 · Cited by 2
Terminology and Lexical Semantics
L’Homme, M.-C. (2022) · Terminology and lexicography research and practice · doi:10.1075/tlrp.23.11lho · Cited by 2
KSAA-RD Shared Task: Arabic Reverse Dictionary
Al-Matham, R. N., Alshammari, W. T., Alosaimy, A., Alhumoud, S. O., Wazrah, A. A. A., Altamimi, A. A., Alharbi, H. M., & Alaifi, A. (2023) · Arabic Natural Language Processing Conference (ArabicNLP) · doi:10.18653/v1/2023.arabicnlp-1.39 · Cited by 2
Characterizing Features of Creative Writing in Older Adults
Orwig, W., & Schacter, D. L. (2024) · The Journals of Gerontology Series B · doi:10.1093/geronb/gbae111 · Cited by 2
Beyond Words: Exploring Co-Attention with BERT in Visual Question Answering
S, V. B., Kamble, N., Karamadi, S., Varur, S., & Desai, P. (2024) · International Conference for Emerging Technology (INCET) · doi:10.1109/incet61516.2024.10593261 · Cited by 2
Assessing Logical Reasoning Capabilities of Encoder-Only Transformer Models
Pirozelli, P., José, M. M., Filho, P. D. T. P. D. C., Brandão, A. A. F., & Cozman, F. G. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-71167-1_2 · Cited by 5
Kosireddy, T. R., Wall, J. D., & Lucas, E. (2024) · Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U) · doi:10.18653/v1/2024.customnlp4u-1.11 · Cited by 3
Shrink the Longest: Improving Latent Space Isotropy with Simplicial Geometry
Kudrjashov, S., Karpik, O. V., & Klyshinsky, E. S. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-88036-0_6 · Cited by 2
Alsuhaibani, M. (2025) · PeerJ Computer Science · doi:10.7717/peerj-cs.2957 · Cited by 2
Wang, S., Bai, Z., & Gai, Y. (2025) · Health Information Science and Systems · doi:10.1007/s13755-025-00416-9 · Cited by 2
Zhang, M., & Xia, Z. (2026) · Humanities and Social Sciences Communications · doi:10.1057/s41599-025-06317-2 · Cited by 2
Transmission With Machine Language Tokens: A Paradigm for Task-Oriented Agent Communication
Xiao, Z., Ye, C., Feng, Y., Hu, Y., Jiao, T., Cai, L., & Liu, G. (2025) · Globecom · doi:10.1109/globecom59602.2025.11431701 · Cited by 2
Fine-grained ranking to boost unsupervised sentence embeddings
Liu, C., Gao, C., Li, Y., & Wang, M. (2026) · World Wide Web · doi:10.1007/s11280-026-01409-8 · Cited by 2
Optimizing of IndoBERT Embedding with Ditto Whitening for Measuring Research Title Similarity
Ishak, R., & Bengnga, A. (2026) · Jambura Journal of Electrical and Electronics Engineering · doi:10.37905/jjeee.v8i1.35554 · Cited by 2
Tail-Aware Adaptive-k: Query-Adaptive Context Selection for Retrieval-Augmented Generation
Song, Z., Fang, J., Li, K., Xia, T., & Wang, C. (2026) · Lecture notes in computer science · doi:10.1007/978-3-032-37667-1_12 · Cited by 2
Terminology and distributional analysis of corpora
Bertels, A. (2022) · Terminology and lexicography research and practice · doi:10.1075/tlrp.23.14ber · Cited by 3
Geometric Analysis and Metric Learning of Instruction Embeddings
Biswas, S., Barao, T., Lazzari, J., McCoy, J., Liu, X., & Kostandarithes, A. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892426 · Cited by 3
Sentiment analysis in SemEval: a review of sentiment identification approaches
Haddaoui, B. E., Chiheb, R., Faizi, R., & Afia, A. E. (2023) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijece.v13i3.pp3322-3338 · Cited by 2
Hämmerl, K., Fastowski, A., Libovický, J., & Fraser, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.439 · Cited by 2
TableSF: A Structural Bias Framework for Table-To-Text Generation
Liu, D., Wang, W., Bao, F., & Gao, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44201-8_33 · Cited by 2
Contextuality of Code Representation Learning
Li, Y., Wang, S., & Nguyen, T. N. (2023) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase56229.2023.00029 · Cited by 4
A Comprehensive Review of Relation Prediction Techniques in Knowledge Graph
Lu, Y., Yang, S., & Tang, B. (2024) · Communications in computer and information science · doi:10.1007/978-981-97-2991-3_2 · Cited by 2
Jose, S., Nguyen, K. T. P., Medjaher, K., Zemouri, R. M., Lévesque, M., & Tahan, A. S. (2024) · IEEE International Conference on Advanced Intelligent Mechatronics (AIM) · doi:10.1109/aim55361.2024.10637229 · Cited by 2
Investigating Idiomaticity in Word Representations
He, W. P., Vieira, T. K., García, M., Scarton, C., Idiart, M., & Villavicencio, A. (2024) · Computational Linguistics · doi:10.1162/coli_a_00546 · Cited by 4
Detecting Text-Based Cybercrimes Using BERT
Manna, A., Al-Fayoumi, M. A., & Al-Fawa’reh, M. (2024) · International Jordanian Cybersecurity Conference (IJCC) · doi:10.1109/ijcc64742.2024.10847273 · Cited by 2
A preliminary exploration of ChatGPT’s potential in medical reasoning and patient care
Siddiqui, Z. H., Azeez, M. A., Sohail, S. S., Ahmed, J., & Madsen, D. Ø. (2025) · Critical Public Health · doi:10.1080/09581596.2025.2484344 · Cited by 2
ESTIME: Estimation of Summary-to-Text Inconsistency by Mismatched Embeddings
Vasilyev, O., & Bohannon, J. (2021) · Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP) · doi:10.18653/v1/2021.eval4nlp-1.10 · Cited by 3
Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models
Ye, Q., Khabsa, M., Lewis, M., Wang, S., Ren, X., & Jaech, A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.169 · Cited by 2
Revealing the Blind Spot of Sentence Encoder Evaluation by HEROS
Chiang, C.-H., Lee, H.-Y., Chuang, Y.-S., & Glass, J. R. (2023) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2023.repl4nlp-1.24 · Cited by 2
Sharoff, S., Rapp, R., & Zweigenbaum, P. (2023) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-31384-4_3 · Cited by 2
Multi-gate Mixture-of-Contrastive-Experts with Graph-based Gating Mechanism for TV Recommendation
Zhang, C., Liu, D., Zuo, L., Feng, J., Deng, C., Sun, J., Zeng, H., & Zhao, Y. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615488 · Cited by 2
Hou, F., Wang, R., Ng, S.-K., Zhu, F., Witbrock, M., Cahan, S. F., Chen, L., & Jia, X. (2024) · Natural Language Engineering · doi:10.1017/s1351324924000019 · Cited by 2
Tangherlini, T. R., & Chen, R. (2024) · Orbis Litterarum · doi:10.1111/oli.12458 · Cited by 3
CAST: Corpus-Aware Self-similarity Enhanced Topic modelling
Ma, Y., Xiao, C., Yuan, C., van der Veer, S. N., Hassan, L., Lin, C., & Nenadić, G. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.386 · Cited by 3
Assessing Lexical Semantic Regularities in Portuguese Word Embeddings.
Oliveira, H. G., Sousa, T., & Alves, A. (2021) · International Journal of Interactive Multimedia and Artificial Intelligence · doi:10.9781/ijimai.2021.02.006 · Cited by 2
ABL-MICRO: Opportunities for Affective AI Built Using a Multimodal Microaggression Dataset.
Washington, G. J., Mance, G., Aryal, S. K., & Kengni, M. (2021) · National Conference on Artificial Intelligence · Cited by 2
Word Embedding based Event Identification and Labeling of Connected Events from Tweets
Das, P. K., Banna, M. A., Fahad, M. A. A., Islam, S., & Mukta, M. S. H. (2021) · International Conference on Software Engineering and Computer Systems · doi:10.1109/icsecs52883.2021.00105 · Cited by 2
Towards Better Evidence Extraction Methods for Fact-Checking Systems
Azevedo, P., Rocha, G., Esteves, D., & Cardoso, H. L. (2021) · IEEE/WIC/ACM International Conference on Web Intelligence · doi:10.1145/3486622.3493930 · Cited by 3
Approximate nearest neighbor for long document relationship labeling in digital libraries
Organisciak, P., Schmidt, B. M., & Durward, M. (2023) · International Journal on Digital Libraries · doi:10.1007/s00799-023-00354-5 · Cited by 2
Overview of the Benefits Deep Learning Can Provide Against Fake News, Cyberbullying and Hate Speech
Al-Dala’in, T., & Zhao, J. H. S. (2023) · Lecture notes in networks and systems · doi:10.1007/978-3-031-35308-6_2 · Cited by 2
Transformer Language Models Handle Word Frequency in Prediction Head
Kobayashi, G., Kuribayashi, T., Yokoi, S., & Inui, K. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.276 · Cited by 6
SAPPHIRE: Simple Aligner for Phrasal Paraphrase with Hierarchical Representation.
Yoshinaka, M., Kajiwara, T., & Arase, Y. (2020) · Language Resources and Evaluation · Cited by 2
Morishita, T., Morio, G., Ozaki, H., & Miyoshi, T. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.36 · Cited by 2
Applying Code Transform Model to Newly Generated Program for Improving Execution Performance
Chang, B. R., Tsai, H. F., & Su, P.-W. (2021) · Scientific Programming · doi:10.1155/2021/6691010 · Cited by 2
Representation and Pre-Activation of Lexical-Semantic Knowledge in Neural Language Models
Derby, S., Devereux, B. J., & Miller, P. C. (2021) · Workshop on Cognitive Modeling and Computational Linguistics (CMCL) · doi:10.18653/v1/2021.cmcl-1.25 · Cited by 5
Algorithmically Exploiting the Knowledge Accumulated in Textual Domains for Technical Support
Pau, D., Tarquini, I., Iannitelli, M., & Allegorico, C. (2021) · PHM Society European Conference · doi:10.36001/phme.2021.v6i1.2900 · Cited by 2
On the differences between BERT and MT encoder spaces and how to address them in translation tasks
Vázquez, R., Çelikkanat, H., Creutz, M., & Tiedemann, J. (2021) · doi:10.18653/v1/2021.acl-srw.35 · Cited by 2
Deriving Word Vectors from Contextualized Language Models using Topic-Aware Mention Selection
Wang, Y., Bouraoui, Z., Espinosa-Anke, L., & Schockaert, S. (2021) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2021.repl4nlp-1.19 · Cited by 3
Do pretrained transformers infer telicity like humans?
Zhao, Y., Ngui, J. G., Hartley, L. H., & Bethard, S. J. (2021) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2021.conll-1.6 · Cited by 3
Using Paraphrases to Study Properties of Contextual Embeddings
Burdick, L., Kummerfeld, J. K., & Mihalcea, R. F. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.338 · Cited by 4
A Weighted Combination of Semantic and Syntactic Word Image Representations
Tüselmann, O., Brandenbusch, K., Chen, M., & Fink, G. A. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21648-0_20 · Cited by 2
Generating Derivational Morphology with BERT.Preprint
Hofmann, V., Pierrehumbert, J. B., & Schütze, H. (2020) · arXiv (Cornell University) · Cited by 2
BERT Busters: Outlier LayerNorm Dimensions that Disrupt BERT.Preprint
Kovaleva, O., Kulshreshtha, S., Rogers, A., & Rumshisky, A. (2021) · arXiv (Cornell University) · Cited by 2
How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on IsotropyPreprint
Rajaee, S., & Pilehvar, M. T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04740 · Cited by 6
Obtaining Better Static Word Embeddings Using Contextual Embedding Models
Gupta, P., & Jaggi, M. (2021) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.5055754 · Cited by 2
Alhoshan, W., Ferrari, A., & Zhao, L. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5472929 · Cited by 2
Mikkelsen, Y. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6497074 · Cited by 2
Models of diachronic semantic change using word embeddings
Montariol, S. (2021) · theses.fr (ABES) · doi:10.70675/c08ef5d0z7732z4caeza42fz68ade712e3cd · Cited by 2
Evaluating Large Language Models: Challenges, Limitations, and Future Directions
Li, L.-H., & Nugroho, A. C. (2025) · Joint International Conference on Digital Arts, Media and Technology with ECTI Northern Section Conference on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI DAMT & NCON) · doi:10.1109/ectidamtncon64748.2025.10961969 · Cited by 1
GLU-PM: A Pretrained Model Toward Generalized Log Understanding
Liu, X., Zheng, S., Zou, H., Yu, H., & Gao, S. (2025) · IEEE Transactions on Reliability · doi:10.1109/tr.2025.3642425 · Cited by 1
Gómez-Moreno, J. M. U., & Ibáñez, F. J. R. D. M. (2022) · Terminology and lexicography research and practice · doi:10.1075/tlrp.23.17gom · Cited by 2
Sengupta, S., Ghosh, S., Nakov, P., & Mitra, P. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i13.27019 · Cited by 1
Meta-learning Pathologies from Radiology Reports using Variance Aware Prototypical Networks
Sehanobish, A., Kannan, K., Abraham, N., Das, A., & Odry, B. L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-industry.34 · Cited by 3
Metadata
- Authors
- Ethayarajh, K.
- Affiliations
- Stanford University (US)
- Corresponding author
- Ethayarajh, Kawin
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 55-65
- OpenAlex
- W2971569798
- Cited by
- 841
- References
- 27
- Citations by year
- 2026: 93 · 2025: 92 · 2024: 115 · 2023: 160 · 2022: 158 · 2021: 157 · 2020: 64
- FWCI
- 36.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- contextualized word representations, word embeddings, BERT, ELMo, GPT-2, embedding geometry, cosine similarity, static embeddings, layer-wise analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
