The DOI matches its Crossref record; 26 of the 27 references it cites were found in a registry or an index.
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
Open accessAbstract
Replacing static word embeddings with contextualized word representations has yielded significant improvements on many NLP tasks.However, just how contextual are the contextualized representations produced by models such as ELMo and BERT?Are there infinitely many context-specific representations for each word, or are words essentially assigned one of a finite number of word-sense representations?For one, we find that the contextualized representations of all words are not isotropic in any layer of the contextualizing model.While representations of the same word in different contexts still have a greater cosine similarity than those of two different words, this self-similarity is much lower in upper layers.This suggests that upper layers of contextualizing models produce more context-specific representations, much like how upper layers of LSTMs produce more task-specific representations.In all layers of ELMo, BERT, and GPT-2, on average, less than 5% of the variance in a word's contextualized representations can be explained by a static embedding for that word, providing some justification for the success of contextualized representations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Precedent exists for using PCA to extract network-level patterns from high-dimensional neural data, including fMRI (Smith et al., 2015), EEG (Blankertz et al., 2011), and even LLM embeddings (Ethayarajh, 2019).”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Ethayarajh, 2019
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Ethayarajh (2019) notes that unidirectional models produce embeddings that vary smoothly across tokens, implying that the difference operation can still capture meaningful transitions within a span.”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Ethayarajh, 2019
Experimental — accuracy not yet measuredIn the citing paper
“Previous literature showed the BERT embeddings54 capture rich semantic information and are widely recognized as effective for sentence similarity tasks55–57.”
From Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation (2025) · cites this paper as [57]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061
Linguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · Cited by 2,913
Neural Word Embedding as Implicit Matrix Factorization
Levy, O., & Goldberg, Y. (2014) · neural information processing systems · Cited by 1,581
Flagged first, then by citations
1 of the 27 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez- Agirre, and Weiwei Guo. 2013. Sem 2013 shared task: Semantic textual similarity, including a pilot on typed-similarity. In SEM 2013: The Second Joint Conference on Lexical and Computational Seman- tics. Association for Computational Linguistics.
- John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word represen- tations. In North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies. Association for Computational Linguistics.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013a. Distributed representa- tions of words and phrases and their compositional- ity. In Advances in Neural Information Processing Systems. pages 3111-3119.
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long Papers). pages 2227-2237.
- Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipan- jan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In Inter- national Conference on Learning Representations. https://openreview.net/forum?id=SJzSgnRcKX.
Cited by
Sorted by citations
Items 601–700 of 898
Representation of Lexical Stylistic Features in Language Models’ Embedding Space
Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.32 · Cited by 4
Podatki v utripu srca: podatkovna platforma za zajem biosenzoričnih podatkov v realnem času
Nered, U., & Jeraj, J. (2023) · OTS Sodobne informacijske tehnologije in storitve (OTS) · doi:10.18690/um.feri.8.2023.19 · Cited by 1
Berro, A., Benatallah, B., Gaci, Y., & Benabdeslem, K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-70341-6_20 · Cited by 1
Kovaleva, O. A., Samokhvalov, A. V., Liashkov, M., & Pchelintsev, S. Y. (2024) · Izvestiya of Saratov University Mathematics Mechanics Informatics · doi:10.18500/1816-9791-2024-24-3-442-451 · Cited by 1
EMCRL: EM-Enhanced Negative Sampling Strategy for Contrastive Representation Learning
Zhang, K., Lv, G., Wu, L., Hong, R., & Wang, M. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2024.3454056 · Cited by 1
Mahajan, Y., Bansal, N., Blanco, E., & Karmaker, S. (2024) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2024.findings-emnlp.436 · Cited by 1
Learning Dynamic Representations in Large Language Models for Evolving Data Streams
Srivastava, A., Bhatnagar, S., Murty, M. N., & Raman, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78169-8_16 · Cited by 1
Poyatos, J. F. (2026) · PNAS Nexus · doi:10.1093/pnasnexus/pgag223 · Cited by 1
Attimonelli, M., Bellis, A. D., Pomo, C., Jannach, D., Sciascio, E. D., & Noia, T. D. (2025) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3705328.3748040 · Cited by 3
GASLITEing the Retrieval: Exploring Vulnerabilities in Dense Embedding-based Search
Ben-Tov, M., & Sharif, M. (2025) · ACM SIGSAC Conference on Computer and Communications Security (CCS) · doi:10.1145/3719027.3765095 · Cited by 1
Efficient Violence Detection Using the CLIP Model
Bartaouch, K., & Jourani, R. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-07915-2_29 · Cited by 1
A Content- and Context-Aware Click Model Based on Dynamic Graph Neural Networks
Yang, L., Mao, J., Zhao, Z., & Yan, Q. (2026) · ACM Transactions on Information Systems · doi:10.1145/3795515 · Cited by 1
Knowledge-enhanced AI models for domain-specific application: a survey
Li, X.-Y., Wang, Y., Liang, J., Wang, X., Zhou, P., Zhao, Q., Zhang, Y., & Song, Q. (2026) · Science China Information Sciences · doi:10.1007/s11432-025-4635-0 · Cited by 1
The Evolution of the Theory and Practice of Artificial Intelligence
Стаднік, А. В., & Mykhaylyshyn, U. B. (2026) · KRPOCH eBooks · doi:10.26697/aids.2026.1 · Cited by 1
Khisa, A., Lia, N. J., Nafis, T. M., Masud, Z., Pial, T., Rayana, S., & Kabir, A. (2025) · Workshop on Bangla Language Processing (BLP) · doi:10.18653/v1/2025.banglalp-1.23 · Cited by 1
Document-level joint entity and relation extraction based on multi-task explicit interaction
Du, M., Zhao, Y., Jin, G., Huang, Z., & Cui, R. (2026) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-026-00624-7 · Cited by 1
Ercan, Ş., & Tanyıldızı, E. (2026) · Fırat Üniversitesi Mühendislik Bilimleri Dergisi · doi:10.35234/fumbd.1727903 · Cited by 1
Grenier, G., Charest, M., Esseiva, P., & Rossy, Q. (2026) · Forensic Science International · doi:10.1016/j.forsciint.2026.112958 · Cited by 1
Liu, Y., Jia, Z., Gao, X., Xu, K., & Xiong, Y. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792700 · Cited by 5
LLM4Log: A Systematic Review of Large Language Model-based Log Analysis
Ma, Z., Yang, J., & Chen, T.-H. P. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3842759 · Cited by 4
mEOL: Training-Free Instruction-Guided Multimodal Embedder for Vector Graphics and Image Retrieval
Kim, K. S., Baek, S.-E., Jung-Mok, L., & Oh, T.-H. (2026) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv61042.2026.00122 · Cited by 2
Density-Guided Response Optimization: Community-Grounded Alignment via Implicit Acceptance Signals
Gerard, P., & Volkova, S. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806449 · Cited by 1
Wang, J., Yang, C., Yong, J., Xu, X., Wang, H., Koshizuka, N., Fukushima, S., Shibasaki, R., & Jiang, R. (2026) · Transportation Research Part C Emerging Technologies · doi:10.1016/j.trc.2026.105862 · Cited by 2
Chu, M., Desikan, B. S., Nadler, E. O., Sardo, D. R. L., Darragh-Ford, E., & Guilbeault, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.492 · Cited by 1
Ma, X., Lu, Y., Lu, Y., & Jiang, M. (2022) · Current Bioinformatics · doi:10.2174/1574893617666220718104518 · Cited by 1
Regularization, Semi-supervision, and Supervision for a Plausible Attention-Based Explanation
Nguyen, D. H., Mallart, C., Gravier, G., & Sébillot, P. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-35320-8_20 · Cited by 1
Can Pretrained Language Models Derive Correct Semantics from Corrupt Subwords under Noise?
Li, X., Liu, M., & Gao, S. B. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.15 · Cited by 1
Attieh, J., Woubie, A., Vlassov, V., Flanagan, A., & Bäckström, T. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-41734-4_8 · Cited by 1
Liu, C., & Chersoni, E. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.28 · Cited by 1
Why Bother with Geometry? On the Relevance of Linear Decompositions of Transformer Embeddings
Mickus, T., & Vázquez, R. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.10 · Cited by 1
Hoeken, S., Alaçam, Ö., Fokkens, A. S., & Sommerauer, P. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.237 · Cited by 1
To Split or Not to Split: Composing Compounds in Contextual Vector Spaces
Jenkins, C., Miletić, F., & Walde, S. S. I. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.1002 · Cited by 1
Gajbhiye, A., Bouraoui, Z., Li, N., Chatterjee, U., Espinosa-Anke, L., & Schockaert, S. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.654 · Cited by 1
Fine-grained Conversational Decoding via Isotropic and Proximal Search
Yao, Y., Wu, H., Xu, Q., & Song, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.5 · Cited by 1
Prefix Data Augmentation for Contrastive Learning of Unsupervised Sentence Embedding
Wang, C., & Lv, S. (2024) · Applied Sciences · doi:10.3390/app14072880 · Cited by 1
Efficient Log Anomaly Detection Based on Dimension Reduction and Attention Aware TCN
Xu, Z., Shi, Y., Su, Z., Song, L., Zhang, J., Wang, X., & Li, H. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-2303-4_33 · Cited by 1
Text Matching Model Combining Ranking Information and Negative Example Smoothing Strategies
Cai, X., Dong, L., Huang, Y., & Chen, M. (2024) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2024.0150679 · Cited by 1
Reinforced Multi-teacher Knowledge Distillation for Unsupervised Sentence Representation
Wang, X., Jin, R., & Qi, S. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72350-6_22 · Cited by 1
A Survey on Deciphering of EEG Waves
Mahajan, G., Divija, L., Jeevan, R., Kumari, P. D., & Narayan, S. (2024) · Smart innovation, systems and technologies · doi:10.1007/978-981-97-2147-4_27 · Cited by 1
Cluster-Mined Negative Samples for Enhanced Unsupervised Sentence Representation Learning
Zhang, Y., Zhang, W., Yang, H., Wang, Z., Song, X., & Wu, X. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-78395-1_29 · Cited by 1
A Dynamic Character and Word Information Fusion Method for Chinese Named Entity Recognition
Luo, Z., & Wu, Z. (2024) · International Conference on Cloud Computing, Big Data Application and Software Engineering (CBASE) · doi:10.1109/cbase64041.2024.10824391 · Cited by 1
Continual Learning With Embedding Layer Surgery and Task-Wise Beam Search Using Whisper
Kwok, C. Y., Yip, J. Q., & Chng, E. S. (2024) · IEEE Spoken Language Technology Workshop (SLT) · doi:10.1109/slt61566.2024.10832213 · Cited by 1
REDIRE: Extreme REduction DImension for extRactivE Summarization
Rodrigues, C., Ortega, M., Bossard, A., & Mellouli, N. (2025) · Data & Knowledge Engineering · doi:10.1016/j.datak.2025.102407 · Cited by 1
Alshamrani, S. (2025) · Electronics · doi:10.3390/electronics14101919 · Cited by 1
Zhang, Y., Wang, H., Zheng, Y., Fei, Z., Zhou, H., & Luo, H. (2025) · Protection and Control of Modern Power Systems · doi:10.23919/pcmp.2024.000406 · Cited by 1
Sippola, V. (2020) · Scholarship@Western (Western University) · Cited by 1
CitiusNLP at SemEval-2020 Task 3: Comparing Two Approaches for Word Vector Contextualization
Gamallo, P. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.34 · Cited by 1
Zhang, W., Campbell, M. S., Yu, Y., & Kumaravel, S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17701 · Cited by 1
An Experimental Comparison of the Geometry of Models Trained on Natural Language and Synthetic Data
Sippola, V., & Mercer, R. E. (2021) · doi:10.21428/594757db.f47d55f8 · Cited by 1
Changing the Basis of Contextual Representations with Explicit Semantics
Ficsor, T., & Berend, G. (2021) · doi:10.18653/v1/2021.acl-srw.25 · Cited by 2
Montiel-Ponsoda, E. (2022) · Terminology and lexicography research and practice · doi:10.1075/tlrp.23.07mon · Cited by 1
Exploring Category Structure with Contextual Language Models and Lexical Semantic Networks
Renner, J., Denis, P., Gilleron, R., & Brunellière, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.167 · Cited by 1
Analyzing the Representational Geometry of Acoustic Word Embeddings
Abdullah, B., & Klakow, D. (2022) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2022.blackboxnlp-1.15 · Cited by 1
On Neurons Invariant to Sentence Structural Changes in Neural Machine Translation
Patel, G., Choshen, L., & Abend, O. (2022) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2022.conll-1.14 · Cited by 1
Dictionary-Assisted Supervised Contrastive Learning
Wu, P. Y., Bonneau, R., Tucker, J. A., & Nagler, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.696 · Cited by 1
To Word Senses and Beyond: Inducing Concepts with Contextualized Language Models
Liétard, B., Denis, P., & Mikaela Keller, A. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.156 · Cited by 1
A Comparative Study on Structural and Semantic Properties of Sentence EmbeddingsPreprint
Kalinowski, A., & An, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.11226 · Cited by 1
Training dynamics of neural language models
Saphra, N. (2021) · ERA · doi:10.7488/era/1421 · Cited by 1
Transformer with a Mixture of Gaussian Keys.Preprint
Nguyen, T., Nguyen, T. M., Le, D. D., Nguyen, K. C. D., Tran, A. N., Baraniuk, R. G., Ho, N., & Osher, S. (2021) · arXiv (Cornell University) · Cited by 1
Reprint: A Randomized Extrapolation Based on Principal Components for Data AugmentationPreprint
Wei, J., Chen, Q., Peng, P., Guedj, B., & Li, L. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4149531 · Cited by 1
Toward context-based text-to-3D scene generation
Henlein, A. (2022) · Publication Server of Goethe University Frankfurt am Main (Goethe University Frankfurt) · doi:10.21248/gups.73448 · Cited by 1
OTS 2023 Sodobne informacijske tehnologije in storitve: Zbornik 26. konference
Pavlič, L., Beranič, T., & Heričko, M. (2023) · doi:10.18690/um.feri.8.2023 · Cited by 1
Periti, F., Picascia, S., Montanelli, S., Ferrara, A., & Tahmasebi, N. (2023) · doi:10.36227/techrxiv.24210915.v1 · Cited by 1
Post-processing Techniques for Word Embedding
Albujasim, Z. M. (2024) · doi:10.22215/etd/2024-15911 · Cited by 1
Liu, X. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.4871946 · Cited by 2
From Transformers to the Future: An In-Depth Exploration of Modern Language Model ArchitecturesPreprint
Xu, H., Bi, Z., Tseng, H.-M., Song, X., & Feng, P. (2024) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/n8r5j · Cited by 1
Neural signatures of impaired semantic contextualization in Autism Spectrum DisorderPreprint
Franch, M. C., Katlowitz, K. A., Mickiewicz, E. A., Belanger, J. L., Mathura, R. K., Zhu, H., Yan, X., Ismail, T., Chavez, A. G., Chericoni, A., et al. (2026) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.64898/2026.03.16.712048 · Cited by 1
Understanding Word Embedding Stability Across Languages and Applications
Burdick, L. (2020) · Deep Blue (University of Michigan)
Metaphor generation through context sensitive distributional semantics
McGregor, S., Purver, M., & Wiggins, G. A. (2020) · Figurative thought and language · doi:10.1075/ftl.10.15mcg
Brendan, C., & Evans, J. A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.05928
Estimation of Summary-to-Text Inconsistency by Mismatched EmbeddingsPreprint
Vasilyev, O., & Bohannon, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.05156
Evaluating the Robustness of Question-Answering Models to Paraphrased Questions
von Geusau, P. A., & Bloem, P. (2021) · Communications in computer and information science · doi:10.1007/978-3-030-76640-5_1 · Cited by 1
Ranking and Retrieval under Semantic Relevance
Chen, T. (2020)
Monolingual Phrase Alignment Based on Word Embedding
Yoshinaka, M., Kajiwara, T., & Arase, Y. (2021) · Journal of Natural Language Processing · doi:10.5715/jnlp.28.508
Language Models as Zero-shot Visual Semantic LearnersPreprint
Jiao, Y., Hare, J. S., & Prügel‐Bennett, A. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.12021
Rare Words Degenerate All Words.Preprint
Yu, S., Song, J., Kim, H., Lee, S.-M., Ryu, W.-J., & Yoon, S. (2021) · arXiv (Cornell University)
On Isotropy Calibration of TransformersPreprint
Ding, Y., Martinkus, K., Pascual, D., Clematide, S., & Wattenhofer, R. P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.13304
Interactively Generating Explanations for Transformer Language Models.Preprint
Schramowski, P., Friedrich, F., Tauchmann, C., & Kersting, K. (2021) · arXiv (Cornell University)
Text analysis and deep learning: A network approachPreprint
Marquart, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.04151
Sparse Distillation: Speeding Up Text Classification by Using Bigger ModelsPreprint
Ye, Q., Khabsa, M., Lewis, M., Wang, S., Ren, X., & Jaech, A. (2021) · arXiv (Cornell University)
MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language ModelsPreprint
Liu, Q., Liu, F., Collier, N., Korhonen, A., & Vulić, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.09237
Distilling Word Meaning in Context from Pre-trained Language Models
Arase, Y., & Kajiwara, T. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.49
HOTTER: Hierarchical Optimal Topic Transport with Explanatory Context Representations
Wehnert, S., Scheel, C., Szakács, S., Nieländer, M., Mielke, P., & Luca, E. W. D. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.418
Using Distributional Principles for the Semantic Study of Contextual Language ModelsPreprint
Ferret, O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.12174
How Do Transformer-Architecture Models Address Polysemy of Korean Adverbial Postpositions?
Mun, S., & Desagulier, G. (2022) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2022.deelio-1.2
When Polysemy Matters: Modeling Semantic Categorization with Word Embeddings
Soper, E., & Koenig, J.-P. (2022) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2022.starsem-1.10 · Cited by 1
How Contextualized Word Embeddings Represent Word Senses
Tripodi, R. (2022) · Accademia University Press eBooks · doi:10.4000/books.aaccademia.10872 · Cited by 1
Evaluation of Word Embeddings for Toxic Span Prediction
Nelatoori, K. B., & Bindu, K. H. (2022) · Cognitive science and technology · doi:10.1007/978-981-19-2350-0_17
Inferring Image Background from Text Description
Chizhikov, D., Efimova, V., Shalamov, V., & Filchenkov, A. A. (2022) · Communications in computer and information science · doi:10.1007/978-3-031-23372-2_1
On generalization of the sense retrofitting model
Lee, Y., Yen, T.-Y., Huang, H., Shiue, Y.-T., & Chen, H. (2023) · Natural Language Engineering · doi:10.1017/s1351324922000523
Chen, J., & Sakti, S. (2023) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp49357.2023.10095119
High-Level Semantic Interpretation of the Russian Static Models Structure
Serikov, O., Ganeeva, V. A., Aksenova, A. A., & Klyshinskiy, E. S. (2023) · Vestnik NSU Series Linguistics and Intercultural Communication · doi:10.25205/1818-7935-2023-21-1-67-82
Can Pretrained Language Models Derive Correct Semantics from Corrupt Subwords under Noise?Preprint
Li, X., Liu, M., & Gao, S. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.15268
A Determinantal Point Process Based Novel Sampling Method of Abstractive Text Summarization
Shen, J., Xuan, J., & Liang, C. (2023) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn54540.2023.10191958
Analyzing Text Representations by Measuring Task Alignment
González-Gutiérrez, C., Primadhanty, A., Cazzaro, F., & Quattoni, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.7
Induction of Bilingual Dictionaries
Sharoff, S., Rapp, R., & Zweigenbaum, P. (2023) · Synthesis lectures on human language technologies · doi:10.1007/978-3-031-31384-4_5
Reconstructing Word Representations from Pre-trained Subword Embeddings
Lauren, P. (2022) · International Conference on Computational Science and Computational Intelligence (CSCI) · doi:10.1109/csci58124.2022.00013 · Cited by 1
Lucas, E. (2023) · Digital Commons - Michigan Tech (Michigan Technological University) · doi:10.37099/mtu.dc.etdr/1599
DEPRESSION DETECTION MODEL USINGWORD AND SENTENCE EMBEDDING WITH DIFFERENT CLASSIFIERS
Rai, K. A., Goyal, R., & Choudhary, M. (2023) · International Journal of Engineering Applied Sciences and Technology · doi:10.33564/ijeast.2023.v08i02.003
Metadata
- Authors
- Ethayarajh, K.
- Affiliations
- Stanford University (US)
- Corresponding author
- Ethayarajh, Kawin
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 55-65
- OpenAlex
- W2971569798
- Cited by
- 841
- References
- 27
- Citations by year
- 2026: 93 · 2025: 92 · 2024: 115 · 2023: 160 · 2022: 158 · 2021: 157 · 2020: 64
- FWCI
- 36.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- contextualized word representations, word embeddings, BERT, ELMo, GPT-2, embedding geometry, cosine similarity, static embeddings, layer-wise analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
