The DOI matches its Crossref record; 26 of the 27 references it cites were found in a registry or an index.
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
Open accessAbstract
Replacing static word embeddings with contextualized word representations has yielded significant improvements on many NLP tasks.However, just how contextual are the contextualized representations produced by models such as ELMo and BERT?Are there infinitely many context-specific representations for each word, or are words essentially assigned one of a finite number of word-sense representations?For one, we find that the contextualized representations of all words are not isotropic in any layer of the contextualizing model.While representations of the same word in different contexts still have a greater cosine similarity than those of two different words, this self-similarity is much lower in upper layers.This suggests that upper layers of contextualizing models produce more context-specific representations, much like how upper layers of LSTMs produce more task-specific representations.In all layers of ELMo, BERT, and GPT-2, on average, less than 5% of the variance in a word's contextualized representations can be explained by a static embedding for that word, providing some justification for the success of contextualized representations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Precedent exists for using PCA to extract network-level patterns from high-dimensional neural data, including fMRI (Smith et al., 2015), EEG (Blankertz et al., 2011), and even LLM embeddings (Ethayarajh, 2019).”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Ethayarajh, 2019
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Ethayarajh (2019) notes that unidirectional models produce embeddings that vary smoothly across tokens, implying that the difference operation can still capture meaningful transitions within a span.”
From Can "consciousness" be observed from large language model (LLM) internal states? Dissecting LLM representations obtained from Theory of Mind test with Integrated Information Theory and Span Representation analysis (2025) · cites this paper as Ethayarajh, 2019
Experimental — accuracy not yet measuredIn the citing paper
“Previous literature showed the BERT embeddings54 capture rich semantic information and are widely recognized as effective for sentence similarity tasks55–57.”
From Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation (2025) · cites this paper as [57]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
XLNet: Generalized Autoregressive Pretraining for Language Understanding
Yang, Z., Dai, Z., Yang, Y., Carbonell, J., Salakhutdinov, R., & Le, Q. V. (2019) · arXiv · doi:10.48550/arxiv.1906.08237 · Cited by 6,687
How transferable are features in deep neural networks?
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014) · arXiv · doi:10.48550/arxiv.1411.1792 · Cited by 6,061
Linguistic Regularities in Continuous Space Word Representations
Mikolov, T., Yih, W.-T., & Zweig, G. (2013) · Cited by 2,913
Neural Word Embedding as Implicit Matrix Factorization
Levy, O., & Goldberg, Y. (2014) · neural information processing systems · Cited by 1,581
Flagged first, then by citations
1 of the 27 references couldn’t be matched to a known paper.
Research works printed but not linked: 5
- Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez- Agirre, and Weiwei Guo. 2013. Sem 2013 shared task: Semantic textual similarity, including a pilot on typed-similarity. In SEM 2013: The Second Joint Conference on Lexical and Computational Seman- tics. Association for Computational Linguistics.
- John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word represen- tations. In North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies. Association for Computational Linguistics.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013a. Distributed representa- tions of words and phrases and their compositional- ity. In Advances in Neural Information Processing Systems. pages 3111-3119.
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume 1 (Long Papers). pages 2227-2237.
- Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipan- jan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In Inter- national Conference on Learning Representations. https://openreview.net/forum?id=SJzSgnRcKX.
Cited by
Sorted by citations
Items 301–400 of 898
Yao, L. (2024) · IEEE Access · doi:10.1109/access.2024.3428829 · Cited by 9
Extracting Semantic Topics About Development in Africa From Social Media
Sibitenda, H., Diattara, A., Traore, A., Hu, R., Zhang, D., Rundensteiner, E. A., & Bâ, C. T. (2024) · IEEE Access · doi:10.1109/access.2024.3466834 · Cited by 9
Liu, F., Jung, J., Feinstein, W. P., D'Ambrogia, J., & Jung, G. (2024) · International Conference on AI-ML Systems (AIMLSystems) · doi:10.1145/3703412.3703434 · Cited by 10
Evaluation of taxonomic and neural embedding methods for calculating semantic similarity
Yang, D., & Yin, Y. (2021) · Natural Language Engineering · doi:10.1017/s1351324921000279 · Cited by 9
Cognitive approaches to the study of term variation
Fernández, S. (2022) · Terminology and lexicography research and practice · doi:10.1075/tlrp.23.20fer · Cited by 10
Creating a stop word dictionary in Serbian
Marovac, U., Avdić, A. M., & Ljajić, A. (2021) · Scientific Publications of the State University of Novi Pazar Series A Applied Mathematics Informatics and mechanics · doi:10.5937/spsunp2101017m · Cited by 9
Cross-Modal Contrastive Learning for Code Search
Shi, Z., Xiong, Y., Zhang, X., Zhang, Y., Li, S., & Zhu, Y. (2022) · IEEE International Conference on Software Maintenance (ICSM) · doi:10.1109/icsme55016.2022.00017 · Cited by 11
Zhang, T., Meng, J., Yang, Y., & Yu, S. (2024) · Applied Sciences · doi:10.3390/app14072817 · Cited by 10
Just-In-Time TODO-Missed Commits Detection
Wang, H., Gao, Z., Hu, X., Lo, D., Grundy, J., & Wang, X. (2024) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2024.3405005 · Cited by 9
AUTH @ CLSciSumm 20, LaySumm 20, LongSumm 20
Gidiotis, A., Stefanidis, S. P., & Tsoumakas, G. (2020) · Workshop on Scholarly Document Processing (SDP) · doi:10.18653/v1/2020.sdp-1.28 · Cited by 10
Of Non-Linearity and Commutativity in BERT
Zhao, S., Pascual, D., Brunner, G., & Wattenhofer, R. P. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9533563 · Cited by 12
Explore Better Relative Position Embeddings from Encoding Perspective for Transformer Models
Qu, A., Niu, J., & Mo, S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.237 · Cited by 9
A Neighborhood Framework for Resource-Lean Content Flagging
Sarwar, S. M., Zlatkova, D., Hardalov, M., Dinkov, Y., Augenstein, I., & Nakov, P. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00472 · Cited by 9
Is anisotropy really the cause of BERT embeddings not being semantic?
Baggetto, A. F., & Fresno, V. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.314 · Cited by 19
A Psycholinguistic Analysis of BERT’s Representations of Compounds
Buijtelaar, L., & Pezzelle, S. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.163 · Cited by 11
Contrastive Visual Semantic Pretraining Magnifies the Semantics of Natural Language Representations
Wolfe, R., & Caliskan, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.217 · Cited by 11
Wang, Y., Wang, Z., Sadeh, G., Zancato, L., Achille, A., Karypis, G., & Rangwala, H. (2024) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2024.10.29.620988 · Cited by 12
Distilled and Contextualized Neural Models Benchmarked for Vulnerable Function Detection
Lin, G., Jia, H., & Wu, D. (2022) · Mathematics · doi:10.3390/math10234482 · Cited by 8
Avornicului, M.-C., Breşfelean, V. P., Popa, S.-C., Forman, N., & Comes, C.-A. (2024) · Electronics · doi:10.3390/electronics13244938 · Cited by 8
Analyzing the Performance of Sentiment Analysis Using BERT, DistilBERT, RoBERTa and GPT-2
Imtiaz, S., & Hashi, E. K. (2024) · International Conference on Sustainable Technologies for Industry 5.0 (STI) · doi:10.1109/sti64222.2024.10951049 · Cited by 8
Optimization techniques for sentiment analysis based on LLM (GPT-3)
Zhan, T., Shi, C., Shi, Y., Li, H., & Lin, Y. (2024) · Applied and Computational Engineering · doi:10.54254/2755-2721/77/2024ma0060 · Cited by 8
Boutyline, A., & Johnston, E. (2026) · Sociological Methods & Research · doi:10.1177/00491241261467930 · Cited by 8
Evaluation of Vector Transformations for Russian Word2Vec and FastText Embeddings
Korogodina, O., Karpik, O. V., & Klyshinsky, E. S. (2020) · Proceedings of the 30th International Conference on Computer Graphics and Machine Vision (GraphiCon 2020). Part 2 · doi:10.51130/graphicon-2020-2-3-18 · Cited by 8
An Automated Approach for the Prediction of the Severity Level of Bug Reports Using GPT-2
kamal, M., Ali, S., Nasir, A., Samad, A., Baseer, S., & Irshad, A. (2022) · Security and Communication Networks · doi:10.1155/2022/2892401 · Cited by 8
RetractedOn Isotropy Calibration of Transformer Models
Ding, Y., Martinkus, K., Pascual, D., Clematide, S., & Wattenhofer, R. P. (2022) · Workshop on Insights from Negative Results in NLP (Insights) · doi:10.18653/v1/2022.insights-1.1 · Cited by 10
Derin öğrenme temelli hibrid altın endeksi (XAU/USD) yön tahmin modeli
KANTAR, O., & Kilimci, Z. H. (2022) · Gazi Üniversitesi Mühendislik-Mimarlık Fakültesi Dergisi · doi:10.17341/gazimmfd.888456 · Cited by 8
Discovering Differences in the Representation of People using Contextualized Semantic Axes
Lucy, L., Tadimeti, D., & Bamman, D. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.228 · Cited by 8
Rosen, Z. P., & Dale, R. A. C. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02267-2 · Cited by 15
A Comparative Study of Deep Learning Models for Detecting Depressive Disorder in Tweets
(2024) · Journal of System and Management Sciences · doi:10.33168/jsms.2024.0318 · Cited by 9
NoTeeline: Supporting Real-Time, Personalized Notetaking with LLM-Enhanced Micronotes
Huq, F., Samee, A., Lin, D. C.-E., Tang, A. X., & Bigham, J. P. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712086 · Cited by 11
Multifocal region-assisted cross-modality learning for chest X-ray report generation
Lian, J., Dong, Z., Zhang, H., Chen, Y., & Liu, J. (2024) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2024.109187 · Cited by 8
LLM-Therapist: A RAG-Based Multimodal Behavioral Therapist as Healthcare Assistant
Shafi, F. R., & Hossain, M. A. (2024) · Globecom · doi:10.1109/globecom52923.2024.10901139 · Cited by 8
CL-IMS @ DIACR-Ita: Volente o Nolente: BERT does not Outperform SGNS on Semantic Change Detection
Laicher, S., Baldissin, G., Castañeda, E., Schlechtweg, D., & Walde, S. S. I. (2020) · Accademia University Press eBooks · doi:10.4000/books.aaccademia.7650 · Cited by 8
Yadav, V., Bethard, S. J., & Surdeanu, M. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.363 · Cited by 9
NCUEE-NLP at SemEval-2022 Task 11: Chinese Named Entity Recognition Using the BERT-BiLSTM-CRF Model
Lee, L., Lu, C.-H., & Lin, T.-M. (2022) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2022.semeval-1.220 · Cited by 8
COVID-19: a disruptive impact on the knowledge support of references
Zhang, Y., Li, H., Mao, J., He, G., Yang, Y., Jiang, Z., & Duan, Y. (2023) · Scientometrics · doi:10.1007/s11192-023-04764-9 · Cited by 8
Transparency Helps Reveal When Language Models Learn Meaning
Wu, Z., Merrill, W., Peng, H. M., Beltagy, I., & Smith, N. A. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00565 · Cited by 8
Aracena, C., Villena, F., Rojas, M. S., & Dunstan, J. (2022) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2022.louhi-1.22 · Cited by 8
Text Augmentation Using Dataset Reconstruction for Low-Resource Classification
Rahamim, A., Uziel, G., Goldbraich, E., & Tavor, A. A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.466 · Cited by 9
Kernel-Whitening: Overcome Dataset Bias with Isotropic Sentence Embedding
Gao, S., Dou, S., Zhang, Q., & Huang, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.275 · Cited by 8
Learning to Perturb for Contrastive Learning of Unsupervised Sentence Representations
Zhou, K., Zhou, Y., Zhao, W. X., & Wen, J.-R. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3304485 · Cited by 8
Wang, T., & Xu, X. (2023) · Behavior Research Methods · doi:10.3758/s13428-023-02274-3 · Cited by 8
MEGClass: Extremely Weakly Supervised Text Classification via Mutually-Enhancing Text Granularities
Kargupta, P., Komarlu, T., Yoon, S., Wang, X., & Han, J. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.708 · Cited by 8
Saturation of moral language predicts lower content engagement on social media
Candia, C., Atari, M., Kteily, N., & Uzzi, B. (2026) · Nature Human Behaviour · doi:10.1038/s41562-026-02560-y · Cited by 8
OoMMix: Out-of-manifold Regularization in Contextual Embedding Space for Text Classification
Lee, S., Lee, D., & Yu, H. (2021) · doi:10.18653/v1/2021.acl-long.49 · Cited by 8
Cat2Type: Wikipedia Category Embeddings for Entity Typing in Knowledge Graphs
Biswas, R., Sofronova, R., Sack, H., & Alam, M. (2021) · International Conference on Knowledge Capture · doi:10.1145/3460210.3493575 · Cited by 8
BERT as a Teacher: Contextual Embeddings for Sequence-Level RewardPreprint
Schmidt, F., & Hofmann, T. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.02738 · Cited by 9
Effect of Post-processing on Contextualized Word RepresentationsPreprint
Sajjad, H., Alam, F., Dalvi, F., & Durrani, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07456 · Cited by 9
Regional bias in monolingual English language models
Lyu, J., Dost, K., Koh, Y. S., & Wicker, J. S. (2024) · Machine Learning · doi:10.1007/s10994-024-06555-6 · Cited by 8
S-NLP at SemEval-2021 Task 5: An Analysis of Dual Networks for Sequence Tagging
Nguyen, V. A., Nguyen, T., Dao, H. Q., & Pham, Q. H. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.120 · Cited by 7
Artene, C.-G., Oprişa, C., Buţincu, C. N., & Leon, F. (2023) · Mathematics · doi:10.3390/math11092053 · Cited by 8
ContraCLM: Contrastive Learning For Causal Language Model
Jain, N., Zhang, D., Ahmad, W. U., Wang, Z., Nan, F., Li, X., Tan, M., Nallapati, R. M., Ray, B., Bhatia, P., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.355 · Cited by 13
Lost in Context? On the Sense-Wise Variance of Contextualized Word Embeddings
Wang, Y., & Zhang, Y. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3337643 · Cited by 9
DenoSent: A Denoising Objective for Self-Supervised Sentence Representation Learning
Wang, X., He, J., Wang, P., Zhou, Y., Sun, T., & Qiu, X. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29886 · Cited by 8
Leveraging natural language processing models to automate speech-intelligibility scoring
Herrmann, B. (2024) · Speech Language and Hearing · doi:10.1080/2050571x.2024.2374160 · Cited by 7
Transfer Learning in Natural Language Processing
Joseph, S., Parthi, A. G., Maruthavanan, D., Jayaram, V., Veerapaneni, P. K., & Parlapalli, V. (2024) · International Conference on Information and Communications Technology (ICOIACT) · doi:10.1109/icoiact64819.2024.10912895 · Cited by 11
Large Language Model can Reduce the Necessity of Using Large Data Samples for Training Models
Rahman, M. A., Francia, G., Shahriar, H., & Ahamed, S. I. (2025) · IEEE Conference on Artificial Intelligence (CAI) · doi:10.1109/cai64502.2025.00173 · Cited by 7
Fine-Tuning Language Models For Semi-Supervised Text Mining
Chen, X., Beaver, I., & Freeman, C. (2020) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata50022.2020.9377810 · Cited by 8
Semantic key generation based on natural language
Wu, Z., Kang, J., & Jiang, Q. (2021) · International Journal of Intelligent Systems · doi:10.1002/int.22711 · Cited by 7
Razumovskaia, E., Vulić, I., & Korhonen, A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.160 · Cited by 7
On the Effect of Isotropy on VAE Representations of Text
Zhang, L., Buntine, W., & Shareghi, E. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.78 · Cited by 7
Wang, J., Li, H., Du, X., Hung, J.-L., & Yang, S. (2023) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2023.101594 · Cited by 7
Hybrid Contrastive Constraints for Multi-Scenario Ad Ranking
Mu, S., Wei, P., Zhao, W. X., Liu, S., Wang, L., & Zheng, B. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3614920 · Cited by 7
Wu, D., & Monz, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.605 · Cited by 8
Unsupervised Word Polysemy Quantification with Multiresolution Grids of Contextual Embeddings
Xypolopoulos, C., Tixier, A. J. ‐., & Vazirgiannis, M. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.297 · Cited by 7
Low Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representations
Bihani, G., & Rayz, J. T. (2021) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2021.deelio-1.9 · Cited by 12
Theoretical perspectives on Terminology
Faber, P., & L’Homme, M.-C. (2022) · Terminology and lexicography research and practice · doi:10.1075/tlrp.23.int · Cited by 7
Optimizing text representations to capture (dis)similarity between political parties
Ceron, T., Blokker, N., & Padó, S. (2022) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2022.conll-1.22 · Cited by 7
Liu, E., & Neubig, G. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.617 · Cited by 7
Probing Cross-Lingual Lexical Knowledge from Multilingual Sentence Encoders
Vulić, I., Glavaš, G., Liu, F., Collier, N., Ponti, E. M., & Korhonen, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.153 · Cited by 9
Samuel, J., Palle, R., & Soares, E. C. (2022) · Journal of Big Data and Artificial Intelligence · doi:10.54116/jbdtp.v1i1.20 · Cited by 7
Intelligent Network Threat Detection Engine Based on Open Source GPT-2 Model
Zhang, X., Chen, T., Wu, J., & Yu, Q. (2023) · International Conference on Computer Science and Automation Technology (CSAT) · doi:10.1109/csat61646.2023.00107 · Cited by 6
Dynamic Boundary Time Warping for sub-sequence matching with few examples
Borchmann, Ł., Jurkiewicz, D., Graliński, F., & Górecki, T. (2020) · Expert Systems with Applications · doi:10.1016/j.eswa.2020.114344 · Cited by 7
Nuopponen, A. (2022) · Terminology and lexicography research and practice · doi:10.1075/tlrp.23.03nuo · Cited by 6
Kunz, J. (2024) · Linköping studies in science and technology. Dissertations · doi:10.3384/9789180754712 · Cited by 7
Diversified realistic face image generation GAN for human subjects in multimedia content creation
Kumar, L., & Singh, D. K. (2024) · Computer Animation and Virtual Worlds · doi:10.1002/cav.2232 · Cited by 6
Neural models for semantic analysis of handwritten document images
Tüselmann, O., & Fink, G. A. (2024) · International Journal on Document Analysis and Recognition (IJDAR) · doi:10.1007/s10032-024-00477-8 · Cited by 6
Word Closure-Based Metamorphic Testing for Machine Translation
Xie, X., Jin, S., Chen, S., & Cheung, S.-C. (2024) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3675396 · Cited by 6
EU trade agreements and external differentiation: a large language model approach
Dür, A. (2024) · West European Politics · doi:10.1080/01402382.2024.2425549 · Cited by 6
How Much Semantic Information is Available in Large Language Model Tokens?
Haslett, D. A., & Cai, Z. G. (2025) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00747 · Cited by 8
A Garbage Detection and Classification Model for Orchards Based on Lightweight YOLOv7
Tian, X., Bai, L., & Mo, D. (2025) · Sustainability · doi:10.3390/su17093922 · Cited by 6
Marjanen, J., Kurunmäki, J., Pivovarova, L., & Zosa, E. (2020) · Journal of Data Mining & Digital Humanities · doi:10.46298/jdmdh.6159 · Cited by 6
Diki-Kidiri, M. (2022) · Terminology and lexicography research and practice · doi:10.1075/tlrp.23.09dik · Cited by 6
León-Araúz, P. (2022) · Terminology and lexicography research and practice · doi:10.1075/tlrp.23.22leo · Cited by 8
General principles of Wüster’s General Theory of Terminology
Candel, D. (2022) · Terminology and lexicography research and practice · doi:10.1075/tlrp.23.02can · Cited by 6
Learning and Evaluating Character Representations in Novels
Inoue, N., Pethe, C., Kim, A., & Skiena, S. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.81 · Cited by 9
UAlberta at LSCDiscovery: Lexical Semantic Change Detection via Word Sense Disambiguation
Teodorescu, D., von der Ohe, S., & Kondrak, G. (2022) · Workshop on Computational Approaches to Historical Language Change (LChange) · doi:10.18653/v1/2022.lchange-1.19 · Cited by 6
RSVN: A RoBERTa Sentence Vector Normalization Scheme for Short Texts to Extract Semantic Information
Gao, L., Zhang, L., Zhang, L., & Huang, J. (2022) · Applied Sciences · doi:10.3390/app122111278 · Cited by 6
Filtering Relevant Comments in Social Media Using Deep Learning
Ramamonjisoa, D., Ikuma, H., & Murakami, R. (2022) · IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT) · doi:10.1109/wi-iat55865.2022.00056 · Cited by 6
HyPe: Better Pre-trained Language Model Fine-tuning with Hidden Representation Perturbation
Yuan, H., Yuan, Z., Tan, C., Huang, F., & Huang, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.182 · Cited by 9
IsoVec: Controlling the Relative Isomorphism of Word Embedding Spaces
Marchisio, K., Verma, N., Duh, K., & Koehn, P. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.404 · Cited by 6
Arici, N., Gerevini, A., Olivato, M., Putelli, L., Sigalini, L., & Serina, I. (2023) · Future Internet · doi:10.3390/fi15080268 · Cited by 7
Investigating the Frequency Distortion of Word Embeddings and Its Impact on Bias Metrics
Valentini, F., Sosa, J., Slezak, D. F., & Altszyler, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.9 · Cited by 6
Ditto: A Simple and Efficient Approach to Improve Sentence Embeddings
Chen, Q., Wang, W., Zhang, Q., Zheng, S., Deng, C., Yu, H., Liu, J., Ma, Y., & Zhang, C. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.359 · Cited by 8
Am I Hurt?: Evaluating Psychological Pain Detection in Hindi Text Using Transformer-based Models
Kaur, R., Bhatia, M. P. S., & Kumar, A. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3650206 · Cited by 6
Systematic Evaluation of Different Approaches on Embedding Search
Aperdannier, R., Koeppel, M., Unger, T., Schacht, S., & Barkur, S. K. (2024) · Lecture notes in networks and systems · doi:10.1007/978-3-031-53963-3_36 · Cited by 8
Identifying the Limits of Cross-Domain Knowledge Transfer for Pretrained Models
Wu, Z., Liu, N. F., & Potts, C. (2022) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2022.repl4nlp-1.11 · Cited by 6
Li, Y., Xu, C., Hu, H., Sha, L., Zhang, Y., & Jiang, D. (2022) · Interspeech (USB) · doi:10.21437/interspeech.2022-10656 · Cited by 8
Founding principles of Socioterminology
Delavigne, V., & Gaudin, F. (2022) · Terminology and lexicography research and practice · doi:10.1075/tlrp.23.08del · Cited by 7
Misogyny Speech Detection Using Long Short-Term Memory and BERT Embeddings
Angeline, R. S., Nurjanah, D., & Nurrahmi, H. (2022) · International Conference on Information and Communications Technology (ICOIACT) · doi:10.1109/icoiact55506.2022.9972171 · Cited by 6
Metadata
- Authors
- Ethayarajh, K.
- Affiliations
- Stanford University (US)
- Corresponding author
- Ethayarajh, Kawin
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 55-65
- OpenAlex
- W2971569798
- Cited by
- 841
- References
- 27
- Citations by year
- 2026: 93 · 2025: 92 · 2024: 115 · 2023: 160 · 2022: 158 · 2021: 157 · 2020: 64
- FWCI
- 36.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Natural Sciences and Engineering Research Council of Canada
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- contextualized word representations, word embeddings, BERT, ELMo, GPT-2, embedding geometry, cosine similarity, static embeddings, layer-wise analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
