SemGloVe: Semantic Co-Occurrences for GloVe From BERT
Open accessAbstract
GloVe learns word embeddings by leveraging statistical information from word co-occurrence matrices. However, word pairs in the matrices are extracted from a predefined local context window, which might lead to limited word pairs and potentially semantic irrelevant word pairs. In this paper, we proposeSemGloVe, which distillssemantic co-occurrencesfrom BERT into static GloVe word embeddings. Particularly, we propose two models to extract co-occurrence statistics based on either the masked language model or the multi-head attention weights of BERT. Our methods can extract word pairs limited by the local window assumption, and can define the co-occurrence weights by directly considering the semantic distance between word pairs. Experiments on several word similarity datasets and external tasks show that SemGloVe can outperform GloVe.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
GloVe utilizes word co-occurrence data or global statistics to infer semantic relationships between words in the corpus, as opposed to word2vec's dependence on local context windows.
“The GloVe algorithm utilizes word co-occurrence data or global statistics to infer semantic relationships between words in the corpus, as opposed to word2vec's dependence on local context windows for deriving these associations (Badri et al. 2022; Gan et al. 2022; Curto et al. 2022).”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Gan et al. 2022
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · see citing works
(2008) · Journal of Machine Learning Research · Cited by 33,051 · see citing works
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · see citing works
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
Distilling the Knowledge in a Neural Network
Hinton, G., Vinyals, O., & Dean, J. (2015) · arXiv · doi:10.48550/arxiv.1503.02531 · Cited by 14,204 · see citing works
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Enriching Word Vectors with Subword Information
Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2017) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00051 · Cited by 9,645 · see citing works
(2010) · Cited by 8,261 · see citing works
Natural Language Processing (almost) from ScratchPreprint
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. P. (2011) · arXiv (Cornell University) · doi:10.48550/arxiv.1103.0398 · Cited by 6,882 · see citing works
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · see citing works
Harris, Z. S. (1954) · <i>WORD</i> · doi:10.1080/00437956.1954.11659520 · Cited by 4,807 · see citing works
A Neural Probabilistic Language Model
Bengio, Y., Ducharme, R., Vincent, P., Jauvin, C., Jaz, K., Hofmann, T., Poggio, T., & Shawe-Taylor, J. S. (2000) · Cited by 3,451 · see citing works
A neural probabilistic language model
BengioYoshua, DucharmeRéjean, VincentPascal, & JanvinChristian (2003) · Journal of Machine Learning Research · doi:10.5555/944919.944966 · Cited by 2,646 · see citing works
TinyBERT: Distilling BERT for natural language understanding
Jiao, X., Yin, Y., Shang, L., Jiang, X., Chen, X., Li, L., Wang, F., & Liu, Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.372 · Cited by 1,786 · see citing works
Dependency-Based Word Embeddings
Levy, O., & Goldberg, Y. (2014) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.3115/v1/p14-2050 · Cited by 1,091 · see citing works
Placing search in context: the concept revisited.
Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G., & Ruppin, E. (2002) · Cited by 939 · see citing works
Ethayarajh, K. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1006 · Cited by 841 · see citing works
Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., & Weischedel, R. (2006) · Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers on XX - NAACL '06 · doi:10.3115/1614049.1614064 · Cited by 806 · see citing works
Retrofitting Word Vectors to Semantic Lexicons
Faruqui, M., Dodge, J., Jauhar, S. K., Dyer, C., Hovy, E. H., & Smith, N. A. (2015) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.3115/v1/n15-1184 · Cited by 785 · see citing works
Patient Knowledge Distillation for BERT Model Compression
Sun, S., Cheng, Y., Gan, Z., & Liu, J. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1441 · Cited by 758 · see citing works
BERT-ATTACK: Adversarial Attack Against BERT Using BERT
Li, L., Ma, R., Guo, Q., Xue, X., & Qiu, X. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.500 · Cited by 623 · see citing works
(2019) · doi:10.18653/v1/n19-1419 · Cited by 601 · see citing works
Dissecting Contextual Word Embeddings: Architecture and Representation
Peters, M., Neumann, M., Zettlemoyer, L., & Yih, W.-T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1179 · Cited by 434 · see citing works
Evgeniy, L. F., Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., & Wolfman, G. (2002) · ACM Transactions on Information Systems · doi:10.1145/503104.503110 · Cited by 426 · see citing works
Simple and Effective Multi-Paragraph Reading Comprehension
Clark, C., & Gardner, M. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1078 · Cited by 402 · see citing works
CCGbank: A Corpus of CCG Derivations and Dependency Structures Extracted from the Penn Treebank
Hockenmaier, J., & Steedman, M. J. (2007) · Computational Linguistics · doi:10.1162/coli.2007.33.3.355 · Cited by 389 · see citing works
Higher-Order Coreference Resolution with Coarse-to-Fine Inference
Lee, K., He, L., & Zettlemoyer, L. (2018) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n18-2108 · Cited by 389 · see citing works
Assessing BERT's Syntactic Abilities
Goldberg, Y. (2019) · arXiv · doi:10.48550/arxiv.1901.05287 · Cited by 385 · see citing works
Counter-fitting Word Vectors to Linguistic Constraints
Mrkšić, N., Séaghdha, D. Ó., Thomson, B., Gašić, M., Rojas-Barahona, L. M., Su, P.-H., Vandyke, D., Wen, T.-H., & Young, S. J. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1018 · Cited by 356 · see citing works
Distilling Task-Specific Knowledge from BERT into Simple Neural NetworksPreprint
Tang, R., Lu, Y., Liu, L., Mou, L., Olga Vechtomova, A., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1903.12136 · Cited by 310 · see citing works
Sentence-State LSTM for Text Representation
Zhang, Y., Liu, Q., & Song, L. (2018) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p18-1030 · Cited by 250 · see citing works
Self-citationAutomated Concatenation of Embeddings for Structured Prediction
Wang, X., Jiang, Y., Bách, N., Wang, T., Huang, Z., Huang, F., & Tu, K. (2021) · doi:10.18653/v1/2021.acl-long.206 · Cited by 178 · see citing works
Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings
Bommasani, R., Davis, K., & Cardie, C. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.431 · Cited by 147 · see citing works
Specializing Word Embeddings for Similarity or Relatedness
Kiela, D., Hill, F., & Clark, S. C. (2015) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d15-1242 · Cited by 139 · see citing works
Distilling Knowledge Learned in BERT for Text Generation
Chen, Y.-C., Gan, Z., Yu, C., Liu, J., & Liu, J. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.705 · Cited by 131 · see citing works
A Geometric Understanding of Deep Learning
Lei, N., An, D., Guo, Y., Su, K., Liu, S., Luo, Z., Yau, S., & Gu, X. D. (2020) · Engineering · doi:10.1016/j.eng.2019.09.010 · Cited by 122 · see citing works
Vashishth, S., Bhandari, M., Yadav, P., Rai, P., Bhattacharyya, C., & Talukdar, P. P. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1320 · Cited by 105 · see citing works
Lewis, M., Lee, K., & Zettlemoyer, L. (2016) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n16-1026 · Cited by 70 · see citing works
Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration
Wang, S., Thompson, L., & Iyyer, M. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.846 · Cited by 64 · see citing works
Specializing Word Embeddings (for Parsing) by Information Bottleneck
Li, X. L., & Eisner, J. M. (2019) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d19-1276 · Cited by 62 · see citing works
Triggerless Backdoor Attack for NLP Tasks with Clean Labels
Gan, L., Li, J., Zhang, T., Li, X., Meng, Y., Wu, F. Q., Yang, Y., Guo, S., & Fan, C. C. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.214 · Cited by 45 · see citing works
Self-citationThe Next Breakthroughs of Artificial Intelligence: The Interdisciplinary Nature of AI
Zhuang, Y., Cai, M., Li, X., Luo, X., Yang, Q., & Wu, F. Q. (2020) · Engineering · doi:10.1016/j.eng.2020.01.009 · Cited by 42 · see citing works
Self-citationMultiple Knowledge Representation of Artificial Intelligence
Pan, Y. (2019) · Engineering · doi:10.1016/j.eng.2019.12.011 · Cited by 37 · see citing works
Jointly learning word embeddings using a corpus and a knowledge base
Alsuhaibani, M., Bollegala, D., Maehara, T., & Kawarabayashi, K. (2018) · PLoS ONE · doi:10.1371/journal.pone.0193094 · Cited by 35 · see citing works
Combining BERT with Static Word Embeddings for Categorizing Social Media
Alghanmi, I., Espinosa-Anke, L., & Schockaert, S. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.5 · Cited by 34 · see citing works
Obtaining Better Static Word Embeddings Using Contextual Embedding Models
Gupta, P., & Jaggi, M. (2021) · doi:10.18653/v1/2021.acl-long.408 · Cited by 33 · see citing works
Improving Skip-Gram Embeddings Using BERT
Wang, Y., Cui, L., & Zhang, Y. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3065201 · Cited by 32 · see citing works
Self-citationUnderstanding Commonsense Inference Aptitude of Deep Contextual Representations
Da, J., & Kasai, J. (2019) · Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing · doi:10.18653/v1/d19-6001 · Cited by 22 · see citing works
Dissecting Contextual Word Embeddings: Architecture and RepresentationPreprint
Peters, M. E., Neumann, M., Zettlemoyer, L., & Yih, W.-T. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.08949 · Cited by 13 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805
Flagged first, then by citations
6 of the 60 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Large language models (LLMs): survey, technical frameworks, and future challenges
Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395 · see citing works
Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AI
Yao, J., Zhang, S., Yao, Y., Wang, F., Ma, J., Zhang, J., Chu, Y., Ji, L., Jia, K., Shen, T., et al. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3178211 · Cited by 156 · see citing works
Self-citationZero-Shot Compound Fault Diagnosis Method Based on Semantic Learning and Discriminative Features
Xu, J., Zhang, H., Zhou, L., & Fan, Y. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3280503 · Cited by 48 · see citing works
Triggerless Backdoor Attack for NLP Tasks with Clean Labels
Gan, L., Li, J., Zhang, T., Li, X., Meng, Y., Wu, F. Q., Yang, Y., Guo, S., & Fan, C. C. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.214 · Cited by 45 · see citing works
Self-citationDai, T., Zhao, J., Li, D., Tian, S., Zhao, X., & Pan, S. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118841 · Cited by 25 · see citing works
A Comprehensive Survey on Multi-modal Conversational Emotion Recognition with Deep Learning
Shou, Y.-T., Meng, T., Ai, W., Fu, F., Yin, N., & Li, K.-Q. (2023) · ACM Trans. Inf. Syst. · doi:10.1145/3786343 · Cited by 73 · see citing works
Generative Zero-Shot Compound Fault Diagnosis Based on Semantic Alignment
Xu, J., Kong, H., Li, K., & Ding, X. (2023) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2023.3345912 · Cited by 20 · see citing works
Overlapping word removal is all you need: revisiting data imbalance in hope speech detection
LekshmiAmmal, H. R., Ravikiran, M., Nisha, G., Balamuralidhar, N., Madhusoodanan, A., Kumar, M. A., & Chakravarthi, B. R. (2023) · Journal of Experimental & Theoretical Artificial Intelligence · doi:10.1080/0952813x.2023.2166130 · Cited by 9 · see citing works
Liu, X., Dong, R., & Yang, H. (2024) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2024.3453919 · Cited by 8 · see citing works
A Comprehensive Survey on Multi-Modal Conversational Emotion Recognition with Deep LearningPreprint
Shou, Y., Meng, T., Ai, W., Yin, N., & Li, K. (2024) · SSRN Electronic Journal · doi:10.2139/ssrn.5017731 · Cited by 8 · see citing works
DPI-ITD: A Dual-Perspective Information-Driven Framework for Insider Threat Detection in IoT Systems
Kong, K., Jin, X., Liu, D., Xu, S., Liu, Z., & Geng, G. (2025) · IEEE Internet of Things Journal · doi:10.1109/jiot.2025.3589636 · Cited by 6 · see citing works
Marappan, S., & Prabha, K. S. S. (2024) · IEEE Access · doi:10.1109/access.2024.3426604 · Cited by 5 · see citing works
Shanthi, P., & Venkatesh, K. (2023) · International Conference on Advances in Computing, Communication and Applied Informatics (ACCAI) · doi:10.1109/accai58221.2023.10199884 · Cited by 3 · see citing works
Methods of natural language processing in solving problems of detecting social engineering attacks
Частикова, В. А., Козачёк, К. В., & Гуляй, В. Г. (2022) · Вестник Адыгейского государственного университета серия «Естественно-математические и технические науки» · doi:10.53598/2410-3225-2021-4-291-95-108 · Cited by 2 · see citing works
GiBERT: Enhancing BERT with Linguistic Information using a Lightweight Gated Injection Method
Peinelt, N., Rei, M., & Liakata, M. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.200 · Cited by 2 · see citing works
An AI-based Approach to Train a Language Model on a Wide Range Corpus Using BERT
Bhutda, A., Sakarkar, G., Shelke, N. M., Paithankar, K., & Panchal, R. (2024) · IEEE International Students' Conference on Electrical, Electronics and Computer Science (SCEECS) · doi:10.1109/sceecs61402.2024.10482037 · Cited by 1 · see citing works
Natural Language Based Corpus Model on a Wide Range of Medical CorpusPreprint
Dutta, P., Sarkar, A., Mahalle, G., Sakarkar, G., & Paithankar, K. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4050337 · Cited by 1 · see citing works
Edge-Cloud Polarization and Collaboration: A Comprehensive SurveyPreprint
Yao, J., Zhang, S., Yang, Y., Wang, F., Ma, J., Zhang, J., Chu, Y., Ji, L., Jia, K., Shen, T., et al. (2021) · arXiv (Cornell University)
Self-citationDheeravath, K., & Saritha, S. J. (2024) · International Journal of Computing and Digital Systems · doi:10.12785/ijcds/160126
HyWE: A Hybrid Word Embedding Method for Smart Contract Vulnerability Detection
Chen, J., Li, Z., & Wang, D. (2024) · IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW) · doi:10.1109/issrew63542.2024.00075
Zhang, A., Zhang, Y., Xie, H., & Zhang, X. (2026) · Discover Applied Sciences · doi:10.1007/s42452-026-08614-0
Self-citationChoi, D., Son, K., Yu, J., Jung, H., & Kim, J. (2026) · Designing Interactive Systems Conference · doi:10.1145/3800645.3813005
- Authors
- Gan, L., Teng, Z., Zhang, Y., Zhu, L., Wu, F., & Yang, Y.
- Affiliations
- Zhejiang University (CN)
- Westlake University (CN)
- University of Technology Sydney (AU)
- Shanghai Artificial Intelligence Laboratory (CN)
- Venue
- IEEE/ACM Transactions on Audio, Speech, and Language Processing
- Published
- 2022
- Volume
- 30
- Pages
- 2696-2704
- ISSN
- 2329-9290 (ISSN-L)
- 2329-9304
- OpenAlex
- W3113816342
- Cited by
- 22
- References
- 60
- Citations by year
- 2026: 2 · 2025: 2 · 2024: 7 · 2023: 4 · 2022: 5 · 2021: 2
- FWCI
- 1.48
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE/ACM Transactions on Audio Speech and Language Processing (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Natural Science Foundation of China (62037001)
- Ministry of Science and Technology (2020 YFC 0832500)
- Zhejiang Province Key Project (2022SDXHDX0003)
- Zhejiang Province Science and Technology Project (2022C01044)
- Starry Night Science Fund of Zhejiang University Shanghai Institute for Advanced Study (SN-ZJU-SIAS-0010)
- CAAI-Huawei MindSpore Open Fund (CAAIXSJLJJ-2021-015A, CAAIXSJLJJ-2021-046A)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- gloves, word embeddings, BERT, masked language models, multi-head attention, word similarity, knowledge distillation, static word embeddings
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
