Knowledge Enhanced Contextual Word Representations
Open accessThe DOI matches its Crossref record; 71 of the 72 references it cites were found in a registry or an index.
Abstract
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A. Smith. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“XLM-R achieves state-of-the-arts results on multiple cross-lingual benchmarks, including XNLI, MLQA, and NER.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [62]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“We compute M m scores using the prior and dot product between the entity-span vectors and entity embeddings, with a two-layer MLP (100 hidden dimensions).”
BackgroundMethodInfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 4×
“, 2018), and BERT (Devlin et al., 2019) have significantly improved the state of the art for a wide range of NLP tasks.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 4×
“, 2018), and BERT (Devlin et al., 2019) have significantly improved the state of the art for a wide range of NLP tasks.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Pretrained word representations Initial work learning word vectors focused on static word embeddings using multi-task learning objectives (Collobert and Weston, 2008) or corpus level cooccurence statistics (Mikolov et al., 2013a; Pennington et al., 2014).”
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 3×
“Pretrained word representations Initial work learning word vectors focused on static word embeddings using multi-task learning objectives (Collobert and Weston, 2008) or corpus level co-occurence statistics (Mikolov et al., 2013a; Pen-nington et al., 2014).”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 3×
“The entity embeddings use a skipgram like objective (Mikolov et al., 2013b) to learn 300-dimensional embeddings of Wikipedia”
BackgroundMethodWordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 1×
“We experiment with integrating both WordNet (Miller, 1995) and Wikipedia, thus explicitly adding word sense knowledge and facts about named entities (including those unseen at training time).”
MethodDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 6×
“Recently the field has shifted toward learning context-sensitive embeddings (Dai and Le, 2015; Peters et al., 2018; Devlin et al., 2019).”
BackgroundMethodInfluentialSelf-citation(2013) · Cited by 5,726
Research works printed but not linked: 5
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013b. Distributed representa- tions of words and phrases and their compositional- ity. In NeurIPS.
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL-HLT.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL.
- Bishan Yang and Tom Michael Mitchell. 2017. Lever- aging knowledge bases in LSTMs for improving ma- chine reading. In ACL.
- Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: En- hanced language representation with informative en- tities. In ACL.
Cited by
Items 201–300 of 705
DAFS: a domain aware few shot generative model for event detection
Xia, N., Yu, H., Wang, Y., Xuan, J., & Luo, X. (2022) · Machine Learning · doi:10.1007/s10994-022-06198-5 · Cited by 24
“ELMO (Peters et al., 2019) gets context sensitive em- beddings by encoding characters with stacked bidirectional LSTM (Hochreiter & Schmidhuber, 1997) and residual structure (He et al.”
BackgroundRobust and explainable identification of logical fallacies in natural language arguments
Sourati, Z., Venkatesh, V. P. P., Deshpande, D. G., Rawlani, H., Ilievski, F., Sandlin, H.-Â., & Mermoud, A. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110418 · Cited by 42
Incorporating Extra Knowledge to Enhance Word Embedding
Roy, A., & Pan, S. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/686 · Cited by 24
“[ Peters et al. , 2019; Zhang et al. , 2019 ] .”
Relational memory-augmented language models
Liu, Q., Yogatama, D., & Blunsom, P. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00476 · Cited by 23
“Using knowledge information for pretraining language models (Peters et al., 2019; Sun et al., 2019; Liu et al., 2020; Guu et al., 2020; Wang et al., 2021b; Agarwal et al., 2021; Verga et al., 2021) has recently grown in popularity and has achieved substantial improvements on knowledge-driven tasks such as question ans…”
BackgroundSentiment Knowledge Enhanced Self-supervised Learning for Multimodal Sentiment Analysis
Fan, Q., Han, J., He, Y., Zheng, T., & Zheng, G. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.821 · Cited by 35
Knowledge Enhanced Fine-Tuning for Better Handling Unseen Entities in Dialogue Generation
Cui, L., Wu, Y., Liu, S.-J., & Zhang, Y. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.179 · Cited by 26
“KnowBERT (Peters et al., 2019) incorporates knowledge base into BERT using the knowledge attention.”
MethodOn the Limitations of Dataset Balancing: The Lost Battle Against Spurious Correlations
Schwartz, R., & Stanovsky, G. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.168 · Cited by 35
Self-citationFusing external knowledge resources for natural language understanding techniques: A survey
Wang, Y., Wang, W., Chen, Q., Huang, K., Nguyen, A., De, S., & Hussain, A. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.11.025 · Cited by 32
Enhancing Multilingual Language Model with Massive Multilingual Knowledge Triples
Liu, L., Li, X., He, R., Bing, L., Joty, S., & Si, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.462 · Cited by 23
“Existing PLMs usually learn universal language representations from general-purpose large-scale corpora but do not concentrate on capturing world’s factual knowledge.”
BackgroundMethodInfluentialLanguage Models with Rationality
Kassner, N., Tafjord, O., Sabharwal, A., Richardson, K., Schuetze, H., & Clark, P. A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.877 · Cited by 50
Wavelet Attention Embedding Networks for Video Super-Resolution
Choi, Y.-J., Lee, Y., & Kim, B. (2021) · Proceedings - International Conference on Pattern Recognition/Proceedings/International Conference on Pattern Recognition · doi:10.1109/icpr48806.2021.9412623 · Cited by 25
“Especially, in natural language processing (NLP) tasks, the latest state-ofthe-art methods [31]–[33] are based on attention mechanism.”
BackgroundSentiLR: Linguistic Knowledge Enhanced Language Representation for Sentiment Analysis.Preprint
Ke, P., Ji, H., Liu, S., Zhu, X., & Huang, M. (2019) · arXiv (Cornell University) · Cited by 23
Pre-training Text-to-Text Transformers for Concept-centric Common SensePreprint
Zhou, W., Lee, D., Selvam, R. K., Lee, S., Lin, B. Y., & Ren, X. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.07956 · Cited by 27
“Towards augmenting PTLMs with more knowledge, prior works mainly focus on training larger models (Brown et al., 2020), adding specific architectures to exploit external knowledge (Peters et al., 2019), or incorporating knowledge bases for pre-training (Xiong et al., 2020).”
BackgroundMethodKnowledge Infused DecodingPreprint
Liu, R., Zheng, G., Gupta, S., Gaonkar, R., Gao, C., Vosoughi, S., Shokouhi, M., & Awadallah, A. H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2204.03084 · Cited by 25
“…et al., 2020; Zhang et al., 2019; Khandelwal et al., 2020) or add special architectures to encode knowledge (Bosselut et al., 2019; Logan et al., 2019; Peters et al., 2019b) from external resources (e.g., knowledge graphs such as C ONCEPT N ET (Speer et al., 2017) and A TOMIC (Sap et al., 2019)).”
BackgroundSentSim: Crosslingual Semantic Evaluation of Machine Translation
Song, Y., Zhao, J., & Specia, L. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.252 · Cited by 36
“Recent studies have showed that the new generation of automatic evaluation metrics, which instead of lexical overlap are based on word semantics using continuous word embedding, such as BERT (Devlin et al., 2019), ElMo (Peters et al., 2019), XL-Net (Yang et al., 2019) or XLM-Roberta (Conneau et al., 2019), have signifi…”
MethodAre we there yet? Exploring clinical domain knowledge of BERT models
Sushil, M., Šuster, S., & Daelemans, W. M. P. (2021) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2021.bionlp-1.5 · Cited by 27
“…al., 2020), by combining relational information from knowledge graphs with text embeddings (Mihaylov and Frank, 2018; Chen et al., 2018; Zhang et al., 2019; Yang et al., 2019a; Liu et al., 2020), or by jointly learning knowledge graph and textual embeddings (Peters et al., 2019; Feng et al., 2020).”
BackgroundPlug-and-Play Knowledge Injection for Pre-trained Language Models
Zhang, Z., Zeng, Z., Lin, Y., Wang, H., Ye, D., Xiao, C., Han, X., Liu, Z., Li, P., Sun, M., et al. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.594 · Cited by 32
Commonsense Knowledge Enhanced Sentiment Dependency Graph for Sarcasm Detection
Yu, Z., Jin, D., Wang, X., Li, Y., Wang, L., & Dang, J. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/269 · Cited by 31
“…NBOW [Tay et al., 2018]; 2) conventional neural networks: TextCNN [Kim, 2014], ATT-LSTM [Yang et al., 2016]; 3) BERT-based models: BERT [Devlin et al., 2018], KnowBERT [Peters et al., 2019]; 4) sarcasm detection methods: SAWS [Pan et al., 2020], ADGCN [Lou et al., 2021], SarDeCK [Li et al., 2021].”
Billion-scale Pre-trained E-commerce Product Knowledge Graph Model
Zhang, W., Wong, C.-M., Ye, G., Wen, B., Zhang, W., & Chen, H. (2021) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde51399.2021.00280 · Cited by 30
“KnowBert [42] uses an integrated entity linker to generate knowledge enhanced entity-span representations via a form of word-to-entity attention.”
MethodIncorporating Word Sense Disambiguation in Neural Language ModelsPreprint
Wahle, J. P., Ruas, T., Meuschke, N., & Gipp, B. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.07967 · Cited by 22
“Current methods (Huang et al., 2019; Du et al., 2019; Peters et al., 2019; Levine et al., 2019) modify the WSD task into a text classification problem, leveraging BERT’s semantic information through”
BackgroundMethodInfluentialHow Can the [MASK] Know? The Sources and Limitations of Knowledge in BERT
Podkorytov, M., Biś, D., & Liu, X. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534299 · Cited by 24
“Peters et al. [18] embed and integrate human-crafted knowledge from multiple KBs (i.e., WordNet, Wikipedia) into BERT using self-supervision on unlabeled data, and report improved factual knowledge recall compared to the original BERT model.”
MethodChen, Q., Li, F.-L., Xu, G., Yan, M., Zhang, J., & Zhang, Y. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/567 · Cited by 31
Transformers in Natural Language Processing
Yvon, F. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-24349-3_6 · Cited by 25
“For this, the combination of statistical models with knowledge graphs seems to be a promising research direction [Peters et al., 2019, Bosselut et al., 2019].”
MethodGao, Y., He, Y., Kan, Z., Han, Y., Qiao, L., & Li, D. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.28 · Cited by 31
“Joint pre-trained language models (PLMs) and knowledge graph approaches can be broadly classified into two categories: one that introduces knowledge from KGs to enhance PLMs on NLP down-stream tasks (Zhang et al., 2019; Peters et al., 2019; He et al., 2020).”
BackgroundLocal and Global: Temporal Question Answering via Information Fusion
Liu, Y., Liang, D., Li, M., Giunchiglia, F., Li, X., Wang, S., Wu, W. B., Huang, L., Feng, X., & Guan, R. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/571 · Cited by 64
“…for comparison on CRONQUESTIONS: (I) pre-trained LMs, including BERT [ Devlin et al. , 2019 ] , RoBERTa [ Liu et al. , 2019 ] and KnowBERT [Peters et al. , 2019]; (II) general KG embedding-based models, including EaE [F´evry et al. , 2020] and Em-bedKGQA [Saxena et al. , 2020]; and (III)…”
Question Calibration and Multi-Hop Modeling for Temporal Question Answering
Xue, C., Liang, D., Wang, P., & Zhang, J. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29903 · Cited by 50
Auxiliary Learning for Relation Extraction
Lyu, S., Cheng, J., Wu, X., Cui, L., Chen, H., & Miao, C. (2020) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2020.3040444 · Cited by 24
Conversational Entity Linking: Problem Definition and Datasets
Joko, H., Hasibi, F., Balog, K., & de Vries, A. P. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3463258 · Cited by 33
“Indeed, augmenting neural language models with information from knowledge graphs has shown to be beneficial in a number of downstream tasks [45, 47, 64].”
BackgroundDistantly Supervised Relation Extraction with Sentence Reconstruction and Knowledge Base Priors
Christopoulou, F., Miwa, M., & Ananiadou, S. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.2 · Cited by 21
Knowledge Mining with Scene Text for Fine-Grained Recognition
Wang, H., Liao, J., Cheng, T., Gao, Z., Líu, H., Ren, B., Bai, X., & Liu, W. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.00458 · Cited by 28
“To confirm this assumption, we first train the models of ViT and KnowBert with image data and scene text, respectively, at the supervision of the classification task.”
BackgroundMethodInfluentialCross-Domain Aspect Extraction using Transformers Augmented with Knowledge Graphs
Howard, P., Ma, A., Lal, V., Simoes, A. P., Korat, D., Pereg, O., Wasserblat, M., & Singer, G. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557275 · Cited by 23
“These approaches have led to a myriad of new BERT-like models such as KnowBERT [26], K-BERT [39], and E-BERT [27] which attempt to curate and inject knowledge from KBs in various ways.”
MethodA Knowledge Enhanced Ensemble Learning Model for Mental Disorder Detection on Social Media
Rao, G., Peng, C., Zhang, L., Wang, X., & Feng, Z. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-55393-7_17 · Cited by 20
Interpretable Entity Representations through Large-Scale Typing
Onoe, Y., & Durrett, G. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.54 · Cited by 26
Dalal, S., Tilwani, D., Gaur, M., Jain, S., Shalin, V. L., & Sheth, A. P. (2025) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2024.3483577 · Cited by 19
“…input embeddings with knowledge using dense passage retrieval (e.g., RAG [35], ISEEQ [36]), (C) customizing and optimizing the attention function of LMs using both input and knowledge semantics (KI-BERT [37], ELMO [38]), and (D) using datasets grounded in additional lexicons or knowledge-graphs.”
MethodDetection, Disambiguation, Re-ranking: Autoregressive Entity Linking as a Multi-Task Problem
Mrini, K., Nie, S., Gu, J., Wang, S., Sanjabi, M., & Firooz, H. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.156 · Cited by 26
Wang, Y., Sun, Y., Fu, Y., Zhu, D., & Tian, Z. (2024) · IEEE Transactions on Instrumentation and Measurement · doi:10.1109/tim.2024.3374300 · Cited by 26
“As the most effective scheme for label deficiency, the “unsupervised pretraining & supervised fine-tuning” paradigm achieves great success in NLP and CV [20], [23], [24], [25], [26], [27], [28].”
BackgroundXu, S., Li, H., Yuan, P., Wang, Y., Wu, Y., He, X., Liu, Y., & Zhou, B. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.1 · Cited by 26
The Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.07616 · Cited by 18
“Another efficient way to enrich PLMs’ text representation with entity knowledge is utilizing word-to-entity attention (Peters et al., 2019; Yamada et al., 2020).”
BackgroundMethodThe Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2024) · Machine Intelligence Research · doi:10.1007/s11633-023-1416-x · Cited by 18
“Another efficient way to enrich PLMs’ text representation with entity knowledge is utilizing word-to-entity attention (Peters et al., 2019; Yamada et al., 2020).”
BackgroundMethodAutoRC: Improving BERT Based Relation Classification Models via Architecture Search
Zhu, W. (2021) · doi:10.18653/v1/2021.acl-srw.4 · Cited by 26
“KnowBert (Peters et al., 2019) explicitly incorporates external KGs.”
BioBERT and Similar Approaches for Relation Extraction
Bhasuran, B. (2022) · Methods in molecular biology · doi:10.1007/978-1-0716-2305-3_12 · Cited by 18
Embracing ambiguity: Improving similarity-oriented tasks with contextual synonym knowledge
Li, Y., Chen, J., Li, Y., Yu, T., Chen, X., & Zheng, H.-T. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126583 · Cited by 23
Document-level relation extraction with three channels
Zhang, Z., Zhao, S., Zhang, H., Wan, Q., & Liu, J. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.111281 · Cited by 19
Dhananjaya, V., Ranathunga, S., & Jayasena, S. (2024) · CAAI Transactions on Intelligence Technology · doi:10.1049/cit2.12333 · Cited by 21
A Survey of Semantic Parsing Techniques
Jiang, P., & Cai, X. (2024) · Symmetry · doi:10.3390/sym16091201 · Cited by 20
Chang, D., Lin, E. C., Brandt, C., & Taylor, R. A. (2021) · JMIR Medical Informatics · doi:10.2196/23101 · Cited by 21
“The attempts to mitigate these shortcomings of language models have often involved the use of graph representation learning techniques [7-9], which provide a natural way for working with knowledge in the form of graphs.”
BERTnesia: Investigating the capture and forgetting of knowledge in BERTPreprint
Wallat, J., Singh, J., & Anand, A. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.09313 · Cited by 24
Xue, B., Wang, W., Wang, H., Mi, F., Wang, R., Wang, Y., Shang, L., Jiang, X., Liu, Q., & Wong, K. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.525 · Cited by 28
CSS-LM: A Contrastive Framework for Semi-Supervised Fine-Tuning of Pre-Trained Language Models
Su, Y., Han, X., Lin, Y., Zhang, Z., Liu, Z., Li, P., Zhou, J., & Sun, M. (2021) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2021.3105013 · Cited by 19
“…designed specific self-supervised learning objectives to help PLMs learn specific abilities in the pre-training phase, such as representing token spans [6] and entities [15], [16], [17], [17], [18], conferential reasoning [19], multi-lingualism [20], multi-modality [21], [22], [23], [24], [25], etc.”
BackgroundZaporojets, K., Deleu, J., Jiang, Y., Demeester, T., & Develder, C. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.88 · Cited by 24
KgPLM: Knowledge-guided Language Model Pre-training via Generative and Discriminative LearningPreprint
He, B., Jiang, X., Xiao, J., & Liu, Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.03551 · Cited by 17
“To further explore their potential, researchers have proposed various approaches to guide the pre-training of the language models by injecting different forms of knowledge into them, such as structured knowledge graph or linguistic knowledge (Zhang et al. 2019; Peters et al. 2019; Xiong et al. 2020; Wang et al. 2020;…”
BackgroundEnhancing text classification with attention matrices based on BERT
Yu, Z., Li, H., & Feng, J. (2023) · Expert Systems · doi:10.1111/exsy.13512 · Cited by 19
“Additionally, to resolve the issue of inconsistency between real-world and training application input, XLNet (Yang et al., 2019) proposes a permutation-based language model, while LIBERT (Lauscher et al., 2019), Know-BERT (Peters et al., 2019), K-BERT (Liu et al., 2020), mBERT (Liu et al., 2020), and other models are…”
MethodPromptLink: Leveraging Large Language Models for Cross-Source Biomedical Concept Linking
Xie, Y., Lu, J., Ho, J. C., Nahab, F. B., Hu, X., & Yang, C. (2024) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3626772.3657904 · Cited by 32
“…LLMs also face challenges including the design of effective and cost-efficient prompts within the context length limits [40], and the NIL prediction capability of reliably rejecting all candidates when correct concepts are absent, instead of returning relatively close but incorrect ones [23].”
Liang, J., Hou, S., Zhao, A., Xu, Q., Xiang, L., Li, R., & Wu, H. (2025) · Geo-spatial Information Science · doi:10.1080/10095020.2025.2483884 · Cited by 22
“Earlier studies have embedded structured knowledge from KGs into the underlying neural network layers of LLMs during the pretraining or fine-tuning process, enabling LLMs to capture both semantic and structural information (Liu et al. 2020; Peters et al. 2019; Tang et al. 2023).”
Lu, Q., Dou, D., & Nguyen, T. H. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.325 · Cited by 33
“There has been a surge of research on knowledge injection for PLMs in recent years (Yao et al., 2019; Zhang et al., 2019; Peters et al., 2019; Kim et al., 2020; Levine et al., 2020; Lauscher et al., 2020; Pereira et al., 2020; Sun et al., 2020; He et al., 2020a; Wang et al., 2021).”
BackgroundCoCoLM: Complex Commonsense Enhanced Language Model with Discourse Relations
Yu, C., Zhang, H., Song, Y., & Ng, W. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.93 · Cited by 28
InforMask: Unsupervised Informative Masking for Language Model Pretraining
Sadeq, N., Xu, C., & McAuley, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.395 · Cited by 16
“Knowledge-Enhanced LMs KnowBERT (Peters et al., 2019) shows that factual recall performance in BERT can be improved significantly by embedding knowledge bases into additional layers of the model.”
BackgroundNeural Knowledge Bank for Pretrained Transformers
Dai, D., Jiang, W., Dong, Q., Lyu, Y., & Sui, Z. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-44696-2_60 · Cited by 27
StereoKG: Data-Driven Knowledge Graph Construction For Cultural Knowledge and Stereotypes
Deshpande, A., Ruiter, D., Mosbach, M., & Klakow, D. (2022) · Workshop on Online Abuse and Harms (WOAH) · doi:10.18653/v1/2022.woah-1.7 · Cited by 23
DictPrompt: Comprehensive dictionary-integrated prompt tuning for pre-trained language model
Cao, R., Wang, Y., Gao, L. C., & Yang, M. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110605 · Cited by 18
Yang, C., Deng, J., Chen, X., & An, Y. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104456 · Cited by 18
ECOLA: Enhancing Temporal Knowledge Embeddings with Contextualized Language Representations
Han, Z. Y., Liao, R., Gu, J., Zhang, Y., Ding, Z., Gu, Y., Koeppl, H., Schütze, H., & Volker Tresp, C. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.335 · Cited by 15
“Zhang et al. (2019) and Peters et al. (2019) focus on enhancing language models using external knowledge.”
BackgroundWTR: A Test Collection for Web Table Retrieval
Chen, Z., Zhang, S., & Davison, B. D. (2021) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3404835.3463260 · Cited by 15
“The recent signs of progress in multi-task learning [20, 26] and pretraining techniques [11, 21] have shown that a model pre-trained or jointly trained on related tasks can help learn more generalized features and improve the model performance on target tasks.”
BackgroundStatic Embeddings as Efficient Knowledge Bases?
Dufter, P., Kassner, N., & Schütze, H. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.186 · Cited by 25
ReadTwice: Reading Very Large Documents with Memories
Zemlyanskiy, Y., Ainslie, J., de Jong, M., Pham, P., Eckstein, I., & Sha, F. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.408 · Cited by 17
Alharbi, A. I., Smith, P., & Lee, M. G. (2021) · Procedia Computer Science · doi:10.1016/j.procs.2021.05.089 · Cited by 17
Chen, H., Deng, S., Zhang, W., Xu, Z., Li, J., & Kharlamov, E. (2021) · Fundamental Research · doi:10.1016/j.fmre.2021.08.013 · Cited by 15
A novel chinese relation extraction method using polysemy rethinking mechanism
Zhao, Q., Gao, T., & Guo, N. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03817-5 · Cited by 15
SGPT: Semantic graphs based pre-training for aspect-based sentiment analysis
Qian, Y., Chen, C., Wang, Z., Xiao, R., & Tang, H. (2023) · World Wide Web · doi:10.1007/s11280-022-01123-1 · Cited by 15
“Recently, to enable PLMs with world knowledge, several attempts (Wang et al., 2019; Peters et al., 2019; Zhang et al., 2019; Liu et al., 2020; Wang et al., 2020) have been made to inject knowledge into BERT leveraging Knowledge Graphs (KGs).”
BackgroundReasoning Over Virtual Knowledge Bases With Open Predicate Relations
Sun, H., Verga, P., Dhingra, B., Salakhutdinov, R., & Cohen, W. W. (2021) · arXiv · doi:10.48550/arxiv.2102.07043 · Cited by 14
“Other work has shown that injecting an external memory constructed from KB into a LM can help training LMs (Peters et al., 2019), improve generation tasks, (Logan et al.”
BackgroundA Brief Survey of Text Classification Methods
Jiao, Q. (2023) · IEEE International Conference on Information Technology, Big Data and Artificial Intelligence (ICIBA) · doi:10.1109/iciba56860.2023.10165621 · Cited by 14
“Meanwhile, some researchers also introduce external knowledge to PTMs, such as SenseBERT [27], ERNIE [49] and KnowBERT [40].”
BackgroundCalixto, I., Raganato, A., & Pasini, T. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.286 · Cited by 24
“Recently, Zhang et al. (2019, ERNIE) and Peters et al. (2019, KnowBERT) devised different meth-ods to incorporate entities from external knowledge graphs into masked language model (LM) training.”
BackgroundKnowledge-aware Named Entity Recognition with Alleviating Heterogeneity
Nie, B., Ding, R., Xie, P., Huang, F., Chen, Q., & Si, L. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17603 · Cited by 20
“In experiment, we leverage KnowBERT (Peters et al. 2019), a powerful knowledge-aware language model to generate every mention-entity pair.”
MethodModeling Multi-Granularity Hierarchical Features for Relation Extraction
Liang, X., Wu, S., Li, M., & Li, Z. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.375 · Cited by 15
Wu, T., Bai, X., Guo, W., Liu, W., Li, S., & Yang, Y. (2023) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3539597.3570415 · Cited by 22
“Moreover, KG-based model and PLM can be jointly optimized to enhance each other, including ERNIE [34], K-BERT [15], KnowBERT [21] and KEPLER [29].”
MethodRevisiting the Knowledge Injection Frameworks
Fu, P., Zhang, Y., Wang, H., Qiu, W., & Zhao, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.677 · Cited by 26
Machine Learning and Knowledge Graphs: Existing Gaps and Future Research Challenges
d’Amato, C., Mahon, L., Monnin, P., & Stamou, G. B. (2023) · Transactions on Graph Data and Knowledge · doi:10.4230/tgdk.1.1.8 · Cited by 27
Joint Biomedical Entity and Relation Extraction with Knowledge-Enhanced Collective InferencePreprint
Lai, T., Ji, H., Zhai, C., & Tran, Q. H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.13456 · Cited by 19
“KnowBertAttention: This baseline uses the Knowledge Attention and Recontextualization (KAR) mechanism of KnowBert (Peters et al., 2019), a state-of-the-art knowledge-enhanced Model Entity (Macro-F1) Relation (Macro-F1)”
BackgroundMethodInfluentialTemporal validity reassessment: commonsense reasoning about information obsoleteness
Hosokawa, T., Jatowt, A., & Sugiyama, K. (2024) · Discover Computing · doi:10.1007/s10791-024-09433-w · Cited by 17
“Especially, commonsense reasoning works commonly incorporate knowledge from large KBs such as Con-ceptNet [61, 62] and WikiData [63] in their architectures [64–66].”
Semantic aware representation learning for optimizing image retrieval systems in radiology
Vagena, Z., Wei, X., Kurtz, C., & Cloppet, F. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.111060 · Cited by 13
Elevating Knowledge-Enhanced Entity and Relationship Understanding for Sarcasm Detection
Wang, X., Wang, Y., He, D., Yu, Z., Li, Y., Wang, L., Dang, J., & Jin, D. (2025) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2025.3547055 · Cited by 40
“(cid:2) Know-BERT [43]: Know-BERT, short for Knowledge BERT model, combines textual representations with knowledge representations through a straightforward concatenation process.”
On the Potential of Logic and Reasoning in Neurosymbolic Systems Using OWL-Based Knowledge Graphs
Herron, D., Jiménez-Ruiz, E., & Weyde, T. (2025) · Neurosymbolic Artificial Intelligence · doi:10.1177/29498732251320043 · Cited by 13
“For example, (Peters et al., 2019; Roy et al., 2023; Yamada et al., 2020; Zhang et al., 2019) explore variations on this theme and all report performance improvements from injecting KG embeddings as background knowledge.”
Design Patterns for Large Language Model Based Neuro-Symbolic Systems
de Boer, M., Smit, Q. T. S., van Bekkum, M. A., Meyer-Vitali, A., & Schmid, T. (2025) · Neurosymbolic Artificial Intelligence · doi:10.1177/29498732251377499 · Cited by 18
“Alternatively, architectures have been proposed using encoder-only (BERT Devlin, 2018) or decoder-only (GPT Brown et al., 2020, BLOOMZ Muennighoff et al., 2022, PaLM Chowdhery et al., 2023) models.”
InfluentialEnhancing Multiple-Choice Question Answering with Causal Knowledge
Dalal, D., Arčan, M., & Buitelaar, P. (2021) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2021.deelio-1.8 · Cited by 17
“Peters et al. (2019) present the Knowledge Attention and Recontextualization”
Islam, S. M., & Bhattacharya, S. (2022) · International World Wide Web Conference (WWW) · doi:10.1145/3485447.3511941 · Cited by 15
“[23, 24, 44] propose a promising line of schemes for incorporating entity knowledge in KGs into BERT embeddings.”
BackgroundMedical knowledge-enhanced prompt learning for diagnosis classification from clinical text
Lu, Y., Zhao, X., & Wang, J. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.33 · Cited by 21
“Know-BERT (Peters et al., 2019) propose Knowledge Attention and Recontextualization (KAR) and entity linking to incorporate knowledge into PLMs. ERNIE-THU (Zhang et al., 2019) introduce an knowledge fusion module, injecting entity embeddings through knowledge encoders.”
SpEL: Structured Prediction for Entity Linking
Shavarani, H. S., & Sarkar, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.686 · Cited by 13
Ji, Z., Dai, L., Pang, J., & Shen, T. (2020) · IEEE Access · doi:10.1109/access.2020.2994247 · Cited by 13
“…to ensure that all of the words in the same entity were masked during word representation training, instead of only one word or character being masked; [20] updated contextual word representations via a form of word-to-entity attention, by inserting prior knowledge into a deep neural model.”
BackgroundFadel, A., Al‐Ayyoub, M., & Cambria, E. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.66 · Cited by 13
“Approaches like KnowBERT (Peters et al., 2019) leverage knowledge bases like WordNet (Miller et al.”
MethodContinual knowledge infusion into pre-trained biomedical language models
Jha, K., & Zhang, A. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab671 · Cited by 17
Does Knowledge Help General NLU? An Empirical StudyPreprint
Xu, R., Fang, Y., Zhu, C., & Zeng, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.00563 · Cited by 13
“In recent years, there has been a rise in the trend of integrating external knowledge into language models (Wang et al., 2020; Yu et al., 2020; Wang et al., 2019b; Liu et al., 2020; Peters et al., 2019; Zhang et al., 2019; Poerner et al., 2020) based on the transformer (Vaswani et al.”
BackgroundInfluentialPiat, G., Allauzen, A., Essafi, H., Bernard, G., Tourille, J., & Semmar, N. (2022) · doi:10.1109/wimob55322.2022 · Cited by 13
Learn to Not Link: Exploring NIL Prediction in Entity Linking
Zhu, F., Yu, J., Jin, H., Hou, L., Li, J., & Sui, Z. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.690 · Cited by 12
Zhang, Y., Zhai, S., Meng, Y., Bi, S., Chen, Y., & Qi, G. (2024) · Information Processing & Management · doi:10.1016/j.ipm.2024.103729 · Cited by 14
Yao, W., Pan, X., Jin, L., Chen, J., Yu, D., & Yu, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.610 · Cited by 12
“Parameters BERT Large (Devlin et al., 2019) 69.6 340M RoBERTa Large (Liu et al., 2019) 69.9 355M KnowBERT W+W (Peters et al., 2019) 70.9 523M SenseBERT Large (Levine et al., 2020) 72.1 380M T5-Large (Raffel et al., 2020) 69.3 770M T5-3B (Raffel et al., 2020) 72.1 3000M BERT ARES (Scarlini et al., 2020b) 72.2 342M SemE…”
InfluentialLanguage cognition and language computation — human and machine language understanding
王, 少., 丁, 鼐., 林, 楠., 张, 家., & 宗, 成. (2022) · Scientia Sinica Informationis · doi:10.1360/ssi-2021-0100 · Cited by 14
A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Models
Ye, D., Lin, Y., Li, P., Sun, M., & Liu, Z. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-short.57 · Cited by 18
Lexical semantics enhanced neural word embeddings
Yang, D., Li, N., Zou, L., & Ma, H. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109298 · Cited by 16
“Another way of joint training is to revise the architecture of neural networks either through training Graph Convolutional Networks with syntactic dependencies and semantic relationships (Vashishth et al. 2019) or by introducing attention mechanisms (Yang and Mitchell 2017, Peters et al. 2019).”
MethodPoetryBERT: Pre-training with Sememe Knowledge for Classical Chinese Poetry
Zhao, J., Bai, T., Wei, Y., & Wu, B. (2022) · Communications in computer and information science · doi:10.1007/978-981-19-8991-9_26 · Cited by 15
Metadata
- Authors
- Peters, M. E., Neumann, M., Logan, R., Schwartz, R., Joshi, V., Singh, S., & Smith, N. A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- University of California, Irvine (US)
- University of Washington (US)
- Corresponding author
- Singh, Sameer
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 43-54
- OpenAlex
- W2970986510
- Cited by
- 661
- References
- 72
- Citations by year
- 2026: 21 · 2025: 54 · 2024: 81 · 2023: 125 · 2022: 115 · 2021: 160 · 2020: 98 · 2019: 6
- FWCI
- 36.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- National Science Foundation (1730158, 1817183)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- contextualized word representations, word embeddings, knowledge graphs, word sense disambiguation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.