Knowledge Enhanced Contextual Word Representations
Open accessThe DOI matches its Crossref record; 71 of the 72 references it cites were found in a registry or an index.
Abstract
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A. Smith. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“XLM-R achieves state-of-the-arts results on multiple cross-lingual benchmarks, including XNLI, MLQA, and NER.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [62]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“We compute M m scores using the prior and dot product between the entity-span vectors and entity embeddings, with a two-layer MLP (100 hidden dimensions).”
BackgroundMethodInfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 4×
“, 2018), and BERT (Devlin et al., 2019) have significantly improved the state of the art for a wide range of NLP tasks.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 4×
“, 2018), and BERT (Devlin et al., 2019) have significantly improved the state of the art for a wide range of NLP tasks.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Pretrained word representations Initial work learning word vectors focused on static word embeddings using multi-task learning objectives (Collobert and Weston, 2008) or corpus level cooccurence statistics (Mikolov et al., 2013a; Pennington et al., 2014).”
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 3×
“Pretrained word representations Initial work learning word vectors focused on static word embeddings using multi-task learning objectives (Collobert and Weston, 2008) or corpus level co-occurence statistics (Mikolov et al., 2013a; Pen-nington et al., 2014).”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 3×
“The entity embeddings use a skipgram like objective (Mikolov et al., 2013b) to learn 300-dimensional embeddings of Wikipedia”
BackgroundMethodWordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 1×
“We experiment with integrating both WordNet (Miller, 1995) and Wikipedia, thus explicitly adding word sense knowledge and facts about named entities (including those unseen at training time).”
MethodDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 6×
“Recently the field has shifted toward learning context-sensitive embeddings (Dai and Le, 2015; Peters et al., 2018; Devlin et al., 2019).”
BackgroundMethodInfluentialSelf-citation(2013) · Cited by 5,726
Research works printed but not linked: 5
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013b. Distributed representa- tions of words and phrases and their compositional- ity. In NeurIPS.
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL-HLT.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL.
- Bishan Yang and Tom Michael Mitchell. 2017. Lever- aging knowledge bases in LSTMs for improving ma- chine reading. In ACL.
- Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: En- hanced language representation with informative en- tities. In ACL.
Cited by
Items 101–200 of 705
Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training
Shi, P., Ng, P., Wang, Z., Zhu, H., Li, A. H., Wang, J., Nogueira dos Santos, C., & Xiang, B. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17627 · Cited by 71
Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph
Sun, J., Xu, C., Tang, L., Wang, S., Lin, C., Gong, Y., Ni, L. M., Shum, H.-Y., & Guo, J. (2023) · arXiv · doi:10.48550/arxiv.2307.07697 · Cited by 74
“Early studies (Peters et al., 2019; Huang et al., 2024; Luo et al., 2024; Zhang et al., 2021; Li et al., 2023b; Liu et al., 2020) embed structured knowledge from KGs into the underlying neural networks during the pretraining or fine-tuning process.”
Beyond word embeddings: A survey
Incitti, F., Urli, F., & Snidaro, L. (2022) · Information Fusion · doi:10.1016/j.inffus.2022.08.024 · Cited by 97
GenericsKB: A Knowledge Base of Generic StatementsPreprint
Bhakthavatsalam, S., Anastasiades, C., & Clark, P. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.00660 · Cited by 74
“…remarkable performance trained on general text, NLP researchers frequently seek out additional repositories of general/commonsense knowledge to boost performance further, e.g., (Icarte et al., 2017; Wang et al., 2018; Yang et al., 2019; Peters et al., 2019; Liu et al., 2019; Paul and Frank, 2019).”
BackgroundTowards Generalizable Neuro-Symbolic Systems for Commonsense Question Answering
Ma, K., Francis, J., Lu, Q., Nyberg, E., & Oltramari, A. (2019) · doi:10.18653/v1/d19-6003 · Cited by 90
“Peters et al. (2019) and Levine et al. (2019) both tried to injecting knowlegde into language models by pretraining on knowledge bases.”
BackgroundJoint Biomedical Entity and Relation Extraction with Knowledge-Enhanced Collective Inference
Lai, T., Ji, H., Zhai, C., & Tran, Q. H. (2021) · doi:10.18653/v1/2021.acl-long.488 · Cited by 84
“KnowBertAttention: This baseline uses the Knowledge Attention and Recontextualization (KAR) mechanism of KnowBert (Peters et al., 2019), a state-of-the-art knowledge-enhanced Model Entity (Macro-F1) Relation (Macro-F1)”
BackgroundMethodInfluentialStructure Aware Negative Sampling in Knowledge Graphs
Ahrabian, K., Feizi, A., Salehi, Y., Hamilton, W. L., & Bose, A. J. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.492 · Cited by 90
SKILL: Structured Knowledge Infusion for Large Language Models
Moiseev, F., Dong, Z., Alfonseca, E., & Jaggi, M. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.113 · Cited by 71
“KnowBert (Peters et al., 2019) incorporates knowledge from Wikipedia and WordNet (Miller, 1995) into a BERT model through entity embeddings with knowledge-attention and recontextualization mechanism.”
MethodMeng, T., Fang, A., Rokhlenko, O., & Malmasi, S. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.118 · Cited by 82
“Recent work has considered directly integrating knowledge into transformers, e.g., KnowBert adds knowledge to BERT layers (Peters et al., 2019), and LUKE is pretrained to predict masked entities (Yamada et al., 2020).”
BackgroundYe, Z.-X., Chen, Q., Wang, W., & Ling, Z.-H. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1908.06725 · Cited by 60
“, 2019b) and embedding multiple knowledge bases (Peters et al., 2019) into language representation”
BackgroundESC: Redesigning WSD with Extractive Sense Comprehension
Barba, E., Pasini, T., & Navigli, R. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.371 · Cited by 113
“…applications such as Neural Machine Translation (Liu et al., 2018; Pu et al., 2018) and Information Extraction (Moro and Navigli, 2013; Delli Bovi et al., 2015), while also being leveraged to enrich the contextual representations of neural models (Peters et al., 2019; Zhang et al., 2019).”
BackgroundEntity Enhanced BERT Pre-training for Chinese NER
Chen, J., Shi, Y., Yang, Q., & Zhang, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.518 · Cited by 84
“Recently, there is an increasing interest to augment such contextualized representation with external knowledge (Zhang et al., 2019; Liu et al., 2019a; Peters et al., 2019).”
BackgroundBERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA
Kassner, N., & Schütze, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.307 · Cited by 63
A Novel Document-Level Relation Extraction Method Based on BERT and Entity Information
Han, X., & Wang, L. (2020) · IEEE Access · doi:10.1109/access.2020.2996642 · Cited by 76
“Another way of contextual representation of each word is to employ the pre-trained language models [9], [25], [26].”
MethodFew-shot Knowledge Graph-to-Text Generation with Pretrained Language Models
Li, J., Tang, T., Zhao, W. X., Wei, Z., Yuan, N. J., & Wen, J.-R. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.136 · Cited by 87
Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning
Shen, T., Mao, Y., He, P., Long, G., Trischler, A., & Chen, W. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.722 · Cited by 63
TSVFN: Two-Stage Visual Fusion Network for multimodal relation extraction
Zhao, Q., Gao, T., & Guo, N. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103264 · Cited by 69
CokeBERT: Contextual knowledge selection and embedding towards enhanced pre-trained language models
Su, Y., Han, X., Zhang, Z., Lin, Y., Li, P., Liu, Z., Zhou, J., & Sun, M. (2021) · AI Open · doi:10.1016/j.aiopen.2021.06.004 · Cited by 72
“For knowledge-enhanced PLMs like ERNIE, KnowBert, and K-BERT, we re-implement them or use their released code for our experiments, and report the results which can match their results in the original papers.”
BackgroundMethodInfluentialBiomedical Event Extraction with Hierarchical Knowledge Graphs
Huang, K.-H., Yang, M., & Peng, N. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.114 · Cited by 92
An empirical study of pre-trained language models in simple knowledge graph question answering
Hu, N., Wu, Y., Qi, G., Min, D., Chen, J., Pan, J. Z., & Ali, Z. (2023) · World Wide Web · doi:10.1007/s11280-023-01166-y · Cited by 78
“More-over, several works utilize structured knowledge to enhance PLMs for more than just self-supervised training on the large-scale corpus, with significant improvements in mention detection [33] and relation detection [34–37] tasks.”
BackgroundRelation Classification with Entity Type Restriction
Lyu, S., & Chen, H. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.34 · Cited by 59
SpaBERT: A Pretrained Language Model from Geographic Data for Geo-Entity Representation
Li, Z., Kim, J., Chiang, Y., & Chen, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.200 · Cited by 90
Enhancing Clinical BERT Embedding using a Biomedical Knowledge Base
Hao, B., Zhu, H., & Paschalidis, I. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.57 · Cited by 48
“Also, the knowledge base information can be utilized during language model pre-training and some works, including (Lauscher et al., 2019; Peters et al., 2019; Yao et al., 2019), have shown promising results in this direction.”
BackgroundKnowledge Enhanced Pretrained Language Models: A Compreshensive SurveyPreprint
Wei, X., Wang, S., Zhang, D., Bhatia, P., & Arnold, A. O. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08455 · Cited by 72
“Recently, an ever-growing body of work aims at explicitly incorporating knowledge into PLMs [100] [108][61][90][96][49][33].”
BackgroundLearning Customized Visual Models with Retrieval-Augmented Knowledge
Liu, H., Son, K., Yang, J., Liu, C., Gao, J., Lee, Y. J., & Li, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01454 · Cited by 106
“In natural language processing, several works augment large language models with external data encoded with structured language and relation representations [3, 15, 22, 26, 31, 39, 66].”
BackgroundLarge Knowledge Model: Perspectives and Challenges
Chen, H. (2024) · Data Intelligence · doi:10.3724/2096-7004.di.2024.0001 · Cited by 111
“…linking technologies, structure correlations among concepts and entities in the KG can be used to expand and enhance CoT in natural language forms, making the textual CoT more in line with human thinking patterns based on associations,thereby enhancing their effectiveness in guiding AI reasoning.”
Bootleg: Chasing the Tail with Self-Supervised Named Entity Disambiguation
Orr, L., Leszczynski, M., Arora, S., Wu, S., Guha, N., Xiao, L., & Ré, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.10363 · Cited by 40
“ootleg transfertotasksbeyondNEDbyextractingBootleg’s learnedembeddingsandusingthemtosetanewSotAby1.0F1pointsontheTACREDrelationextraction task[2,53],wherethepriorSotAmodelalsousesentity-basedknowledge[38]. Bootleg representations furtherprovidean8%performanceliftonhighlyoptimizedindustrialsearchandassistanttasksatamaj…”
InfluentialSelf-Supervised Knowledge Triplet Learning for Zero-Shot Question Answering
Banerjee, P., & Baral, C. R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.11 · Cited by 75
Wang, X., Zhang, X., Hoo, V., Shao, Z., & Zhang, X. (2024) · IEEE Access · doi:10.1109/access.2024.3496666 · Cited by 75
“In the context of LLMs, [15] proposed knowledge-enhanced language models that incorporate external knowledge bases during pre-training.”
Relational World Knowledge Representation in Contextual Language Models: A Review
Safavi, T., & Koutra, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.81 · Cited by 68
“…choose where (i.e., at which layer) the entity embeddings are integrated, and how the entity embeddings are learned and re-contextualized into the LM. Peters et al. (2019) integrate externally pre-trained, frozen entity embeddings into BERT’s final hidden layers using a word-to-entity attention…”
BackgroundMethodInfluentialJoint Entity and Relation Extraction for Legal Documents with Legal Feature Enhancement
Chen, Y., Sun, Y., Yang, Z., & Lin, H. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.137 · Cited by 69
“Works on embedding the knowledge bases into pre-trained language models (Liu et al., 2020; Peters et al., 2019) contribute to introducing domain knowledge.”
BackgroundAdaptable and Interpretable Neural MemoryOver Symbolic Knowledge
Verga, P., Sun, H., Baldini Soares, L., & Cohen, W. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.288 · Cited by 41
“…has explored techniques to improve the performance of large LMs in answering factual probes, by adding additional supervision in pre-training (Xiong et al., 2019; Wang et al., 2020b) or by adding entity embeddings into an extended LM (Peters et al., 2019; Zhang et al., 2019; Févry et al., 2020).”
MethodModifying Memories in Transformer Models
Zhu, C., Rawat, A. S., Zaheer, M., Bhojanapalli, S., Li, D., Yu, F., & Kumar, S. (2020) · arXiv · doi:10.48550/arxiv.2012.00363 · Cited by 48
“Another line of work (Sun et al., 2019; Zhang et al., 2019; Peters et al., 2019) focuses on leveraging the readily available structured KBs to further complement the knowledge possessed by language models.”
BackgroundSummarization as Indirect Supervision for Relation Extraction
Lu, K., Hsu, I.-H., Zhou, W., Ma, M. D., & Chen, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.490 · Cited by 70
Debiasing knowledge graph embeddings
Fisher, J., Mittal, A., Palfrey, D., & Christodoulopoulos, C. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.595 · Cited by 64
“Alternatively, if graph embed-dings are used as input to a Transformer (Vaswani et al., 2017) encoder as in (Peters et al., 2019), the same effects on coreference resolution, entity linking and other downstream task as have been observed with word embeddings will re-occur.”
BackgroundMethodZhang, T., Cai, Z., Wang, C., Qiu, M., Yang, B., & He, X. (2021) · doi:10.18653/v1/2021.acl-long.457 · Cited by 61
“KEPLMs: We employ two SOTA KEPLMs continually pre-trained on our medical corpora as our baseline models, including ERNIE-THU (Zhang et al., 2019) and KnowBERT (Peters et al., 2019).”
InfluentialZhou, Y., Shen, T., Geng, X., Long, G., & Jiang, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.183 · Cited by 113
Embedding Refinement Framework for Targeted Aspect-Based Sentiment Analysis
Liang, B., Yin, R., Du, J., Gui, L., He, Y., Yang, M., & Xu, R. (2021) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2021.3071388 · Cited by 47
“Word embeddings, which are usually learned from contextual semantic information, play an important role in many NLP tasks [12], [13], [14], [15], [16], [17], [18], [19], [20].”
BackgroundE-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commercePreprint
Zhang, D., Yuan, Z., Liu, Y., Zhuang, F., Chen, H., & Xiong, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.02835 · Cited by 55
“Knowledge Enhanced PLMs Recently, to enable PLMs with world knowledge, several attempts (Wang et al. 2019; Peters et al. 2019; Zhang et al. 2019; Liu et al. 2020; Wang et al. 2020) have been made to inject knowledge into BERT leveraging Knowledge Graphs (KGs).”
BackgroundUnsupervised Pre-training for Biomedical Question AnsweringPreprint
Kommaraju, V., Gunasekaran, K. P., Li, K., Bansal, T., McCallum, A., Williams, I., & Istrate, A.-M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.12952 · Cited by 51
“Recently, some pre-training approaches have been proposed to incorporate factual knowledge into pre-trained models [16,29] which also require extracting such named entities from text.”
BackgroundLi, X., Henriksson, A., Duneld, M., Nouri, J., & Wu, Y. (2023) · Future Internet · doi:10.3390/fi16010012 · Cited by 52
“Another approach involves a deep integration through knowledge attention and a recontextualization mechanism [48].”
BackgroundCiatto, G., Sabbatini, F., Agiollo, A., Magnini, M., & Omicini, A. (2024) · ACM Computing Surveys · doi:10.1145/3645103 · Cited by 65
DKPLM: Decomposable Knowledge-Enhanced Pre-trained Language Model for Natural Language Understanding
Zhang, T., Wang, C., Hu, N., Qiu, M., Tang, C., He, X., & Huang, J. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21425 · Cited by 45
A survey on knowledge-enhanced multimodal learning
Lymperaiou, M., & Stamou, G. B. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10825-z · Cited by 59
“, 2020), and models such as KnowBert (Peters et al., 2019), E-BERT (Poerner et al.”
BackgroundMethodGAP: A novel Generative context-Aware Prompt-tuning method for relation extraction
Chen, Z., Li, Z., Zeng, Y., Zhang, C., & Ma, H. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.123478 · Cited by 52
Highly Parallel Autoregressive Entity Linking with Discriminative Correction
Cao, N. D., Aziz, W. F., & Titov, I. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.604 · Cited by 56
KALA: Knowledge-Augmented Language Model Adaptation
Kang, M., Baek, J., & Hwang, S. J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.379 · Cited by 36
Diao, S., Xu, T., Xu, R., Wang, J., & Zhang, T. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.280 · Cited by 78
mLUKE: The Power of Entity Representations in Multilingual Pretrained Language Models
Ri, R., Yamada, I., & Tsuruoka, Y. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.505 · Cited by 62
Wang, Q., Xu, H., Wen, Z., Liang, B., Yang, M., Qin, B., & Xu, R. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3333200 · Cited by 53
“Due to its usefulness and availability, researchers proposed several symbolic knowledge integrated models [60], [61], [62], which combine symbolic knowledge with deep learning technology, to address the challenges of knowledge learning and model interpretability in various tasks.”
BackgroundPre-training Text-to-Text Transformers for Concept-centric Common SensePreprint
Zhou, W., Lee, D., Selvam, R. K., Lee, S., & Ren, X. (2021) · arXiv (Cornell University) · Cited by 43
ESCOXLM-R: Multilingual Taxonomy-driven Pre-training for the Job Market Domain
Zhang, M., van der Goot, R., & Plank, B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.662 · Cited by 65
What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge
Richardson, K., & Sabharwal, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00331 · Cited by 48
“…explicitly, could serve as a cheaper alternative to recent attempts to build architectures that explicitly incorporate structured knowledge (Peters et al., 2019); we see many areas where our inoculation strategy could be improved for such purposes, including having more complex loss…”
BackgroundMethodMention Memory: incorporating textual knowledge into Transformers through entity mention attentionPreprint
de Jong, M., Zemlyanskiy, Y., FitzGerald, N., Sha, F., & Cohen, W. W. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.06176 · Cited by 53
“KnowBERT (Peters et al., 2019) is similar to EaE, but with entity embeddings generated from a KB instead of trained end-to-end with a text corpus.”
MethodCalibrating Factual Knowledge in Pretrained Language Models
Dong, Q., Dai, D., Song, Y., Xu, J., Sui, Z., & Li, L. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.438 · Cited by 39
Framing Word Sense Disambiguation as a Multi-Label Problem for Model-Agnostic Knowledge Integration
Conia, S., & Navigli, R. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.286 · Cited by 57
Local-to-global GCN with knowledge-aware representation for distantly supervised relation extraction
Huang, W., Mao, Y., Liu, Y., Yang, Z., & Long, J. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107565 · Cited by 39
SpellBERT: A Lightweight Pretrained Model for Chinese Spelling Check
Ji, T., Yan, H. R., & Qiu, X. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.287 · Cited by 60
“It has been shown that external information can be better integrated into BERT by pre-training alike tasks (Peters et al., 2019; Zhang et al., 2019; Sun et al., 2020; Ma et al., 2020).”
BackgroundAP-BERT: enhanced pre-trained model through average pooling
Zhao, S., Zhang, T., Hu, M., Chang, W., & You, F. (2022) · Applied Intelligence · doi:10.1007/s10489-022-03190-3 · Cited by 42
“And LIBERT [28], KnowBERT [29], K-BERT [30], mBERT [31].”
MethodAn Overview of Vision Transformers for Image Processing: A Survey
Kameswari, C. S., Kavitha, J., Reddy, T. S., Chinthaguntla, B., Jagatheesaperumal, S. K., Gaftandzhieva, S. N., & Doneva, R. Z. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0140830 · Cited by 42
Probing Linguistic Features of Sentence-Level Representations in Relation Extraction
Alt, C., Gabryszak, A., & Hennig, L. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.140 · Cited by 46
CHOLAN: A Modular Approach for Neural Entity Linking on Wikipedia and Wikidata
Ravi, M. P. K., Singh, K., Mulang, I. O., Shekarpour, S., Hoffart, J., & Lehmann, J. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.40 · Cited by 38
Syntax-Enhanced Pre-trained Model
Xu, Z., Guo, D., Tang, D., Su, Q., Shou, L., Gong, M., Zhong, W., Quan, X., Jiang, D., & Duan, N. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.420 · Cited by 59
“, 2019), KnowBERT (Peters et al., 2019), WKLM (Xiong et al.”
BackgroundMethodInfluentialKnowledge Prompting in Pre-trained Language Model for Natural Language Understanding
Wang, J., Huang, W., Qiu, M., Shi, Q., Wang, H., Li, X., & Gao, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.207 · Cited by 53
Multi-Grained Multimodal Interaction Network for Entity Linking
Luo, P., Xu, T., Wu, S., Zhu, C., Xu, L., & Chen, E. (2023) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3580305.3599439 · Cited by 37
“Motivated by the popularity of Transformer [32], Peters et al. [28] designed a projection layer over mention spans, and recontextualized these spans with cross-attention to link entity as well as integrate knowledge into BERT [9].”
MethodLarge Language Model Enhanced Knowledge Representation Learning: A Survey
Wang, X., Chen, Z., Wang, H., Leong Hou, U., Li, Z., & Guo, W. (2025) · Data Science and Engineering · doi:10.1007/s41019-025-00285-y · Cited by 48
“Overcoming these obstacles often calls for specialized modeling architectures and benchmarking strategies that explicitly measure performance in dynamic, multimodal contexts.”
Constructing Taxonomies from Pretrained Language Models
Chen, C., Lin, K., & Klein, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.373 · Cited by 48
“Tax-onomies are also used as a resource for building knowledge and systematicity into neural models (Peters et al., 2019; Geiger et al., 2020; Talmor et al., 2020).”
MethodBenchmark and Best Practices for Biomedical Knowledge Graph Embeddings
Chang, D., Balažević, I., Allen, C., Chawla, D. G., Brandt, C., & Taylor, A. (2020) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.18653/v1/2020.bionlp-1.18 · Cited by 63
Verlinden, S., Zaporojets, K., Deleu, J., Demeester, T., & Develder, C. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.171 · Cited by 45
Relation Extraction as Open-book Examination: Retrieval-enhanced Prompt Tuning
Chen, X., Li, L., Zhang, N., Tan, C., Huang, F., Si, L., & Chen, H. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531746 · Cited by 50
“We leverage full trainsets to conduct experiments and compare with several knowledge-enhanced models, including SpanBERT [22], KnowBERT [30], LUKE [38], andMTB [4], which are strong baselines for our method.”
MethodMeasuring Social Bias in Knowledge Graph EmbeddingsPreprint
Fisher, J., Palfrey, D., Christodoulopoulos, C., & Mittal, A. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.02761 · Cited by 38
“…first step in utilizing knowledge graphs for a range of tasks, from missing link prediction, (Bordes et al., 2013; Trouillon et al., 2016), to more recent methods integrating learned embeddings into ∗Corresponding author language models, (Zhang et al., 2019; IV et al., 2019; Peters et al., 2019).”
BackgroundMethodInfluentialKnowledge grounded medical dialogue generation using augmented graphs
Varshney, D., Zafar, A., Behera, N. K., & Ekbal, A. (2023) · Scientific Reports · doi:10.1038/s41598-023-29213-8 · Cited by 55
“There are also findings Peters et al. 37 , He et al. 38 that focused on inserting prior knowledge by using entity and relation embedding into deep neural language models.”
BackgroundWhat Are You Trying to Do? Semantic Typing of Event Processes
Chen, M., Zhang, H., Wang, H., & Roth, D. (2020) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2020.conll-1.43 · Cited by 38
Entity-aware Transformers for Entity Search
Gerritse, E. J., Hasibi, F., & de Vries, A. P. (2022) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3477495.3531971 · Cited by 48
“In KnowBert [34], contextual word embeddings are also enhanced by integrating information about entities.”
BackgroundLu, Y., Lu, H., Fu, G., & Liu, Q. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04223 · Cited by 41
Bonfigli, A., Bacco, L., Merone, M., & Dell’Orletta⋄, F. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.103003 · Cited by 36
Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies
Pergola, G., Kochkina, E., Gui, L., Liakata, M., & He, Y. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.169 · Cited by 50
A Custom Word Embedding Model for Clustering of Maintenance Records
Bhardwaj, A. S., Deep, A., Veeramani, D., & Zhou, S. (2021) · IEEE Transactions on Industrial Informatics · doi:10.1109/tii.2021.3079521 · Cited by 32
Drop Redundant, Shrink Irrelevant: Selective Knowledge Injection for Language Pretraining
Zhang, N., Deng, S., Cheng, X., Chen, X., Zhang, Y., Zhang, W., & Chen, H. (2021) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/552 · Cited by 37
KEBLM: Knowledge-Enhanced Biomedical Language Models
Lai, T., Zhai, C., & Ji, H. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104392 · Cited by 43
Contextualized Representations Using Textual Encyclopedic KnowledgePreprint
Joshi, M., Lee, K., Luan, Y., & Toutanova, K. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.12006 · Cited by 34
“Recent work focuses on refining pretrained contextualized representations using entity or triple embeddings from structured knowledge graphs (KGs) (Peters et al., 2019; Yang et al., 2019; Zhang et al., 2019).”
BackgroundInfluentialKnowledge Perceived Multi-modal Pretraining in E-commerce
Zhu, Y., Zhao, H., Zhang, W., Ye, G., Chen, H., Zhang, N., & Chen, H. (2021) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3474085.3475648 · Cited by 30
Panchendrarajan, R., & Zubiaga, A. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.124097 · Cited by 37
Structure-inducing pre-training
McDermott, M. B. A., Yap, B., Szolovits, P., & Žitnik, M. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00647-z · Cited by 33
“Their usage of automatically attending over entities within their language model (without explicit constraints on those matches) is motivated by [23]’s work in [23] and has similarities to Know-BERT [77].”
BackgroundMethodInfluentialLi, R., Jiang, Z., Wang, L., Lu, X., Zhao, M., & Chen, D. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106936 · Cited by 28
LMMS reloaded: Transformer-based sense embeddings for disambiguation and beyond
Loureiro, D., Jorge, A., & Camacho-Collados, J. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2022.103661 · Cited by 43
Unified Semantic Typing with Meaningful Label Inference
Huang, J. Y., Li, B., Xu, J., & Chen, M. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.190 · Cited by 40
Huang, Z., Bonab, H., Sarwar, S. M., Rahimi, R., & Allan, J. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482452 · Cited by 34
“There has been a number of efforts to inject structured world knowledge into unsupervised pretraining and contextualized representations [14, 19, 20, 32, 44, 51].”
An Embarrassingly Simple Model for Dialogue Relation Extraction
Xue, F., Sun, A., Zhang, H., Ni, J., & Chng, E. S. (2022) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp43922.2022.9747486 · Cited by 27
“SimpleRE outperforms all baselines including KnowBERT [17] on both datasets.”
MethodExplaining Relationships Between Scientific Documents
Luu, K., Wu, X., Koncel-Kedziorski, R., Lo, K., Cachola, I. A., & Smith, N. A. (2021) · doi:10.18653/v1/2021.acl-long.166 · Cited by 57
“Combining knowledge graphs with language models and generation is an active research area that has shown promise in other domains (Bosselut et al., 2019; Koncel-Kedziorski et al., 2019; Peters et al., 2019).”
BackgroundSelf-citationReasonBERT: Pre-trained to Reason with Distant Supervision
Deng, X., Su, Y., Lees, A., Wu, Y., Yu, C., & Sun, H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.494 · Cited by 35
“More recently, self-supervised pre-training has also shown promising results on modalities other than plain text, such as tables (Herzig et al., 2020; Deng et al., 2020; Iida et al., 2021), knowledge bases (Zhang et al., 2019; Peters et al., 2019) and image-text (Su et al., 2020).”
BackgroundPre-train or Annotate? Domain Adaptation with a Constrained Budget
Bai, F., Ritter, A., & Xu, W. H. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.409 · Cited by 43
“We use the same five random seeds as Peters et al. (2019) for all our experiments in §5 and §6.”
MethodKLMo: Knowledge Graph Enhanced Pretrained Language Model with Fine-Grained Relationships
He, L., Zheng, S., Yang, T., & Zhang, F. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.384 · Cited by 32
“…incorporate KG information into PLMs (Devlin et al., 2018; Radford et al.) to enhance language representations, such as ERNIETHU (Zhang et al., 2019), WKLM (Xiong et al., 2019), KEPLER (Wang et al., 2019), KnowBERT (Peters et al., 2019), BERT-MK (He et al., 2019) and KALM (Rosset et al., 2020), .”
Evaluation of Fake News Detection with Knowledge-Enhanced Language Models
Whitehouse, C., Weyde, T., Madhyastha, P., & Komninos, N. (2022) · Proceedings of the International AAAI Conference on Web and Social Media · doi:10.1609/icwsm.v16i1.19400 · Cited by 38
Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT
Meng, Z., Liu, F., Clark, T. H., Shareghi, E., & Collier, N. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.383 · Cited by 38
“…on a computationally expensive joint training of an underlying masked language model (MLM) along with a knowledge-infusion objective function to minimize the risk of catastrophic forgetting (Xiong et al., 2019; Zhang et al., 2019; Wang et al., 2021, 2020; Peters et al., 2019; Yuan et al., 2021).”
BackgroundMethodExposing Shallow Heuristics of Relation Extraction Models with Challenge Data
Rosenman, S., Jacovi, A., & Goldberg, Y. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.302 · Cited by 34
CausalBERT: Injecting Causal Knowledge Into Pre-trained Models with Minimal SupervisionPreprint
Li, Z., Ding, X., Liao, K., Qin, B., & Liu, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.09852 · Cited by 33
“KnowBERT (Peters et al., 2019) incorporates knowledge bases into BERT using Knowledge attention and recontextualization, where the knowledge comes from synset-synset and lemma-lemma relationships in WordNet.”
MethodScientific Language Models for Biomedical Knowledge Base Completion: An Empirical StudyPreprint
Nadkarni, R., Wadden, D., Beltagy, I., Smith, N. A., Hajishirzi, H., & Hope, T. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2106.09700 · Cited by 31
Self-citationProtein Representation Learning via Knowledge Enhanced Primary Structure ModelingPreprint
Zhou, H.-Y., Fu, Y., Zhang, Z., Bian, C., & Yu, Y. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.26.525795 · Cited by 38
“However, as pointed out by (Peters et al., 2019; Zhang et al., 2019; Sun et al., 2020; Wang et al., 2021), pre-trained language models often suffer from a lack of factual knowledge.”
BackgroundAdd-Vit: CNN-Transformer Hybrid Architecture for Small Data Paradigm Processing
Chen, J., Wu, P., Zhang, X., Xu, R., & Liang, J. (2024) · Neural Processing Letters · doi:10.1007/s11063-024-11643-8 · Cited by 28
Metadata
- Authors
- Peters, M. E., Neumann, M., Logan, R., Schwartz, R., Joshi, V., Singh, S., & Smith, N. A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- University of California, Irvine (US)
- University of Washington (US)
- Corresponding author
- Singh, Sameer
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 43-54
- OpenAlex
- W2970986510
- Cited by
- 661
- References
- 72
- Citations by year
- 2026: 21 · 2025: 54 · 2024: 81 · 2023: 125 · 2022: 115 · 2021: 160 · 2020: 98 · 2019: 6
- FWCI
- 36.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- National Science Foundation (1730158, 1817183)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- contextualized word representations, word embeddings, knowledge graphs, word sense disambiguation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.