Knowledge Enhanced Contextual Word Representations
Open accessThe DOI matches its Crossref record; 71 of the 72 references it cites were found in a registry or an index.
Abstract
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A. Smith. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“XLM-R achieves state-of-the-arts results on multiple cross-lingual benchmarks, including XNLI, MLQA, and NER.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [62]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“We compute M m scores using the prior and dot product between the entity-span vectors and entity embeddings, with a two-layer MLP (100 hidden dimensions).”
BackgroundMethodInfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 4×
“, 2018), and BERT (Devlin et al., 2019) have significantly improved the state of the art for a wide range of NLP tasks.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 4×
“, 2018), and BERT (Devlin et al., 2019) have significantly improved the state of the art for a wide range of NLP tasks.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Pretrained word representations Initial work learning word vectors focused on static word embeddings using multi-task learning objectives (Collobert and Weston, 2008) or corpus level cooccurence statistics (Mikolov et al., 2013a; Pennington et al., 2014).”
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 3×
“Pretrained word representations Initial work learning word vectors focused on static word embeddings using multi-task learning objectives (Collobert and Weston, 2008) or corpus level co-occurence statistics (Mikolov et al., 2013a; Pen-nington et al., 2014).”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 3×
“The entity embeddings use a skipgram like objective (Mikolov et al., 2013b) to learn 300-dimensional embeddings of Wikipedia”
BackgroundMethodWordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 1×
“We experiment with integrating both WordNet (Miller, 1995) and Wikipedia, thus explicitly adding word sense knowledge and facts about named entities (including those unseen at training time).”
MethodDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 6×
“Recently the field has shifted toward learning context-sensitive embeddings (Dai and Le, 2015; Peters et al., 2018; Devlin et al., 2019).”
BackgroundMethodInfluentialSelf-citation(2013) · Cited by 5,726
Research works printed but not linked: 5
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013b. Distributed representa- tions of words and phrases and their compositional- ity. In NeurIPS.
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL-HLT.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL.
- Bishan Yang and Tom Michael Mitchell. 2017. Lever- aging knowledge bases in LSTMs for improving ma- chine reading. In ACL.
- Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: En- hanced language representation with informative en- tities. In ACL.
Cited by
Items 401–500 of 705
Generating Accurate Electronic Health Assessment from Medical Graph
Yang, Z., & Yu, H. Q. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.336 · Cited by 10
“Even though pre-trained language models are able to answer queries structured as “fillin-the-blank” cloze statements, and Petroni et al. (2019) have shown that factual relational knowledge already presents within these pretrained models, however, Poerner et al. (2019) have demonstrated that these pre-trained language…”
BackgroundInfluentialInstilling Type Knowledge in Language Models via Multi-Task QA
Li, S. X., Sridhar, M., Prakash, C. S., Cao, J., Hamza, W., & McAuley, J. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.45 · Cited by 9
Syntax Controlled Knowledge Graph-to-Text Generation with Order and Semantic Consistency
Liu, J., Fan, C., Fengyu, Z., & Xu, H. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.95 · Cited by 9
Relation Extraction Based on Prompt Information and Feature Reuse
Feng, P., Zhang, X., Zhao, J. M., Wang, Y., & Huang, B. (2023) · Data Intelligence · doi:10.1162/dint_a_00192 · Cited by 6
“[19] embed the contents of multiple knowledge bases into BERT, and the knowledge-enhanced BERT also achieves better results in downstream tasks such as relation extraction.”
BackgroundA Joint Domain-Specific Pre-Training Method Based on Data Enhancement
Gan, Y., Lu, G., Su, Z., Lei, W., Zhou, J., Jiang, J., & Chen, D. (2023) · Applied Sciences · doi:10.3390/app13074115 · Cited by 7
Research on Relation Classification Tasks Based on Cybersecurity Text
Shi, Z., Li, H., Zhao, D., & Pan, C. (2023) · Mathematics · doi:10.3390/math11122598 · Cited by 6
Commonsense Knowledge Transfer for Pre-trained Language Models
Zhou, W., Bras, R. J. L., & Choi, Y. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.368 · Cited by 12
What Has Been Enhanced in my Knowledge-Enhanced Language Model?
Hou, Y., Fu, G., & Sachan, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.102 · Cited by 6
“KI in LMs can be implemented by aligning phrases in text to entities (Peters et al., 2019; Zhang et al., 2019) or triples (Liu et al., 2020; Wang et al., 2021a) and incorporating corresponding entity or the triple embeddings in the LM. KI methods also include modifications to the Trans-former…”
MethodMining Knowledge for Natural Language Inference from Wikipedia Categories
Chen, M., Chu, Z., Stratos, K., & Gimpel, K. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.313 · Cited by 6
Knowledge-Enhanced Evidence Retrieval for Counterargument Generation
Jo, Y., Yoo, H., Bak, J., Oh, A. H., Reed, C., & Hovy, E. H. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.264 · Cited by 6
Multi-Faceted Knowledge-Driven Pre-Training for Product Representation Learning
Zhang, D., Liu, Y., Yuan, Z., Fu, Y., Chen, H., & Xiong, H. (2022) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3200921 · Cited by 6
“On the other hand, to enable PLMs with world knowledge, several attempts [18], [26], [32], [33], [40] have been made to inject knowledge from external KGs into BERT.”
BackgroundVerb Classification Across Languages
Majewska, O., & Korhonen, A. (2022) · Annual Review of Linguistics · doi:10.1146/annurev-linguistics-030521-043632 · Cited by 12
Gold standard, multi-genre dataset for named entity recognition and linking
Olewniczak, S., & Szymański, J. (2025) · Scientific Data · doi:10.1038/s41597-025-05274-4 · Cited by 6
“Although there are some EL systems that perform MD and ED jointly ( 14,15,22 among others) they cannot be used reliably in our case.”
CMCN: Chinese medical concept normalization using continual learning and knowledge-enhanced
Han, P., Li, X., Zhang, Z., Zhong, Y., Liang, G., Hua, Y., & Li, X. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102965 · Cited by 5
Shi, X., Xia, Z., Li, Y., Wang, X., & Niu, Y. (2024) · Knowledge and Information Systems · doi:10.1007/s10115-024-02235-y · Cited by 5
“KnowBERT [33] uses a knowledge attention and recontextualization component that integrates entity linkers and entity embeddings into a pre-trained model BERT.”
Swarup, A., Bhandarkar, A., Dizon-Paradis, O. P., Wilson, R., & Woodard, D. L. (2024) · IEEE Access · doi:10.1109/access.2024.3494737 · Cited by 6
“Subsequently, algorithms started incorporating more advanced forms of knowledge by mapping entities to their KB counterparts [7], [40]–[42].”
A prompt tuning method based on relation graphs for few-shot relation extraction
Zhang, Z., Yang, Y., & Chen, B. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107214 · Cited by 8
The long COVID research literature
Porter, A. L., Markley, M., & Newman, N. C. (2023) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2023.1149091 · Cited by 6
“, word2vec and BERT) have advanced notably (c.f., Ethayarajh et al., 2019; Peters et al., 2019; Reimers et al., 2019).”
BackgroundSelective UMLS knowledge infusion for biomedical question answering
Park, H., Son, J., Min, J., & Choi, J. (2023) · Scientific Reports · doi:10.1038/s41598-023-41423-8 · Cited by 11
A Knowledge Enhanced Pre-Training Model for Chinese Weibo Sentiment Analysis
Yao, L. (2024) · International Conference on Cloud Computing and Big Data Analytics (ICCCBDA) · doi:10.1109/icccbda61447.2024.10569319 · Cited by 5
“This study draws inspiration from ENIRE [24] and Knowbert [25] to propose a method for integrating sentiment-aware knowledge within transformer encoder layers.”
MethodKA-NER: Knowledge Augmented Named Entity Recognition
Nie, B., Li, C., & Wang, H. (2021) · Communications in computer and information science · doi:10.1007/978-981-16-6471-7_5 · Cited by 6
Bruera, A., & Poesio, M. (2022) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2022.796793 · Cited by 8
“…distributional models, the lack of precise entity-level information; and for knowledge bases, their structural rigidity and their difficult integration with generic linguistic knowledge, as well as their costly creation and maintenance (Peters et al., 2019; Sun et al., 2020b; Yamada et al., 2020b).”
BackgroundA semantics-aware approach for multilingual natural language inference
Le-Hong, P., & Cambria, E. (2023) · Language Resources and Evaluation · doi:10.1007/s10579-023-09635-6 · Cited by 6
Temporal Natural Language Inference: Evidence-Based Evaluation of Temporal Text Validity
Hosokawa, T., Jatowt, A., & Sugiyama, K. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28244-7_28 · Cited by 11
EntityCS: Improving Zero-Shot Cross-lingual Transfer with Entity-Centric Code Switching
Whitehouse, C., Christopoulou, F., & Iacobacci, I. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.499 · Cited by 15
Driving Context into Text-to-Text Privatization
Arnold, S., Yesilbas, D., & Weinzierl, S. (2023) · Workshop on Trustworthy Natural Language Processing (TrustNLP) · doi:10.18653/v1/2023.trustnlp-1.2 · Cited by 14
From Fake to Hyperpartisan News Detection Using Domain Adaptation
Smădu, R.-A., Echim, S.-V., Cercel, D.-C., Marin, I., & Pop, F. (2023) · International conference Recent advances in natural language processing · doi:10.26615/978-954-452-092-2_117 · Cited by 5
Contrastive Language-knowledge Graph Pre-training
Yuan, X.-W., Liu, K., & Wang, Y.-Q. (2024) · ACM Trans. Asian Low Resour. Lang. Inf. Process. · doi:10.1145/3644820 · Cited by 5
“They mostly utilize a pre-trained embedding model like TransE [1] for KG, necessitating additional alignment for heterogeneous embedding spaces and inevitably introducing biases [29, 32, 60].”
MethodPadhi, T., Kursuncu, U., Kumar, Y., Shalin, V. L., & Fronczek, L. P. (2024) · IEEE International Conference on Big Data (BigData) · doi:10.1109/bigdata62323.2024.10825252 · Cited by 8
“…proposed to utilize KGs to enhance these models for various tasks, including Language Modeling [34], [42], [44], [85], [91], Natural Language Inference (NLI) [53], Language Generation [56], [87], leading to developments of E-BERT [79], ERICA [57] ERNIE [90], ERNIE-NLI [4], and LUKE [81].”
World Knowledge Representation
Liu, Z., Lin, Y., & Sun, M. (2020) · Springer eBooks · doi:10.1007/978-981-15-5573-2_7 · Cited by 6
Bayes-enhanced Lifelong Attention Networks for Sentiment Classification
Wang, H., Wang, S., Mazumder, S., Liu, B., Yang, Y. T., & Li, T. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.50 · Cited by 6
“Knowledge-enhanced attention models have been introduced in (Chen et al., 2017; Zhou et al., 2018; Yang et al., 2019; Peters et al., 2019; Chen et al., 2019; Huang et al., 2020).”
DESCGEN: A Distantly Supervised Datasetfor Generating Entity Descriptions
Shi, W., Joshi, M., & Zettlemoyer, L. (2021) · doi:10.18653/v1/2021.acl-long.35 · Cited by 5
“Entity knowledge has been shown to play an important role in various applications including language modeling (Peters et al., 2019), open-domain question answering (Xu et al., 2016), and dialogue generation (Qin et al., 2019).”
BackgroundExplainable Natural Language Inference via Identifying Important Rationales
Yang, Z., Dong, S., & Hu, J. (2022) · IEEE Transactions on Artificial Intelligence · doi:10.1109/tai.2022.3178391 · Cited by 6
“However, there are also some pretrained models combined with external knowledge, such as ERNIE [11], Know-BERT [12], and K-BERT [13].”
BackgroundA marker collaborating model for entity and relation extraction
Yi-zhao, W., Chen, Y., Qin, Y., Huang, R., & Tang, R. (2022) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2022.08.038 · Cited by 7
Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling
Berend, G. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.876 · Cited by 16
Unsupervised Improvement of Factual Knowledge in Language Models
Sadeq, N., Kang, B., Lamba, P., & McAuley, J. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.215 · Cited by 6
“However, a traditional MLM objective may not be optimal for knowledge-intensive tasks (Peters et al., 2019).”
Benchmarking and Analyzing Generative Data for Visual Recognition
Li, B., Liu, H., Chen, L., Lee, Y. J., Li, C., & Liu, Z. (2025) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2025.3572476 · Cited by 6
“In NLP, several works augment large language models with external knowledge [5], [20], [26], [32], [37], [46], [64].”
Data-driven models and computational tools for neurolinguistics: a language technology perspectivePreprint
Artemova, E., Bakarov, A., Artemov, A., Burnaev, E. V., & Sharaev, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2003.10540 · Cited by 7
“, 2018) and language models, such as KnowBERT (Peters et al., 2019).”
BackgroundMethodImproving Relation Extraction by Leveraging Knowledge Graph Link PredictionPreprint
Stoica, G., Platanios, E. A., & Póczos, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2012.04812 · Cited by 5
Knowledge Based Multilingual Language ModelPreprint
Liu, L., Li, X., He, R., Bing, L., Joty, S., & Si, L. (2021) · arXiv (Cornell University) · Cited by 5
Gómez-Pérez, J. M., García-Silva, A., Leone, R., Albani, M., Fontaine, M., Poncet, C., Summerer, L., Donati, A., Roma, I., & Scaglioni, S. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.03640 · Cited by 5
“Finally, we may also opt for a hybrid approach, e.g. to inject structured knowledge about entities and the relations between them from a pre-existing knowledge base into a pre-trained language model (Peters et al., 2019; Wang et al., 2021b), enabling domain adaptation at a limited cost.”
BackgroundPromptSED: An evolving topic-enhanced prompting framework for incremental social event detection
Yu, X., Ren, J., Jiang, L., Peng, H., Hao, Z., Sun, L., Peng, K., Zhu, L., & Yu, P. S. (2025) · Neural Networks · doi:10.1016/j.neunet.2025.107772 · Cited by 6
Deep Learning Strategy to Recognize Kannada Named Entities
Pushpalatha, E. A. M. (2021) · Türk bilgisayar ve matematik eğitimi dergisi · doi:10.17762/turcomat.v12i10.5387 · Cited by 4
Rawal, K. V., Vinay, A., Krishna, M. H., Singh, N., Almusawi, M., & Alzobidy, S. A. M. (2023) · IEEE Uttar Pradesh Section International Conference on Electrical, Electronics and Computer Engineering (UPCON) · doi:10.1109/upcon59197.2023.10434672 · Cited by 4
“However, it takes a lot of time and money to produce high-quality annotated data [14].”
BackgroundTinarrage, R., Ennes, H., Resck, L., Gomes, L. T., Ponciano, J. R., & Poco, J. (2025) · Artificial Intelligence and Law · doi:10.1007/s10506-025-09458-6 · Cited by 5
Contrastive learning-based few-shot relation extraction with open-book datastore
Gong, W., & Zhou, Q. (2024) · Applied Soft Computing · doi:10.1016/j.asoc.2024.112302 · Cited by 4
Finding NeMo: Negative-Mined Mosaic Augmentation for Referring Image Segmentation
Ha, S., Kim, C., Kim, D., Lee, J., Lee, S., & Lee, J. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72890-7_8 · Cited by 9
Integrating Lexical Information into Entity Neighbourhood Representations for Relation Prediction
Wood, I., Johnson, M., & Wan, S. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.268 · Cited by 8
“: (Cohen et al., 2020; Wang et al., 2019; Peters et al., 2019)2) including the creation of many”
BackgroundWhen Few-Shot Learning Meets Large-Scale Knowledge-Enhanced Pre-training: Alibaba at FewCLUE
Xu, Z., Wang, C., Li, P., Li, Y., Wang, M., Hou, B., Qiu, M., Tang, C., & Huang, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-88483-3_34 · Cited by 7
“The first approach is to inject factual knowledge from Knowledge Graphs (KGs) into PLMs. PLMs pre-trained on large-scale unstructured corpora pay little attention to semantic information of important entity mentions and their relations expressed in texts.”
BackgroundMethodInfluentialAutoRAG: Grounding text and symbols
Schulz, T., Luttermann, M., & Möller, R. (2024) · KI - Künstliche Intelligenz · doi:10.1007/s13218-024-00850-z · Cited by 4
“Some approaches introduce special architecture changes to tightly steer generation using a knowledge base [6, 22, 41].”
Effective Sentence-Level Relation Extraction Model Using Entity-Centric Dependency Tree
Park, S., & Kim, H. (2022) · PeerJ Computer Science · doi:10.2139/ssrn.4272160 · Cited by 5
RelBERT: Embedding relations with language models
Ushio, A., Camacho-Collados, J., & Schockaert, S. (2025) · Artificial Intelligence · doi:10.1016/j.artint.2025.104359 · Cited by 5
Goodwill Hunting: Analyzing and Repurposing Off-the-Shelf Named Entity Linking Systems
Goel, K., Orr, L., Rajani, N. F., Vig, J., & Ré, C. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-industry.26 · Cited by 4
“As NEL models have direct impact on the success of downstream products (Peters et al., 2019), ∗ E-mail: kgoel@cs.stanford.edu all major technology companies deploy large-scale NEL systems; e.g., in Google Search, Apple Siri and Salesforce Einstein.”
BackgroundA Comparative Study of Pre-trained Encoders for Low-Resource Named Entity Recognition
Chen, Y., Mikkelsen, J., Binder, A., Alt, C., & Hennig, L. (2022) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2022.repl4nlp-1.6 · Cited by 4
Enhancing Natural Language Representation with Large-Scale Out-of-Domain Commonsense
Cui, W., & Chen, X. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.138 · Cited by 6
“These methods include (1) OK-Transformer, (2) KnowBERT (Peters et al., 2019), (3) using the original [CLS] token instead of the proposed knowledge token, and (4) abandoning the knowledge token and instead calculating the csemb of each verb phrase of the target sentence separately, and adding them to these verb phrases…”
BackgroundMethodInfluentialEnriching Contextualized Representations with Biomedical Ontologies: Extending KnowBert to UMLS
Piat, G., Semmar, N., Allauzen, A., Essafi, H., & Tourille, J. (2022) · Lecture notes in networks and systems · doi:10.1007/978-3-031-10464-0_52 · Cited by 4
“We follow the method described by Peters et al. [2] known as KnowBert to integrate knowledge derived from the UMLS Knowledge Base into a BERT-based language model.”
InfluentialLi, L., Fu, Y., Zhu, D., Li, X., Sun, Y., Ding, J., Wu, M., Cao, N., & Higgs, R. J. (2023) · Intelligent Automation & Soft Computing · doi:10.32604/iasc.2023.035257 · Cited by 4
Multi-feature Fusion for Relation Extraction using Entity Types and Word Dependencies
Zhang, P., Li, J., Chen, S., Zhang, J., & Tang, L. (2023) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2023.0140731 · Cited by 4
“3) KnowBERT [13]: To enhance text representations with structured knowledge, the knowledge-enhanced BERT (KnowBERT) incorporates multiple knowledge bases (KBs) into the BERT model and obtains knowledge-enhanced representations that can be used for a variety of downstream tasks.”
MethodInfluentialZhang, T., Xu, R., Wang, C., Duan, Z., Chen, C., Qiu, M., Cheng, D., He, X., & Qian, W. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.969 · Cited by 7
Oh, J., Iida, R., Kloetzer, J., & Torisawa, K. (2021) · doi:10.18653/v1/2021.acl-long.164 · Cited by 4
CoCoLM: COmplex COmmonsense Enhanced Language ModelPreprint
Yu, C., Zhang, H., Song, Y., & Ng, W. (2020) · arXiv (Cornell University) · Cited by 4
Focusing Knowledge-based Graph Argument Mining via Topic ModelingPreprint
Abels, P., Ahmadi, Z., Burkhardt, S., Schiller, B., Gurevych, I., & Krämer, S. (2021) · TUbilio (Technical University of Darmstadt) · doi:10.48550/arxiv.2102.02086 · Cited by 4
“…approaches build upon pre-trained language models, other approaches enhance language models, more specifically BERT, with external knowledge (Guan et al. 2020; He et al. 2019; Levine et al. 2019; Zhang et al. 2019; Xiong et al. 2019; Peters et al. 2019; Wang et al. 2020; Lauscher et al. 2019).”
BackgroundResultTowards Multi-Sense Cross-Lingual Alignment of Contextual EmbeddingsPreprint
Liu, L., Nguyen, T. H., Joty, S., Bing, L., & Si, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.06459 · Cited by 4
“Peters et al. (2019) embed WordNet knowledge into BERT with attention mechanism.”
MethodDoes Entity Abstraction Help Generative Transformers Reason?
Gontier, N., Reddy, S., & Pal, C. (2022) · Transactions on machine learning research · doi:10.48550/arxiv.2201.01787 · Cited by 5
“A flurry of recent work has also examined ways to augment TLMs with entities from external knowledge bases (Zhang et al., 2019; Peters et al., 2019; Févry et al., 2020; Verga et al., 2020).”
BackgroundEnriching large language models with semantic lexicons and analogies
Zervakis, G. (2023) · theses.fr (ABES) · doi:10.70675/4f4cd0e5z5987z422az813bzc6e431d1a918 · Cited by 7
ConceptFormer: Towards Efficient Use of Knowledge-Graph Embeddings in Large Language Models
Barmettler, J., Bernstein, A., & Rossetto, L. (2025) · arXiv · doi:10.48550/arxiv.2504.07624 · Cited by 3
“K-Bert [26], KnowBert [32], and KP-PLM [48] each propose new attention or injection layers, partially or fully overriding a model’s internal architecture.”
Zhang, W., & Chiong, R. (2025) · Applied Intelligence · doi:10.1007/s10489-024-06141-2 · Cited by 3
Topic-Guided Knowledge Graph Construction for Argument Mining
Li, W., Abels, P., Ahmadi, Z., Burkhardt, S., Schiller, B., Gurevych, I., & Krämer, S. (2021) · IEEE International Conference on Big Knowledge (ICBK) · doi:10.1109/ickg52313.2021.00049 · Cited by 3
Temporal Word Meaning Disambiguation using TimeLMs
Godbole, M. S., Dandavate, P., & Kane, A. (2022) · doi:10.18653/v1/2022.evonlp-1.8 · Cited by 3
XLORE 3: A Large-Scale Multilingual Knowledge Graph from Heterogeneous Wiki Knowledge Resources
Zeng, K., Jin, H., Lv, X., Zhu, F., Hou, L., Zhang, Y., Pang, F., Qi, Y., Liu, D., Li, J., et al. (2024) · ACM Transactions on Information Systems · doi:10.1145/3660521 · Cited by 4
“Nowadays, more and more knowledge-enhanced PLMs are emerging, integrating different kinds of knowledge in different ways and dealing with a variety of NLP tasks [39].”
When automated fact-checking meets argumentation: Unveiling fake news through argumentative evidence
Wang, X., Cabrio, E., & Villata, S. (2025) · Argument & Computation · doi:10.1177/19462174251330980 · Cited by 9
“47 However, we use KnowBert as the textual feature extractor (best-performing knowledge-enhanced LLM for FNC) instead of the BERT base model used in Ma et al. 24 A softmax layer is applied to the final embedding (textual embedding for KB , and concatenation of graph and textual embeddings for KGB ) to get the logits fo…”
InfluentialYang, X., Li, H., Zhang, J., Deng, Y., & Yuan, H. (2025) · Applied Intelligence · doi:10.1007/s10489-025-07011-1 · Cited by 3
“Representative efforts include K-BERT [ 37] and knowBERT [51], which inject KG triples or entities into Transformer-based models to enrich token representations, while ERNIE [ 34 ] incorporates entity-level knowledge into pretrained embeddings.”
Colon-Hernandez, P., Xin, Y., Lieberman, H. A., Havasi, C., Breazeal, C., & Chin, P. S. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.183 · Cited by 3
“, KnowBERT (Peters et al., 2019), Align-mask-select (Ye et al.”
BackgroundMultilevel Entity-Informed Business Relation Extraction
Khaldi, H., Benamara, F., Abdaoui, A., Aussenac-Gilles, N., & Kang, E.-B. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-80599-9_10 · Cited by 3
ReadOnce Transformers: Reusable Representations of Text for Transformers
Lin, S.-T., Sabharwal, A., & Khot, T. (2021) · doi:10.18653/v1/2021.acl-long.554 · Cited by 6
“We set the learning rate to 1e-5 for the weights initialized from the BART model, and to 1e-4 for randomly initialized newly added weights, which is shown beneficial in Peters et al. (2019).”
MethodRanjbar, N., & Zeinali, H. (2021) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2021.semeval-1.95 · Cited by 3
“There are several published works that used BERT for WSD task (Wiedemann et al., 2019; Du et al., 2019; Yap et al., 2020; Guo et al., 2020) and also some other papers that combined BERT and Word-Net for this task (Levine et al., 2020; Peters et al., 2019).”
MethodTrans-KBLSTM: An External Knowledge Enhanced Transformer BiLSTM Model for Tabular Reasoning
Varun, Y., Sharma, A., & Gupta, V. (2022) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2022.deelio-1.7 · Cited by 3
“Other approaches include, pretraining on external knowledge corpus to inject knowledge (Wang et al., 2021; Peters et al., 2019; Umair and Ferraro, 2021), better knowledge representations (Bauer et al.”
BackgroundMethodMetamorphic Testing of Relation Extraction Models
Yuhe, S., Ding, Z., Huang, H., Zou, S., & Jiang, M. (2023) · Algorithms · doi:10.3390/a16020102 · Cited by 4
“In light of its importance, various deep learning models have been developed for RE, including models based on attention mechanisms [6], models combining external information sources [7], and pretrained language models [8].”
MethodDMLM: Descriptive Masked Language Modeling
Barba, E., Campolungo, N., & Roberto Navigli, A. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.808 · Cited by 3
“…has proven to model both syntactic and semantic information (Rogers et al., 2020), its formulation provides no explicit ties to the real world (Peters et al., 2019; Zhang et al., 2019), a limitation that several works have tried to overcome through the injection of information coming from…”
BackgroundMethodSeeking Clozure: Robust Hypernym extraction from BERT with Anchored Prompts
Liu, C., Cohn, T., & Frermann, L. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.18 · Cited by 5
“…fundamental role, the automatic extraction of hypernym knowledge from large texts (Hearst, 1992; Roller et al., 2018) or pre-trained language models (PLMs) (Takeoka et al., 2021; Jain and Espinosa Anke, 2022), and its injection into NLP methods are active areas of research (Peters et al., 2019).”
BackgroundCLG: Contrastive Label Generation with Knowledge for Few-Shot Learning
Ma, H., Fan, B., Ng, B. K., & Lam, C. (2024) · Mathematics · doi:10.3390/math12030472 · Cited by 3
Quan, Z., Vong, C., Zeng, W., & Yang, W. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3364610 · Cited by 3
“In the machine learning and natural language processing community, existing works (e.g., CWE [4], MGE [5], JWE [6], KnowBert [23], MVP-BERT [24], KinyaBERT [25], Day-oBERT [26], and MarkBERT [27]) show that many natural language processing tasks can benefit from morphological knowledge [28], [29],…”
BackgroundA Knowledge-Injected Curriculum Pretraining Framework for Question Answering
Lin, X., Su, T., Huang, Z., Xue, S., Liu, H., & Chen, E. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645406 · Cited by 6
“As the pretrained LMs have shown its weakness on knowledge-based tasks [21, 23], researchers have tried many efforts to enhance LMs with knowledge from KGs, including the explicit methods [21, 26, 31, 50] and implicit methods [9, 20, 34, 36, 37, 42].”
BackgroundFine-Grained Scene Image Classification with Modality-Agnostic Adapter
Wang, Y., Zhou, Z., Du, X., Wu, X., Zheng, Y., & Jin, C. (2024) · IEEE International Conference on Multimedia and Expo (ICME) · doi:10.1109/icme57554.2024.10687875 · Cited by 3
“For better comparison with previous models, we first use the output of ViT [16] as the global visual embedding and KnowBert [23] as the text embedding, without using local visual feature.”
BackgroundMethodInfluentialZhu, E., Yu, Z., Huang, Y., Gao, S., & Xian, Y. (2024) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02367-1 · Cited by 3
“…by knowledge graphs [49, 53, 54], entities [49, 55], n-grams [56], knowledge embedding [57], synonym and hyponym-hypernym relations in WordNet [19], word-supersense knowledge [58], and knowledge bases [59–61] can be injected into PLMs by feeding knowledge inputs and designing new objectives.”
BackgroundPandit, O., Denis, P., & Ralaivola, L. (2020) · HAL · Cited by 3
Improving Knowledge Graph Representation Learning by Structure Contextual Pre-training
Ye, G., Zhang, W., Bi, Z., Wong, C. M., Hui, C., & Chen, H. (2021) · International Joint Conference on Knowledge Graphs (IJCKG) · doi:10.1145/3502223.3502244 · Cited by 3
“Later studies have proposed variants of BERT combined with KG, like K-BERT [11], KG-BERT [9], ERNIE [18] and KnowBert [12], paying attention to the combination of knowledge graph and pre-trained language model.”
MethodBoukkouri, H. E., Ferret, O., Lavergne, T., & Zweigenbaum, P. (2022) · International Workshop on Health Text Mining and Information Analysis (LOUHI) · doi:10.18653/v1/2022.louhi-1.9 · Cited by 4
“…language models have also been shown to benefit from external knowledge injection, with approaches like LIBERT (Lauscher et al., 2020), KnowBERT (Peters et al., 2019), and KEPLER (Wang et al., 2021b) in the general domain, or (Hao et al., 2020) and (Lu et al., 2021) in the medical domain.”
MethodWaLDORf: Wasteless Language-model Distillation On Reading-comprehensionPreprint
Tian, J. Y., Kreuzer, A. P., Chen, P.-H., & Will, H.-M. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.06638 · Cited by 3
“In academia, a plethora of research has arisen not only towards advancing and augmenting these state-of-the-art VLLMs[9, 10, 21, 22, 35, 45] but also towards studying their inner workings[1, 8, 17, 32, 33, 43, 47, 50–52] - a eld sometimes called ”BERTology”.”
BackgroundIslam, S. M., & Bhattacharya, S. (2021) · arXiv (Cornell University) · Cited by 3
Chen, X., He, L., Hu, W., & Yi, S. (2025) · AI · doi:10.3390/ai6110290 · Cited by 2
“Architectures such as K-BERT [22], KnowBERT [30], LUKE [40], and various K-GCN variants (e.g., those employing multi-head attention GCNs like [20,21]) exemplify these efforts.”
Prompt Learning with Structured Semantic Knowledge Makes Pre-Trained Language Models Better
Zheng, H.-T., Xie, Z., Liu, W., Huang, D., Wu, B., & Kim, H. (2023) · Electronics · doi:10.3390/electronics12153281 · Cited by 6
“KnowBERT [26] integrates WordNet [18] and a subset of Wikipedia into BERT and uses the knowledge attention and recontextualization mechanism to explicitly model entity spans in the input text.”
MethodTRAFMEL: Multimodal Entity Linking Based on Transformer Reranking and Multimodal Co-Attention Fusion
Zhang, X., Kaikai, M., & Wang, H. (2024) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s021819402450013x · Cited by 2
Efficient Textual Similarity using Semantic MinHashing
Nawaz, W., Baig, M., & Khan, K. U. (2024) · IEEE International Conference on Big Data and Smart Computing (BigComp) · doi:10.1109/bigcomp60711.2024.00048 · Cited by 2
“KnowBert [29] is a generic and efficient model for incorporating prior knowledge into a deep neural network.”
Wang, C. (2025) · Springer eBooks · doi:10.1007/978-981-96-5194-8_7 · Cited by 2
Guiding Generative Recommender Systems with Structured Human Priors via Multi-head Decoding
Zhang, Y., Zhang, Q., Yang, D., Lin, R., Qiu, R., Zhang, B., Yu, H., Liu, J., Xia, Y., Zhao, Z., et al. (2026) · International World Wide Web Conference (WWW) · doi:10.1145/3774904.3792100 · Cited by 5
Wang, C., Cao, Z. G., & Zhou, K. (2025) · ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing (SNPD) · doi:10.1109/snpd65828.2025.11253421 · Cited by 2
Fengrui, Y., & Du, Y. (2026) · Cybersecurity · doi:10.1186/s42400-025-00413-1 · Cited by 4
“Early research (Zhang et al. 2019; Peters et al. 2019; Yamada et al. 2020; Wang et al. 2021) primarily focused on converting knowledge base data into vectors using embedding methods, facilitating model training and recognition.”
Neurosymbolic Large Language Models: A Survey of Symbolic Integration, Reasoning and Explainability
Rani, M., Mishra, B. K., & Thakker, D. (2026) · Information Systems Frontiers · doi:10.1007/s10796-026-10794-4 · Cited by 2
Teach the Rules, Provide the Facts: Targeted Relational-knowledge Enhancement for Textual Inference
Rozen, O., Amar, S., Shwartz, V., & Dagan, I. (2021) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2021.starsem-1.8 · Cited by 2
“KnowBert (Peters et al., 2019) incorporated knowledge from Wikipedia and WordNet into a BERT model through entity embeddings, improving performance on relation extraction and entity typing.”
MethodInfluential
Metadata
- Authors
- Peters, M. E., Neumann, M., Logan, R., Schwartz, R., Joshi, V., Singh, S., & Smith, N. A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- University of California, Irvine (US)
- University of Washington (US)
- Corresponding author
- Singh, Sameer
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 43-54
- OpenAlex
- W2970986510
- Cited by
- 661
- References
- 72
- Citations by year
- 2026: 21 · 2025: 54 · 2024: 81 · 2023: 125 · 2022: 115 · 2021: 160 · 2020: 98 · 2019: 6
- FWCI
- 36.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- National Science Foundation (1730158, 1817183)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- contextualized word representations, word embeddings, knowledge graphs, word sense disambiguation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.