Knowledge Enhanced Contextual Word Representations
Open accessThe DOI matches its Crossref record; 71 of the 72 references it cites were found in a registry or an index.
Abstract
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A. Smith. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“XLM-R achieves state-of-the-arts results on multiple cross-lingual benchmarks, including XNLI, MLQA, and NER.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [62]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“We compute M m scores using the prior and dot product between the entity-span vectors and entity embeddings, with a two-layer MLP (100 hidden dimensions).”
BackgroundMethodInfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 4×
“, 2018), and BERT (Devlin et al., 2019) have significantly improved the state of the art for a wide range of NLP tasks.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 4×
“, 2018), and BERT (Devlin et al., 2019) have significantly improved the state of the art for a wide range of NLP tasks.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Pretrained word representations Initial work learning word vectors focused on static word embeddings using multi-task learning objectives (Collobert and Weston, 2008) or corpus level cooccurence statistics (Mikolov et al., 2013a; Pennington et al., 2014).”
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 3×
“Pretrained word representations Initial work learning word vectors focused on static word embeddings using multi-task learning objectives (Collobert and Weston, 2008) or corpus level co-occurence statistics (Mikolov et al., 2013a; Pen-nington et al., 2014).”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 3×
“The entity embeddings use a skipgram like objective (Mikolov et al., 2013b) to learn 300-dimensional embeddings of Wikipedia”
BackgroundMethodWordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 1×
“We experiment with integrating both WordNet (Miller, 1995) and Wikipedia, thus explicitly adding word sense knowledge and facts about named entities (including those unseen at training time).”
MethodDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 6×
“Recently the field has shifted toward learning context-sensitive embeddings (Dai and Le, 2015; Peters et al., 2018; Devlin et al., 2019).”
BackgroundMethodInfluentialSelf-citation(2013) · Cited by 5,726
Research works printed but not linked: 5
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013b. Distributed representa- tions of words and phrases and their compositional- ity. In NeurIPS.
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL-HLT.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL.
- Bishan Yang and Tom Michael Mitchell. 2017. Lever- aging knowledge bases in LSTMs for improving ma- chine reading. In ACL.
- Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: En- hanced language representation with informative en- tities. In ACL.
Cited by
Items 501–600 of 705
BERTBooster: A knowledge enhancement method jointing incremental training and gradient optimization
Jiang, W., Jiarong, L., Liang, T., Xiao, H. Y., Lu, J., & Wu, T. (2022) · International Journal of Intelligent Systems · doi:10.1002/int.22998 · Cited by 2
Adapting without forgetting: KnowBert-UMLS
Piat, G., Semmar, N., Allauzen, A., Essafi, H., Bernard, G., & Tourille, J. (2022) · IEEE International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob) · doi:10.1109/wimob55322.2022.9941720 · Cited by 2
“One of the main categories of approaches is to rely on the Transformer’s attention mechanism to combine entity and word information, as do ERNIE [6] and KnowBert [3].”
A data-centric way to improve entity linking in knowledge-based question answering
Liu, S., Zhou, G., Xia, Y., Wu, H., & Li, Z. (2023) · PeerJ Computer Science · doi:10.7717/peerj-cs.1233 · Cited by 2
“The entity linking model proposed in (Logeswaran et al., 2019; Peters et al., 2019; Chen et al., 2020; Wu et al., 2020) exploited pre-trained BERT to capture as many multidimensional features as possible to improve mention and entity encoding.”
MethodDeep Dive Text Analytics and Natural Language Understanding
Gómez-Pérez, J. M., García-Silva, A., Berrío, C., Rigau, G., Soroa, A., Lieske, C., Hoffart, J., Sasaki, F., Dahlmeier, D., Skadiņa, I., et al. (2023) · Cognitive technologies · doi:10.1007/978-3-031-28819-7_42 · Cited by 3
Text-To-KG Alignment: Comparing Current Methods on Classification Tasks
Wold, S., Øvrelid, L., & Velldal, E. (2023) · doi:10.18653/v1/2023.matching-1.1 · Cited by 4
AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model
Yan, Y., Xue, K., Shi, X., Ye, Q., Liu, J., & Ruan, T. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385733 · Cited by 6
“Recent studies have demonstrated that injecting extra knowledge information can enhance the model [26], such as knowledge acquisition [27].”
BackgroundYoo, S., Jeong, M., Seomun, S., Kim, K., & Han, Y. (2024) · IEEE Transactions on Computational Biology and Bioinformatics · doi:10.1109/tcbb.2024.3368046 · Cited by 2
“Recent advances in NLP have demonstrated that self-supervised learning can be a powerful tool for extracting useful information from unlabeled sequence data [29], [30], [31].”
BackgroundAdaptive Prompt Construction Method for Relation Extraction
Chen, Z., Li, Z., Huang, Y., & Tang, Z. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448452 · Cited by 2
Word Sense Linking: Disambiguating Outside the Sandbox
Bejgu, A. S., Barba, E., Procopio, L., Fernández-Castro, A., & Roberto Navigli, A. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.851 · Cited by 2
Research of Knowledge-Enhanced Large Language Model Based on Knowledge Graph
Meng, Q., Wu, Z., Zhao, Z., & Lian, X. (2024) · International Conference on Artificial Intelligence and Computer Engineering (ICAICE) · doi:10.1109/icaice63571.2024.10863933 · Cited by 2
“…inject knowledge or embed entities into LLM, such as K-BERT (Knowledge-enhanced BERT) [16], which enhances the understanding ability of the model by integrating the triple information of knowledge graph; Know-BERT [17] links the entity of knowledge base to models to realize knowledge enhancement.”
Explainable AI for Large Language Models via Context-Aware Word Embeddings
Ecker, J. E. (2025) · AIAA SciTech Forum · doi:10.2514/6.2025-1916 · Cited by 2
A Knowledge-Enhanced Cyber Threat Intelligence Entity and Relation Extraction Method
Li, K., Yin, X., Liu, J., & Dai, J. (2024) · IEEE International Conference on Frontiers Technology of Information and Computer (ICFTIC) · doi:10.1109/icftic64248.2024.10912987 · Cited by 2
Knowledge injection methods in question answering
Radyush, D. (2025) · Russian Technological Journal · doi:10.32362/2500-316x-2025-13-3-21-43 · Cited by 3
Generalized Knowledge-Enhanced Framework for Biomedical Entity and Relation Extraction
Nguyen, M., & Le, P. (2025) · Lecture notes in computer science · doi:10.1007/978-981-96-6585-3_19 · Cited by 2
Verbalization diversification for relation extraction with long-tailed entity augmentation
Liang, Y., Jiang, T., Xin, Y., Yan, C., Wang, L., Wang, X., & He, L. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130771 · Cited by 2
Sakhovskiy, A., & Tutubalina, E. V. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3729901 · Cited by 5
Pessutto, L. R. C., de Melo, T., Moreira, V. P., & da Silva, A. S. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.5 · Cited by 2
JBNU at SemEval-2020 Task 4: BERT and UniLM for Commonsense Validation and Explanation
Na, S., & Lee, J. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.65 · Cited by 2
“…pretrained models (Sanh et al., 2019), analyzing BERT’s syntactic knowledge (Hewitt and Manning, 2019), knowledge-enhanced language models (Peters et al., 2019), cross-lingual pretrained language models (Conneau and Lample, 2019), few-shot learning using language models (Radford et al.,…”
BackgroundMethodGiBERT: Enhancing BERT with Linguistic Information using a Lightweight Gated Injection Method
Peinelt, N., Rei, M., & Liakata, M. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.200 · Cited by 6
“Peters et al. (2019) suggested a word-to-entity attention mechanism to incorporate external knowledge into BERT and Wang et al. (2020) proposed to inject factual and linguistic knowledge through separate adapter modules.”
BackgroundMethodInfluentialWang, Z., Zhang, G., & Lashkarashvili, N. (2022) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2022.semeval-1.2 · Cited by 2
How to be Helpful on Online Support Forums?
Wang, Z., & Torres, P. E. (2022) · doi:10.18653/v1/2022.wnu-1.3 · Cited by 6
Distantly Supervised Named Entity Recognition with Self-Adaptive Label Correction
Nie, B., & Li, C. (2022) · Applied Sciences · doi:10.3390/app12157659 · Cited by 2
EREC: Enhanced Language Representations with Event Chains
Wang, H., & Wang, Y. (2022) · Information · doi:10.3390/info13120582 · Cited by 2
Chinese Named Entity Recognition Based on Improved K-BERT
Li, J., Yilahun, H., & Hamdulla, A. (2022) · International Conference on Virtual Reality, Human-Computer Interaction and Artificial Intelligence (VRHCIAI) · doi:10.1109/vrhciai57205.2022.00050 · Cited by 2
Lovón-Melgarejo, J., Moreno, J. G., Besançon, R., Ferret, O., & Tamine, L. (2023) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2023.semeval-1.121 · Cited by 2
“Similarly, previous works proposed pre-training enhanced-PLMs by fusing knowledge base representations with PLMs to improve token representations, which has been helpful in related NLP tasks (Zhang et al., 2019; Peters et al., 2019).”
BackgroundPre-trained Models for Representation Learning
Lin, Y., Ding, N., Liu, Z., & Sun, M. (2023) · Springer eBooks · doi:10.1007/978-981-99-1600-9_5 · Cited by 2
Textual Entailment for Effective Triple Validation in Object Prediction
García-Silva, A., Berrío, C., & Gómez-Pérez, J. M. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47240-4_5 · Cited by 4
“Other works such as KnowBERT [30], and E-BERT [32] enhance BERT with pretrained entity embeddings, showing that such enhanced versions improve BERT results on LAMA [31].”
MethodCao, B., Tang, Q., Lin, H., Han, X., & Sun, L. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.143 · Cited by 2
“Popular strategies include: 1) Embedding combined methods (Zhang et al., 2019; Peters et al., 2019; He et al., 2020b), which encode structured knowledge via knowledge embedding algorithms and then enhance PLMs’ text representation with knowledge graph embedding; 2) Knowledge supervised meth-ods…”
MethodCross-Lingual Fine-Grained Entity TypingPreprint
Selvaraj, N., Onoe, Y., & Durrett, G. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07837 · Cited by 2
“Another related line of work attempts to add entity information into BERT (Zhang et al., 2019; Peters et al., 2019; Poerner et al., 2020; Févry et al., 2020, inter alia); our model could potentially benefit from this, but given our focus on generalizing to new entities, a model that more easily…”
BackgroundInterpretable Sentence Representation with Variational Autoencoders and AttentionPreprint
Felhi, G. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.02810 · Cited by 3
Models of diachronic semantic change using word embeddings
Montariol, S. (2021) · theses.fr (ABES) · doi:10.70675/c08ef5d0z7732z4caeza42fz68ade712e3cd · Cited by 10
Boukkouri, H. E. (2021) · theses.fr (ABES) · doi:10.70675/4c63eb46zd8cfz4a5dz80e9z666563e756d8 · Cited by 10
Representation, information extraction, and summarization for automatic multimedia understanding
Harrando, I. (2022) · theses.fr (ABES) · doi:10.70675/702b8ba3z3acaz4ce3za9b9zf2cb1609e328 · Cited by 2
Joint Learning-based Multiple Documents Heterogeneous Graph Inference for Biomedical Entity Linking
Dai, Q., Zhong, J., Wang, C., Yin, H., Lei, Q., Li, X., & Li, R. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385533 · Cited by 1
“In [26], a mention span is used to represent the mention.”
MethodConstructing Chinese taxonomy trees from understanding and generative pretrained language models
Guo, J., Chen, J., Li, R., Zhou, H., Xu, W., & Jia, H. (2024) · PeerJ Computer Science · doi:10.7717/peerj-cs.2358 · Cited by 1
“Based on Chen, Lin & Klein (2021) and Peters et al. (2019) , gloss collected from the web improves the model ’ s performance by 3.2%.”
InfluentialUnified Contextualized Knowledge Embedding Method for Static and Temporal Knowledge Graph
Gao, Y., Qiao, L., Huang, Z., Kan, Z., He, Y., & Li, D. (2024) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3507557 · Cited by 1
“The first incorporates knowledge from KGs to enhance PLMs in downstream NLP application tasks[46],[47],[48].”
The integration of large language model and knowledge graph and its application in flight safety
Chao, G., Liu, X., & Dongfang, Z. (2024) · IEEE International Conference on Civil Aviation Safety and Information Technology (ICCASIT) · doi:10.1109/iccasit62299.2024.10827945 · Cited by 1
WIP: Comparison of Large Language Models for Applied Mathematics Questions in Engineering Courses
Merlos, C., Hussain, F., Kar, S., Shri, L., Olugbenle, O., Banavar, M. K., & AlMomani, A. A. R. (2024) · IEEE Frontiers in Education Conference (FIE) · doi:10.1109/fie61694.2024.10893476 · Cited by 1
“In RAG, a retriever component is used to retrieve relevant context or information from a large knowledge base when presented with a query, which is then incorporated into the generation process of the language model [11, 12].”
Wang, S., Zhu, E., Zhao, F., Bu, D., Li, J., & Zhao, Y. (2025) · IEEE Transactions on Audio Speech and Language Processing · doi:10.1109/taslpro.2025.3579311 · Cited by 1
“Additionally, several studies have explored incorporating external knowledge—such as knowledge graphs, synonym nets, Wikipedia anchor texts [32], [33], [34], [35], [36], or language-specific lexical knowledge (e.g., Chinese word information) [37], [38]—to further enhance model performance,…”
Contrastive Learning Fusing Dual Coding for Knowledge Graph-to-Text Generation
Luo, X., Wang, X., Wang, X., & Liu, R. (2025) · International Conference on Information Science, Electrical and Automation Engineering (ISEAE) · doi:10.1109/iseae64934.2025.11041722 · Cited by 1
Yang, D., Zhang, X., Han, T., & Liu, Y. (2025) · Natural language processing. · doi:10.1017/nlp.2025.10005 · Cited by 1
“…approach to enhancing NWEs is through the utilization of more intricate neural architectures such as training graph convolutional networks with syntactic dependencies and semantic relationships (Vashishth et al. 2019) and employing attention mechanisms (Yang and Mitchell, 2017; Peters et al. 2019).”
CRWMTR: Causal Reasoning and Weighting Mechanisms of Thought Reasoning in Large Language Models
Yang, W., Zeng, Q., Lin, Z., Zhong, X., Luo, H., Yuan, J., Zhang, S., Zhang, J., & Liu, Y. (2024) · International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE) · doi:10.1109/icbaie63306.2024.11117006 · Cited by 1
“Due to the hallucinations, explainability and transparency in LLMs, CoTs Incorporate causal reasoning to enhance the reasoning in Knowledge Graphs (KGs) [3].”
Lee, H., Yoo, S., Loh, W.-K., & Jeong, O. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.114390 · Cited by 1
Brenda, G., & Fred, A. L. (2025) · Earth Science Informatics · doi:10.1007/s12145-025-02005-6 · Cited by 1
“…a transformer model that only uses the self-attention process to make the global connection of input and output: it now is the most widely used model for natural language processing (Devlin et al. 2018; Radford et al. 2018; Brown et al. 2020; Yang et al. 2019; Peters et al. 2019; Liu et al. 2019).”
Lin, L., Liu, Y., Xia, K., & Wang, S. (2025) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3746252.3761093 · Cited by 1
IKG-Agent: Intent-driven Knowledge Graph Agent for Adaptive Workflow Reasoning
Liang, Y., Tao, S., Teng, H., Yao, H., Wang, X., Guo, X., Wang, L., & Ji, X. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11228190 · Cited by 2
Knowledge Weighted Method for Enabling Language Representation
Xing, T., Chen, F., Liu, X., Li, T., Wang, J., Zhang, K., Bian, L., & Hou, M. (2025) · IEEE International Conference on Civil Aviation Safety and Information Technology (ICCASIT) · doi:10.1109/iccasit66611.2025.11348754 · Cited by 1
“THU-ERNIE [10] modified BERT’s encoder into an aggregator for mutual integration of word and entity information.”
Chen, Z., Yan, H., Du, J., Xue, M., & Zhao, S. (2026) · Electronics · doi:10.3390/electronics15030631 · Cited by 1
Industrial agents: Architecture, key technologies, and future perspectives
Lei, R. E. N., Wang, H., Dong, J., Zhang, J., Yan, J., Cao, Z., Li, S., Lai, L., Zhang, L., & Li, B. (2026) · Scientia Sinica Technologica · doi:10.1360/sst-2025-0364 · Cited by 2
Empowering LLMs with disentangled structure awareness for prompt-based knowledge graph completion
Kang, T., Li, S., Tang, J., Long, K., Yin, H., & Wang, T. (2026) · Information Systems · doi:10.1016/j.is.2026.102701 · Cited by 1
Can Knowledge Enhance Reading Comprehension? An Integrated Approach with Semantic Lexicon
Wei Kun, K. O. N. G., Racharak, T., & Nguyen, L.-M. (2020) · International Conference on Knowledge and Systems Engineering/Proceedings · doi:10.1109/kse50997.2020.9287218 · Cited by 1
Patel, R., & Ferraro, F. (2020) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2020.deelio-1.11 · Cited by 2
TR at SemEval-2020 Task 4: Exploring the Limits of Language-model-based Common Sense Validation
Teo, D. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.76 · Cited by 2
“The methods vary by the choice of knowledge sources and the training objective (Zhang et al., 2019; Lauscher et al., 2019; Levine et al., 2019; Peters et al., 2019; Xiong et al., 2019; He et al., 2019; Wang et al., 2020b).”
MethodPaths to Relation Extraction through Semantic Structure
Yellin, J., & Abend, O. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.231 · Cited by 3
“With the increased pre-dominance of end-to-end neural network architectures in NLP practice, it is not surprising that recent work on supervised relation extraction has focused on adapting end-to-end neural systems for the task (Peters et al., 2019; Yamada et al., 2020).”
BackgroundLV-BERT: Exploiting Layer Variety for BERT
Yu, W., Jiang, Z., Chen, F., Hou, Q., & Feng, J. (2021) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2021.findings-acl.2 · Cited by 2
Simultaneously Self-Attending to Text and Entities for Knowledge-Informed Text Representations
Thai, D., Thirukovalluru, R., Bansal, T., & McCallum, A. (2021) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2021.repl4nlp-1.25 · Cited by 2
“However, existing meth-ods typically assume access to entity-linked data for training (Zhang et al., 2019; Peters et al., 2019), which is scarce and expensive to annotate, preventing large scale pre-training.”
BackgroundMethodInfluentialResearch on Pre-training of Tibetan Natural Language Processing
Li, Z., Zhu, J., & Cao, H. (2021) · IEEE International Conference on Pattern Recognition and Machine Learning (PRML) · doi:10.1109/prml52754.2021.9520714 · Cited by 1
Interacting Knowledge Sources, Inspection and Analysis: Case-studies on Biomedical text processing
Bagherzadeh, P., & Bergler, S. (2021) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2021.blackboxnlp-1.35 · Cited by 1
“For instance, (Peters et al., 2019) developed KnowBERT (an extension to BERT), to inject information from WordNet into an intermediate layer of BERT, which requires re-training the BERT model.”
MethodA text matching model based on dynamic multi‐mask and augmented adversarial
Zhong, L., Zeng, J., Yang, Y., Tao, H., Jiang, W., & Cheng, L. (2022) · Expert Systems · doi:10.1111/exsy.13165 · Cited by 1
“, 1995), TF-IDF feature (Salton & Buckley, 1988) and ELMO feature (Peters et al., 2019).”
Chinese Machine Reading Comprehension Based on Language Model Containing Knowledge
Chen, W., Fan, C., Wu, Y., & Wang, Y. (2022) · International Conference on Computer Science and Application Engineering (CSAE) · doi:10.1145/3565387.3565405 · Cited by 1
Cohen, T., Patel, V. L., & Shortliffe, E. H. (2022) · Cognitive informatics in biomedicine and healthcare · doi:10.1007/978-3-031-09108-7_20 · Cited by 1
Positive Unlabeled Learning by Sample Selection and Prototype Refinement
Wang, Z., & Long, G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-22064-7_23 · Cited by 1
Exploring Category Structure with Contextual Language Models and Lexical Semantic Networks
Renner, J., Denis, P., Gilleron, R., & Brunellière, A. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.167 · Cited by 1
Zhang, F., Qin, J., & Cheng, J. (2023) · Knowledge-Based Systems · doi:10.1016/j.knosys.2023.110471 · Cited by 1
Symbolic Semantic Memory in Transformer Language Models
Morain, R., Vargas, K., & Ventura, D. A. (2022) · International Conference on Machine Learning and Applications · doi:10.1109/icmla55696.2022.00166 · Cited by 2
Foundation Models for Information Extraction
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_5 · Cited by 1
Free Lunch for Efficient Textual Commonsense Integration in Language Models
Cui, W., & Chen, X. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.208 · Cited by 1
Language Model Pre-Training with Sparse Latent Typing
Ren, L., Zhang, Z., Wang, H., Voss, C. R., Zhai, C., & Ji, H. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.96 · Cited by 1
Knowledge Representation Learning and Knowledge-Guided NLP
Han, X., Chen, W., Liu, Z., Lin, Y., & Sun, M. (2023) · Springer eBooks · doi:10.1007/978-981-99-1600-9_9 · Cited by 1
Entity linking for Chinese short texts based on multi-task learning
Liu, Z., & Yu, M. (2023) · International Conference on Advanced Algorithms and Neural Networks (AANN) · doi:10.1117/12.3005028 · Cited by 1
“Peters et al. [7] utilized the BERT (Bidirectional Encoder Representation from Transformers) pre-training model and max-pooling layer to encode the text, where candidate entities' encoding also features an attention mechanism.”
MethodInfluentialRethinking the Role of Entity Type in Relation Classification
Dai, X., Karimi, S., & Wan, S. (2023) · International Joint Conference on Natural Language Processing and Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL) · doi:10.18653/v1/2023.ijcnlp-main.24 · Cited by 1
“One research direction continues to improve pre-trained models via injecting factual and linguistic knowledge, usually with the help of external knowledge base consisting of relation tuples (Peters et al., 2019; Soares et al., 2019; Zhang et al., 2019b; Yamada et al., 2020; Wang et al., 2021).”
BackgroundMixCL: Mixed Contrastive Learning for Relation Extraction
Zhang, J., Li, B., Cao, X., Zhang, M., & Zhao, W. (2024) · Lecture notes in computer science · doi:10.1007/978-981-97-2238-9_7 · Cited by 1
Liu, X., Rzepka, R., & Araki, K. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111883 · Cited by 1
Refining Entity Descriptions with Relation Embeddings for Scientific Relation Classification
Li, C. H., Liu, X., Li, J. L., Wang, J. T.-L., Feng, Z., & Li, Q. (2024) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn60899.2024.10650696 · Cited by 1
“To enrich the entity knowledge of PLMs, references [2], [3] have designed pre-training tasks that inject entity knowledge from external corpora and Knowledge Graphs (KGs) into PLMs for entity knowledge enhancement.”
BackgroundMethodEffective and Reliable Domain-Specific Knowledge Question Answering
Fang, Y., Chen, Y., Jiang, Z., Xiao, J., & Ge, Y. (2024) · IEEE International Conference on e-Business Engineering (ICEBE) · doi:10.1109/icebe62490.2024.00044 · Cited by 2
“The emergence and popularity of the pre-trained language models such as BERT [17], XLNET [18] and BART [19] have significantly enhanced the performance of these Natural Language Processing (NLP) tasks [20, 21].”
A New Multimodal Large Model Framework for Knowledge-enhanced Image Caption Generation
Wan, T., Liu, H., & Geng, L. (2024) · IEEE International Conference on Advances in Electrical Engineering and Computer Applications (AEECA) · doi:10.1109/aeeca62331.2024.00102 · Cited by 2
“Peters et al. [22] developed a deep learning framework that dynamically acquires information from multiple knowledge sources to enhance the predictive power of language models, particularly effective in handling specialized or rare topics.”
Geta, C. B., & Gereme, F. B. (2025) · SN Computer Science · doi:10.1007/s42979-025-04286-6 · Cited by 1
Knowledge-Infused Pre-trained Models for KG Completion
Yu, H., Jiang, R., Zhou, B., & Li, A. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-62005-9_20 · Cited by 1
Towards humanized open-domain conversational agents
Zhong, P. (2021) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/147791 · Cited by 2
Refining latent multi-view graph for relation extraction
Xue, F. (2021) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/151541 · Cited by 1
Towards Continual Entity Learning in Language Models for Conversational AgentsPreprint
Gadde, R. T., & Bulyko, I. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.00082 · Cited by 1
“Unlike [15, 28, 44], that are desgined to improve the representation of fact-based entities, in this paper, we only focus on preference-based entities.”
BackgroundEnhancing Language Models with Plug-and-Play Large-Scale CommonsensePreprint
Cui, W., & Chen, X. (2021) · arXiv (Cornell University) · Cited by 1
Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation ExtractionPreprint
Chen, X., Zhang, N., Xie, X. Y., Deng, S., Yao, Y., Tan, C., Huang, F., Si, L., & Chen, H. (2021) · arXiv (Cornell University) · Cited by 21
Metadata Shaping: Natural Language Annotations for the TailPreprint
Arora, S., Wu, S., Liu, E., & Ré, C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08430 · Cited by 1
“Incorporating Knowledge in LMs As previously discussed, significant prior work incorporates knowledge into language modeling through changing the LM architecture or loss [24, 25, 32, 37, 39, 41, 46, 47, 51], in 8Per subject and object entity for FewRel, and per main entity for OpenEntity.”
BackgroundMethodInfluentialCross-Domain Neural Entity LinkingPreprint
Soliman, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2210.15616 · Cited by 1
“(2019) [25] also falls into this category and is discussed in more detail in the next section.”
BackgroundInfluentialA Cohesive Distillation Architecture for Neural Language ModelsPreprint
Wahle, J. P. (2023) · Authorea · doi:10.22541/au.167528147.79728645/v1 · Cited by 1
Foolad, S., & Kiani, K. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.10443 · Cited by 1
“ERNIE [4] and Know-BERT [20] learned static entity embeddings from a Knowledge Base (KB), while LUKE (Language Understanding with Knowledge-based Embeddings) applied a new pretraining task to learn entity representations.”
MethodDeveloping novel triple embeddings for scalable alignment of knowledge graphs and natural language
Kalinowski, A., & An, Y. (2024) · iDEA - Drexel University Institutional Repository · doi:10.17918/00001906 · Cited by 1
Knowledge Enhancement of Small-Scale Models in Medical Question Answering (Preprint)Preprint
Li, X., Peng, S., Wakamiya, S., & Aramaki, E. (2024) · doi:10.2196/preprints.68320 · Cited by 1
Leveraging linguistic and semantic information for relation extraction from domain-specific texts
Tang, A. (2023) · theses.fr (ABES) · doi:10.70675/5e2de3fdz3bb1z4e21z877ez380f3d91a22a · Cited by 3
Evaluation and Integration of Structural Knowledge for Pre-trained Language Models
Lovón-Melgarejo, J. (2024) · theses.fr (ABES) · doi:10.70675/9b2d6bf6za13az4dfcza45bzca75ad5b25a6 · Cited by 2
Graph Representation Learning Network via Adaptive SamplingPreprint
de Andrade, A., & Liu, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.04637
“The information contained in many knowledge graphs (KG) has been used in several machine learning tasks in natural language understanding [20], computer vision [16], and recommendation systems [29].”
MethodIXA-NER-RE at eHealth-KD Challenge 2020.
Andrés, E., Sainz, O., Atutxa, A., & de Lacalle, O. L. (2020) · IberLEF@SEPLN · Cited by 2
A Brief Review of Relation Extraction Based on Pre-Trained Language Models
Xu, T., & Zhang, F. (2020) · Frontiers in artificial intelligence and applications · doi:10.3233/faia200755 · Cited by 2
A Deep Learning Approach to Interest Analysis
van der Meer, T. G. L. A. (2020) · Utrecht University Repository (Utrecht University)
Defx at SemEval-2020 Task 6: Joint Extraction of Concepts and Relations for Definition Extraction
Hübner, M., Alt, C., Schwarzenberg, R., & Hennig, L. (2020) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/2020.semeval-1.92 · Cited by 1
Tendulkar’s Cat and Schrodinger’s Bat—Knowledge-Enhanced Real-Word Error Correction
Sridhar, S., Saahitya, E., Sundaram, S. S., & Das, B. (2021) · Lecture notes in networks and systems · doi:10.1007/978-981-15-9774-9_37
Explorations In Curriculum Learning Methods For Training Language Models
Campos, D. (2020) · ResearchWorks at the University of Washington (University of Washington)
Self-Attention Network for Text Representation Learning
Shen, T. (2021) · UTS ePRESS (University of Technology Sydney)
Semantische Technologien für Enterprise Intelligence am Beispiel von Lieferkettenbeobachtung
Hennig, L., & Uszkoreit, H. (2021) · Springer eBooks · doi:10.1007/978-3-658-31938-0_6
Metadata
- Authors
- Peters, M. E., Neumann, M., Logan, R., Schwartz, R., Joshi, V., Singh, S., & Smith, N. A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- University of California, Irvine (US)
- University of Washington (US)
- Corresponding author
- Singh, Sameer
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 43-54
- OpenAlex
- W2970986510
- Cited by
- 661
- References
- 72
- Citations by year
- 2026: 21 · 2025: 54 · 2024: 81 · 2023: 125 · 2022: 115 · 2021: 160 · 2020: 98 · 2019: 6
- FWCI
- 36.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- National Science Foundation (1730158, 1817183)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- contextualized word representations, word embeddings, knowledge graphs, word sense disambiguation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.