Knowledge Enhanced Contextual Word Representations
Open accessThe DOI matches its Crossref record; 71 of the 72 references it cites were found in a registry or an index.
Abstract
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A. Smith. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“XLM-R achieves state-of-the-arts results on multiple cross-lingual benchmarks, including XNLI, MLQA, and NER.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [62]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“We compute M m scores using the prior and dot product between the entity-span vectors and entity embeddings, with a two-layer MLP (100 hidden dimensions).”
BackgroundMethodInfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 4×
“, 2018), and BERT (Devlin et al., 2019) have significantly improved the state of the art for a wide range of NLP tasks.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 4×
“, 2018), and BERT (Devlin et al., 2019) have significantly improved the state of the art for a wide range of NLP tasks.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Pretrained word representations Initial work learning word vectors focused on static word embeddings using multi-task learning objectives (Collobert and Weston, 2008) or corpus level cooccurence statistics (Mikolov et al., 2013a; Pennington et al., 2014).”
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 3×
“Pretrained word representations Initial work learning word vectors focused on static word embeddings using multi-task learning objectives (Collobert and Weston, 2008) or corpus level co-occurence statistics (Mikolov et al., 2013a; Pen-nington et al., 2014).”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 3×
“The entity embeddings use a skipgram like objective (Mikolov et al., 2013b) to learn 300-dimensional embeddings of Wikipedia”
BackgroundMethodWordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 1×
“We experiment with integrating both WordNet (Miller, 1995) and Wikipedia, thus explicitly adding word sense knowledge and facts about named entities (including those unseen at training time).”
MethodDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 6×
“Recently the field has shifted toward learning context-sensitive embeddings (Dai and Le, 2015; Peters et al., 2018; Devlin et al., 2019).”
BackgroundMethodInfluentialSelf-citation(2013) · Cited by 5,726
Research works printed but not linked: 5
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013b. Distributed representa- tions of words and phrases and their compositional- ity. In NeurIPS.
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL-HLT.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL.
- Bishan Yang and Tom Michael Mitchell. 2017. Lever- aging knowledge bases in LSTMs for improving ma- chine reading. In ACL.
- Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: En- hanced language representation with informative en- tities. In ACL.
Cited by
Items 301–400 of 705
Conception: Multilingually-Enhanced, Human-Readable Concept Vector Representations
Conia, S., & Navigli, R. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.291 · Cited by 17
“Contextualized word embeddings are able to capture the many facets of a polysemous word in a context (Pilehvar and Camacho-Collados, 2019), but their implicitly encoded meanings are still disconnected from human-curated knowledge bases, even if more recent efforts showed promising results in imparting structured seman…”
BackgroundVerb Knowledge Injection for Multilingual Event Processing
Majewska, O., Vulić, I., Glavaš, G., Ponti, E. M., & Korhonen, A. (2021) · doi:10.18653/v1/2021.acl-long.541 · Cited by 15
“Expanding a line of research on injecting ex-a ternal linguistic knowledge into (LM-)pretrained models (Peters et al., 2019; Levine et al., 2020; Lauscher et al., 2020b), we integrate verb knowledge into contextualised representations for the first time.”
BackgroundLow Anisotropy Sense Retrofitting (LASeR) : Towards Isotropic and Sense Enriched Representations
Bihani, G., & Rayz, J. T. (2021) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2021.deelio-1.9 · Cited by 12
Improving Sentence-Level Relation Extraction through Curriculum LearningPreprint
Park, S., & Kim, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2107.09332 · Cited by 12
“…with TRACRED and Re-TACRED showed that the proposed method is simple but effective to Models P R F1 SpanBERT-large [10] 70.8 70.9 70.8 KnowBERT-W [11] 71.6 71.4 71.5 BERT-MTB [12] - - 71.5 DeNERT-KG [13] 71.8 73.1 72.4 LUKE [14] 70. increasing the performance of sentence-level relation…”
InfluentialLM-CORE: Language Models with Contextually Relevant External Knowledge
Kaur, J. N., Bhatia, S., Aggarwal, M., Bansal, R., & Krishnamurthy, B. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.57 · Cited by 16
“This is consistent with the initialization in Peters et al. (2019); Zhang et al. (2019); Wang et al. (2021b) and is a common practice to save up on pre-training time.”
BackgroundInfluentialOverview of indoor scene recognition and representation methods based on multimodal knowledge graphs
Li, J., Si, G., Tian, P., An, Z., & Zhou, F. (2023) · Applied Intelligence · doi:10.1007/s10489-023-05235-7 · Cited by 13
“However, KnowBert [23] provides a distinctive approach by applying architecture injection and output injection in the fine-tuning phase to explicitly model the span of entities in the input.”
An Entity Ontology-Based Knowledge Graph Embedding Approach to News Credibility Assessment
Liu, Q., Jin, Y., Cao, X., Liu, X., Zhou, X., Zhang, Y., Xu, X., & Qi, L. (2024) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2023.3342873 · Cited by 12
“Peters et al. [22] propose a method to enhance contextual word representations with structured knowledge from multiple knowledge bases.”
MethodKW-ATTN: Knowledge Infused Attention for Accurate and Interpretable Text Classification
Jang, H., Bang, S., Xiao, W., Carenini, G., Ng, R. T., & Lee, Y. J. (2021) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2021.deelio-1.10 · Cited by 13
“KnowBERT (Peters et al., 2019) trained BERT for entity linkers and language modeling in a multitask setting to incorporate entity representation.”
MethodEnabling Language Representation with Knowledge Graph and Structured Semantic Information
Xu, W., Fang, M., Li, Y., Jiang, H., Geng, L., & Zuo, C. (2021) · International Conference on Computer Communication and Artificial Intelligence (CCAI) · doi:10.1109/ccai50917.2021.9447453 · Cited by 11
“KnowBERT [19], this method mainly releases the pre-training process of structured knowledge graph data through entity recognition and entity connection, which is essentially a multi-task learning process.”
MethodTowards Automatic Bias Detection in Knowledge Graphs
Keidar, D., Zhong, M., Zhang, C., Shrestha, Y. R., & Paudel, B. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.321 · Cited by 15
Integrating Transformers and Knowledge Graphs for Twitter Stance Detection
Clark, T. H., Conforti, C., Liu, F., Meng, Z., Shareghi, E., & Collier, N. (2021) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2021.wnut-1.34 · Cited by 11
“This can be contrasted with approaches like K-Adapter and KnowBERT, which infuse knowledge by modifying the model architecture, rather than by adding context to the input (Wang et al., 2020; Peters et al., 2020).”
MethodTheory-Guided Deep Learning Algorithms: An Experimental Evaluation
Monaco, S., Apiletti, D., & Malnati, G. (2022) · Electronics · doi:10.3390/electronics11182850 · Cited by 12
“Graph layers can be the CNN’s counterpart to graph-structured data [29] or even improve representation over knowledge graphs in image processing or natural language processing [30].”
BackgroundMulti-hop Evidence Retrieval for Cross-document Relation Extraction
Lu, K., Hsu, I.-H., Zhou, W., Ma, M. D., & Chen, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.657 · Cited by 21
EDIN: An End-to-end Benchmark and Pipeline for Unknown Entity Discovery and Indexing
Kassner, N., Petroni, F., Plekhanov, M., Riedel, S., & Cancedda, N. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.593 · Cited by 11
Wang, G., Liu, S., & Wei, F. (2021) · Applied Intelligence · doi:10.1007/s10489-021-02596-9 · Cited by 11
“…further improve the performance of many RE tasks: fine-tuning of language representations pretrained on the token level [18, 19] or span level [20], fine-tuning of knowledge-enhanced word representations that are pretrained on entity-linked text [21, 22], and “matching the blanks” pretraining [23].”
BackgroundTweeki: Linking Named Entities on Twitter to a Knowledge Graph
Harandizadeh, B., & Singh, S. (2020) · Workshop on Noisy User-generated Text (W-NUT) · doi:10.18653/v1/2020.wnut-1.29 · Cited by 21
“…text and KG is a valuable resource for learning representations that enable better reasoning about entities and how they are expressed in text (tweets in this case), such as pretrained language models that use such resources for better contextual modeling (Peters et al., 2019; Logan et al., 2019).”
BackgroundSelf-citationDOCENT: Learning Self-Supervised Entity Representations from Large Document Collections
Zemlyanskiy, Y., Gandhe, S., He, R., Kanagal, B., Ravula, A., Gottweis, J., Sha, F., & Eckstein, I. (2021) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2021.eacl-main.217 · Cited by 11
Representation Learning via Variational Bayesian Networks
Barkan, O., Caciularu, A., Rejwan, I., Katz, O., Weill, J., Malkiel, I., & Koenigstein, N. (2021) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3459637.3482363 · Cited by 11
“Recently, contextualized word embedding models were expanded to include external linguistic information during pretraining [1, 18, 28, 34, 49].”
BackgroundGaze-infused BERT: Do human gaze signals help pre-trained language models?
Wang, B., Liang, B., Zhou, L., & Xu, R. (2024) · Neural Computing and Applications · doi:10.1007/s00521-024-09725-8 · Cited by 16
Bridging Gaps in Content and Knowledge for Multimodal Entity Linking
Luo, P., Xu, T., Liu, C., Zhang, S., Xu, L., Li, M., & Chen, E. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681661 · Cited by 10
TempEL: Linking Dynamically Evolving and Newly Emerging Entities
Zaporojets, K., Kaffee, L.-A., Deleu, J., Demeester, T., Develder, C., & Augenstein, I. (2022) · neural information processing systems · doi:10.52202/068431-0135 · Cited by 28
Bauer, L., Deng, L., & Bansal, M. (2021) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2021.deelio-1.7 · Cited by 10
“Peters et al. (2019) investigated embedding knowledge bases into large-scale models in a multitask setup, seeing improvements on relationship extraction, entity typing, and word sense disambiguation.”
BackgroundUMLS-KGI-BERT: Data-Centric Knowledge Integration in Transformers for Biomedical Entity Recognition
Mannion, A., Schwab, D., & Goeuriot, L. (2023) · Clinical Natural Language Processing Workshop (ClinicalNLP) · doi:10.18653/v1/2023.clinicalnlp-1.35 · Cited by 10
Retrieval-Augmented Domain Adaptation of Language Models
Xu, B., Zhao, C., Jiang, W., Zhu, P., Dai, S., Pang, C., Sun, Z., Wang, S., & Sun, Y. (2023) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2023.repl4nlp-1.5 · Cited by 22
“Different from many existing methods that resort to integrate structured knowledge (Zhang et al., 2019; Peters et al., 2019), the retrieved textual domain corpus are much less knowledge-intensive and thus might contain irrelevant noise distractions.”
BackgroundRobust Integration of Contextual Information for Cross-Target Stance Detection
Beck, T., Waldis, A., & Gurevych, I. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.43 · Cited by 22
“Within the era of PLMs, many approaches rely on extensive pretraining using data from knowledge bases (Peters et al., 2019; Zhang et al., 2019) (KB) or supervision from knowledge completion tasks (Wang et al., 2021; Rozen et al., 2021).”
Vision–Language–Knowledge Co-Embedding for Visual Commonsense Reasoning
Lee, J., & Kim, I. (2021) · Sensors · doi:10.3390/s21092911 · Cited by 10
ANNA”:" Enhanced Language Representation for Question Answering
Jun, C., Jang, H., Sim, M., Kim, H. J., Choi, J., Min, K., & Bae, K. (2022) · Workshop on Representation Learning for NLP (RepL4NLP) · doi:10.18653/v1/2022.repl4nlp-1.13 · Cited by 10
Metadata Shaping: A Simple Approach for Knowledge-Enhanced Language Models
Arora, S., Wu, S., Liu, E., & Ré, C. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.137 · Cited by 10
“Explainability Existing knowledge-aware LMs use metadata (Peters et al., 2019; Alt et al., 2020), but do not explain when and why different meta-data help.”
BackgroundMethodInfluentialDynamic Knowledge Integration for Natural Language Inference
Guo, M., Chen, Y., Xu, J., & Zhang, Y. (2022) · 2022 4th International Conference on Natural Language Processing (ICNLP) · doi:10.1109/icnlp55136.2022.00066 · Cited by 10
“Additionally, knowledge has been applied in other fields of natural language processing, such as text classification, machine reading comprehension, natural language generation [24]–[27].”
BackgroundFeng, S., Tan, Z., Zhang, W., Lei, Z., & Tsvetkov, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.118 · Cited by 16
Knowledge-integrated Multi-modal Movie Turning Point Identification
Wang, D., Xu, R., Cheng, L., & Wang, Z. (2023) · ACM Transactions on Multimedia Computing Communications and Applications · doi:10.1145/3638557 · Cited by 9
“Peters et al. [24] embed structured, human-curated knowledge from WordNet and Wikipedia into BERT to enhance contextual word representations.”
BackgroundKG-prompt: Interpretable knowledge graph prompt for pre-trained language models
Chen, L., Liu, J., Duan, Y., & Wang, R. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113118 · Cited by 12
Relevance-aware prompt-tuning method for multimodal social entity and relation extraction
Chen, Z., Li, Z., Liu, M., Zhang, C., & Ma, H. (2025) · Neurocomputing · doi:10.1016/j.neucom.2025.130316 · Cited by 9
Knowledge-Guided Paraphrase Identification
Wang, H., Ma, F., Wang, Y., & Gao, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.72 · Cited by 11
“Recent works (Zhang et al., 2019; Liu et al., 2020; Xiong et al., 2019; Peters et al., 2019; Cui et al., 2020; Song et al., 2021; Hu et al., 2019; Ye et al., 2019) explore how to introduce knowledge graphs to enhance language models for downstream tasks.”
BackgroundKnowledge-Augmented Methods for Natural Language Processing
Zhu, C., Yi‐chong, X., Ren, X., Lin, B. Y., Jiang, M., & Yu, W. (2022) · doi:10.18653/v1/2022.acl-tutorials.3 · Cited by 19
Diversity matters: Robustness of bias measurements in Wikidata
Das, P., Karnam, S. K., Panda, A., Guda, B. P. R., Sarkar, S., & Mukherjee, A. (2023) · ACM Web Science Conference (WebSci) · doi:10.1145/3578503.3583620 · Cited by 13
“Academic research on KGs has been gaining significant interest for exploring knowledge in the light of representation of large-scale graph-structure data, reasoning, and application in a variety of automated tasks - recommendation [29], chatbot question answering [13], language modelling [23, 35] etc.”
BackgroundEnhancing text representations separately with entity descriptions
Zhao, Q., Lei, Y., Wang, Q., Kang, Z., & Liu, J. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126511 · Cited by 10
Knowledge-Enhanced Prompt Learning for Few-Shot Text Classification
Liu, J., & Yang, L. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8040043 · Cited by 13
“Although the research methods for knowledge-enhanced prompt learning are not numerous, there is a considerable amount of research on knowledge-enhanced pre-trained language models [32–34].”
RTA: A reinforcement learning-based temporal knowledge graph question answering model
Zhu, Y., Ma, T., Sun, S., Rong, H., Bian, Y., & Huang, K. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128994 · Cited by 11
Aejas, B., Belhi, A., & Bouras, A. (2025) · Sustainability · doi:10.3390/su17094215 · Cited by 13
Graph Representation Learning in Document Wikification
Saeidi, M., Milios, E., & Zeh, N. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86159-9_37 · Cited by 9
Prix-LM: Pretraining for Multilingual Knowledge Base Construction
Zhou, W., Liu, F., Vulić, I., Collier, N., & Chen, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.371 · Cited by 15
Introducing Semantic Information for Numerical Attribute Prediction over Knowledge Graphs
Xue, B., Li, Y., & Zou, L. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19433-7_1 · Cited by 13
A novel self-attention enriching mechanism for biomedical question answering
Kaddari, Z., & Bouchentouf, T. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120210 · Cited by 12
Improving Pre-trained Language Models
Paaß, G., & Giesselbach, S. (2023) · Artificial intelligence: foundations, theory, and algorithms/Artificial intelligence: Foundations, theory, and algorithms · doi:10.1007/978-3-031-23190-2_3 · Cited by 10
Foolad, S., & Kiani, K. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126786 · Cited by 9
“Recently, researchers have proposed methods such as ERNIE [10], Know-BERT [21], and LUKE, which treat entities as independent tokens besides input words.”
MethodNLIRE: A Natural Language Inference method for Relation Extraction
Hu, W., Liu, L., Sun, Y., Wu, Y., Liu, Z., Zhang, R., & Peng, T. (2021) · Journal of Web Semantics · doi:10.1016/j.websem.2021.100686 · Cited by 9
Enhancing Self-Attention with Knowledge-Assisted Attention Maps
Bai, J., Wang, Y., Sun, H., Wu, R., Yang, T., Tang, P., Cao, D., Zhang, M., Tong, Y., Yang, Y., et al. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.8 · Cited by 12
“KnowBERT (Peters et al., 2019) incorporates knowledge bases into BERT through Knowledge attention and re-contextualization.”
MethodDecouple knowledge from paramters for plug-and-play language modeling
Cheng, X., Lin, Y., Chen, X., Zhao, D., & Yan, R. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.901 · Cited by 21
LERCause: Deep learning approaches for causal sentence identification from nuclear safety reports
Kim, J., Kim, J., Lee, A., Kim, J., & Diesner, J. (2024) · PLoS ONE · doi:10.1371/journal.pone.0308155 · Cited by 8
Causality extraction model based on two-stage GCN
Zhu, G., Sun, Z., Zhang, S., Wei, S., & Li, K. (2022) · Soft Computing · doi:10.1007/s00500-022-07370-8 · Cited by 9
“The Bi-LSTM context (Peters et al. 2019) coding is mainly to improve the pre-training process.”
BackgroundLearning Sentiment-Enhanced Word Representations by Fusing External Hybrid Sentiment Knowledge
Li, Y., Lin, Z., Lin, Y., Yin, J., & Chang, L. (2023) · Cognitive Computation · doi:10.1007/s12559-023-10164-1 · Cited by 8
“Peters et al. [24] and Liu et al. [25] enhanced the pretrained language model by fusing manually constructed semantic knowledge graphs.”
MethodAPRE: Annotation-Aware Prompt-Tuning for Relation Extraction
Wei, C. Y., Chen, Y., Wang, K., Qin, Y., Huang, R., & Zheng, Q. (2024) · Neural Processing Letters · doi:10.1007/s11063-024-11437-y · Cited by 8
Grounded Compositional Outputs for Adaptive Language Modeling
Pappas, N., Mulcaire, P., & Smith, N. A. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.96 · Cited by 9
Self-citationEnhancing the Context Representation in Similarity-based Word Sense Disambiguation
Wang, M., Zhang, J., & Wang, Y. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.706 · Cited by 8
“Many systems are proposed to tackle WSD by employing BERT either by extracting features (Vial et al., 2019; Loureiro and Jorge, 2019) or fine-tuning (Peters et al., 2019; Levine et al., 2020).”
Bridging the Language Gap: Knowledge Injected Multilingual Question Answering
Duan, Z., Li, X., Zhang, Z., Li, Z., Liu, N., & Wang, J. (2021) · IEEE International Conference on Big Knowledge (ICBK) · doi:10.1109/ickg52313.2021.00052 · Cited by 14
Park, J., & Cho, S. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121021 · Cited by 8
A Multilingual Bag-of-Entities Model for Zero-Shot Cross-Lingual Text Classification
Nishikawa, S., Yamada, I., Tsuruoka, Y., & Echizen, I. (2022) · Conference on Computational Natural Language Learning · doi:10.18653/v1/2022.conll-1.1 · Cited by 8
Zhang, T., Dong, J., Wang, J., Wang, C., Wang, A., Liu, Y., Huang, J. G., Li, Y., & He, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-industry.57 · Cited by 12
Fan, C., Tian, J., Li, Y., Chen, W., He, H., & Jin, Y. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.913 · Cited by 11
TCohPrompt: task-coherent prompt-oriented fine-tuning for relation extraction
Long, J., Yin, Z., Liu, C., & Huang, W. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01563-4 · Cited by 8
“KnowBERT [22], an extension of BERT, incorporates knowledge graph information during pre-training, enabling the model to better capture semantic relationships between entities and concepts, and leading to improved performance on knowledge-intensive natural language processing tasks.”
Enhancing Online Knowledge Graph Population with Semantic Knowledge
Fernàndez-Cañellas, D., Rimmek, J. M., Espadaler, J., Garolera, B., Barja, A., Codina, M., Sastre, M., Giró-i-Nieto, X., Riveiro, J. C., & Bou‐Balust, E. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-62419-4_11 · Cited by 11
“Other works like ERNIE [41], KG-BERT [37] or KnowBert [24] propose enhanced language representations by incorporating external knowledge from KGs.”
MethodZhao, Z., Papalexakis, E. E., & Ma, X. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.266 · Cited by 10
“Knowledge graphs have been shown to be useful for many NLP tasks, such as contextual word embedding (Peters et al., 2019), text classification (K M et al., 2018), and language generation (Zhou et al., 2018).”
BackgroundTyping Errors in Factual Knowledge Graphs: Severity and Possible Ways Out
Yao, P., & Barbosa, D. (2021) · The Web Conference · doi:10.1145/3442381.3449977 · Cited by 9
“3449977 recommendation systems [6], and contextualized language models [25].”
BackgroundExploring the Combination of Contextual Word Embeddings and Knowledge Graph EmbeddingsPreprint
Dieudonat, L., Han, K., Leavitt, P., & Marquer, E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.08371 · Cited by 8
“s the task of predicting the type of an entity represented by a KG node or a word. Entity typing is typically used for KB embeddings as type information is usually included in the knowledge base, but [22] presents a contextual embeddings version. This family of tasks can be interpreted as a measure of whether the mode…”
BackgroundMethodInfluentialComputational Job Market Analysis with Natural Language ProcessingPreprint
Zhang, M. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2404.18977 · Cited by 12
Multi-level Mixture of Experts for Multimodal Entity Linking
Hu, Z., Gutiérrez-Basulto, V., Xiang, Z., Li, R., & Pan, J. Z. (2025) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3711896.3737060 · Cited by 7
From Symbolic to Neural and Back: Exploring Knowledge Graph–Large Language Model Synergies
Škrlj, B., Koloski, B., Pollak, S., & Lavrač, N. (2025) · Lecture notes in computer science · doi:10.1007/978-3-032-03028-3_11 · Cited by 9
Knowledge Enhanced BERT Based on Corpus Associate Generation
Jiarong, L., Hong, X., Jiang, W., Yang, J., & Tao, W. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-20102-8_41 · Cited by 7
Content-based and Knowledge-enriched Representations for Classification Across Modalities: A Survey
Pittaras, N., Giannakopoulos, G., Stamatopoulos, P., & Karkaletsis, V. (2023) · ACM Computing Surveys · doi:10.1145/3583682 · Cited by 8
“Entity-aware language models for text are investigated in knowBERT [Peters et al. 2019], providing approaches for integrating pretrained models with generic knowledge bases structured as triplets or graphs.”
MethodRelation-Guided Pre-Training for Open-Domain Question Answering
Hu, Z., Sun, Y., & Chang, K.-W. (2021) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2021.findings-emnlp.292 · Cited by 7
Lifelong topic modeling with knowledge-enhanced adversarial network
Zhang, X., Rao, Y., & Li, Q. (2021) · World Wide Web · doi:10.1007/s11280-021-00984-2 · Cited by 8
“Recently, developing specialized architectures is widely studied to process knowledge, including attention network based methods [7, 13, 18, 46], graph neural network based methods [61, 62], and memory network based methods [34, 58].”
BackgroundPositive Unlabeled Learning by Semi-Supervised Learning
Wang, Z., Jiang, J. B., & Long, G. (2022) · Proceedings - International Conference on Image Processing · doi:10.1109/icip46576.2022.9897738 · Cited by 7
“For 20NEWS, we preprocess the raw text data into 9216-dimensional feature vectors using ELMo [18] following [17].”
MethodSelf-Attention Over Tree for Relation Extraction With Data-Efficiency and Computational Efficiency
Lyu, S., Zhou, X., Wu, X., Chen, Q., & Chen, H. (2023) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2023.3286268 · Cited by 7
“KnowBert-W+W [54] inserted WordNet [55] and a subset of Wikipedia into BERT to produce knowledge enhanced entity-span representations.”
MethodAdapters for Enhanced Modeling of Multilingual Knowledge and Text
Hou, Y., Jiao, W., Liu, M., Allen, C., Tu, Z., & Sachan, M. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.287 · Cited by 7
PURE: a Prompt-based framework with dynamic Update mechanism for educational Relation Extraction
Cui, X., Yang, Y., Li, D., Cui, J., Qu, X., Song, C. L., Liu, H., & Ke, S. (2024) · Complex & Intelligent Systems · doi:10.1007/s40747-024-01692-w · Cited by 9
“• KnowBERT [15] incorporates knowledge bases into BERT using the knowledge attention and recontextual-ization mechanism.”
Exploring Neural Entity Representations for Semantic Information
Runge, A., & Hovy, E. H. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.20 · Cited by 8
An empirical analysis of existing systems and datasets toward general simple question answering
Han, N., Topić, G., Noji, H., Takamura, H., & Miyao, Y. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.465 · Cited by 9
“…on SimpleQuestion has not outperformed a simpler baseline so far (Lukovnikov et al., 2019), and it is also nontrivial to integrate BERT with knowledge graph embeddings, which is necessary for KEQA-based approach and is currently actively studied (Peters et al., 2019; Weijie et al., 2020).”
BackgroundDo Models Learn the Directionality of Relations? A New Evaluation: Relation Direction Recognition
Lyu, S., Wu, X., Li, J., Chen, Q., & Chen, H. (2022) · IEEE Transactions on Emerging Topics in Computational Intelligence · doi:10.1109/tetci.2021.3136598 · Cited by 8
Jin, W., Lee, D.-H., Zhu, C., Pujara, J., & Ren, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.196 · Cited by 13
GraphCache: Message Passing as Caching for Sentence-Level Relation Extraction
Wang, Y., Chen, M., Zhou, W., Cai, Y., Liang, Y., & Hooi, B. (2022) · Findings of the Association for Computational Linguistics: NAACL · doi:10.18653/v1/2022.findings-naacl.128 · Cited by 11
A hybrid attention mechanism for multi-target entity relation extraction using graph neural networks
Javeed, A. (2022) · Machine Learning with Applications · doi:10.1016/j.mlwa.2022.100444 · Cited by 7
Incremental BERT with commonsense representations for multi-choice reading comprehension
Li, R., Wang, L., Jiang, Z., Liu, D., Zhao, M., & Lu, X. (2021) · Multimedia Tools and Applications · doi:10.1007/s11042-021-11197-0 · Cited by 7
“Methods in the first group design auxiliary knowledge-driven objectives and updating parameters in a multi-task learning manner [24, 37, 39, 48], which requires pre-calculating knowledge representation and even pre-training BERT from scratch.”
BackgroundLiu, Z., Lin, Y., & Sun, M. (2020) · Springer eBooks · doi:10.1007/978-981-15-5573-2_4 · Cited by 8
Enhancing Targeted Minority Class Prediction in Sentence-Level Relation Extraction
Baek, H.-R., & Choi, Y. S. (2022) · Sensors · doi:10.3390/s22134911 · Cited by 10
Zhang, T., Cai, Z., Wang, C., Qiu, M., Yang, B., & He, X. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.08983 · Cited by 12
“KEPLMs: We employ two SOTA KEPLMs continually pre-trained on our medical corpora as our baseline models, including ERNIE-THU (Zhang et al., 2019) and KnowBERT (Peters et al., 2019).”
InfluentialLearning Sense-Specific Static Embeddings using Contextualised Word Embeddings as a ProxyPreprint
Zhou, Y., & Bollegala, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.02204 · Cited by 7
SYNERGY: Building Task Bots at Scale Using Symbolic Knowledge and Machine TeachingPreprint
Peng, B., Li, C., Zhang, Z., Li, J., Zhu, C., & Gao, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.11514 · Cited by 7
“In the natural language processing community, several works have explored leveraging knowledge for learning contextual word representations (Peters et al., 2019; Wang et al., 2020), open-domain response generation (Ghazvininejad et al., 2018), commonsense reasoning (Xu et al., 2020), etc.”
BackgroundMixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERTPreprint
Meng, Z., Liu, F., Clark, T. H., Shareghi, E., & Collier, N. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.04810 · Cited by 9
“…on a computationally expensive joint training of an underlying masked language model (MLM) along with a knowledge-infusion objective function to minimize the risk of catastrophic forgetting (Xiong et al., 2019; Zhang et al., 2019; Wang et al., 2021, 2020; Peters et al., 2019; Yuan et al., 2021).”
BackgroundMethodPitchclass2vec: Symbolic Music Structure Segmentation with Chord EmbeddingsPreprint
Lazzari, N., Poltronieri, A., & Presutti, V. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.15306 · Cited by 7
“To obtain a semantically richer representations we plan on enhancing pitchclass2vec by using deep contextual word embeddings [50] along with knowledge enhancement techniques [51] that combines domain-specific ontologies, such as [52], with deep contextual word embeddings.”
MethodLLM-Based Text Style Transfer: Have We Taken a Step Forward?
Toshevska, M., & Gievska, S. (2025) · IEEE Access · doi:10.1109/access.2025.3548967 · Cited by 22
TEQA: Temporal knowledge graph enhanced question answering
Liu, Q., Feng, S., & Huang, M. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.113916 · Cited by 10
Enhanced semantic representation model for multisource point of interest attribute alignment
Li, P., Wang, Y., Liu, J., Luo, A., Xu, S., & Zhang, Z. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101852 · Cited by 7
M3TQA: Multi-View, Multi-Hop and Multi-Stage Reasoning for Temporal Question Answering
Zha, Z., Qi, P., Bao, X., Tian, M., & Qin, B. (2024) · IEEE International Conference on Acoustics Speech and Signal Processing · doi:10.1109/icassp48485.2024.10448071 · Cited by 10
“(I) PLMs includes BERT, RoBERTa, KnowBERT[23] and T5.”
MethodGenKP: generative knowledge prompts for enhancing large language models
Li, X., Peng, S., Yada, S., Wakamiya, S., & Aramaki, E. (2025) · Applied Intelligence · doi:10.1007/s10489-025-06318-3 · Cited by 6
“The practice of adopting knowledge-inserted parameter optimization training in small-scale models, such as BERT [19] and its variants, has achieved improvement [9, 16, 17, 20, 21].”
Towards Semantics-Enhanced Pre-Training: Can Lexicon Definitions Help Learning Sentence Meanings?
Ren, X., Sun, X., Wang, H., & Liu, Q. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17619 · Cited by 6
“To remedy this, several extensions are proposed to incorporate world knowledge (Sun et al. 2019; Zhang et al. 2019; Peters et al. 2019) or commonsense (Ye et al. 2019; Levine et al. 2020) into BERT, but most of them focused on specific downstream tasks, such as commonsense question answering (Ye et…”
BackgroundEnhancing Semantic Relation Classification With Shortest Dependency Path Reasoning
Li, J., Shuang, K., Guo, J., Shi, Z., & Wang, H. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3265205 · Cited by 15
“[22] pre-train relation representations using distant supervision from the entity-linked text.”
BackgroundKnowledge Graph Enhanced Language Models for Sentiment Analysis
Li, J., Li, X., Hu, L., Zhang, Y., & Wang, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47240-4_24 · Cited by 9
Metadata
- Authors
- Peters, M. E., Neumann, M., Logan, R., Schwartz, R., Joshi, V., Singh, S., & Smith, N. A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- University of California, Irvine (US)
- University of Washington (US)
- Corresponding author
- Singh, Sameer
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 43-54
- OpenAlex
- W2970986510
- Cited by
- 661
- References
- 72
- Citations by year
- 2026: 21 · 2025: 54 · 2024: 81 · 2023: 125 · 2022: 115 · 2021: 160 · 2020: 98 · 2019: 6
- FWCI
- 36.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- National Science Foundation (1730158, 1817183)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- contextualized word representations, word embeddings, knowledge graphs, word sense disambiguation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.