Knowledge Enhanced Contextual Word Representations
Open accessThe DOI matches its Crossref record; 71 of the 72 references it cites were found in a registry or an index.
Abstract
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A. Smith. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“XLM-R achieves state-of-the-arts results on multiple cross-lingual benchmarks, including XNLI, MLQA, and NER.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [62]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“We compute M m scores using the prior and dot product between the entity-span vectors and entity embeddings, with a two-layer MLP (100 hidden dimensions).”
BackgroundMethodInfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 4×
“, 2018), and BERT (Devlin et al., 2019) have significantly improved the state of the art for a wide range of NLP tasks.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 4×
“, 2018), and BERT (Devlin et al., 2019) have significantly improved the state of the art for a wide range of NLP tasks.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Pretrained word representations Initial work learning word vectors focused on static word embeddings using multi-task learning objectives (Collobert and Weston, 2008) or corpus level cooccurence statistics (Mikolov et al., 2013a; Pennington et al., 2014).”
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 3×
“Pretrained word representations Initial work learning word vectors focused on static word embeddings using multi-task learning objectives (Collobert and Weston, 2008) or corpus level co-occurence statistics (Mikolov et al., 2013a; Pen-nington et al., 2014).”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 3×
“The entity embeddings use a skipgram like objective (Mikolov et al., 2013b) to learn 300-dimensional embeddings of Wikipedia”
BackgroundMethodWordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 1×
“We experiment with integrating both WordNet (Miller, 1995) and Wikipedia, thus explicitly adding word sense knowledge and facts about named entities (including those unseen at training time).”
MethodDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 6×
“Recently the field has shifted toward learning context-sensitive embeddings (Dai and Le, 2015; Peters et al., 2018; Devlin et al., 2019).”
BackgroundMethodInfluentialSelf-citation(2013) · Cited by 5,726
Research works printed but not linked: 5
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013b. Distributed representa- tions of words and phrases and their compositional- ity. In NeurIPS.
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL-HLT.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL.
- Bishan Yang and Tom Michael Mitchell. 2017. Lever- aging knowledge bases in LSTMs for improving ma- chine reading. In ACL.
- Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: En- hanced language representation with informative en- tities. In ACL.
Cited by
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
“114M Yes KnowBERT [94] Transformer Enc.”
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E. H., Feng, J., & Yan, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00060 · Cited by 2,415
“Transformers in Vision Transformers [37] are the models that entirely rely on the self-attention mechanism to draw global dependencies between input and output, which has dominated in natural language modelling [10, 30, 2, 46, 29, 23].”
BackgroundTokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E., Feng, J., & Yan, S. (2021) · arXiv · doi:10.48550/arxiv.2101.11986 · Cited by 2,415
“Transformers in Vision Transformers [37] are the models that entirely rely on the self-attention mechanism to draw global dependencies between input and output, which has dominated in natural language modelling [10, 30, 2, 46, 29, 23].”
BackgroundPre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
“The external knowledge ranges from linguistic [94, 83, 136, 191], semantic [99], commonsense [52], factual [214, 136, 111, 202, 195], to domain-specific knowledge [58, 111].”
BackgroundMethodInfluentialA primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552
How Can We Know What Language Models Know?
Jiang, Z., Xu, F. F., Araki, J., & Neubig, G. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00324 · Cited by 1,002
“Similar extensions have been applied to pre-trained LMs like BERT, where contextualized representations are enhanced with entity embeddings (Zhang et al., 2019; Peters et al., 2019; Pörner et al., 2019).”
BackgroundMethodResultInfluentialA comprehensive survey on pretrained foundation models: a history from BERT to ChatGPT
Zhou, C., Li, Q., Li, C., Yu, J., Liu, Y., Wang, G., Zhang, K., Ji, C., Yan, Q., He, L., et al. (2025) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-024-02443-6 · Cited by 557
von Rueden, L., Mayer, S., Beckh, K., Georgiev, B., Giesselbach, S., Heese, R., Kirsch, B., Walczak, M., Pfrommer, J., Pick, A., et al. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3079836 · Cited by 793
“ERNIE [86], which are fed into language models like BERT [94], [119].”
MethodInfluentialPre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994
LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
Yamada, I., Asai, A., Shindo, H., Takeda, H., & Matsumoto, Y. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.523 · Cited by 590
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., & Tang, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00360 · Cited by 676
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Wang, R., Tang, D., Duan, N., Wei, Z., Huang, X., Ji, J., Cao, G., Jiang, D., & Zhou, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.121 · Cited by 456
A Frustratingly Easy Approach for Entity and Relation Extraction
Zhong, Z., & Chen, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.5 · Cited by 477
“This idea of using additional markers to high-light the subject and object is not entirely new as it has been studied recently in relation classification (Zhang et al., 2019; Soares et al., 2019; Peters et al., 2019).”
BackgroundREALM: Retrieval-Augmented Language Model Pre-Training
Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M.-W. (2020) · arXiv · doi:10.48550/arxiv.2002.08909 · Cited by 574
“We are particularly optimistic about generalizations of this work to (1) structured knowledge, which would result in a generalization of Peters et al. (2019) where we would also learn the decision of which entities are informative, (2) the multi-lingual setting, e.g., retrieving knowledge in a…”
BackgroundPTR: Prompt Tuning with Rules for Text Classification
Han, X., Zhao, W., Ding, N., Liu, Z., & Sun, M. (2022) · AI Open · doi:10.1016/j.aiopen.2022.11.003 · Cited by 467
A Frustratingly Easy Approach for Entity and Relation Extraction
Zhong, Z., & Chen, D. (2020) · arXiv · doi:10.48550/arxiv.2010.12812 · Cited by 531
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
Yin, P., Neubig, G., Yih, W.-T., & Riedel, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.745 · Cited by 487
How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence
Zhong, H., Xiao, C., Tu, C., Zhang, T., Liu, Z., & Sun, M. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.466 · Cited by 367
PTR: Prompt Tuning with Rules for Text Classification
Han, X., Zhao, W., Ding, N., Liu, Z., & Sun, M. (2021) · arXiv · doi:10.48550/arxiv.2105.11259 · Cited by 478
“, 2020), KNOWBERT (Peters et al., 2019), LUKE (Yamada et al.”
Pre-Trained Language Models and Their Applications
Wang, H., Li, J., Wu, H., Hovy, E., & Sun, Y. (2023) · Engineering · doi:10.1016/j.eng.2022.04.024 · Cited by 385
KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction
Chen, X., Zhang, N., Xie, X., Deng, S., Yao, Y., Tan, C., Huang, F., Si, L., & Chen, H. (2022) · Proceedings of the ACM Web Conference 2022 · doi:10.1145/3485447.3511998 · Cited by 389
“SPANBERT [30] adopt knowledge to enhance learning objectives, KNOWBERT [38] propose to incorporate knowledge into input features, and LUKE [52] leverage knowledge to improve model architectures.”
MethodInfluentialOn the Opportunities and Challenges of Foundation Models for GeoAI (Vision Paper)
Mai, G., Huang, W., Sun, J., Song, S., Mishra, D. R., Liu, N., Gao, S., Liu, T., Cong, G., Hu, Y., et al. (2024) · ACM Transactions on Spatial Algorithms and Systems · doi:10.1145/3653070 · Cited by 444
“— BLIP : We use the pre-trained BLIP-2 model [101] provided by Huggingface 10 that consists of a CLIP-like image encoder, a Querying Transformer (Q-Former), and a large language model (Flan T5-xl).”
MethodAMMUS : A Survey of Transformer-based Pretrained Models in Natural Language ProcessingPreprint
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.05542 · Cited by 426
“Some of the examples of knowledge enriched T-PTLMs are CasualBERT [197], KnowBERT [189], SenseBERT [204], LIMIT-BERT [199], LiBERT [198], SentiLARE [200], ERNINE [188], E-BERT [186] in the general domain and Clincal Kb-BERT [201], Clinical Kb-ALBERT [201], SapBERT [202], Sap-XLMR [202], UmlsBERT [127], CoderBERT [203]…”
BackgroundMethodInfluentialA Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
Patil, R., & Gudivada, V. (2024) · Applied Sciences · doi:10.3390/app14052074 · Cited by 290
“In KnowBert [70], multiple knowledge bases were used during pretraining to enhance the representations further.”
MethodVOLO: Vision Outlooker for Visual Recognition
Yuan, L., Hou, Q., Jiang, Z., Feng, J., & Yan, S. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2022.3206108 · Cited by 288
Multimodal Foundation Models: From Specialists to General-Purpose Assistants
Li, C., Gan, Z., Yang, Z., Yang, J., Li, L., Wang, L., & Gao, J. (2024) · Foundations and Trends® in Computer Graphics and Vision · doi:10.1561/0600000110 · Cited by 128
A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
Patil, R., & Gudivada, A. V. (2024) · doi:10.20944/preprints202402.0357.v1 · Cited by 296
A Survey of Text Representation and Embedding Techniques in NLP
Patil, R., Boit, S., Gudivada, V., & Nandigam, J. (2023) · IEEE Access · doi:10.1109/access.2023.3266377 · Cited by 214
“KnowBert [83] proposes a general method to embed multiple knowledge bases (KBs) into large-scale models and thereby enhance their representations with structured, human-curated”
MethodLarge-scale Multi-modal Pre-trained Models: A Comprehensive Survey
Wang, X., Chen, G., Qian, G., Gao, P., Wei, X.-Y., Wang, Y., Tian, Y., & Gao, W. (2023) · Machine Intelligence Research · doi:10.1007/s11633-022-1410-8 · Cited by 223
“Similar work named KnowBert is also proposed for largescale models to embed multiple knowledge bases with entity linkers, which retrieves relevant entity embeddings and updates contextual word representations by the word-to-entity attention [223].”
MethodA Survey of Knowledge Enhanced Pre-Trained Language Models
Hu, L., Liu, Z., Zhao, Z., Hou, L., Nie, L., & Li, J. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3310002 · Cited by 196
“KnowBert [78] retrieves entity embeddings related to the input text via an entity linker and then updates contextual word representations through word-to-entity attention, allowing the long-range interactions between words and all entities.”
BackgroundMethodYang, L., Chen, H., Li, Z., Ding, X., & Wu, X. (2023) · arXiv · doi:10.48550/arxiv.2306.11489 · Cited by 239
“For instance, KnowBERT [95] incorporates a knowledge attention recontextualization module to integrate multiple KGs into a PLM.”
MethodYang, L., Chen, H., Li, Z., Ding, X., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3360454 · Cited by 238
“For instance, KnowBERT [95] incorporates a knowledge attention recontextualization module to integrate multiple KGs into a PLM.”
MethodMeasuring and improving consistency in pretrained language models
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00410 · Cited by 220
“The recent rise of PLMs has sparked a discussion about whether these models can be used as Knowledge Bases (KBs) (Petroni et al., 2019, 2020; Davison et al., 2019; Peters et al., 2019; Jiang et al., 2020; Roberts et al., 2020).”
BackgroundLearning from Context or Names? An Empirical Study on Neural Relation Extraction
Peng, H., Gao, T., Han, X., Lin, Y., Li, P., Liu, Z., Sun, M., & Zhou, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.298 · Cited by 185
Neural entity linking: A survey of models based on deep learning
Sevgili, Ö., Shelmanov, A., Arkhipov, M., Panchenko, A., & Biemann, C. (2022) · Semantic Web · doi:10.3233/sw-222986 · Cited by 286
“gs in the unified architecture. Most of the state-of-the-art studies compare mention and entity representations using a dot product (Ganea and Hofmann, 2017; Gupta et al., 2017; Kolitsas et al., 2018; Peters et al., 2019; Wu et al., 2020) or cosine similarity (Sun et al., 2015; Francis-Landau et al., 2016; Gillick et a…”
BackgroundMethodInfluentialTURL: table understanding through representation learning
Deng, X., Sun, H., Lees, A., Wu, Y., & Yu, C. (2020) · Proceedings of the VLDB Endowment · doi:10.14778/3430915.3430921 · Cited by 271
SenseBERT: Driving Some Sense into BERT
Levine, Y., Lenz, B., Dagan, O., Ram, O., Padnos, D., Sharir, O., Shalev-Shwartz, S., Shashua, A., & Shoham, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.423 · Cited by 162
SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge
Ke, P., Ji, H., Liu, S., Zhu, X., & Huang, M. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.567 · Cited by 143
“…variants of BERT have been proposed, which mainly fall into four aspects: 1) Knowledge enhancement: ERNIE-Tsinghua (Zhang et al., 2019) / KnowBERT (Peters et al., 2019) explicitly introduces knowledge graph / knowledge base to BERT, while ERNIE-Baidu (Sun et al., 2019b) designs entity-specific…”
BackgroundPretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model
Xiong, W., Du, J., Wang, W. Y., & Stoyanov, V. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1912.09637 · Cited by 23
“Compared with previous work (Zhang et al., 2019; Peters et al., 2019) that utilize external knowledge base to incorporate entity knowledge, our method is able to directly derive real-world knowledge from unstructured text.”
BackgroundMethodCoLAKE: Contextualized Language and Knowledge Embedding
Sun, T., Shao, Y., Qiu, X., Guo, Q., Hu, Y., Huang, X., & Zhang, Z. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.327 · Cited by 185
Coreferential Reasoning Learning for Language Representation
Ye, D., Lin, Y., Du, J., Liu, Z., Li, P., Sun, M., & Liu, Z. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.582 · Cited by 163
Yamada, I., Asai, A., Sakuma, J., Shindo, H., Takeda, H., Takefuji, Y., & Matsumoto, Y. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.4 · Cited by 164
“Notably, recent studies have also shown that these embeddings can be used to enhance the performance of state-of-the-art contextualized word embeddings (i.e., BERT (Devlin et al., 2019)) on downstream tasks (Zhang et al., 2019; Peters et al., 2019; Poerner et al., 2019).”
BackgroundA comprehensive overview of knowledge graph completion
Shen, T., Zhang, F., & Cheng, J. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109597 · Cited by 209
TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task
Alt, C., Gabryszak, A., & Hennig, L. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.142 · Cited by 132
GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing
Yu, T., Wu, C.-S., Lin, X., Wang, B., Tan, Y. C., Yang, X., Radev, D., Socher, R., & Xiong, C. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.13845 · Cited by 54
“This triggered an exciting line of research work under the themes of (1) cross-modal pre-training that involves text (Lu et al., 2019; Peters et al., 2019; Yin et al., 2020a; Herzig et al., 2020a) and (2) pre-training architectures and objectives catering subsets of NLP tasks (Lewis et al.,…”
BackgroundKnowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training
Agarwal, O., Ge, H., Shakeri, S., & Al-Rfou, R. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.278 · Cited by 161
“…KGs Most prior works on incorporating KG with text often learn KG entity representations and add them to the mention spans linked to the entity (Peters et al., 2019; Yu et al., 2020; Févry et al., 2020) or create subgraphs relevant to the query that are expanded with text in the embedding…”
BackgroundConvBERT: Improving BERT with Span-based Dynamic Convolution
Jiang, Z., Yu, W., Zhou, D., Chen, Y., Feng, J., & Yan, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.02496 · Cited by 126
“There are also some works that extend pre-training by incorporating knowledge [73, 47, 37, 68], multiple languages [22, 9, 6], and multiple modalities [39, 58, 5, 59, 7].”
BackgroundEntities as Experts: Sparse Memory Access with Entity Supervision
Févry, T., Baldini Soares, L., FitzGerald, N., Choi, E., & Kwiatkowski, T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.400 · Cited by 130
Complex Knowledge Base Question Answering: A Survey
Lan, Y., He, G., Jiang, J., Jiang, J., Zhao, W. X., & Wen, J.-R. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3223858 · Cited by 117
“Similar ideas were also adopted in knowledge-enhanced language model pre-training [98], [99].”
ResultBevilacqua, M., & Navigli, R. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.255 · Cited by 229
“…into neural networks for Natural Language Processing (NLP) tasks, be it through pretraining on self-supervised tasks such as language modeling (Peters et al., 2018; Devlin et al., 2019), or through the incorporation of information from knowledge bases (Peters et al., 2019; Logan et al., 2019).”
BackgroundHittER: Hierarchical Transformers for Knowledge Graph Embeddings
Chen, S., Liu, X., Gao, J., Jiao, J., Zhang, R., & Ji, Y. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.812 · Cited by 197
“Unlike the previous shallow KGE methods that cannot be trivially utilized by widely used Transformer-based models for language tasks (Peters et al., 2019), our approach benefits from the unified Transformer architecture and its extensibility.”
MethodE-BERT: Efficient-Yet-Effective Entity Embeddings for BERT
Poerner, N., Waltinger, U., & Schütze, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.71 · Cited by 146
“Figure 3 shows that gains of E-BERT-concat over BERT, KnowBert and ERNIE in terms of mean Hits@k are especially big for k > 1 .”
BackgroundMethodInfluentialZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations
Diao, S., Bai, J., Song, Y., Zhang, T., & Wang, Y. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.425 · Cited by 137
Zhao, K., Xu, H., Cheng, Y., Li, X., & Gao, K. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.106888 · Cited by 157
Packed Levitated Marker for Entity and Relation Extraction
Ye, D., Lin, Y., Li, P., & Sun, M. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.337 · Cited by 144
“Zhang et al. (2019); Peters et al. (2019); 151 Wang et al. (2021a) learn to use the external en- 152 tity embedding from the knowledge graph or the 153 synonym net to acquire knowledge.”
BackgroundJAKET: Joint Pre-training of Knowledge Graph and Language Understanding
Yu, D., Zhu, C., Yang, Y., & Zeng, M. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21417 · Cited by 126
Multi-Task Learning for Knowledge Graph Completion with Pre-trained Language Models
Kim, B., Hong, T., Ko, Y., & Seo, J. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.153 · Cited by 181
“In recent years, Natural Language Processing (NLP) has benefited from utilizing KGs in various applications such as language modeling (Peters et al., 2019; Liu et al., 2019a), question answering (Zhang et al., 2019; Huang et al., 2019), and machine reading (Yang and Mitchell, 2017).”
Evaluating the Ripple Effects of Knowledge Editing in Language Models
Cohen, R., Biran, E., Yoran, O., Globerson, A., & Geva, M. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00644 · Cited by 134
“…(Roberts et al., 2020; Zhang et al., 2021), semi-parametric methods, where external information is added from a knowledge-base (Zhang et al., 2019; Peters et al., 2019; Lewis et al., 2020; Zhang et al., 2022), using adapters to store knowledge (Wang et al., 2021a), or extending the MLP layers…”
MethodVision Transformer with Progressive Sampling
Yue, X., Sun, S., Kuang, Z., Wei, M., Torr, P. H. S., Zhang, W., & Lin, D. (2021) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00044 · Cited by 159
“Benefiting from their powerful global relation modeling abilities, and highly ef-ficient training, transformers achieve significant improvements and become the de-facto standard in many Natural Language Processing (NLP) tasks [11, 3, 30, 29, 48].”
BackgroundDialogue-Based Relation Extraction
Yu, D., Sun, K., Cardie, C., & Yu, D. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.444 · Cited by 160
Qin, Y., Lin, Y., Takanobu, R., Liu, Z., Li, P., Ji, H., Huang, M., Sun, M., & Zhou, J. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.260 · Cited by 92
“Others modify the internal structures of PLMs to fuse both textual and KG’s information (Zhang et al., 2019; Peters et al., 2019; Wang et al., 2020; He et al., 2020).”
BackgroundCalibration of Transformer-Based Models for Identifying Stress and Depression in Social Media
Ilias, L., Mouzakitis, S., & Askounis, D. (2023) · IEEE Transactions on Computational Social Systems · doi:10.1109/tcss.2023.3283009 · Cited by 154
“For this reason, there have been proposed methods for enhancing these models with external information or additional modalities [11], [12], [13], [14].”
MethodHe, Y., Zhu, Z., Zhang, Y., Chen, Q., & Caverlee, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.372 · Cited by 103
Affective and Contextual Embedding for Sarcasm Detection
Babanejad, N., Davoudi, H., An, A., & Papagelis, M. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.20 · Cited by 156
“While there are different ways to combine these two embeddings to achieve a meta-embedding (Kiela et al., 2018; Coates and Bollegala, 2018; Peters et al., 2019), we combine these two contextual embeddings by a simple concatenation to obtain the final embedding C = A ⊕ B for a token.”
BackgroundCPM: A large-scale generative Chinese Pre-trained language model
Zhang, Z., Han, X., Zhou, H., Ke, P., Gu, Y., Ye, D., Qin, Y., Su, Y., Ji, H., Guan, J., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.07.001 · Cited by 106
Transformers: “The End of History” for Natural Language Processing?
Chernyavskiy, A., Ilvovsky, D., & Nakov, P. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-86523-8_41 · Cited by 147
A Survey of Knowledge Enhanced Pre-trained Language Models
Yang, J., Hu, X., Xiao, G., & Shen, Y. (2024) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3631392 · Cited by 47
“The line thickness of the graph represents the quantity of each type.”
BackgroundInfluentialOntoProtein: Protein Pretraining With Gene Ontology Embedding
Zhang, N., Bi, Z., Liang, X., Cheng, S., Hong, H., Deng, S., Lian, J., Zhang, Q., & Chen, H. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2201.11147 · Cited by 49
“…which has inspired knowledge-enhanced models including ERNIE (Tsinghua) (Zhang et al. (2019b)), ERNIE (Baidu) (Sun et al. (2019)), KnowBERT (Peters et al. (2019)), WKLM (Xiong et al. (2020)), LUKE (Yamada et al. (2020)), KEPLER (Wang et al. (2021b)), K-BERT (Liu et al. (2020)), K-Adaptor…”
Improving Biomedical Pretrained Language Models with Knowledge
Yuan, Z., Liu, Y., Tan, C., Huang, S., & Huang, F. (2021) · Proceedings of the 20th Workshop on Biomedical Language Processing · doi:10.18653/v1/2021.bionlp-1.20 · Cited by 79
Scarlini, B., Pasini, T., & Navigli, R. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.285 · Cited by 141
“More-over, BERT ARES performs better than or on a par with its closest competitors, i.e., KnowBert, SenseBERT LARGE and T5 (Large, and 3B), which, instead, rely on more complex architectures, spe-cific pre-training phases and between 3000 M and 40 M more parameters.”
BackgroundMethodInfluentialRepresentation Learning for Natural Language Processing
Liu, Z., Lin, Y., & Sun, M. (2020) · Springer eBooks · doi:10.1007/978-981-15-5573-2 · Cited by 140
Survey of BERT-Base Models for Scientific Text Classification: COVID-19 Case Study
Khadhraoui, M., Bellâaj, H., Ammar, M. B., Hamam, H., & Jmaïel, M. (2022) · Applied Sciences · doi:10.3390/app12062891 · Cited by 117
Analysis and Evaluation of Language Models for Word Sense Disambiguation
Loureiro, D., Rezaee, K., Pilehvar, M. T., & Camacho-Collados, J. (2021) · Computational Linguistics · doi:10.1162/coli_a_00405 · Cited by 140
“Finally, a few works have attempted to induce semantic priors coming from knowledge resources like WordNet to improve the generalization of pre-trained language models like BERT (Levine et al. 2020; Peters et al. 2019).”
BackgroundArtificial Cognition for Early Leaf Disease Detection using Vision Transformers
Thai, H.-T., Tran-Van, N.-Y., & Le, K.-H. (2021) · International Conference on Advanced Technologies for Communications · doi:10.1109/atc52653.2021.9598303 · Cited by 106
JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs
Ke, P., Ji, H., Ran, Y., Cui, X., Wang, L., Song, L., Zhu, X., & Huang, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.223 · Cited by 92
Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge
Chen, Z., Li, G., & Wan, X. (2022) · Proceedings of the 30th ACM International Conference on Multimedia · doi:10.1145/3503161.3547948 · Cited by 82
“According to the knowledge injection schemes, existing studies can be classified into four categories: embeddings combination [41, 59], data structure compatibility [16, 32, 45], knowledge supervision [46, 49], and neural-symbolic methods [2].”
BackgroundQuestion Answering Over Temporal Knowledge Graphs
Saxena, A., Chakrabarti, S., & Talukdar, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.520 · Cited by 91
“BERT: We experiment with BERT, RoBERTa (Liu et al., 2019) and KnowBERT (Peters et al., 2019) which is a variant of BERT where information from knowledge bases such as WikiData and WordNet has been injected into BERT.”
MethodInfluentialInteractive Natural Language ProcessingPreprint
Wang, Z., Zhang, G., Yang, K., Shi, N., Zhou, W., Hao, S., Xiong, G., Li, Y., Sim, M. Y., Chen, X., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.13246 · Cited by 112
Conceptualized Representation Learning for Chinese Biomedical Text MiningPreprint
Zhang, N., Jia, Q., Yin, K., Liang, D., Gao, F., & Hua, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.10813 · Cited by 103
“Moreover, recent studies [6] have shown that injecting extra knowledge information can significantly enhance original models, such as knowledge acquisition [10].”
From static to dynamic word representations: a survey
Wang, Y., Hou, Y., Che, W., & Liu, T. (2020) · International Journal of Machine Learning and Cybernetics · doi:10.1007/s13042-020-01069-8 · Cited by 102
“ERNIE (THU) [128] and KnowBERT [88] incorporated BERT with entity representations.”
MethodKnowledge-Aware Language Model Pretraining
Rosset, C., Xiong, C., Phan, M., Song, X., Bennett, P. N., & Tiwary, S. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.00655 · Cited by 40
“To address this, previous research mainly grounds the language model to some forms of explicit semantic structures, for example, additional entity-oriented transformer layers [16, 18], or explicitly stored knowledge graph triples [19, 40, 41].”
BackgroundMethodInfluentialLauscher, A., Majewska, O., Ribeiro, L. F. R., Gurevych, I., Rozanov, N., & Glavaš, G. (2020) · Workshop on Knowledge Extraction and Integration for Deep Learning Architectures: Deep Learning Inside Out (DeeLIO) · doi:10.18653/v1/2020.deelio-1.5 · Cited by 102
Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity
Lauscher, A., Vulić, I., Ponti, E. M., Korhonen, A., & Glavaš, G. (2020) · International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.118 · Cited by 90
“Peters et al. (2019) fuse language modeling with an end-to-end entity linker, updating contextual word representations with word-to-entity attention.”
BackgroundRe-TACRED: Addressing Shortcomings of the TACRED Dataset
Stoica, G., Platanios, E. A., & Poczos, B. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17631 · Cited by 31
SentiX: A Sentiment-Aware Pre-Trained Model for Cross-Domain Sentiment Analysis
Zhou, J., Tian, J., Wang, R., Wu, Y., Xiao, W., & He, L. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.49 · Cited by 72
“ERINE (Zhang et al., 2019b) and KnowBERT (Peters et al., 2019) integrated entity representation into BERT.”
MethodYamada, I., Asai, A., Sakuma, J., Shindo, H., Takeda, H., Takefuji, Y., & Matsumoto, Y. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.06280 · Cited by 95
GDPNet: Refining Latent Multi-View Graph for Relation Extraction
Xue, F., Sun, A., Zhang, H., & Chng, E. S. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i16.17670 · Cited by 106
“Meanwhile, we report the results of KnowBERT, which incorporates external resources for training (Peters et al. 2019).”
ResultLearning Rich Representation of Keyphrases from Text
Kulkarni, M., Mahata, D., Arora, R., & Bhowmik, R. (2022) · Findings of the Association for Computational Linguistics: NAACL 2022 · doi:10.18653/v1/2022.findings-naacl.67 · Cited by 55
Language Models are Open Knowledge Graphs
Wang, C., Liu, X., & Song, D. (2020) · arXiv · doi:10.48550/arxiv.2010.11967 · Cited by 79
“LMs are further enhanced by KGs (Zhang et al., 2019; Peters et al., 2019) to improve knowledge-driven tasks.”
BackgroundKnowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Ma, K., Ilievski, F., Francis, J., Bisk, Y., Nyberg, E., & Oltramari, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17593 · Cited by 120
“Finally, complete KGs can be incorporated directly in training by introducing additional modeling objectives, to teach a model about general commonsense regardless of the task at hand (Peters et al. 2019; Levine et al. 2020; Liu et al. 2020; Zhang et al. 2019; Talmor et al. 2020).”
BackgroundHeinzerling, B., & Inui, K. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.153 · Cited by 78
KinGDOM: Knowledge-Guided DOMain Adaptation for Sentiment Analysis
Ghosal, D., Hazarika, D., Roy, A., Majumder, N., Mihalcea, R. F., & Poria, S. (2020) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2020.acl-main.292 · Cited by 84
Incorporating medical knowledge in BERT for clinical relation extraction
Roy, A., & Pan, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.435 · Cited by 61
“…PLMs. Based on the types of knowledge added, we can group the work into two categories: integrating domain text in PLMs (Lee et al., 2020; Peng et al., 2019a; Gu et al., 2020) and integrating domain-specific knowledge graphs into PLMs (Wang et al., 2021; Zhang et al., 2019; Peters et al., 2019).”
An Improved Baseline for Sentence-level Relation Extraction
Zhou, W., & Chen, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2102.01373 · Cited by 47
“• KnowBERT (Peters et al., 2019) jointly trains a language model and an entity linker, which allows the subtokens to attend to entity embedding that is pre-trained on knowledge bases.”
BackgroundMethodResultInfluentialDWIE: An entity-centric dataset for multi-task document-level information extraction
Zaporojets, K., Deleu, J., Develder, C., & Demeester, T. (2021) · Information Processing & Management · doi:10.1016/j.ipm.2021.102563 · Cited by 110
A bibliometric analysis of generative AI in education: current status and development
Liu, J., Wang, C., Liu, Z., Gao, M., Xu, Y., Chen, J., & Cheng, Y. (2024) · Asia Pacific Journal of Education · doi:10.1080/02188791.2024.2305170 · Cited by 60
“…2018; Liu et al., 2019);The fourth paper explored the importance of contextually enhanced word representations in natural language processing tasks (Peters et al., 2019); The fifth paper emphasized the effects of generative pre-training methods in enhancing language understanding (Radford et al.,…”
BackgroundLibKGE - A knowledge graph embedding library for reproducible research
Broscheit, S., Ruffinelli, D., Kochsiek, A., Betz, P., & Gemulla, R. (2020) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2020.emnlp-demos.22 · Cited by 107
“Some uses of KGE models are: enhancing the knowledge representation in language models (Peters et al., 2019), drug discovery in bio-medical KGs (Mohamed et al., 2019), as part of recommender systems (Wang et al., 2017), or for visual relationship detection (Baier et al., 2017).”
MethodA Survey on Multimodal Knowledge Graphs: Construction, Completion and Applications
Chen, Y., Ge, X., Yang, S., Hu, L., Li, J., & Zhang, J. (2023) · Mathematics · doi:10.3390/math11081815 · Cited by 89
“Common ways to carry out the NIL prediction include setting a threshold for the best linking probability [140,141], introducing an additional "NIL" entity in the ranking stage [142] and training an additional binary classifier [120,143].”
MethodLearning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training
Shi, P., Ng, P., Wang, Z., Zhu, H., Li, A. H., Wang, J., Nogueira dos Santos, C., & Xiang, B. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17627 · Cited by 71
Metadata
- Authors
- Peters, M. E., Neumann, M., Logan, R., Schwartz, R., Joshi, V., Singh, S., & Smith, N. A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- University of California, Irvine (US)
- University of Washington (US)
- Corresponding author
- Singh, Sameer
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 43-54
- OpenAlex
- W2970986510
- Cited by
- 661
- References
- 72
- Citations by year
- 2026: 21 · 2025: 54 · 2024: 81 · 2023: 125 · 2022: 115 · 2021: 160 · 2020: 98 · 2019: 6
- FWCI
- 36.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- National Science Foundation (1730158, 1817183)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- contextualized word representations, word embeddings, knowledge graphs, word sense disambiguation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.