KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
Open accessAbstract
Pre-trained language representation models (PLMs) cannot well capture factual knowledge from text. In contrast, knowledge embedding (KE) methods can effectively represent the relational facts in knowledge graphs (KGs) with informative entity embeddings, but conventional KE models cannot take full advantage of the abundant textual information. In this paper, we propose a unified model for Knowledge Embedding and Pre-trained LanguagERepresentation (KEPLER), which can not only better integrate factual knowledge into PLMs but also produce effective text-enhanced KE with the strong PLMs. In KEPLER, we encode textual entity descriptions with a PLM as their embeddings, and then jointly optimize the KE and language modeling objectives. Experimental results show that KEPLER achieves state-of-the-art performances on various NLP tasks, and also works remarkably well as an inductive KE model on KG link prediction. Furthermore, for pre-training and evaluating KEPLER, we construct Wikidata5M1 , a large-scale KG dataset with aligned entity descriptions, and benchmark state-of-the-art KE methods on it. It shall serve as a new KE benchmark and facilitate the research on large KG, inductive KE, and KG with text. The source code can be obtained from https://github.com/THU-KEG/KEPLER.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
UltraEditBench is constructed using entity-relation-object triples from the Wikidata5M knowledge base.
“We construct UltraEditBench using entity-relation-object triples from the Wikidata5M (Wang et al., 2021) knowledge base.”
From UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models (2025) · cites this paper as Wang et al., 2021
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515 · see citing works
WordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 1× · see citing works
“(2) KEPLER-WordNet WordNet (Miller, 1995) is an English lexical graph, where nodes are lemmas and synsets, and edges are their relations.”
BackgroundDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem
McCloskey, M., & Cohen, N. J. (1989) · Psychology of Learning and Motivation · doi:10.1016/s0079-7421(08)60536-8 · Cited by 3,770 · see citing works
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264 · see citing works
Complex Embeddings for Simple Link Prediction
Trouillon, T., Welbl, J., Riedel, S., Gaussier, É., & Bouchard, G. (2016) · arXiv · doi:10.48550/arxiv.1606.06357 · Cited by 2,100 · see citing works
RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space
Sun, Z., Deng, Z.-H., Nie, J.-Y., & Tang, J. (2019) · arXiv · doi:10.48550/arxiv.1902.10197 · Cited by 1,414 · see citing works
Self-citationK-BERT: Enabling Language Representation with Knowledge Graph
Liu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H., & Wang, P. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i03.5681 · Cited by 797 · see citing works
TuckER: Tensor Factorization for Knowledge Graph Completion
Balazevic, I., Allen, C., & Hospedales, T. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1522 · Cited by 696 · cited 1× · see citing works
“This may be because it uses the TuckER (Balazevic et al., 2019) KE model while ERNIE and KEPLER follow TransE (Bordes et al.”
MethodSimplE Embedding for Link Prediction in Knowledge Graphs
Kazemi, S. M., & Poole, D. (2018) · arXiv · doi:10.48550/arxiv.1802.04868 · Cited by 476 · see citing works
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT
Poerner, N., Waltinger, U., & Schütze, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.71 · Cited by 146 · see citing works
Latent Relation Language Models
Hayashi, H., Hu, Z., Xiong, C., & Neubig, G. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6298 · Cited by 44 · see citing works
Flagged first, then by citations
Research works printed but not linked: 49
- Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the Blanks: Distributional Similarity for Rela- tion Learning. In Proceedings of ACL, pages 2895–2905.
- Aleksandar Bojchevski and Stephan Günnemann. 2018. Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking. In Proceedings of ICLR.
- Antoine Bordes, Nicolas Usunier, Alberto Garcia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating Embeddings for Modeling Multi-relational Data. In Advances in Neural Information Processing Systems (NIPS), pages 2787–2795.
- Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu, Chengjiang Li, Xu Chen, and Tiansi Dong. 2018. Joint Representation Learning of Cross-lingual Words and Entities via Attentive Distant Supervi- sion. In Proceedings of EMNLP, pages 227–237.
- Yixin Cao, Lifu Huang, Heng Ji, Xu Chen, and Juanzi Li. 2017. Bridge Text and Knowledge by Learning Multi-Prototype Entity Mention Em- bedding. In Proceedings of ACL, pages 1623– 1633.
- Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer. 2018. Ultra-Fine Entity Typing. In Proceedings of ACL, pages 87–96.
- Ronan Collobert and Jason Weston. 2008. A uni- fied architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of ICML, pages 160–167.
- Andrew M Dai and Quoc V Le. 2015. Semi- supervised sequence learning. In Advances in Neural Information Processing Systems (NIPS), pages 3079–3087.
- Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre- training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of NAACL-HLT, pages 4171–4186.
- Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019. FewRel 2.0: Towards More Challenging Few-Shot Rela- tion Classification. In Proceedings of EMNLP- IJCNLP, pages 6251–6256.
- Takuo Hamaguchi, Hidekazu Oiwa, Masashi Shimbo, and Yuji Matsumoto. 2017. Knowledge Transfer for Out-of-Knowledge-Base Entities: A Graph Neural Network Approach. In Proceed- ings of IJCAI, pages 1802–1808.
- William L. Hamilton, Rex Ying, and Jure Leskovec. 2017. Inductive Representation Learning on Large Graphs. In Advances in Neural Infor- mation Processing Systems (NIPS), pages 1025– 1035.
- Xu Han, Zhiyuan Liu, and Maosong Sun. 2018a. Neural Knowledge Acquisition via Mutual At- tention Between Knowledge Graph and Text. In Proceedings of AAAI, pages 4832–4839.
- Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018b. FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of- the-Art Evaluation. In Proceedings of EMNLP, pages 4803–4809.
- Hiroaki Hayashi, Zecong Hu, Chenyan Xiong, and Graham Neubig. 2020. Latent Relation Lan- guage Models. In Proceedings of AAAI, pages 7911–7918.
- Jeremy Howard and Sebastian Ruder. 2018. Uni- versal Language Model Fine-tuning for Text Classification. In Proceedings of ACL, pages 328–339.
- Nora Kassner and Hinrich Schütze. 2020. Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly. In Proceedings of ACL, pages 7811–7818.
- Zhenzhong Lan, Mingda Chen, Sebastian Good- man, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for Self- supervised Learning of Language Representa- tions. In Proceedings of ICLR.
- Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic Knowledge and Transferability of Contextual Representations. In Proceedings of NAACL-HLT, pages 1073–1094.
- Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2020. K- BERT: Enabling Language Representation with Knowledge Graph. In Proceedings of AAAI, pages 2901–2908.
- Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019b. Multi-Task Deep Neural Networks for Natural Language Understanding. In Proceedings of ACL, pages 4487–4496.
- Robert Logan, Nelson F. Liu, Matthew E. Pe- ters, Matt Gardner, and Sameer Singh. 2019. Barack's Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling. In Proceedings of ACL, pages 5962–5971.
- Lajanugen Logeswaran, Ming-Wei Chang, Ken- ton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-Shot Entity Linking by Reading Entity Descriptions. In Proceedings of ACL, pages 3449–3460.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Gre- gory S. Corrado, and Jeffrey Dean. 2013. Dis- tributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems (NIPS), pages 3111–3119.
- Myle Ott, Sergey Edunov, Alexei Baevski, An- gela Fan, Sam Gross, Nathan Ng, David Grang- ier, and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence Modeling. In Proceedings of NAACL-HLT (Demonstrations), pages 48–53.
- Jeffrey Pennington, Richard Socher, and Christo- pher Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of EMNLP, pages 1532–1543.
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextual- ized Word Representations. In Proceedings of NAACL-HLT, pages 2227–2237.
- Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019. Knowledge Enhanced Contextual Word Representations. In Proceed- ings of EMNLP-IJCNLP, pages 43–54.
- Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language Models as Knowledge Bases? In Proceedings of EMNLP- IJCNLP, pages 2463–2473.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceed- ings of ACL, pages 1715–1725.
- Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical Networks for Few-shot Learning. In Advances in Neural Information Processing Systems (NIPS), pages 4077–4087.
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. At- tention is All you Need. In Advances in Neural Information Processing Systems (NIPS), pages 5998–6008.
- Alex Wang, Jan Hula, Patrick Xia, Raghaven- dra Pappagari, R. Thomas McCoy, Roma Pa- tel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Ben- jamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman. 2019a. Can You Tell Me How to Get Past Sesame Street? Sentence- Level Pretraining Beyond Language Modeling. In Proceedings of ACL, pages 4465–4476.
- Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019b. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Under- standing. In Proceedings of ICLR.
- PeiFeng Wang, Jialong Han, Chenliang Li, and Rong Pan. 2019c. Logic attention based neigh- borhood aggregation for inductive knowledge graph embedding. In Proceedings of AAAI, pages 7152–7159.
- Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Cuihong Cao, Daxin Jiang, and Ming Zhou. 2020. K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters. CoRR, cs.CL/2002.01808v3.
- Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge Graph and Text Jointly Embedding. In Proceedings of EMNLP, pages 1591–1601.
- Adina Williams, Nikita Nangia, and Samuel Bow- man. 2018. A Broad-Coverage Challenge Cor- pus for Sentence Understanding through Infer- ence. In Proceedings of NAACL-HLT, pages 1112–1122.
- Ledell Wu, Fabio Petroni, Martin Josifoski, Sebas- tian Riedel, and Luke Zettlemoyer. 2020. Scal- able Zero-shot Entity Linking with Dense En- tity Retrieval. In Proceedings of EMNLP, pages 6397–6407.
- Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, and Maosong Sun. 2016. Representation Learn- ing of Knowledge Graphs with Entity Descrip- tions. In Proceedings of AAAI, pages 2659– 2665.
- Wenhan Xiong, Jingfei Du, William Yang Wang, and Stoyanov Veselin. 2019. Pretrained En- cyclopedia: Weakly Supervised Knowledge- Pretrained Language Model. In Proceedings of ICLR.
- Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016. Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation. In Proceedings of CoNLL, pages 250–259.
- Bishan Yang and Tom Mitchell. 2017. Leveraging Knowledge Bases in LSTMs for Improving Ma- chine Reading. In Proceedings of ACL, pages 1436–1446.
- Bishan Yang, Scott Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding Entities and Relations for Learning and Infer- ence in Knowledge Bases. In Proceedings of ICLR.
- Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized Autoregressive Pretraining for Language Understanding. In Ad- vances in Neural Information Processing Sys- tems (NeurIPS), pages 5754–5764.
- Poorya Zaremoodi, Wray Buntine, and Gholam- reza Haffari. 2018. Adaptive Knowledge Shar- ing in Multi-Task Learning: Improving Low- Resource Neural Machine Translation. In Pro- ceedings of ACL, pages 656–661. https://proceedings.neurips.cc/paper/2017/file/cb8da6767461f2812ae4290eac7cbc42-Paper.pdf https://proceedings.neurips.cc/paper/2017/file/cb8da6767461f2812ae4290eac7cbc42-Paper.pdf https://openreview.net/pdf?id=HkgEQnRqYQ https://openreview.net/pdf?id=HkgEQnRqYQ https://openreview.net/pdf?id=HkgEQnRqYQ http://proceedings.mlr.press/v48/trouillon16.pdf http://proceedings.mlr.press/v48/trouillon16.pdf https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf https://doi.org/10.18653/v1/P19-1439 https://doi.org/10.18653/v1/P19-1439 https://doi.org/10.18653/v1/P19-1439 https://openreview.net/pdf?id=rJ4km2R5t7 https://openreview.net/pdf?id=rJ4km2R5t7 https://openreview.net/pdf?id=rJ4km2R5t7 https://doi.org/10.1609/aaai.v33i01.33017152 https://doi.org/10.1609/aaai.v33i01.33017152 https://doi.org/10.1609/aaai.v33i01.33017152 https://arxiv.org/abs/2002.01808v3 https://arxiv.org/abs/2002.01808v3 https://arxiv.org/abs/2002.01808v3 https://doi.org/10.3115/v1/D14-1167 https://doi.org/10.3115/v1/D14-1167 http://aclweb.org/anthology/N18-1101 http://aclweb.org/anthology/N18-1101 http://aclweb.org/anthology/N18-1101 https://www.aclweb.org/anthology/2020.emnlp-main.519 https://www.aclweb.org/anthology/2020.emnlp-main.519 https://www.aclweb.org/anthology/2020.emnlp-main.519 http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12216 http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12216 http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12216 https://openreview.net/pdf?id=BJlzm64tDH https://openreview.net/pdf?id=BJlzm64tDH https://openreview.net/pdf?id=BJlzm64tDH https://doi.org/10.18653/v1/K16-1025 https://doi.org/10.18653/v1/K16-1025 https://doi.org/10.18653/v1/K16-1025 https://doi.org/10.18653/v1/P17-1132 https://doi.org/10.18653/v1/P17-1132 https://doi.org/10.18653/v1/P17-1132 https://www.microsoft.com/en-us/research/publication/embedding-entities-and-relations-for-learning-and-inference-in-knowledge-bases/ https://www.microsoft.com/en-us/research/publication/embedding-entities-and-relations-for-learning-and-inference-in-knowledge-bases/ https://www.microsoft.com/en-us/research/publication/embedding-entities-and-relations-for-learning-and-inference-in-knowledge-bases/ http://papers.nips.cc/paper/8812-xlnet-generalized-autoregressive-pretraining-for-language-understanding http://papers.nips.cc/paper/8812-xlnet-generalized-autoregressive-pretraining-for-language-understanding https://doi.org/10.18653/v1/P18-2104 https://doi.org/10.18653/v1/P18-2104 https://doi.org/10.18653/v1/P18-2104 · DOI
- Yuhao Zhang, Victor Zhong, Danqi Chen, Ga- bor Angeli, and Christopher D. Manning. 2017. Position-aware Attention and Supervised Data Improve Slot Filling. In Proceedings of EMNLP, pages 35–45.
- Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced Language Representation with Infor- mative Entities. In Proceedings of ACL, pages 1441–1451.
- Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Tor- ralba, and Sanja Fidler. 2015. Aligning Books and Movies: Towards Story-Like Visual Expla- nations by Watching Movies and Reading Books. In Proceedings of ICCV, pages 19–27.
Cited by
Sorted by citations
A Survey on Knowledge Graphs: Representation, Acquisition, and Applications
Ji, S., Pan, S., Cambria, E., Marttinen, P., & Yu, P. S. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3070843 · Cited by 2,947 · see citing works
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676 · see citing works
Unifying Large Language Models and Knowledge Graphs: A Roadmap
Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3352100 · Cited by 1,376 · see citing works
Pre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994 · see citing works
Self-citationCompute Trends Across Three Eras of Machine Learning
Sevilla, J., Heim, L., Ho, A., Besiroglu, T., Hobbhahn, M., & Villalobos, P. (2022) · 2022 International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9891914 · Cited by 363 · see citing works
CoLAKE: Contextualized Language and Knowledge Embedding
Sun, T., Shao, Y., Qiu, X., Guo, Q., Hu, Y., Huang, X., & Zhang, Z. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.327 · Cited by 185 · see citing works
Self-citationE-BERT: Efficient-Yet-Effective Entity Embeddings for BERT
Poerner, N., Waltinger, U., & Schütze, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.71 · Cited by 146 · see citing works
Packed Levitated Marker for Entity and Relation Extraction
Ye, D., Lin, Y., Li, P., & Sun, M. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.337 · Cited by 144 · see citing works
From Word Embeddings to Pre-Trained Language Models: A State-of-the-Art Walkthrough
Mars, M. (2022) · Applied Sciences · doi:10.3390/app12178805 · Cited by 91 · see citing works
JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs
Ke, P., Ji, H., Ran, Y., Cui, X., Wang, L., Song, L., Zhu, X., & Huang, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.223 · Cited by 92 · see citing works
A Review of Current Trends, Techniques, and Challenges in Large Language Models (LLMs)
Patil, R., & Gudivada, A. V. (2024) · doi:10.20944/preprints202402.0357.v1 · Cited by 296 · see citing works
KoLA: Carefully Benchmarking World Knowledge of Large Language Models
Yu, J., Wang, X., Tu, S., Cao, S., Zhang-Li, D., Lv, X., Peng, H., Yao, Z., Zhang, X., Li, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2306.09296 · Cited by 29 · see citing works
“Given a triplet in Wikidata5M (Wang et al., 2021a), we transform it into a sentence with a relation-specific template and let LLMs complete its tail entity.”
MethodInfluentialSelf-citationKnowledge Graph-Enhanced Large Language Models via Path Selection
Liu, H., Wang, S., Zhu, Y., Dong, Y., & Li, J. (2024) · Findings of the Association for Computational Linguistics ACL 2024 · doi:10.18653/v1/2024.findings-acl.376 · Cited by 28
Self-citationRelational memory-augmented language models
Liu, Q., Yogatama, D., & Blunsom, P. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00476 · Cited by 23 · see citing works
The Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.07616 · Cited by 18
Chen, X., He, L., Hu, W., & Yi, S. (2025) · AI · doi:10.3390/ai6110290 · Cited by 2 · see citing works
UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Models
Gu, X., Huang, Z., Gu, J.-C., & Zhang, K. (2025) · arXiv · doi:10.48550/arxiv.2505.14679
- Authors
- Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., & Tang, J.
- Affiliations
- Department of CST, BNRist. wangxz20@mails.tsinghua.edu.cn
- Princeton University (US)
- Mila - Quebec Artificial Intelligence Institute (CA)
- Beijing National Research Center for Information Science and Technology (CN)
- Tsinghua University (CN)
- HEC Montréal (CA)
- Corresponding author
- Wang, Xiaozhi
- Gao, Tianyu
- Zhu, Zhaocheng
- Zhang, Zhengyan
- Liu, Zhiyuan
- Li, Juanzi
- Tang, Jian
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2021-03
- Volume
- 9
- Pages
- 176-194
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W3151929433
- Cited by
- 676
- References
- 14
- Citations by year
- 2026: 75 · 2025: 104 · 2024: 148 · 2023: 155 · 2022: 112 · 2021: 68 · 2020: 14
- FWCI
- 40.85
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Aeronautics and Space Administration
- Canadian Institute for Advanced Research
- National Natural Science Foundation of China (U1736204, 61732008, 61533018, 61772302)
- Tsinghua University
- Natural Sciences and Engineering Research Council of Canada
- National Key Research and Development Program of China (61732008, 2018YFB1004503)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- knowledge embedding, knowledge graphs, entity embeddings, knowledge graph link prediction, knowledge graph benchmark, entity descriptions, joint optimization
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
