ERNIE: Enhanced Language Representation with Informative Entities
Open accessThe DOI matches its Crossref record; 58 of the 58 references it cites were found in a registry or an index.
Abstract
Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks.However, the existing pre-trained language models rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better language understanding.We argue that informative entities in KGs can enhance language representation with external knowledge.In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously.The experimental results have demonstrated that ERNIE achieves significant improvements on various knowledge-driven tasks, and meanwhile is comparable with the state-of-the-art model BERT on other common NLP tasks.The source code and experiment details of this paper can be obtained from https:// github.com/thunlp/ERNIE.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“However, akin to question answering (QA), modality interaction is frequently confined to feeding knowledge into language (Zhang et al. 2019; Shen et al. 2020; Donghan et al. 2022), rather than fostering interactions across multiple layers.”
From Large language models (LLMs): survey, technical frameworks, and future challenges (2024) · cites this paper as Zhang et al. 2019
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“THU-ERNIE (Zhang et al. 2019) is a pioneer in this direction by fusing entity information, but the relations between entities are ignored by it.”
From K-BERT: Enabling Language Representation with Knowledge Graph (2020) · cites this paper as Zhang et al. 2019
Experimental — accuracy not yet measuredIn the citing paper
“Experiments demonstrate that mBART produces significant performance gains across a wide variety of machine translation (MT) tasks.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [61]
In the citing paper
“Experiments demonstrated that mBART could significantly improve both supervised and unsupervised machine translation at both the sentence level and document level.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [61]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“For the large cost of training ERNIE from scratch, we adopt the parameters of BERT released by Google to initialize the Trans-former blocks for encoding tokens.”
BackgroundInfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 6×
“More details of these pre-training tasks can be found from Devlin et al. (2018).”
BackgroundInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 6×
“More details of these pre-training tasks can be found from Devlin et al. (2018).”
BackgroundInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“The early work (Collobert and Weston, 2008; Mikolov et al., 2013; Pennington et al., 2014) focuses on adopting feature-based approaches to transform words into distributed representations.”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 4×
“There are three important directions remain for future research: (1) inject knowledge into feature-based pre-training models such as ELMo (Peters et al., 2018); (2) introduce diverse structured knowledge into language representation models such as Con-ceptNet (Speer and Havasi, 2012) which is different from world know…”
BackgroundMethodInfluentialExtracting and composing robust features with denoising autoencoders
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390294 · Cited by 7,436
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878 · cited 1×
“, 2018) is a collection of diverse natural language understanding tasks (Warstadt et al., 2018; Socher et al., 2013; Dolan and Brockett, 2005; Agirre et al., 2007; Williams et al., 2018; Rajpurkar et al., 2016; Dagan et al., 2006; Levesque et al., 2011), which is the main benchmark used in Devlin et al.”
BackgroundSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 2×
“…including feature-based (Mikolov et al., 2013; Pen-nington et al., 2014; Peters et al., 2017, 2018) and fine-tuning (Dai and Le, 2015; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2018) approaches, can capture rich language information from text and then benefit many NLP applications.”
Background(2013) · Cited by 5,726
A unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 1×
“The early work (Collobert and Weston, 2008; Mikolov et al., 2013; Pennington et al., 2014) focuses on adopting feature-based approaches to transform words into distributed representations.”
BackgroundGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222 · cited 2×
“…named entity recognition (Sang and De Meulder, 2003), question ∗ indicates equal contribution † Corresponding author: Z.Liu(liuzy@tsinghua.edu.cn) answering (Rajpurkar et al., 2016; Zellers et al., 2018), natural language inference (Bowman et al., 2015), and text classification (Wang et al., 2018).”
BackgroundUniversal Language Model Fine-tuning for Text Classification
Howard, J., & Ruder, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1031 · Cited by 4,100 · cited 3×
“Howard and Ruder (2018) present AWD-LSTM (Merity et al., 2017) to build a universal language model (ULMFiT).”
BackgroundMethodGaussian Error Linear Units (GELUs)
Hendrycks, D., & Gimpel, K. (2016) · arXiv · doi:10.48550/arxiv.1606.08415 · Cited by 3,537 · cited 1×
“σ(·) is the non-linear activation function, which usually is the GELU function (Hendrycks and Gimpel, 2016).”
BackgroundA large annotated corpus for learning natural language inference
Bowman, S. R., Angeli, G., Potts, C., & Manning, C. D. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1075 · Cited by 3,377 · cited 1×
“…named entity recognition (Sang and De Meulder, 2003), question ∗ indicates equal contribution † Corresponding author: Z.Liu(liuzy@tsinghua.edu.cn) answering (Rajpurkar et al., 2016; Zellers et al., 2018), natural language inference (Bowman et al., 2015), and text classification (Wang et al., 2018).”
BackgroundA Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
Williams, A., Nangia, N., & Bowman, S. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1101 · Cited by 3,342
Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition
Tjong Kim Sang, E. F., & De Meulder, F. (2003) · Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - · doi:10.3115/1119176.1119195 · Cited by 2,788
(2010) · Cited by 1,906
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture Notes in Computer Science · doi:10.1007/11736790_9 · Cited by 1,628 · cited 1×
“, 2018) is a collection of diverse natural language understanding tasks (Warstadt et al., 2018; Socher et al., 2013; Dolan and Brockett, 2005; Agirre et al., 2007; Williams et al., 2018; Rajpurkar et al., 2016; Dagan et al., 2006; Bar Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009; Levesque et al…”
BackgroundIntroduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition
Sang, E. T. K. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0209010 · Cited by 1,560
Neural Network Acceptability Judgments
Warstadt, A., Singh, A., & Bowman, S. R. (2019) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00290 · Cited by 1,215 · cited 1×
“, 2018) is a collection of diverse natural language understanding tasks (Warstadt et al., 2018; Socher et al., 2013; Dolan and Brockett, 2005; Agirre et al., 2007; Williams et al., 2018; Rajpurkar et al., 2016; Dagan et al., 2006; Bar Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009; Levesque et al…”
BackgroundDistant Supervision for Relation Extraction via Piecewise Convolutional Neural Networks
Zeng, D., Liu, K., Chen, Y., & Zhao, J. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1203 · Cited by 1,149 · cited 2×
“These extra mark tokens play a similar role like position embeddings in the conventional relation classification models (Zeng et al., 2015).”
BackgroundAutomatically Constructing a Corpus of Sentential Paraphrases.
Dolan, W. B., & Brockett, C. (2005) · Cited by 1,095
Graph Convolution over Pruned Dependency Trees Improves Relation Extraction
Zhang, Y., Qi, P., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1244 · Cited by 1,048 · cited 2×
“The results of CNN, PA-LSTM, and C-GCN come from the paper by Zhang et al. (2018), which are the best results of CNN, RNN, and GCN respectively.”
MethodNeural Relation Extraction with Selective Attention over Instances
Lin, Y., Shen, S., Liu, Z., Luan, H., & Sun, M. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1200 · Cited by 1,021 · cited 1×
“are introduced into CNN (Zeng et al., 2015; Lin et al., 2016; Wu et al., 2017; Han et al., 2018b).”
Self-citationSemi-supervised Sequence Learning
Dai, A. M., & Le, Q. V. (2015) · arXiv · doi:10.48550/arxiv.1511.01432 · Cited by 994 · cited 3×
“Different from the above-mentioned feature-based language approaches only using the pre-trained language representations as input features, Dai and Le (2015) train auto-encoders on unlabeled text, and then use the pre-trained model architecture and parameters as a starting point for other specific…”
BackgroundMethodHan, X., Zhu, H., Yu, P., Wang, Z., Yao, Y., Liu, Z., & Sun, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1514 · Cited by 914 · cited 8×
“Since FewRel is built by checking whether the sentences contain facts in Wiki-data, we drop the related facts in KGs before pre-training for fair comparison.”
BackgroundMethodInfluentialSelf-citationLevesque, H. J., Davis, E., & Morgenstern, L. (2011) · Cited by 885
Position-aware Attention and Supervised Data Improve Slot Filling
Zhang, Y., Zhong, V., Chen, D., Angeli, G., & Manning, C. D. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1004 · Cited by 798 · cited 3×
“Zhang et al. (2017) propose PA-LSTM introducing a position-aware attention mechanism over an LSTM network, which evaluates the relative contribution of each word in the sequence for the final sentence representation.”
BackgroundMethodRepresenting Text for Joint Embedding of Text and Knowledge Bases
Toutanova, K., Chen, D., Pantel, P., Poon, H., Choudhury, P., & Gamon, M. (2015) · Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d15-1174 · Cited by 750 · cited 1×
“In fact, some work has attempted to joint representation learning of words and entities for effectively leveraging external KGs and achieved promising results (Wang et al., 2014; Toutanova et al., 2015; Han et al., 2016; Yamada et al., 2016; Cao et al., 2017, 2018).”
BackgroundTAGME: on-the-fly annotation of short text fragments (by wikipedia entities)
Ferragina, P., & Scaiella, U. (2010) · Proceedings of the 19th ACM international conference on Information and knowledge management · doi:10.1145/1871437.1871689 · Cited by 743 · cited 1×
“We also fine-tune ERNIE on the distant supervised dataset, i.e., FIGER (Ling et al., 2015 we use TAGME (Ferragina and Scaiella, 2010) to extract the entity mentions in the sentences and link them to their corresponding entities in KGs.”
MethodERNIE: Enhanced Representation through Knowledge Integration
Sun, Y., Wang, S., Li, Y., Feng, S., Chen, X., Zhang, H., Tian, X., Zhu, D., Tian, H., & Wu, H. R. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09223 · Cited by 721 · cited 1×
“It is a coincidence that both Sun et al. (2019) and we chose ERNIE as the model names, which follows the interesting naming habits like ELMo and BERT.”
MethodRegularizing and Optimizing LSTM Language Models
Merity, S., Keskar, N. S., & Socher, R. (2017) · arXiv · doi:10.48550/arxiv.1708.02182 · Cited by 697 · cited 1×
“Howard and Ruder (2018) present AWD-LSTM (Merity et al., 2017) to build a universal language model (ULMFiT).”
BackgroundSWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference
Zellers, R., Bisk, Y., Schwartz, R., & Choi, Y. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1009 · Cited by 690 · cited 1×
“…named entity recognition (Sang and De Meulder, 2003), question ∗ indicates equal contribution † Corresponding author: Z.Liu(liuzy@tsinghua.edu.cn) answering (Rajpurkar et al., 2016; Zellers et al., 2018), natural language inference (Bowman et al., 2015), and text classification (Wang et al., 2018).”
BackgroundSemi-supervised sequence tagging with bidirectional language models
Peters, M., Ammar, W., Bhagavatula, C., & Power, R. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1161 · Cited by 655 · cited 1×
“Pre-trained language representation models, including feature-based (Mikolov et al., 2013; Pen-nington et al., 2014; Peters et al., 2017, 2018) and fine-tuning (Dai and Le, 2015; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2018) approaches, can capture rich language information from…”
MethodRepresenting General Relational Knowledge in ConceptNet 5
Speer, R., & Havasi, C. (2012) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/3zycidvg6dgg · Cited by 407 · cited 2×
“There are three important directions remain for future research: (1) inject knowledge into feature-based pre-training models such as ELMo (Peters et al., 2018); (2) introduce diverse structured knowledge into language representation models such as Con-ceptNet (Speer and Havasi, 2012) which is different from world know…”
BackgroundKnowledge Graph and Text Jointly Embedding
Wang, Z., Zhang, J., Feng, J., & Chen, Z. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1167 · Cited by 406 · cited 1×
“In fact, some work has attempted to joint representation learning of words and entities for effectively leveraging external KGs and achieved promising results (Wang et al., 2014; Toutanova et al., 2015; Han et al., 2016; Yamada et al., 2016; Cao et al., 2017, 2018).”
BackgroundJoint Learning of the Embedding of Words and Entities for Named Entity Disambiguation
Yamada, I., Shindo, H., Takeda, H., & Takefuji, Y. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1025 · Cited by 332 · cited 1×
“In fact, some work has attempted to joint representation learning of words and entities for effectively leveraging external KGs and achieved promising results (Wang et al., 2014; Toutanova et al., 2015; Han et al., 2016; Yamada et al., 2016; Cao et al., 2017, 2018).”
BackgroundMem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems
Madotto, A., Wu, C.-S., & Fung, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1136 · Cited by 302 · cited 1×
“…significantly enhance original models, such as reading comprehension (Mihaylov and Frank, 2018; Zhong et al., 2018), machine translation (Zaremoodi et al., 2018), natural language inference (Chen et al., 2018), knowledge acquisition (Han et al., 2018a), and dialog systems (Madotto et al., 2018).”
BackgroundNeural Natural Language Inference Models Enhanced with External Knowledge
Chen, Q., Zhu, X., Ling, Z.-H., Inkpen, D., & Wei, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1224 · Cited by 253 · cited 1×
“…significantly enhance original models, such as reading comprehension (Mihaylov and Frank, 2018; Zhong et al., 2018), machine translation (Zaremoodi et al., 2018), natural language inference (Chen et al., 2018), knowledge acquisition (Han et al., 2018a), and dialog systems (Madotto et al., 2018).”
BackgroundAdversarial Training for Relation Extraction
Wu, Y., Bamman, D., & Russell, S. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1187 · Cited by 224 · cited 1×
“are introduced into CNN (Zeng et al., 2015; Lin et al., 2016; Wu et al., 2017; Han et al., 2018b).”
Design Challenges for Entity Linking
Ling, X., Singh, S., & Weld, D. S. (2015) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00141 · Cited by 217 · cited 7×
“Following the same evaluation criteria used in the previous work, we compare NFGEC, BERT, ERNIE on FIGER, and adopt strict accuracy, loose macro, loose micro scores for evaluation.”
MethodInfluentialChoi, E., Levy, O., Choi, Y., & Zettlemoyer, L. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1009 · Cited by 198 · cited 10×
“UFET is proposed with the Open Entity dataset, which uses a Bi-LSTM for context representation instead of two Bi-LSTMs separated by entity mentions in NFGEC.”
MethodInfluentialMihaylov, T., & Frank, A. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1076 · Cited by 192 · cited 1×
“…it outputs new token embeddings and entity embeddings for the next layer. significantly enhance original models, such as reading comprehension (Mihaylov and Frank, 2018; Zhong et al., 2018), machine translation (Zaremoodi et al., 2018), natural language inference (Chen et al., 2018),…”
BackgroundNeural Knowledge Acquisition via Mutual Attention Between Knowledge Graph and Text
Han, X., Liu, Z., & Sun, M. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11927 · Cited by 155 · cited 1×
“…significantly enhance original models, such as reading comprehension (Mihaylov and Frank, 2018; Zhong et al., 2018), machine translation (Zaremoodi et al., 2018), natural language inference (Chen et al., 2018), knowledge acquisition (Han et al., 2018a), and dialog systems (Madotto et al., 2018).”
BackgroundSelf-citationHierarchical Relation Extraction with Coarse-to-Fine Grained Attention
Han, X., Yu, P., Liu, Z., Sun, M., & Li, P. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1247 · Cited by 144 · cited 7×
“Since FewRel is built by checking whether the sentences contain facts in Wiki-data, we drop the related facts in KGs before pre-training for fair comparison.”
BackgroundMethodInfluentialSelf-citationTAGME: on-the-fly annotation of short text fragments (by Wikipedia entities)Preprint
Ferragina, P., & Scaiella, U. (2010) · arXiv (Cornell University) · Cited by 143
An Attentive Neural Architecture for Fine-grained Entity Type Classification
Shimaoka, S., Stenetorp, P., Inui, K., & Riedel, S. (2016) · Automated Knowledge Base Construction · doi:10.18653/v1/w16-1313 · Cited by 103 · cited 2×
“As previous typing models make full use of both context embeddings and entity mention embeddings (Shimaoka et al., 2016), we argue that the modified input sequence with the mention mark token [ENT] can guide ERNIE to combine both context information and entity mention information attentively.”
MethodBridge Text and Knowledge by Learning Multi-Prototype Entity Mention Embedding
Cao, Y., Huang, L., Ji, H., Chen, X., & Li, J. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1149 · Cited by 98
Improving Question Answering by Commonsense-Based Pre-training
Zhong, W., Tang, D., Duan, N., Zhou, M., Wang, J., & Yin, J. P. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32233-5_2 · Cited by 84
Adaptive Knowledge Sharing in Multi-Task Learning: Improving Low-Resource Neural Machine Translation
Zaremoodi, P., Buntine, W., & Haffari, G. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2104 · Cited by 53 · cited 1×
“…next layer. significantly enhance original models, such as reading comprehension (Mihaylov and Frank, 2018; Zhong et al., 2018), machine translation (Zaremoodi et al., 2018), natural language inference (Chen et al., 2018), knowledge acquisition (Han et al., 2018a), and dialog systems (Madotto et…”
BackgroundJoint Representation Learning of Text and Knowledge for Knowledge Graph CompletionPreprint
Han, X., Liu, Z., & Sun, M. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.04125 · Cited by 50 · cited 1×
“In fact, some work has attempted to joint representation learning of words and entities for effectively leveraging external KGs and achieved promising results (Wang et al., 2014; Toutanova et al., 2015; Han et al., 2016; Yamada et al., 2016; Cao et al., 2017, 2018).”
BackgroundSelf-citationMulti-level Representations for Fine-Grained Typing of Knowledge Base Entities
Yaghoobzadeh, Y., & Schütze, H. (2017) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/e17-1055 · Cited by 46
Joint Representation Learning of Cross-lingual Words and Entities via Attentive Distant Supervision
Cao, Y., Hou, L., Li, J., Liu, Z., Li, C., Chen, X., & Dong, T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1021 · Cited by 41
Self-citationPut It Back: Entity Typing with Language Model Enhancement
Xin, J., Zhu, H., Han, X., Liu, Z., & Sun, M. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1121 · Cited by 24 · cited 1×
“As previous typing models make full use of both context embeddings and entity mention embeddings (Shimaoka et al., 2016; Yaghoobzadeh and Schütze, 2017; Xin et al., 2018), we argue that the modified input sequence with the mention mark token [ENT] can guide ERNIE to combine both context information and entity mention…”
MethodSelf-citationImproving Question Answering by Commonsense-Based Pre-TrainingPreprint
Zhong, W., Tang, D., Duan, N., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03568 · Cited by 13 · cited 2×
“…embeddings and entity embeddings for the next layer. significantly enhance original models, such as reading comprehension (Mihaylov and Frank, 2018; Zhong et al., 2018), machine translation (Zaremoodi et al., 2018), natural language inference (Chen et al., 2018), knowledge acquisition (Han et…”
Neural Network Acceptability Judgments
Warstadt, A., Singh, A., & Bowman, S. R. (2018) · arXiv · doi:10.48550/arxiv.1805.12471 · cited 1×
“, 2018) is a collection of diverse natural language understanding tasks (Warstadt et al., 2018; Socher et al., 2013; Dolan and Brockett, 2005; Agirre et al., 2007; Williams et al., 2018; Rajpurkar et al., 2016; Dagan et al., 2006; Bar Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009; Levesque et al…”
Background
Research works printed but not linked: 4
- Eneko Agirre, Llu'is M'arquez, and Richard Wicen- towski. 2007. Proceedings of the fourth interna- tional workshop on semantic evaluations (semeval- 2007). In Proceedings of SemEval-2007.
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositional- ity. In Proceedings of NIPS, pages 3111-3119.
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In Proceedings of NAACL-HLT, pages 2227-2237.
- Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language under- standing by generative pre-training. In Proceedings of Technical report, OpenAI.
Cited by
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
A Survey on Knowledge Graphs: Representation, Acquisition, and Applications
Ji, S., Pan, S., Cambria, E., Marttinen, P., & Yu, P. S. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3070843 · Cited by 2,947
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Self-supervised Learning: Generative or Contrastive
Liu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J., & Tang, J. (2021) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2021.3090866 · Cited by 1,087
“ERNIE [114], [149] NLP G AE Masked words Sentence”
BackgroundSpanBERT: Improving Pre-training by Representing and Predicting Spans
Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., & Levy, O. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00300 · Cited by 1,827
Metadata
- Authors
- Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q.
- Affiliations
- State Key Laboratory of Intelligent Technology and Systems (CN)
- Tsinghua University (CN)
- Huawei Technologies (China) (CN)
- Corresponding author
- Zhang, Zhengyan
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 1441-1451
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2953356739
- Cited by
- 1,482
- References
- 56
- Citations by year
- 2026: 74 · 2025: 142 · 2024: 236 · 2023: 292 · 2022: 250 · 2021: 284 · 2020: 174 · 2019: 29
- FWCI
- 70.49
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- China Association for Science and Technology (2016QNRC001)
- Deutsche Forschungsgemeinschaft (DFG TRR 169, TRR169, 61621136008, NSFC 61621136008)
- National Natural Science Foundation of China (61621136008, NSFC 61621136008, TRR-169, 2016QNRC001, 61572273)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Graph Neural Networks
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- ERNIE, linguistic representation, knowledge graphs, pre-trained language models, BERT, external knowledge integration
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.