Knowledge Enhanced Contextual Word Representations
Open accessThe DOI matches its Crossref record; 71 of the 72 references it cites were found in a registry or an index.
Abstract
Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, Noah A. Smith. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“XLM-R achieves state-of-the-arts results on multiple cross-lingual benchmarks, including XNLI, MLQA, and NER.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [62]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 4×
“We compute M m scores using the prior and dot product between the entity-span vectors and entity embeddings, with a two-layer MLP (100 hidden dimensions).”
BackgroundMethodInfluentialExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 4×
“, 2018), and BERT (Devlin et al., 2019) have significantly improved the state of the art for a wide range of NLP tasks.”
BackgroundMethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 4×
“, 2018), and BERT (Devlin et al., 2019) have significantly improved the state of the art for a wide range of NLP tasks.”
BackgroundMethodInfluentialGlove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071 · cited 1×
“Pretrained word representations Initial work learning word vectors focused on static word embeddings using multi-task learning objectives (Collobert and Weston, 2008) or corpus level cooccurence statistics (Mikolov et al., 2013a; Pennington et al., 2014).”
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 3×
“Pretrained word representations Initial work learning word vectors focused on static word embeddings using multi-task learning objectives (Collobert and Weston, 2008) or corpus level co-occurence statistics (Mikolov et al., 2013a; Pen-nington et al., 2014).”
BackgroundDistributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533 · cited 3×
“The entity embeddings use a skipgram like objective (Mikolov et al., 2013b) to learn 300-dimensional embeddings of Wikipedia”
BackgroundMethodWordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 1×
“We experiment with integrating both WordNet (Miller, 1995) and Wikipedia, thus explicitly adding word sense knowledge and facts about named entities (including those unseen at training time).”
MethodDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 6×
“Recently the field has shifted toward learning context-sensitive embeddings (Dai and Le, 2015; Peters et al., 2018; Devlin et al., 2019).”
BackgroundMethodInfluentialSelf-citation(2013) · Cited by 5,726
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
Wu, Y., Schuster, M., Chen, Z., Le, Q. V., Norouzi, M., Macherey, W., Krikun, M., Cao, Y., Gao, Q., Macherey, K., et al. (2016) · arXiv · doi:10.48550/arxiv.1609.08144 · Cited by 5,319 · cited 2×
“…left-to-right and rightto-left LMs such as ELMo and GPT. Formally, BERT accepts as input a sequence of N WordPiece tokens (Sennrich et al., 2016; Wu et al., 2016), (x1, . . . , xN ), and computes L layers of D-dimensional contextual representations Hi ∈ RN×D by successively applying non-linear…”
MethodA unified architecture for natural language processing: deep neural networks with multitask learning
Collobert, R., & Weston, J. (2008) · Proceedings of the 25th international conference on Machine learning - ICML '08 · doi:10.1145/1390156.1390177 · Cited by 5,058 · cited 2×
“Pretrained word representations Initial work learning word vectors focused on static word embeddings using multi-task learning objectives (Collobert and Weston, 2008) or corpus level co-occurence statistics (Mikolov et al., 2013a; Pen-nington et al., 2014).”
BackgroundThe unified medical language system (UMLS): integrating biomedical terminology
Bodenreider, O. (2004) · Nucleic Acids Research · doi:10.1093/nar/gkh061 · Cited by 4,356
Knowledge Graph Embedding by Translating on Hyperplanes
Wang, Z., Zhang, J., Feng, J., & Chen, Z. (2014) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v28i1.8870 · Cited by 3,915 · cited 3×
“These methods broadly fall into two categories: translational distance models (Bordes et al., 2013; Wang et al., 2014b; Lin et al., 2015; Xiao et al., 2016) which use a distance-based scoring function, and linear models (Nickel et al., 2011; Yang et al., 2014; Trouillon et al., 2016; Dettmers et…”
BackgroundMethodLearning Entity and Relation Embeddings for Knowledge Graph Completion
Lin, Y., Liu, Z., Sun, M., Liu, Y., & Zhu, X. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9491 · Cited by 3,749 · cited 2×
“These methods broadly fall into two categories: translational distance models (Bordes et al., 2013; Wang et al., 2014b; Lin et al., 2015; Xiao et al., 2016) which use a distance-based scoring function, and linear models (Nickel et al., 2011; Yang et al., 2014; Trouillon et al., 2016; Dettmers et…”
MethodWikidata: a free collaborative knowledgebase
Vrandečić, D., & Krötzsch, M. (2014) · Communications of the ACM · doi:10.1145/2629489 · Cited by 3,342 · cited 3×
“Early experiments with embeddings derived from Wikidata relations 4 did not improve results.”
MethodConvolutional 2D Knowledge Graph Embeddings
Dettmers, T., Minervini, P., Stenetorp, P., & Riedel, S. (2018) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v32i1.11573 · Cited by 2,486
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
Speer, R., Chin, J., & Havasi, C. (2017) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v31i1.11164 · Cited by 2,312
Zhu, Y., Kiros, R., Zemel, R., Salakhutdinov, R., Urtasun, R., Torralba, A., & Fidler, S. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.11 · Cited by 2,264
Embedding Entities and Relations for Learning and Inference in Knowledge Bases
Yang, B., Yih, W.-T., He, X., Gao, J., & Deng, L. (2014) · arXiv · doi:10.48550/arxiv.1412.6575 · Cited by 2,252 · cited 2×
“…distance models (Bordes et al., 2013; Wang et al., 2014b; Lin et al., 2015; Xiao et al., 2016) which use a distance-based scoring function, and linear models (Nickel et al., 2011; Yang et al., 2014; Trouillon et al., 2016; Dettmers et al., 2018) which use a similarity-based scoring function.”
MethodComplex Embeddings for Simple Link Prediction
Trouillon, T., Welbl, J., Riedel, S., Gaussier, É., & Bouchard, G. (2016) · arXiv · doi:10.48550/arxiv.1606.06357 · Cited by 2,100 · cited 2×
“…distance models (Bordes et al., 2013; Wang et al., 2014b; Lin et al., 2015; Xiao et al., 2016) which use a distance-based scoring function, and linear models (Nickel et al., 2011; Yang et al., 2014; Trouillon et al., 2016; Dettmers et al., 2018) which use a similarity-based scoring function.”
Method5.7 The WORDNET lexical database of English
(2019) · Computer Corpus Lexicography · doi:10.1515/9781474471459-050 · Cited by 1,865
A Three-Way Model for Collective Learning on Multi-Relational Data
Nickel, M., Volker Tresp, C., & Kriegel, H. (2011) · Cited by 1,806
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482 · cited 4×
“As shown in Table 7, KnowBert-W+W shows an improvement of 0.6% F 1 over ERNIE and 2.5% over BERT BASE .”
MethodInfluentialSuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019) · arXiv · doi:10.48550/arxiv.1905.00537 · Cited by 1,051
Graph Convolution over Pruned Dependency Trees Improves Relation Extraction
Zhang, Y., Qi, P., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1244 · Cited by 1,048
Semi-supervised Sequence Learning
Dai, A. M., & Le, Q. V. (2015) · arXiv · doi:10.48550/arxiv.1511.01432 · Cited by 994 · cited 1×
“Recently the field has shifted toward learning context-sensitive embed-dings (Dai and Le, 2015; Peters et al., 2018; De-vlin et al., 2019).”
BackgroundRobust Disambiguation of Named Entities in Text
Hoffart, J., Yosef, M. A., Bordino, I., Fürstenau, H., Pinkal, M., Spaniol, M., Taneva, B., Thater, S., & Weikum, G. (2011) · Cited by 925
End-to-end Neural Coreference Resolution
Lee, K., He, L., Lewis, M., & Zettlemoyer, L. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1018 · Cited by 869 · cited 1×
“4.1) with a linear projection, (1) Then, the KAR computes C mention-span representations s m ∈ R E , one for each candidate mention, by pooling over all word pieces in a mention-span using the self-attentive span pooling from Lee et al. (2017).”
InfluentialPosition-aware Attention and Supervised Data Improve Slot Filling
Zhang, Y., Zhong, V., Chen, D., Angeli, G., & Manning, C. D. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1004 · Cited by 798 · cited 4×
“Relation extraction Our first task is relation extraction using the TACRED (Zhang et al., 2017) and SemEval 2010 Task 8 (Hendrickx et al., 2009) datasets.”
MethodInfluentialTAGME: on-the-fly annotation of short text fragments (by wikipedia entities)
Ferragina, P., & Scaiella, U. (2010) · Proceedings of the 19th ACM international conference on Information and knowledge management · doi:10.1145/1871437.1871689 · Cited by 743 · cited 3×
“, 2019) uses TAGME (Ferragina and Scaiella, 2010) to link entities to Wikidata, retrieves the associated entity embeddings, and fuses them into BERTBASE by fine-tuning.”
MethodMatching the Blanks: Distributional Similarity for Relation Learning
Baldini Soares, L., FitzGerald, N., Ling, J., & Kwiatkowski, T. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1279 · Cited by 731
Representation Learning of Knowledge Graphs with Entity Descriptions
Xie, R., Liu, Z., Jia, J., Luan, H., & Sun, M. (2016) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v30i1.10329 · Cited by 725 · cited 2×
“Other meth-ods combine entity metadata with the graph (Xie et al., 2016), use entity contexts (Chen et al., 2014; Ganea and Hofmann, 2017), or a combination of contexts and the KB (Wang et al., 2014a; Gupta et al., 2017).”
BackgroundMethodOpen Domain Question Answering Using Early Fusion of Knowledge Bases and Text
Sun, H., Dhingra, B., Zaheer, M., Mazaitis, K., Salakhutdinov, R., & Cohen, W. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1455 · Cited by 716 · cited 2×
“Task-specific KB architectures Other work has focused on integrating KBs into neural architectures for specific downstream tasks (Yang and Mitchell, 2017; Sun et al., 2018; Chen et al., 2018; Bauer et al., 2018; Mihaylov and Frank, 2018; Wang and Jiang, 2019; Yang et al., 2019).”
BackgroundConvolutional 2D Knowledge Graph EmbeddingsPreprint
Dettmers, T., Minervini, P., Stenetorp, P., et al. (2017) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.833288 · Cited by 713
Felbo, B., Mislove, A. E., Søgaard, A., Rahwan, I., & Lehmann, S. (2017) · Research at the University of Copenhagen (University of Copenhagen) · Cited by 712
TuckER: Tensor Factorization for Knowledge Graph Completion
Balažević, I., Allen, C., & Hospedales, T. M. (2019) · arXiv · doi:10.48550/arxiv.1901.09590 · Cited by 702 · cited 5×
“We experiment with TuckER (Balazevic et al., 2019) embeddings, a recent linear model which generalizes many of the aforecited models.”
MethodInfluentialWiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
Pilehvar, M. T., & Camacho-Collados, J. (2019) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/n19-1128 · Cited by 698
Convolutional 2D Knowledge Graph Embeddings
Dettmers, T., Minervini, P., Stenetorp, P., & Riedel, S. (2017) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.833287 · Cited by 639
context2vec: Learning Generic Context Embedding with Bidirectional LSTM
Melamud, O., Goldberger, J., & Dagan, I. (2016) · Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning · doi:10.18653/v1/k16-1006 · Cited by 513 · cited 1×
“The ELMo and BERT baselines use a nearest neighbor approach trained on the SemCor dataset, similar to the evaluation in Melamud et al. (2016), which has previously been shown to be competitive with task-specific architectures (Raganato et al., 2017 match the performance of KnowBert-WordNet despite…”
MethodImproving efficiency and accuracy in multilingual entity extraction
Daiber, J., Jakob, M., Hokamp, C., & Mendes, P. N. (2013) · Proceedings of the 9th International Conference on Semantic Systems · doi:10.1145/2506182.2506198 · Cited by 502 · cited 2×
“, 2011) and DBpedia Spotlight (Daiber et al., 2013).”
MethodRelation Classification via Multi-Level Attention CNNs
Wang, L., Cao, Z., de Melo, G., & Liu, Z. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1123 · Cited by 487
SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals
Hendrickx, I., Kim, S. N., Kozareva, Z., Nakov, P., Séaghdha, D. Ó., Padó, S., Marco Pennacchiotti, B., Romano, L., & Śzpakowicz, S. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1911.10422 · Cited by 444
Knowledge Graph and Text Jointly Embedding
Wang, Z., Zhang, J., Feng, J., & Chen, Z. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1167 · Cited by 406 · cited 3×
“These methods broadly fall into two categories: translational distance models (Bordes et al., 2013; Wang et al., 2014b; Lin et al., 2015; Xiao et al., 2016) which use a distance-based scoring function, and linear models (Nickel et al., 2011; Yang et al., 2014; Trouillon et al., 2016; Dettmers et…”
BackgroundMethodDeep Joint Entity Disambiguation with Local Neural Attention
Ganea, O.-E., & Hofmann, T. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1277 · Cited by 327
Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison
Raganato, A., Camacho-Collados, J., & Navigli, R. (2017) · Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers · doi:10.18653/v1/e17-1010 · Cited by 316 · cited 1×
“Table 2 displays fine-grained WSD F 1 using the evaluation framework from Navigli et al. (2017) and the ALL dataset (combing SemEval 2007, 2013, 2015 and Senseval 2 and 3).”
MethodA Unified Model for Word Sense Representation and Disambiguation
Chen, X., Liu, Z., & Sun, M. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1110 · Cited by 265 · cited 2×
“, 2016), use entity contexts (Chen et al., 2014; Ganea and Hofmann, 2017), or a combination of contexts and the KB (Wang et al.”
BackgroundLeveraging Knowledge Bases in LSTMs for Improving Machine Reading
Yang, B., & Mitchell, T. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1132 · Cited by 261 · cited 3×
“Task-specific KB architectures Other work has focused on integrating KBs into neural architectures for specific downstream tasks (Yang and Mitchell, 2017; Sun et al., 2018; Chen et al., 2018; Bauer et al., 2018; Mihaylov and Frank, 2018; Wang and Jiang, 2019; Yang et al., 2019).”
BackgroundResultNeural Natural Language Inference Models Enhanced with External Knowledge
Chen, Q., Zhu, X., Ling, Z.-H., Inkpen, D., & Wei, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1224 · Cited by 253 · cited 4×
“Task-specific KB architectures Other work has focused on integrating KBs into neural architectures for specific downstream tasks (Yang and Mitchell, 2017; Sun et al., 2018; Chen et al., 2018; Bauer et al., 2018; Mihaylov and Frank, 2018; Wang and Jiang, 2019; Yang et al., 2019).”
BackgroundResultInfluentialUsing a semantic concordance for sense identification
Miller, G. A., Chodorow, M., Landes, S., Leacock, C., & Thomas, R. G. (1994) · Proceedings of the workshop on Human Language Technology - HLT '94 · doi:10.3115/1075812.1075866 · Cited by 251 · cited 3×
“For supervision, we combined the SemCor word sense disambiguation (WSD) dataset (Miller et al., 1994) with all lemma example usages from WordNet6 and link directly to synsets.”
MethodInfluentialEnd-to-End Neural Entity Linking
Kolitsas, N., Ganea, O.-E., & Hofmann, T. (2018) · Proceedings of the 22nd Conference on Computational Natural Language Learning · doi:10.18653/v1/k18-1050 · Cited by 228 · cited 2×
“1 Following Kolitsas et al. (2018), S e is used to score each of the candidate entities while incorporating the candidate entity prior from the KB.”
MethodEntity Linking via Joint Encoding of Types, Descriptions, and Context
Gupta, N., Singh, S., & Roth, D. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1284 · Cited by 227
Self-citationDesign Challenges for Entity Linking
Ling, X., Singh, S., & Weld, D. S. (2015) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00141 · Cited by 217 · cited 1×
“Ling et al. (2015) showed that incorporating candidate priors into entity linkers can be a powerful signal, so we optionally allow for the candidate selector to return an associated prior probability for each entity candidate.”
BackgroundSelf-citationLearning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
Subramanian, S., Trischler, A., Bengio, Y., & Pal, C. J. (2018) · arXiv · doi:10.48550/arxiv.1804.00079 · Cited by 207 · cited 2×
“Then, we extracted the gloss for each synset and used an off-the-shelf state-of-the-art sentence embedding method (Subramanian et al., 2018) to produce 2048-dimensional vectors.”
MethodChoi, E., Levy, O., Choi, Y., & Zettlemoyer, L. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1009 · Cited by 198 · cited 3×
“Entity typing We also evaluated KnowBertW+W using the entity typing dataset from Choi et al. (2018). To directly compare to ERNIE, we adopted the evaluation protocol in Zhang et al.”
MethodNeural Sequence Learning Models for Word Sense Disambiguation
Raganato, A., Delli Bovi, C., & Navigli, R. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1120 · Cited by 197 · cited 2×
“…neighbor approach trained on the SemCor dataset, similar to the evaluation in Melamud et al. (2016), which has previously been shown to be competitive with task-specific architectures (Raganato et al., 2017 match the performance of KnowBert-WordNet despite incorporating both Wikipedia and WordNet.”
BackgroundMethodMihaylov, T., & Frank, A. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1076 · Cited by 192
Simple BERT Models for Relation Extraction and Semantic Role Labeling
Shi, P., & Lin, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.05255 · Cited by 170
Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling
Logan, R., Liu, N. F., Peters, M. E., Gardner, M., & Singh, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1598 · Cited by 168 · cited 4×
“Entity-aware language models Some previous work has focused on adding KBs to generative language models (LMs) (Ahn et al., 2017; Yang et al., 2017; Logan et al., 2019) or building entity-centric LMs (Ji et al., 2017).”
BackgroundInfluentialSelf-citationA Cross-Lingual Dictionary for English Wikipedia Concepts
Spitkovsky, V. I., & Chang, A. X. (2012) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/44o84k2bkpd9 · Cited by 164 · cited 5×
“The candidate selectors and priors are a combination of CrossWikis, a large, precomputed dictionary that combines statistics from Wikipedia and a web corpus (Spitkovsky and Chang, 2012), and the YAGO dictionary (Hoffart et al., 2011).”
BackgroundMethodInfluentialCommonsense for Generative Multi-Hop Question Answering Tasks
Bauer, L., Wang, Y., & Bansal, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1454 · Cited by 161 · cited 2×
“Task-specific KB architectures Other work has focused on integrating KBs into neural architectures for specific downstream tasks (Yang and Mitchell, 2017; Sun et al., 2018; Chen et al., 2018; Bauer et al., 2018; Mihaylov and Frank, 2018; Wang and Jiang, 2019; Yang et al., 2019).”
BackgroundFrom One Point to A Manifold: Knowledge Graph Embedding For Precise Link PredictionPreprint
Xiao, H., Huang, M., & Zhu, X. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1512.04792 · Cited by 161 · cited 2×
“These methods broadly fall into two categories: translational distance models (Bordes et al., 2013; Wang et al., 2014b; Lin et al., 2015; Xiao et al., 2016) which use a distance-based scoring function, and linear models (Nickel et al., 2011; Yang et al., 2014; Trouillon et al., 2016; Dettmers et…”
MethodEnhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension
Yang, A., Wang, Q., Liu, J., Liu, K., Lyu, Y., Wu, H., She, Q., & Li, S. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1226 · Cited by 155 · cited 1×
“Task-specific KB architectures Other work has focused on integrating KBs into neural architectures for specific downstream tasks (Yang and Mitchell, 2017; Sun et al., 2018; Chen et al., 2018; Bauer et al., 2018; Mihaylov and Frank, 2018; Wang and Jiang, 2019; Yang et al., 2019).”
BackgroundTAGME: on-the-fly annotation of short text fragments (by Wikipedia entities)Preprint
Ferragina, P., & Scaiella, U. (2010) · arXiv (Cornell University) · Cited by 143
Extracting Multiple-Relations in One-Pass with Pre-Trained Transformers
Wang, H., Tan, M., Yu, M., Chang, S., Wang, D., Xu, K., Guo, X., & Potdar, S. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1132 · Cited by 125
A Neural Knowledge Language Model
Ahn, S., Choi, H., Pärnamaa, T., & Bengio, Y. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.00318 · Cited by 101 · cited 2×
“Entity-aware language models Some previous work has focused on adding KBs to generative language models (LMs) (Ahn et al., 2017; Yang et al., 2017; Logan et al., 2019) or building entity-centric LMs (Ji et al., 2017).”
BackgroundDynamic Entity Representations in Neural Language Models
Ji, Y., Tan, C., Martschat, S., Choi, Y., & Smith, N. A. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1195 · Cited by 94 · cited 3×
“Entity-aware language models Some previous work has focused on adding KBs to generative language models (LMs) (Ahn et al., 2017; Yang et al., 2017; Logan et al., 2019) or building entity-centric LMs (Ji et al., 2017).”
BackgroundSelf-citationImproving Relation Extraction by Pre-trained Language RepresentationsPreprint
Alt, C., Hübner, M., & Hennig, L. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1906.03088 · Cited by 93 · cited 1×
“4 Alt et al. (2019) GPT 70.”
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
Pilehvar, M. T., & Camacho-Collados, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1808.09121 · Cited by 87
Reference-Aware Language Models
Yang, Z., Blunsom, P., Dyer, C., & Ling, W. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1197 · Cited by 68 · cited 2×
“Entity-aware language models Some previous work has focused on adding KBs to generative language models (LMs) (Ahn et al., 2017; Yang et al., 2017; Logan et al., 2019) or building entity-centric LMs (Ji et al., 2017).”
BackgroundExplicit Utilization of General Knowledge in Machine Reading Comprehension
Wang, C., & Jiang, H. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1219 · Cited by 50 · cited 2×
“Task-specific KB architectures Other work has focused on integrating KBs into neural architectures for specific downstream tasks (Yang and Mitchell, 2017; Sun et al., 2018; Chen et al., 2018; Bauer et al., 2018; Mihaylov and Frank, 2018; Wang and Jiang, 2019; Yang et al., 2019).”
Background
Research works printed but not linked: 5
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013b. Distributed representa- tions of words and phrases and their compositional- ity. In NeurIPS.
- Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word rep- resentations. In NAACL-HLT.
- Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL.
- Bishan Yang and Tom Michael Mitchell. 2017. Lever- aging knowledge bases in LSTMs for improving ma- chine reading. In ACL.
- Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: En- hanced language representation with informative en- tities. In ACL.
Cited by
Han (2023) · doi:10.1109/tpami.2022.3152247 · Cited by 4,682
“114M Yes KnowBERT [94] Transformer Enc.”
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E. H., Feng, J., & Yan, S. (2021) · 2021 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv48922.2021.00060 · Cited by 2,415
“Transformers in Vision Transformers [37] are the models that entirely rely on the self-attention mechanism to draw global dependencies between input and output, which has dominated in natural language modelling [10, 30, 2, 46, 29, 23].”
BackgroundTokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Yuan, L., Chen, Y., Wang, T., Yu, W., Shi, Y., Jiang, Z., Tay, F. E., Feng, J., & Yan, S. (2021) · arXiv · doi:10.48550/arxiv.2101.11986 · Cited by 2,415
“Transformers in Vision Transformers [37] are the models that entirely rely on the self-attention mechanism to draw global dependencies between input and output, which has dominated in natural language modelling [10, 30, 2, 46, 29, 23].”
BackgroundPre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
“The external knowledge ranges from linguistic [94, 83, 136, 191], semantic [99], commonsense [52], factual [214, 136, 111, 202, 195], to domain-specific knowledge [58, 111].”
BackgroundMethodInfluential
Metadata
- Authors
- Peters, M. E., Neumann, M., Logan, R., Schwartz, R., Joshi, V., Singh, S., & Smith, N. A.
- Affiliations
- Allen Institute for Artificial Intelligence (US)
- University of California, Irvine (US)
- University of Washington (US)
- Corresponding author
- Singh, Sameer
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 43-54
- OpenAlex
- W2970986510
- Cited by
- 661
- References
- 72
- Citations by year
- 2026: 21 · 2025: 54 · 2024: 81 · 2023: 125 · 2022: 115 · 2021: 160 · 2020: 98 · 2019: 6
- FWCI
- 36.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Funders
- National Science Foundation (1730158, 1817183)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Sentiment Analysis and Opinion Mining
- Keywords
- contextualized word representations, word embeddings, knowledge graphs, word sense disambiguation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.