Enhancing Pre-Trained Language Representations with Rich Knowledge for Machine Reading Comprehension
Open accessThe DOI matches its Crossref record; 47 of the 47 references it cites were found in a registry or an index.
Abstract
Machine reading comprehension (MRC) is a crucial and challenging task in NLP.Recently, pre-trained language models (LMs), especially BERT, have achieved remarkable success, presenting new state-of-the-art results in MRC.In this work, we investigate the potential of leveraging external knowledge bases (KBs) to further improve BERT for MRC.We introduce KT-NET, which employs an attention mechanism to adaptively select desired knowledge from KBs, and then fuses selected knowledge with BERT to enable context-and knowledgeaware predictions.We believe this would combine the merits of both deep LMs and curated KBs towards better MRC.Experimental results indicate that KT-NET offers significant and consistent improvements over BERT, outperforming competitive baselines on ReCoRD and SQuAD1.1 benchmarks.Notably, it ranks the 1st place on the ReCoRD leaderboard, and is also the best single model on the SQuAD1.1 leaderboard at the time of submission (March 4th, 2019). 1
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures



Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Besides, Tenney et al. [122] analyzed the roles of BERT's layers in di erent tasks and found that BERT solves tasks in a similar order to that in NLP pipelines.”
From Pre-trained models for natural language processing: A survey (2020) · cites this paper as [122]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“We refer readers to (Devlin et al., 2018; Vaswani et al., 2017) for details.”
MethodAdam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · cited 1×
“We use the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 3e-5 and a batch size of 24.”
MethodExploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173 · cited 6×
“Finally, we follow Devlin et al. (2018) and use the FullTokenizer built in BERT to segment words into wordpieces.”
MethodInfluentialBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 6×
“Finally, we follow Devlin et al. (2018) and use the FullTokenizer built in BERT to segment words into wordpieces.”
MethodInfluentialWordNet: a lexical database for English
Miller, G. A. (1995) · Communications of the ACM · doi:10.1145/219717.219748 · Cited by 14,015 · cited 3×
“We use two KBs: WordNet and NELL.”
MethodDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · cited 3×
“…government has extended its review into whether trade sanctions against Sudan should be repealed. are pre-trained on unlabeled text and then applied to MRC, in either a feature-based (Peters et al., 2018a) or a fine-tuning (Radford et al., 2018) manner, both offering substantial performance boosts.”
BackgroundMethodThe Stanford CoreNLP Natural Language Processing Toolkit
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S., & McClosky, D. (2014) · Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.3115/v1/p14-5010 · Cited by 7,037 · cited 1×
“Entity mentions are given as answer candidates on ReCoRD, and recognized by Stanford CoreNLP (Manning et al., 2014) on SQuAD1.”
MethodSQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 4×
“1 (Rajpurkar et al., 2016) is a well-known extractive MRC dataset that consists of questions created by crowdworkers for Wikipedia articles.”
BackgroundMethodInfluentialNLTK: the Natural Language Toolkit
Loper, E., & Bird, S. (2002) · Proceedings of the ACL-02 Workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics - · doi:10.3115/1118108.1118117 · Cited by 3,218
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2019) · North American Chapter of the Association for Computational Linguistics · doi:10.18653/v1/n19-1421 · Cited by 2,835 · cited 1×
“Incorporating KBs Several MRC datasets that require external knowledge have been proposed, such as ReCoRD (Zhang et al., 2018), ARC (Clark et al., 2018), MCScript (Ostermann et al., 2018), OpenBookQA (Mihaylov et al., 2018) and Com-monsenseQA (Talmor et al., 2018).”
MethodTeaching Machines to Read and Comprehend
Hermann, K. M., Kočiský, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., & Blunsom, P. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.03340 · Cited by 2,521 · cited 1×
“Machine Reading Comprehension In the last few years, a number of datasets have been created for MRC, e.g., CNN/DM (Hermann et al., 2015), SQuAD (Rajpurkar et al., 2016, 2018), SearchQA (Dunn et al., 2017), TriviaQA (Joshi et al., 2017), and MS-MARCO (Nguyen et al., 2016).”
MethodMS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Bajaj, P., Campos, D., Craswell, N., Deng, L., Gao, J., Liu, X., Majumder, R., McNamara, A., Mitra, B., Nguyen, T., et al. (2016) · arXiv · doi:10.48550/arxiv.1611.09268 · Cited by 2,473 · cited 2×
“Machine Reading Comprehension In the last few years, a number of datasets have been created for MRC, e.g., CNN/DM (Hermann et al., 2015), SQuAD (Rajpurkar et al., 2016, 2018), SearchQA (Dunn et al., 2017), TriviaQA (Joshi et al., 2017), and MS-MARCO (Nguyen et al., 2016).”
BackgroundMethodEmbedding Entities and Relations for Learning and Inference in Knowledge Bases
Yang, B., Yih, W.-T., He, X., Gao, J., & Deng, L. (2014) · arXiv · doi:10.48550/arxiv.1412.6575 · Cited by 2,252 · cited 3×
“We adopt the BILINEAR model (Yang et al., 2015) which measures the validity via a bilinear function f ( s, r, o ) = s (cid:62) diag( r ) o .”
BackgroundMethodInfluentialKnow What You Don't Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · arXiv · doi:10.48550/arxiv.1806.03822 · Cited by 2,242 · cited 3×
“0 (Rajpurkar et al., 2018).”
BackgroundMethodKnow What You Don’t Know: Unanswerable Questions for SQuAD
Rajpurkar, P., Jia, R., & Liang, P. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · doi:10.18653/v1/p18-2124 · Cited by 2,240 · cited 3×
“0 (Rajpurkar et al., 2018).”
BackgroundMethodToward an architecture for never-ending language learning
Carlson, A., Betteridge, J., Kisiel, B., Settles, B., Hruschka, E., & Mitchell, T. (2010) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v24i1.7519 · Cited by 2,000 · cited 13×
“In this work, we treat WordNet synsets and NELL concepts as knowledge to be retrieved from KBs, similar to (Yang and Mitchell, 2017).”
MethodInfluential5.7 The WORDNET lexical database of English
(2019) · Computer Corpus Lexicography · doi:10.1515/9781474471459-050 · Cited by 1,865
NLTK: The Natural Language Toolkit
Loper, E., & Bird, S. (2002) · arXiv (Cornell University) · doi:10.48550/arxiv.cs/0205028 · Cited by 1,844
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D., & Zettlemoyer, L. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1147 · Cited by 1,690 · cited 2×
“Machine Reading Comprehension In the last few years, a number of datasets have been created for MRC, e.g., CNN/DM (Hermann et al., 2015), SQuAD (Rajpurkar et al., 2016, 2018), SearchQA (Dunn et al., 2017), TriviaQA (Joshi et al., 2017), and MS-MARCO (Nguyen et al., 2016).”
BackgroundMethodTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
Joshi, M., Choi, E., Weld, D. S., & Zettlemoyer, L. (2017) · arXiv · doi:10.48550/arxiv.1705.03551 · Cited by 1,650 · cited 2×
“Machine Reading Comprehension In the last few years, a number of datasets have been created for MRC, e.g., CNN/DM (Hermann et al., 2015), SQuAD (Rajpurkar et al., 2016, 2018), SearchQA (Dunn et al., 2017), TriviaQA (Joshi et al., 2017), and MS-MARCO (Nguyen et al., 2016).”
BackgroundMethodBidirectional Attention Flow for Machine Comprehension
Seo, M. J., Kembhavi, A., Farhadi, A., & Hajishirzi, H. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01603 · Cited by 1,445 · cited 2×
“(Wang and Jiang, 2017), BiDAF (Seo et al., 2017), AoA Reader (Cui et al.”
BackgroundMethodGated Self-Matching Networks for Reading Comprehension and Question Answering
Wang, W., Yang, N., Wei, F., Chang, B., & Zhou, M. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1018 · Cited by 666 · cited 1×
“These datasets have led to advances like Match-LSTM (Wang and Jiang, 2017), BiDAF (Seo et al., 2017), AoA Reader (Cui et al., 2017), DCN (Xiong et al., 2017), R-Net (Wang et al., 2017), and QANet (Yu et al., 2018).”
MethodCan a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
Mihaylov, T., Clark, P., Khot, T., & Sabharwal, A. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1260 · Cited by 649 · cited 1×
“Incorporating KBs Several MRC datasets that require external knowledge have been proposed, such as ReCoRD (Zhang et al., 2018), ARC (Clark et al., 2018), MCScript (Ostermann et al., 2018), OpenBookQA (Mihaylov et al., 2018) and Com-monsenseQA (Talmor et al., 2018).”
MethodA Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Chen, D., Bolton, J., & Manning, C. D. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1223 · Cited by 491 · cited 1×
“However, as widely recognized, genuine reading comprehension requires not only language understanding, but also knowledge that supports sophisticated reasoning (Chen et al., 2016; Mi-haylov and Frank, 2018; Bauer et al., 2018; Zhong et al., 2018).”
BackgroundThink you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. (2018) · arXiv · doi:10.48550/arxiv.1803.05457 · Cited by 462 · cited 2×
“Machine Reading Comprehension In the last few years, a number of datasets have been created for MRC, e.g., CNN/DM (Hermann et al., 2015), SQuAD (Rajpurkar et al., 2016, 2018), SearchQA (Dunn et al., 2017), TriviaQA (Joshi et al., 2017), and MS-MARCO (Nguyen et al., 2016).”
MethodDynamic Coattention Networks For Question Answering
Xiong, C., Zhong, V., & Socher, R. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1611.01604 · Cited by 450 · cited 1×
“These datasets have led to advances like Match-LSTM (Wang and Jiang, 2017), BiDAF (Seo et al., 2017), AoA Reader (Cui et al., 2017), DCN (Xiong et al., 2017), R-Net (Wang et al., 2017), and QANet (Yu et al., 2018).”
MethodDissecting Contextual Word Embeddings: Architecture and Representation
Peters, M., Neumann, M., Zettlemoyer, L., & Yih, W.-T. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1179 · Cited by 434 · cited 3×
“…government has extended its review into whether trade sanctions against Sudan should be repealed. are pre-trained on unlabeled text and then applied to MRC, in either a feature-based (Peters et al., 2018a) or a fine-tuning (Radford et al., 2018) manner, both offering substantial performance boosts.”
BackgroundMethodMachine Comprehension Using Match-LSTM and Answer Pointer
Wang, S., & Jiang, J. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1608.07905 · Cited by 408 · cited 1×
“These datasets have led to advances like Match-LSTM (Wang and Jiang, 2017), BiDAF (Seo et al., 2017), AoA Reader (Cui et al., 2017), DCN (Xiong et al., 2017), R-Net (Wang et al., 2017), and QANet (Yu et al., 2018).”
MethodSimple and Effective Multi-Paragraph Reading Comprehension
Clark, C., & Gardner, M. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1078 · Cited by 396
Assessing BERT's Syntactic Abilities
Goldberg, Y. (2019) · arXiv · doi:10.48550/arxiv.1901.05287 · Cited by 385 · cited 2×
“1 leaderboard at the time of submission (March 4th, 2019).”
BackgroundQANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
Yu, A. W., Dohan, D., Luong, M.-T., Zhao, R., Chen, K., Norouzi, M., & Le, Q. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1804.09541 · Cited by 368 · cited 1×
“These datasets have led to advances like Match-LSTM (Wang and Jiang, 2017), BiDAF (Seo et al., 2017), AoA Reader (Cui et al., 2017), DCN (Xiong et al., 2017), R-Net (Wang et al., 2017), and QANet (Yu et al., 2018).”
MethodSearchQA: A New Q&A Dataset Augmented with Context from a Search Engine
Dunn, M., Sagun, L., Higgins, M., Guney, V. U., Cirik, V., & Cho, K. (2017) · arXiv · doi:10.48550/arxiv.1704.05179 · Cited by 359 · cited 1×
“Machine Reading Comprehension In the last few years, a number of datasets have been created for MRC, e.g., CNN/DM (Hermann et al., 2015), SQuAD (Rajpurkar et al., 2016, 2018), SearchQA (Dunn et al., 2017), TriviaQA (Joshi et al., 2017), and MS-MARCO (Nguyen et al., 2016).”
MethodCommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
Talmor, A., Herzig, J., Lourie, N., & Berant, J. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1811.00937 · Cited by 296 · cited 1×
“Incorporating KBs Several MRC datasets that require external knowledge have been proposed, such as ReCoRD (Zhang et al., 2018), ARC (Clark et al., 2018), MCScript (Ostermann et al., 2018), OpenBookQA (Mihaylov et al., 2018) and Com-monsenseQA (Talmor et al., 2018).”
MethodLeveraging Knowledge Bases in LSTMs for Improving Machine Reading
Yang, B., & Mitchell, T. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1132 · Cited by 261 · cited 5×
“In this work, we treat WordNet synsets and NELL concepts as knowledge to be retrieved from KBs, similar to (Yang and Mitchell, 2017).”
BackgroundMethodInfluentialNeural Natural Language Inference Models Enhanced with External Knowledge
Chen, Q., Zhu, X., Ling, Z.-H., Inkpen, D., & Wei, S. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1224 · Cited by 253 · cited 1×
“Weissenborn et al. (2017), Bauer et al. (2018), Mihaylov and Frank (2018), Pan et al. (2019), Chen et al. (2018), Wang et al. (2018) follow a retrieve-then-encode paradigm, i.e., they first retrieve relevant knowledge from KBs, and only the retrieved knowledge relevant locally to the reading text…”
BackgroundReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
Zhang, S., Liu, X., Liu, J., Gao, J., Duh, K., & Durme, B. V. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.12885 · Cited by 200 · cited 5×
“We evaluate our approach on two benchmarks: ReCoRD (Zhang et al., 2018) and SQuAD1.”
BackgroundMethodInfluentialSelf-citationStochastic Answer Networks for Machine Reading Comprehension
Liu, X., Shen, Y., Duh, K., & Gao, J. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1157 · Cited by 197
Mihaylov, T., & Frank, A. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1076 · Cited by 192 · cited 1×
“Weissenborn et al. (2017), Bauer et al. (2018), Mihaylov and Frank (2018), Pan et al. (2019), Chen et al. (2018), Wang et al. (2018) follow a retrieve-then-encode paradigm, i.e., they first retrieve relevant knowledge from KBs, and only the retrieved knowledge relevant locally to the reading text…”
BackgroundAttention-over-Attention Neural Networks for Reading Comprehension
Cui, Y., Chen, Z., Wei, S., Wang, S., Liu, T., & Hu, G. (2017) · Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p17-1055 · Cited by 171 · cited 1×
“These datasets have led to advances like Match-LSTM (Wang and Jiang, 2017), BiDAF (Seo et al., 2017), AoA Reader (Cui et al., 2017), DCN (Xiong et al., 2017), R-Net (Wang et al., 2017), and QANet (Yu et al., 2018).”
MethodCommonsense for Generative Multi-Hop Question Answering Tasks
Bauer, L., Wang, Y., & Bansal, M. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1454 · Cited by 161 · cited 3×
“Advantages Previous attempts that leverage extra knowledge for MRC (Bauer et al., 2018; Mi-haylov and Frank, 2018) usually follow a retrieve-then-encode paradigm, i.e., they first retrieve relevant knowledge from KBs, and only the retrieved knowledge—which is relevant locally to the reading…”
BackgroundSemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge
Ostermann, S., Roth, M., Modi, A., Thater, S., & Pinkal, M. (2018) · Proceedings of The 12th International Workshop on Semantic Evaluation · doi:10.18653/v1/s18-1119 · Cited by 114 · cited 1×
“Incorporating KBs Several MRC datasets that require external knowledge have been proposed, such as ReCoRD (Zhang et al., 2018), ARC (Clark et al., 2018), MCScript (Ostermann et al., 2018), OpenBookQA (Mihaylov et al., 2018) and Com-monsenseQA (Talmor et al., 2018).”
MethodWang, L., Sun, M., Zhao, W., Shen, K., & Liu, J. (2018) · International Workshop on Semantic Evaluation (SemEval) · doi:10.18653/v1/s18-1120 · Cited by 96 · cited 1×
“Weissenborn et al. (2017), Bauer et al. (2018), Mihaylov and Frank (2018), Pan et al. (2019), Chen et al. (2018), Wang et al. (2018) follow a retrieve-then-encode paradigm, i.e., they first retrieve relevant knowledge from KBs, and only the retrieved knowledge relevant locally to the reading text…”
BackgroundSelf-citationImproving Question Answering by Commonsense-Based Pre-training
Zhong, W., Tang, D., Duan, N., Zhou, M., Wang, J., & Yin, J. P. (2019) · Lecture notes in computer science · doi:10.1007/978-3-030-32233-5_2 · Cited by 86
Dynamic Integration of Background Knowledge in Neural NLU SystemsPreprint
Weissenborn, D., Kočiský, T., & Dyer, C. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1706.02596 · Cited by 79 · cited 1×
“Weissenborn et al. (2017), Bauer et al. (2018), Mihaylov and Frank (2018), Pan et al. (2019), Chen et al. (2018), Wang et al. (2018) follow a retrieve-then-encode paradigm, i.e., they first retrieve relevant knowledge from KBs, and only the retrieved knowledge relevant locally to the reading text…”
BackgroundImproving Question Answering with External KnowledgePreprint
Pan, X., Sun, K., Yu, D., Chen, J., Ji, H., Cardie, C., & Yu, D. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1902.00993 · Cited by 33 · cited 1×
“Weissenborn et al. (2017), Bauer et al. (2018), Mihaylov and Frank (2018), Pan et al. (2019), Chen et al. (2018), Wang et al. (2018) follow a retrieve-then-encode paradigm, i.e., they first retrieve relevant knowledge from KBs, and only the retrieved knowledge relevant locally to the reading text…”
BackgroundImproving Question Answering by Commonsense-Based Pre-TrainingPreprint
Zhong, W., Tang, D., Duan, N., et al. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1809.03568 · Cited by 13 · cited 2×
“Zhong et al. (2018) try to leverage pre-trained KB embeddings to solve the multi-choice MRC task.”
Research works printed but not linked: 1
- Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word rep- resentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227- 2237. Association for Computational Linguistics.
Cited by
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
“ows injecting factual knowledge during fine-tuning on downstream tasks. Guan et al. [52] employed commonsense knowledge bases, ConceptNet and ATOMIC, to enhance GPT-2 for story generation. Yang et al. [207] proposed a knowledge-text fusion model to acquire related linguistic and factual knowledge for machine reading co…”
BackgroundKnowledge Enhanced Contextual Word Representations
Peters, M. E., Neumann, M., Logan, R., Schwartz, R., Joshi, V., Singh, S., & Smith, N. A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1005 · Cited by 661
“Task-specific KB architectures Other work has focused on integrating KBs into neural architectures for specific downstream tasks (Yang and Mitchell, 2017; Sun et al., 2018; Chen et al., 2018; Bauer et al., 2018; Mihaylov and Frank, 2018; Wang and Jiang, 2019; Yang et al., 2019).”
BackgroundQA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
Yasunaga, M., Ren, H., Bosselut, A., Liang, P., & Leskovec, J. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.45 · Cited by 522
Domain-specific knowledge graphs: A survey
Abu-Salih, B. (2021) · Journal of Network and Computer Applications · doi:10.1016/j.jnca.2021.103076 · Cited by 379
Large language models (LLMs): survey, technical frameworks, and future challenges
Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395
Metadata
- Authors
- Yang, A., Wang, Q., Liu, J., Liu, K., Lyu, Y., Wu, H., She, Q., & Li, S.
- Affiliations
- Baidu (China) (CN)
- Peking University (CN)
- Venue
- Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics
- Published
- 2019
- Pages
- 2346-2357
- ISSN
- 0736-587X (ISSN-L)
- OpenAlex
- W2951561177
- Cited by
- 155
- References
- 45
- Citations by year
- 2026: 3 · 2025: 5 · 2024: 16 · 2023: 22 · 2022: 24 · 2021: 47 · 2020: 29 · 2019: 9
- FWCI
- 10.43
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Proceedings of the conference - Association for Computational Linguistics. Meeting (publishedVersion, cc-by)
- Funders
- Baidu
- National Natural Science Foundation of China (61876009, 61533018, 61876223)
- Peking University
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- machine reading comprehension, pre-trained language models, BERT, knowledge base, external knowledge integration, knowledge fusion, records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.