Language Models as Knowledge Bases?
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Tracing back to the work [53], large language models have shown their impressive power in storing factual information, such as entities and common sense, and then commonsense knowledge can be reliably transferred to downtown tasks.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While pure generative LLMs are often capable of answering basic factual questions (Petroni et al., 2019), simply by reciting information memorized from their training sets and stored in the model parameters”
From Deciphering the Interplay of Parametric and Non-parametric Memory in Retrieval-augmented Language Models (2024) · cites this paper as Petroni et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It is well-accepted, even among skeptical researchers, that the parameters of modern LLMs encode a remarkable amount of world knowledge [2, 12, 23, 28, 31, 39, 62, 79], which can be leveraged by an LLM in flexible and robust ways [81, 83, 93].”
From HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models (2024) · cites this paper as [62]
The cited source says what the claim attributes to it.
In the citing paper
“Pre-training of models on a large corpus is known to result in the model memorizing knowledge in its parameters [169, 201, 221].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [201]
The cited source says what the claim attributes to it.
In the citing paper
“The aforementioned example shows that AI long-term memory can be implicitly stored in the parameters of AI models [248, 249].”
From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [248]
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“BERT (base) (Devlin et al., 2018a) Transformer 110M Wikipedia (en) & BookCorpus 3.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“SQuAD (Rajpurkar et al., 2016) is a popular question answering dataset.”
Background(2013) · Cited by 5,726
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · cited 3×
“…2017), recurrent neural networks (Zaremba et al., 2014; Merity et al., 2016; Melis et al., 2017) or self-attention mechanisms (Radford et al., 2018; Dai et al., 2019; Radford et al., 2019). fairseq-fconv: Instead of commonly used recurrent neural networks, Dauphin et al. (2017) use multiple layers…”
BackgroundMethodInfluential
Research works printed but not linked: 3
- [1]1. Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. 2019. Learning to understand goal specifications by 7 https://github.com/allenai/allennlp 8 https://github.com/pytorch/fairseq 9 https://github.com/huggingface/ pytorch-transformers modelling reward. In International Conference on Learning Representations (ICLR).
- [12]12. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh Interna- tional Conference on Language Resources and Eval- uation (LREC-2018).
- [24]24. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word rep- resentations. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, NAACL-HLT 2018, New Or- leans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 2227-2237.
Cited by
Items 201–300 of 1,061
RARR: Researching and revising what language models say, using language models
Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A. T., Fan, Y., Zhao, V., Lao, N., Lee, H., Juan, D.-C., et al. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.910 · Cited by 141
Prompt engineering for healthcare: Methodologies and applications
Wang, J., Shi, E., Yu, S., Wu, Z., Hu, H., Ma, C., Dai, H., Yang, Q., Kang, Y., Wu, J., et al. (2026) · Meta-Radiology · doi:10.1016/j.metrad.2025.100190 · Cited by 139
SoK: Memorization in General-Purpose Large Language ModelsPreprint
Hartmann, V., Suri, A., Bindschaedler, V., Evans, D. M., Tople, S., & West, R. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.18362 · Cited by 139
Turning a CLIP Model into a Scene Text Detector
Yu, W., Liu, Y., Hua, W., Jiang, D., Ren, B., & Bai, X. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00674 · Cited by 138
Should You Fine-Tune BERT for Automated Essay Scoring?
Mayfield, E., & Black, A. W. (2020) · Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications · doi:10.18653/v1/2020.bea-1.15 · Cited by 137
Inducing Relational Knowledge from BERT
Bouraoui, Z., Camacho-Collados, J., & Schockaert, S. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6242 · Cited by 137
Towards Tracing Knowledge in Language Models Back to the Training Data
Akyürek, E., Bolukbasi, T., Liu, F., Xiong, B., Tenney, I., Andreas, J., & Guu, K. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.180 · Cited by 138
Prototypical Verbalizer for Prompt-based Few-shot Tuning
Cui, G., Hu, S., Ding, N., Huang, L., & Liu, Z. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.483 · Cited by 136
Continual Learning of Large Language Models: A Comprehensive Survey
Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang, Y., Wang, Z., Ebrahimi, S., & Wang, H. (2024) · arXiv · doi:10.48550/arxiv.2404.16789 · Cited by 135
DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering
Neeman, E., Aharoni, R., Honovich, O., Choshen, L., Szpektor, I., & Abend, O. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.559 · Cited by 139
Evaluating the Ripple Effects of Knowledge Editing in Language Models
Cohen, R., Biran, E., Yoran, O., Globerson, A., & Geva, M. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00644 · Cited by 134
“Modern language models (LMs) capture a large volume of factual knowledge in their parameters, which can be effectively utilized in down-stream tasks (Petroni et al., 2019; Roberts et al., 2020; Shin et al., 2020; Razniewski et al., 2021; Heinzerling and Inui, 2021; Kadavath et al., 2022; Cohen et…”
BackgroundReasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.294 · Cited by 134
Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit KnowledgePreprint
Talmor, A., Tafjord, O., Clark, P. E., Goldberg, Y., & Berant, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.06609 · Cited by 134
Investigating Pretrained Language Models for Graph-to-Text Generation
Ribeiro, L. F. R., Schmitt, M., Schütze, H., & Gurevych, I. (2021) · Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI · doi:10.18653/v1/2021.nlp4convai-1.20 · Cited by 133
Prompt-learning for Fine-grained Entity Typing
Ding, N., Chen, Y., Han, X., Xu, G., Wang, X., Xie, P., Zheng, H., Liu, Z., Li, J., & Kim, H.-G. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.512 · Cited by 133
A Review of Knowledge Graph Completion
Zamini, M., Reza, H., & Rabiei, M. (2022) · Information · doi:10.3390/info13080396 · Cited by 133
A Systematic Investigation of Commonsense Knowledge in Large Language Models
Li, X. L., Kuncoro, A., Hoffmann, J., d’Autume, C. D. M., Blunsom, P., & Nematzadeh, A. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.812 · Cited by 133
Fang, X., Wang, F., Liu, L., He, J., Lin, D., Xiang, Y., Zhu, K., Zhang, X., Wu, H., Li, H., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00721-6 · Cited by 132
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning
Deng, M., Wang, J., Hsieh, C.-P., Wang, Y., Guo, H., Shu, T., Song, M., Xing, E., & Hu, Z. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.222 · Cited by 132
That is a Known Lie: Detecting Previously Fact-Checked Claims
Shaar, S., Babulkov, N., Da San Martino, G., & Nakov, P. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.332 · Cited by 132
Extracting Cultural Commonsense Knowledge at Scale
Nguyen, T.-P., Razniewski, S., Varde, A. S., & Weikum, G. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583535 · Cited by 133
Word meaning in minds and machines.
Lake, B. M., & Murphy, G. L. (2023) · Psychological Review · doi:10.1037/rev0000297 · Cited by 131
Large Language Models are Edge-Case Fuzzers: Testing Deep Learning Libraries via FuzzGPTPreprint
Deng, Y.-L., Xia, C., Yang, C., Zhang, S., Yang, S., & Zhang, L.-M. (2023) · arXiv.org · doi:10.48550/arxiv.2304.02014 · Cited by 122
Entities as Experts: Sparse Memory Access with Entity Supervision
Févry, T., Baldini Soares, L., FitzGerald, N., Choi, E., & Kwiatkowski, T. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.400 · Cited by 130
Language models are not naysayers: an analysis of language models on negation benchmarks
Truong, T. H., Baldwin, T., Verspoor, K. M., & Cohn, T. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.10 · Cited by 130
Language Models Implement Simple Word2Vec-style Vector Arithmetic
Merullo, J., Eickhoff, C., & Pavlick, E. (2024) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2024.naacl-long.281 · Cited by 132
Chiu, Y. Y., Jiang, L., Lin, B. Y., Park, C. Y., Li, S. S., Ravi, S., Bhatia, M., Antoniak, M., Tsvetkov, Y., Shwartz, V., et al. (2025) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2025.acl-long.1247 · Cited by 131
Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks
Patil, V., Hase, P., & Bansal, M. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.17410 · Cited by 128
EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models
Wang, P., Zhang, N., Tian, B., Xi, Z., Yao, Y., Xu, Z., Wang, M., Mao, S., Wang, X., Cheng, S., et al. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-demos.9 · Cited by 129
Formal Mathematical Reasoning: A New Frontier in AIPreprint
Yang, K., Poesia, G., He, J., Li, W., Lauter, K., Chaudhuri, S., & Song, D. X. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2412.16075 · Cited by 129
Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
Elazar, Y., Ravfogel, S., Jacovi, A., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00359 · Cited by 126
“, 2019a), world knowledge (Petroni et al., 2019; Jiang et al., 2020), reasoning (Talmor et al.”
BackgroundGet Your Vitamin C! Robust Fact Verification with Contrastive Evidence
Schuster, T., Fisch, A., & Barzilay, R. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.52 · Cited by 126
Reframing Instructional Prompts to GPTk’s Language
Mishra, S., Khashabi, D., Baral, C., Choi, Y., & Hajishirzi, H. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.50 · Cited by 126
Lin, B. Y., Lee, S., Qiao, X., & Ren, X. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.102 · Cited by 125
Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering
Baek, J., Aji, A. F., & Saffari, A. (2023) · Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE) · doi:10.18653/v1/2023.nlrse-1.7 · Cited by 124
Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training
Li, M., Roller, S., Kulikov, I., Welleck, S., Boureau, Y.-L., Cho, K., & Weston, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.428 · Cited by 124
LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development
Chalkidis, I., Garneau, N., Goanță, C., Katz, D. M., & Søgaard, A. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.865 · Cited by 124
GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models
Yin, D., Bansal, H., Monajatipoor, M., Li, L. H., & Chang, K.-W. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.132 · Cited by 123
Adapting Large Language Models to Domains via Reading Comprehension
Cheng, D., Huang, S.-H., & Wei, F. (2023) · International Conference on Learning Representations · doi:10.48550/arxiv.2309.09530 · Cited by 60
Large Language Models can Learn RulesPreprint
Zhu, Z., Xue, Y., Chen, X., Zhou, D., Tang, J., Schuurmans, D., & Dai, H. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.07064 · Cited by 123
ProofWriter: Generating implications, proofs, and abductive statements over natural language
Tafjord, O., Dalvi, B., & Clark, P. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.317 · Cited by 122
Memory-efficient Transformers via Top-k Attention
Gupta, A., Dar, G., Goodman, S., Ciprut, D., & Berant, J. (2021) · Workshop on Simple and Efficient Natural Language Processing (SustaiNLP) · doi:10.18653/v1/2021.sustainlp-1.5 · Cited by 123
Symbolic Knowledge Distillation: from General Language Models to Commonsense ModelsPreprint
West, P., Bhagavatula, C., Hessel, J., Hwang, J. D., Jiang, L., Bras, R. J. L., Lu, X., Welleck, S., & Choi, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.07178 · Cited by 123
Can Multiple-choice Questions Really Be Useful in Detecting the Abilities of LLMs?
Li, W., Li, L., Xiang, T., Liu, X., Deng, W., & García, N. (2024) · Language Resources and Evaluation Conference (LREC) · doi:10.63317/2bvpg8s5dnii · Cited by 121
Topic Modeling with Contextualized Word Representation ClustersPreprint
Thompson, L., & Mimno, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.12626 · Cited by 123
Towards Faithful Model Explanation in NLP: A Survey
Lyu, Q., Apidianaki, M., & Callison-Burch, C. (2024) · Computational Linguistics · doi:10.1162/coli_a_00511 · Cited by 120
On the Explainability of Natural Language Processing Deep Models
Zini, J. E., & Awad, M. (2023) · ACM Computing Surveys · doi:10.1145/3529755 · Cited by 120
Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering
Ma, K., Ilievski, F., Francis, J., Bisk, Y., Nyberg, E., & Oltramari, A. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i15.17593 · Cited by 124
Where to Hide a Stolen Elephant: Leaps in Creative Writing with Multimodal Machine Intelligence
Singh, N., Bernal, G., Savchenko, D., & Glassman, E. L. (2023) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3511599 · Cited by 119
“In parallel, models with implicitly richer knowledge bases [74] may also extend more diverse suggestions to users, inding interesting relations with aspects of their writing, and assisting them in performing contextually appropriate and creatively fulilling integrations.”
BackgroundLanguage Models as Knowledge Embeddings
Wang, X., He, Q., Liang, J., & Xiao, Y. (2022) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2022/318 · Cited by 119
Prompt to Transfer: Sim-to-Real Transfer for Traffic Signal Control with Prompt Learning
Da, L., Gao, M., Mei, H., & Wei, H. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i1.27758 · Cited by 123
The Prompt Report: A Systematic Survey of Prompt Engineering Techniques
Schulhoff, S., Ilie, M., Balepur, N., Kahadze, K., Liu, A., Si, C., Li, Y., Gupta, A., Han, H., Schulhoff, S., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.06608 · Cited by 118
Few-shot Learning with Multilingual Generative Language Models
Lin, X. V., Mihaylov, T., Artetxe, M., Wang, T., Chen, S., Simig, D., Ott, M., Goyal, N., Bhosale, S., Du, J., et al. (2022) · Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2022.emnlp-main.616 · Cited by 118
Lin, B. Y., Lee, S., Khanna, R., & Ren, X. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.557 · Cited by 117
Complex Knowledge Base Question Answering: A Survey
Lan, Y., He, G., Jiang, J., Jiang, J., Zhao, W. X., & Wen, J.-R. (2023) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2022.3223858 · Cited by 117
BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?
Ushio, A., Espinosa-Anke, L., Schockaert, S., & Camacho-Collados, J. (2021) · doi:10.18653/v1/2021.acl-long.280 · Cited by 119
NormBank: A Knowledge Bank of Situational Social Norms
Ziems, C., Dwivedi-Yu, J., Wang, Y., Halevy, A. Y., & Yang, D. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.429 · Cited by 116
Pro-Tuning: Unified Prompt Tuning for Vision Tasks
Nie, X., Ni, B., Chang, J., Meng, G., Huo, C., Xiang, S., & Tian, Q. C. (2023) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2023.3327605 · Cited by 117
Deng, Y., Xia, C. S., Yang, C., Zhang, S. D., Yang, S., & Zhang, L. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3623343 · Cited by 115
Hurdles to Progress in Long-form Question Answering
Krishna, K., Roy, A., & Iyyer, M. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.393 · Cited by 114
Time Masking for Temporal Language Models
Rosin, G. D., Guy, I., & Radinsky, K. (2022) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3488560.3498529 · Cited by 114
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances
Zhang, Z., Fang, M., Chen, L., Namazi‐Rad, M., & Wang, Y.-X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.516 · Cited by 117
An Information-theoretic Approach to Prompt Engineering Without Ground Truth Labels
Sorensen, T., Robinson, J., Rytting, C., Shaw, A., Rogers, K., Delorey, A., Khalil, M., Fulda, N., & Wingate, D. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.60 · Cited by 113
“…pre-trained language023 models (LMs) learn substantial linguistic (Liu et al.,024 2019; Amrami and Goldberg, 2018) and factual025 world knowledge (Petroni et al., 2020; Bosselut026 et al.; Bouraoui et al.; Zuo et al., 2018), achiev-027 ing state-of-the-art performance on classic NLP028 tasks…”
Auto-Debias: Debiasing Masked Language Models with Automated Biased Prompts
Guo, Y., Yang, Y., & Abbasi, A. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.72 · Cited by 113
NaturalProofs: Mathematical Theorem Proving in Natural LanguagePreprint
Welleck, S., Liu, J., Bras, R. J. L., Hajishirzi, H., Choi, Y., & Cho, K. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.01112 · Cited by 116
Few-Shot Bot: Prompt-Based Learning for Dialogue SystemsPreprint
Madotto, A., Lin, Z., Winata, G. I., & Fung, P. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.08118 · Cited by 114
Interactive Natural Language ProcessingPreprint
Wang, Z., Zhang, G., Yang, K., Shi, N., Zhou, W., Hao, S., Xiong, G., Li, Y., Sim, M. Y., Chen, X., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2305.13246 · Cited by 113
What Artificial Neural Networks Can Tell Us about Human Language Acquisition
Warstadt, A., & Bowman, S. R. (2022) · Algebraic Structures in Natural Language · doi:10.1201/9781003205388-2 · Cited by 111
Hardalov, M., Mihaylov, T., Zlatkova, D., Dinkov, Y., Koychev, I. K., & Nakov, P. (2020) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.438 · Cited by 113
Flooding spread of manipulated knowledge in LLM-based multi-agent communities
Ju, T., Wang, Y., Hua, Y., Ma, X., Cheng, P., Zhao, H., Wang, Y., Liu, L., Xie, J., Zhang, Z., et al. (2026) · Science China Information Sciences · doi:10.1007/s11432-024-4663-2 · Cited by 111
PromDA: Prompt-based Data Augmentation for Low-Resource NLU Tasks
Wang, Y., Xu, C., Sun, Q., Hu, H., Tao, C., Geng, X., & Jiang, D. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.292 · Cited by 111
Cheng, P., Yang, X., Smith, K. E., Yu, Z., Chen, A., Bian, J., & Wu, Y. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104630 · Cited by 111
Self-citationGood Examples Make A Faster Learner: Simple Demonstration-based Learning for Low-resource NER
Lee, D., Kadakia, A., Tan, K., Agarwal, M., Feng, X., Shibuya, T., Mitani, R., Sekiya, T., Pujara, J., & Ren, X. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.192 · Cited by 110
Massively Multi-Cultural Knowledge Acquisition & LM BenchmarkingPreprint
Fung, Y. R., Zhao, R., Doo, J., Sun, C., & Ji, H. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.09369 · Cited by 109
Zhang, Z., Fang, M., & Chen, L. (2024) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2024.findings-acl.415 · Cited by 108
CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeamingPreprint
Chiu, Y. Y., Jiang, L., Lin, B. Y., Park, C. Y., Li, S. S., Ravi, S., Bhatia, M., Antoniak, M., Tsvetkov, Y., Shwartz, V., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2410.02677 · Cited by 108
Few-Shot Text Generation with Natural Language Instructions
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.32 · Cited by 107
“The idea to rephrase tasks as cloze questions is commonly used to probe the knowledge contained within masked language models (e.g., Petroni et al., 2019; Wang et al., 2019; Talmor et al., 2020; Schick and Schütze, 2020; Ettinger, 2020; Kassner and Schütze, 2020; Sakaguchi et al., 2020).”
MethodDimensions of commonsense knowledge
Ilievski, F., Oltramari, A., Ma, K., Zhang, B., McGuinness, D. L., & Szekely, P. A. (2021) · Knowledge-Based Systems · doi:10.1016/j.knosys.2021.107347 · Cited by 106
Understanding by Understanding Not: Modeling Negation in Language Models
Hosseini, A., Reddy, S., Bahdanau, D., Hjelm, R. D., Sordoni, A., & Courville, A. C. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.102 · Cited by 106
Pre-trained Language Model with Prompts for Temporal Knowledge Graph Completion
Xu, W., Liu, B., Peng, M., Xu, J., & Peng, M. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.493 · Cited by 106
Are Large Language Models Geospatially Knowledgeable?
Bhandari, P., Anastasopoulos, A., & Pfoser, D. (2023) · ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (SIGSPATIAL GIS) · doi:10.1145/3589132.3625625 · Cited by 106
Understanding LLMs: A comprehensive overview from training to inference
Liu, Y., He, H., Han, T., Zhang, X., Liu, M., Tian, J., Zhang, Y., Wang, J., Gao, X., Zhong, T., et al. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.129190 · Cited by 111
Open challenges and opportunities in federated foundation models towards biomedical healthcare
Li, X., Peng, L., Wang, Y., & Zhang, W. (2025) · BioData Mining · doi:10.1186/s13040-024-00414-9 · Cited by 108
SentiPrompt: Sentiment Knowledge Enhanced Prompt-Tuning for Aspect-Based Sentiment AnalysisPreprint
Li, C., Gao, F., Bu, J., Xu, L., Chen, X., Gu, Y., Shao, Z., Zheng, Q., Zhang, N., Wang, Y., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.08306 · Cited by 107
Language Model Behavior: A Comprehensive Survey
Chang, T. A., & Bergen, B. K. (2024) · Computational Linguistics · doi:10.1162/coli_a_00492 · Cited by 105
“countries and locations, popular products, historical figures, and even genres of books, movies, and music (Petroni et al. 2019; Penha and Hauff 2020).”
BackgroundDeepRAG: Thinking to Retrieve Step by Step for Large Language Models
Guan, X., Zeng, J., Meng, F., Xin, C., Lu, Y., Lin, H., Han, X., Sun, L., & Zhou, J. (2026) · International Conference on Learning Representations · doi:10.48550/arxiv.2502.01142 · Cited by 105
UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models
Dong, Y. R., Lin, H., Belkin, M., Huerta, R., & Vulić, I. (2025) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2025.naacl-long.444 · Cited by 106
TransOMCS: From Linguistic Graphs to Commonsense Knowledge
Zhang, H., Khashabi, D., Song, Y., & Roth, D. (2020) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2020/554 · Cited by 104
Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey
Bhargava, P., & Ng, V. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i11.21496 · Cited by 104
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP
Ye, Q., Lin, B. Y., & Ren, X. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.572 · Cited by 103
Image captioning for effective use of language models in knowledge-based visual question answering
Salaberria, A., Azkune, G., de Lacalle, O. L., Soroa, A., & Agirre, E. (2022) · Expert Systems with Applications · doi:10.1016/j.eswa.2022.118669 · Cited by 103
Solving Aspect Category Sentiment Analysis as a Text Generation Task
Liu, J., Teng, Z., Cui, L., Liu, H., & Zhang, Y. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.361 · Cited by 103
Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction
Bae, I., Lee, J., & Jeon, H. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.00078 · Cited by 105
Are Large Pre-Trained Language Models Leaking Your Personal Information?
Huang, J., Shao, H., & Chang, K. C.-C. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.148 · Cited by 102
Jin, W., Zhao, B., Yu, H., Tao, X., Yin, R., & Liu, G. (2022) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-022-00891-8 · Cited by 103
Explainable artificial intelligence (XAI): From inherent explainability to large language models
Mumuni, F., & Mumuni, A. (2026) · Array · doi:10.1016/j.array.2026.101133 · Cited by 102
Entity-Based Knowledge Conflicts in Question Answering
Longpre, S., Perisetla, K., Chen, A., Ramesh, N., DuBois, C., & Singh, S. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.565 · Cited by 101
“, their learned weights) instilled by pre-training and fine-tuning (Petroni et al., 2019).”
BackgroundFactual Confidence of LLMs: on Reliability and Robustness of Current Estimators
Mahaut, M., Aina, L., Czarnowska, P., Hardalov, M., Müller, T., & Màrquez, L. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.250 · Cited by 102
Housekeep: Tidying Virtual Households Using Commonsense Reasoning
Kant, Y., Ramachandran, A., Yenamandra, S., Gilitschenski, I., Batra, D., Szot, A., & Agrawal, H. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-19842-7_21 · Cited by 101
Tan, H., Sun, F., Yang, W., Wang, Y., Cao, Q., & Cheng, X. (2024) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2024.acl-long.337 · Cited by 102
Metadata
- Authors
- Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A.
- Affiliations
- Meta (United States) (US)
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 2463-2473
- OpenAlex
- W2970476646
- Cited by
- 1,834
- References
- 42
- Citations by year
- 2026: 129 · 2025: 230 · 2024: 300 · 2023: 469 · 2022: 334 · 2021: 246 · 2020: 121 · 2019: 5
- FWCI
- 67.54
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- language models, knowledge base, knowledge representation, knowledge extraction, question answering, commonsense knowledge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.