Language Models as Knowledge Bases?
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Tracing back to the work [53], large language models have shown their impressive power in storing factual information, such as entities and common sense, and then commonsense knowledge can be reliably transferred to downtown tasks.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While pure generative LLMs are often capable of answering basic factual questions (Petroni et al., 2019), simply by reciting information memorized from their training sets and stored in the model parameters”
From Deciphering the Interplay of Parametric and Non-parametric Memory in Retrieval-augmented Language Models (2024) · cites this paper as Petroni et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It is well-accepted, even among skeptical researchers, that the parameters of modern LLMs encode a remarkable amount of world knowledge [2, 12, 23, 28, 31, 39, 62, 79], which can be leveraged by an LLM in flexible and robust ways [81, 83, 93].”
From HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models (2024) · cites this paper as [62]
The cited source says what the claim attributes to it.
In the citing paper
“Pre-training of models on a large corpus is known to result in the model memorizing knowledge in its parameters [169, 201, 221].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [201]
The cited source says what the claim attributes to it.
In the citing paper
“The aforementioned example shows that AI long-term memory can be implicitly stored in the parameters of AI models [248, 249].”
From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [248]
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“BERT (base) (Devlin et al., 2018a) Transformer 110M Wikipedia (en) & BookCorpus 3.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“SQuAD (Rajpurkar et al., 2016) is a popular question answering dataset.”
Background(2013) · Cited by 5,726
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · cited 3×
“…2017), recurrent neural networks (Zaremba et al., 2014; Merity et al., 2016; Melis et al., 2017) or self-attention mechanisms (Radford et al., 2018; Dai et al., 2019; Radford et al., 2019). fairseq-fconv: Instead of commonly used recurrent neural networks, Dauphin et al. (2017) use multiple layers…”
BackgroundMethodInfluential
Research works printed but not linked: 3
- [1]1. Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. 2019. Learning to understand goal specifications by 7 https://github.com/allenai/allennlp 8 https://github.com/pytorch/fairseq 9 https://github.com/huggingface/ pytorch-transformers modelling reward. In International Conference on Learning Representations (ICLR).
- [12]12. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh Interna- tional Conference on Language Resources and Eval- uation (LREC-2018).
- [24]24. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word rep- resentations. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, NAACL-HLT 2018, New Or- leans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 2227-2237.
Cited by
Items 101–200 of 1,061
SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models
Wang, L., Zhao, W., Wei, Z., & Liu, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.295 · Cited by 264
A Survey of Knowledge-enhanced Text Generation
Yu, W., Zhu, C., Li, Z., Hu, Z., Wang, Q., Ji, H., & Jiang, M. (2022) · ACM Computing Surveys · doi:10.1145/3512467 · Cited by 257
Holistic Evaluation of Language Models
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2211.09110 · Cited by 255
Self-citationLarge language models for human–robot interaction: A review
Zhang, C., Chen, J., Li, J., Peng, Y., & Mao, Z. (2023) · Biomimetic Intelligence and Robotics · doi:10.1016/j.birob.2023.100131 · Cited by 254
Visual-Language Prompt Tuning with Knowledge-Guided Context Optimization
Yao, H., Zhang, R., & Xu, C. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.00653 · Cited by 252
Information-Theoretic Probing with Minimum Description Length
Voita, E., & Titov, I. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.14 · Cited by 251
Lu, Y., Liu, J., Zhang, Y., Liu, Y., & Tian, X. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.00514 · Cited by 246
Exploring the role of large language models in the scientific method: from hypothesis to discovery
Zhang, Y., Khan, S. A., Mahmud, A., Yang, H., Lavin, A., Levin, M. G., Frey, J. G., Dunnmon, J., Evans, J. A., Bundy, A., et al. (2025) · npj Artificial Intelligence · doi:10.1038/s44387-025-00019-5 · Cited by 247
Yang, L., Chen, H., Li, Z., Ding, X., & Wu, X. (2023) · arXiv · doi:10.48550/arxiv.2306.11489 · Cited by 239
“As a result, some researchers [8]– [10] argue that LLMs can serve as parameterized knowledge bases to replace structured knowledge bases like knowledge”
Yang, L., Chen, H., Li, Z., Ding, X., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3360454 · Cited by 238
“For example, Petroni et al. [8] proposed LAMA, a knowledge probe consisting of cloze-style queries, to measure relational knowledge contained in PLMs.”
BackgroundEfficient Methods for Natural Language Processing: A Survey
Treviso, M., Lee, J.-U., Ji, T., van Aken, B., Cao, Q., Ciosici, M. R., Hassid, M., Heafield, K., Hooker, S., Raffel, C., et al. (2023) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00577 · Cited by 242
CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say No
Wang, H., Li, Y., Yao, H., & Li, X. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00173 · Cited by 240
Locating and Editing Factual Associations in GPT
Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022) · arXiv · doi:10.48550/arxiv.2202.05262 · Cited by 230
True Few-Shot Learning with Language Models
Perez, E., Kiela, D., & Cho, K. (2021) · arXiv · doi:10.48550/arxiv.2105.11447 · Cited by 226
“1 (left) shows that the worst prompts perform far worse than average, so it would be useful if prompt selection helped to avoid the worst The chance of various accuracy gains on LAMA over the average prompt, when using prompts chosen by CV, and ( Right ) conservative estimates of CV that also minimize variance in CV;…”
BackgroundMethodInfluentialCOLD: A Benchmark for Chinese Offensive Language Detection
Deng, J., Zhou, J., Sun, H., Zheng, C., Mi, F., Meng, H. M. L., & Huang, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.796 · Cited by 226
Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
Yang, C., Zhu, Y. T., Wang, L., Wang, Y., Chen, Q., Gao, C., Yan, B., & Chen, Y. (2025) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3699518 · Cited by 226
Efficient Large Scale Language Modeling with Mixtures of Experts
Artetxe, M., Bhosale, S., Goyal, N., Mihaylov, T., Ott, M., Shleifer, S., Lin, X. V., Du, J., Iyer, S., Pasunuru, R., et al. (2021) · arXiv · doi:10.48550/arxiv.2112.10684 · Cited by 223
Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly
Kassner, N., & Schütze, H. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.698 · Cited by 223
Measuring and improving consistency in pretrained language models
Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., & Goldberg, Y. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00410 · Cited by 220
“For each relation, we sample up to five paraphrases, comparing each of the new patterns to the base-pattern from LAMA.”
MethodUniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual PretrainingPreprint
Chung, H. W., Constant, N., Garcia, X., Roberts, A., Tay, Y., Narang, S., & Fırat, O. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.09151 · Cited by 224
Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models
Schramowski, P., Brack, M., Deiseroth, B., & Kersting, K. (2023) · 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52729.2023.02157 · Cited by 219
“they retain general knowledge implicitly present in the data [27].”
BackgroundInfluentialHuang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2023) · arXiv · doi:10.48550/arxiv.2311.05232 · Cited by 2,516
“While significant efforts have been undertaken in exploring their knowledge storage (Geva et al., 2021; Meng et al., 2022)and probing (Petroni et al., 2019; Zhong et al., 2021; Yu et al., 2023c), the exact mechanism by which LLMs capture the factual knowledge remains elusive.”
ShapeLLM: Universal 3D Object Understanding for Embodied Interaction
Qi, Z., Dong, R., Zhang, S., Geng, H., Han, C., Ge, Z., Yi, L., & Ma, K. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72775-7_13 · Cited by 221
Foundation metrics for evaluating effectiveness of healthcare conversations powered by generative AI
Abbasian, M., Khatibi, E., Azimi, I., Oniani, D., Shakeri Hossein Abad, Z., Thieme, A., Sriram, R., Yang, Z., Wang, Y., Lin, B., et al. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01074-z · Cited by 213
Time-aware language models as temporal knowledge bases
Dhingra, B., Cole, J. R., Eisenschlos, J. M., Gillick, D., Eisenstein, J., & Cohen, W. W. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00459 · Cited by 212
“Petroni et al. (2019) introduced the LAMA benchmark for probing the factual knowledge mem-orized by LMs, which consists of cloze queries about facts, e.g. “Dante was born in __X__”.”
BackgroundDense Text Retrieval Based on Pretrained Language Models: A Survey
Zhao, W. X., Liu, J., Ren, R., & Wen, J.-R. (2024) · ACM Transactions on Information Systems · doi:10.1145/3637870 · Cited by 211
Topic-Driven and Knowledge-Aware Transformer for Dialogue Emotion Detection
Zhu, L., Pergola, G., Gui, L., Zhou, D., & He, Y. (2021) · doi:10.18653/v1/2021.acl-long.125 · Cited by 210
Ibrahim, N., AboulEla, S., Ibrahim, A. A., & Kashef, R. F. (2024) · Discover Artificial Intelligence · doi:10.1007/s44163-024-00175-8 · Cited by 209
CLIP Models are Few-Shot Learners: Empirical Studies on VQA and Visual Entailment
Song, H., Dong, L., Zhang, W., Liu, T., & Wei, F. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.421 · Cited by 211
Annotating Columns with Pre-trained Language Models
Suhara, Y., Li, J., Li, Y., Zhang, D., Demiralp, Ç., Chen, C., & Tan, W.-C. (2022) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3514221.3517906 · Cited by 206
Do Prompt-Based Models Really Understand the Meaning of Their Prompts?
Webson, A., & Pavlick, E. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.167 · Cited by 203
Machine Knowledge: Creation and Curation of Comprehensive Knowledge Bases
Weikum, G., Dong, X. L., Razniewski, S., & Suchanek, F. M. (2021) · Foundations and Trends in Databases · doi:10.1561/1900000064 · Cited by 205
Prompting for Multimodal Hateful Meme Classification
Cao, R., Lee, R. K.-W., Chong, W.-H., & Jiang, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.22 · Cited by 201
Domain Adaptation via Prompt Learning
Ge, C., Huang, R., Xie, M., Lai, Z., Song, S., Li, S., & Huang, G. (2025) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2023.3327962 · Cited by 199
VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena
Pârcălăbescu, L., Cafagna, M., Muradjan, L., Frank, A., Calixto, I., & Gatt, A. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.567 · Cited by 200
A Survey of Knowledge Enhanced Pre-Trained Language Models
Hu, L., Liu, Z., Zhao, Z., Hou, L., Nie, L., & Li, J. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2023.3310002 · Cited by 196
Text Classification Using Label Names Only: A Language Model Self-Training Approach
Meng, Y., Zhang, Y., Huang, J., Xiong, C., Ji, H., Zhang, C., & Han, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.724 · Cited by 196
Autoregressive Entity Retrieval
Cao, N. D., Izacard, G., Riedel, S., & Petroni, F. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2010.00904 · Cited by 193
Self-citationInferFix: End-to-End Program Repair with LLMs
Jin, M., Shahriar, S., Tufano, M., Shi, X., Lu, S., Sundaresan, N., & Svyatkovskiy, A. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3613892 · Cited by 191
“With the ability to propose high-quality fixes for critical bugs, InferFix has the potential to greatly enhance the productivity and reliability of the software development process.”
BackgroundGenerated Knowledge Prompting for Commonsense Reasoning
Liu, J., Liu, A., Lu, X., Welleck, S., West, P., Le Bras, R., Choi, Y., & Hajishirzi, H. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.225 · Cited by 191
LAnoBERT: System log anomaly detection based on BERT masked language model
Lee, Y., Kim, J., & Kang, P. (2023) · Applied Soft Computing · doi:10.1016/j.asoc.2023.110689 · Cited by 191
Friha, O., Amine Ferrag, M., Kantarci, B., Cakmak, B., Ozgun, A., & Ghoualmi-Zine, N. (2024) · IEEE Open Journal of the Communications Society · doi:10.1109/ojcoms.2024.3456549 · Cited by 189
Knowledge Neurons in Pretrained Transformers
Dai, D., Dong, L., Hao, Y., Sui, Z., Chang, B., & Wei, F. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.581 · Cited by 187
ISR-LLM: Iterative Self-Refined Large Language Model for Long-Horizon Sequential Task Planning
Zhou, Z., Song, J., Yao, K., Shu, Z., & Ma, L. (2024) · Proceedings - IEEE International Conference on Robotics and Automation/Proceedings · doi:10.1109/icra57147.2024.10610065 · Cited by 187
CoLAKE: Contextualized Language and Knowledge Embedding
Sun, T., Shao, Y., Qiu, X., Guo, Q., Hu, Y., Huang, X., & Zhang, Z. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.327 · Cited by 185
KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA
Marino, K., Chen, X., Parikh, D., Gupta, A., & Rohrbach, M. (2021) · 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr46437.2021.01389 · Cited by 185
“Language models, and especially transformer-based language models, have shown to contain common sense and factual knowledge [58, 30].”
BackgroundOpenPrompt: An Open-source Framework for Prompt-learning
Ding, N., Hu, S., Zhao, W., Chen, Y., Liu, Z., Zheng, H., & Sun, M. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2022.acl-demo.10 · Cited by 185
Learning from Context or Names? An Empirical Study on Neural Relation Extraction
Peng, H., Gao, T., Han, X., Lin, Y., Li, P., Liu, Z., Sun, M., & Zhou, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.298 · Cited by 185
Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification
Schick, T., Schmid, H., & Schütze, H. (2020) · Proceedings of the 28th International Conference on Computational Linguistics · doi:10.18653/v1/2020.coling-main.488 · Cited by 184
Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models
Qi, J., Fernández, R., & Bisazza, A. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.658 · Cited by 182
Editing Factual Knowledge in Language Models
De Cao, N., Aziz, W., & Titov, I. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.522 · Cited by 180
“Petroni et al. (2019) show that pre-trained language models re-call factual knowledge without fine-tuning, which they do by feeding specific prompts to LMs. Hand-crafted prompts have been found not to be the best option to extract knowledge from LMs, and various solutions have been proposed to…”
BackgroundUnsupervised Commonsense Question Answering with Self-Talk
Shwartz, V., West, P., Le Bras, R., Bhagavatula, C., & Choi, Y. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.373 · Cited by 179
Can Foundation Models Wrangle Your Data?
Narayan, A., Chami, I., Orr, L., & Ré, C. (2022) · Proceedings of the VLDB Endowment · doi:10.14778/3574245.3574258 · Cited by 178
Jiang, Z., Araki, J., Ding, H., & Neubig, G. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00407 · Cited by 177
“…demonstrating that modern large-scale LMs also capture a significant amount of knowledge, including factual knowledge about real-world entities (Petroni et al., 2019; Jiang et al., 2020b; Roberts et al., 2020; Bouraoui et al., 2020), commonsense knowledge (Trinh and Le, 2018; Kocijan et al.,…”
BackgroundChia, Y. K., Bing, L., Poria, S., & Si, L. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.5 · Cited by 178
When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment
Jin, Z., Levine, S., Adauto, F. G., Kamal, O., Sap, M., Sachan, M., Mihalcea, R. F., Tenenbaum, J. B., & Schölkopf, B. (2022) · neural information processing systems · doi:10.52202/068431-2063 · Cited by 179
Applications of transformer-based language models in bioinformatics: a survey
Zhang, S., Fan, R., Liu, Y., Chen, S., Liu, Q., & Zeng, W. (2023) · Bioinformatics Advances · doi:10.1093/bioadv/vbad001 · Cited by 176
Unveiling the Implicit Toxicity in Large Language Models
Wen, J., Ke, P., Sun, H., Zhang, Z., Li, C., Bai, J., & Huang, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.84 · Cited by 176
SBERT-WK: A Sentence Embedding Method by Dissecting BERT-Based Word Models
Wang, B., & Kuo, C.-C. J. (2020) · IEEE/ACM Transactions on Audio, Speech, and Language Processing · doi:10.1109/taslp.2020.3008390 · Cited by 175
Fan, H., Liu, X., Fuh, J. Y. H., Lu, W. F., & Li, B. (2025) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-023-02294-y · Cited by 173
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models
Jang, J., Ye, S., Lee, C.-H., Yang, S., Shin, J., Han, J., Kim, G., & Seo, M. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.418 · Cited by 174
SEGA: Instructing Text-to-Image Models using Semantic Guidance
Brack, M., Friedrich, F., Hintersdorf, D., Struppek, L., Schramowski, P., & Kersting, K. (2023) · neural information processing systems · doi:10.52202/075280-1102 · Cited by 178
Knowledge Graph Embedding: A Survey from the Perspective of Representation Spaces
Cao, J., Fang, J., Meng, Z., & Liang, S. (2024) · ACM Computing Surveys · doi:10.1145/3643806 · Cited by 171
DeepStruct: Pretraining of Language Models for Structure Prediction
Wang, C., Liu, X., Chen, Z., Hong, H., Tang, J., & Song, D. X. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.67 · Cited by 172
AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models
Fang, J., Jiang, H., Wang, K., Ma, Y., Jie, S., Wang, X., He, X., & Chua, T.-S. (2024) · arXiv · doi:10.48550/arxiv.2410.02355 · Cited by 170
“Large language models (LLMs) have demonstrated the capability to store extensive knowledge during pre-training and recall it during inference (Brown et al., 2020; Petroni et al., 2019; Roberts et al., 2020; Radford et al., 2019).”
MetaICL: Learning to Learn In Context
Min, S., Lewis, M., Zettlemoyer, L., & Hajishirzi, H. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.201 · Cited by 169
Pre-trained Language Models in Biomedical Domain: A Systematic Survey
Wang, B., Xie, Q., Pei, J., Chen, Z., Tiwari, P., Li, Z., & Fu, J. (2024) · ACM Computing Surveys · doi:10.1145/3611651 · Cited by 169
Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text ClassificationPreprint
Hu, S., Ding, N., Wang, H., Liu, Z., Wang, J., Li, J., Wu, W., & Sun, M. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.02035 · Cited by 169
CPT: Colorful Prompt Tuning for pre-trained vision-language models
Yao, Y., Zhang, A., Zhang, Z., Liu, Z., Chua, T.-S., & Sun, M. (2024) · AI Open · doi:10.1016/j.aiopen.2024.01.004 · Cited by 167
Open-Domain, Content-based, Multi-modal Fact-checking of Out-of-Context Images via Online Resources
Abdelnabi, S., Hasan, R., & Fritz, M. (2022) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52688.2022.01452 · Cited by 170
Streaming Dense Video Captioning
Zhou, X., Arnab, A., Buch, S., Shen, Y., Myers, A., Xiong, X., Nagrani, A., & Schmid, C. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01727 · Cited by 165
Injecting Numerical Reasoning Skills into Language Models
Geva, M., Gupta, A., & Berant, J. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.89 · Cited by 162
Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive SurveyPreprint
Chen, Z., Zhang, Y., Fang, Y., Geng, Y., Guo, L., Chen, X., Li, Q., Zhang, W., Chen, J., Zhu, Y., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2402.05391 · Cited by 162
Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training
Agarwal, O., Ge, H., Shakeri, S., & Al-Rfou, R. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.278 · Cited by 161
“For LAMA probe, we first evaluated the pre-trained REALM, reporting the results on the different sub-corpora in Table 6 (row Wikipedia under REALM).”
BackgroundMethodInfluentialBehind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
Cao, J., Gan, Z., Cheng, Y., Yu, L., Chen, Y.-C., & Liu, J. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58539-6_34 · Cited by 161
Knowledge Graphs: A Practical Review of the Research Landscape
Kejriwal, M. (2022) · Information · doi:10.3390/info13040161 · Cited by 160
Finding Skill Neurons in Pre-trained Transformer-based Language Models
Wang, X., Wen, K., Zhang, Z., Hou, L., Liu, Z., & Li, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.765 · Cited by 161
A Survey on Complex Knowledge Base Question Answering: Methods, Challenges and Solutions
Lan, Y., He, G., Jiang, J., Jiang, J., Zhao, W. X., & Wen, J.-R. (2021) · Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2021/611 · Cited by 159
Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?
Tsai, Y., Hsu, C., Xie, C., Lin, C.-H., Chen, J.-Y., Li, B., Chen, P., Yu, C.-M., & Huang, C. (2024) · International Conference on Learning Representations · doi:10.48550/arxiv.2310.10012 · Cited by 165
Limitations of transformers on clinical text classification
Gao, S., Alawad, M., Young, M. T., Gounley, J., Schaefferkoetter, N., Yoon, H. J., Wu, X.-C., Durbin, E. B., Doherty, J., Stroup, A., et al. (2021) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2021.3062322 · Cited by 157
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models
West, P., Bhagavatula, C., Hessel, J., Hwang, J., Jiang, L., Le Bras, R., Lu, X., Welleck, S., & Choi, Y. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.341 · Cited by 156
Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models
Zhao, S., Wen, J., Luu, A. N., Zhao, J., & Fu, J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.757 · Cited by 158
A Survey on Green Deep LearningPreprint
Xu, J., Zhou, W., Fu, Z., Zhou, H., & Li, L. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.05193 · Cited by 160
LLM Fine-Tuning: Concepts, Opportunities, and Challenges
Wu, X.-K., Chen, M., Li, W., Wang, R., Lu, L., Liu, J., Hwang, K., Hao, Y., Pan, Y., Meng, Q., et al. (2025) · Big Data and Cognitive Computing · doi:10.3390/bdcc9040087 · Cited by 158
Expel: LLM agents are experiential learners
Zhao, A., Huang, D., Xu, Q., Lin, M., Liu, Y.-J., & Huang, G. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i17.29936 · Cited by 1,451
HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models
Gutiérrez, B. J., Shu, Y., Gu, Y., Yasunaga, M., & Su, Y. (2024) · arXiv · doi:10.48550/arxiv.2405.14831 · Cited by 152
“It is well-accepted, even among skeptical researchers, that the parameters of modern LLMs encode a remarkable amount of world knowledge [2, 10, 17, 21, 24, 31, 47, 62], which can be leveraged by an LLM in flexible and robust ways [64, 65, 74].”
FactKG: Fact Verification via Reasoning on Knowledge Graphs
Kim, J., Park, S., Kwon, Y., Jo, Y., Thorne, J., & Choi, E. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.895 · Cited by 152
Prompt Engineering for Healthcare: Methodologies and Applications
Wang, J., Shi, E., Yu, S., Wu, Z., Ma, C., Dai, H., Yang, Q., Kang, Y., Wu, J., Hu, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2304.14670 · Cited by 150
“In prompt engineering, LLMs are of great importance as they enable the design of appropriate prompts for specific tasks and can generate high-quality results even with limited training data [16].”
Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Era
Wu, X., Zhao, H., Zhu, Y., Shi, Y., Yang, F., Hu, L., Liu, T., Zhaı, X., Yao, W., Li, J., et al. (2026) · ACM Transactions on Knowledge Discovery from Data · doi:10.1145/3816150 · Cited by 150
Feng, S., Park, C. Y., Liu, Y., & Tsvetkov, Y. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.656 · Cited by 149
Locating and Editing Factual Associations in GPT
Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1262 · Cited by 149
Prompt Learning for News Recommendation
Zhang, Z., & Wang, B. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591752 · Cited by 153
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT
Poerner, N., Waltinger, U., & Schütze, H. (2019) · arXiv · doi:10.48550/arxiv.1911.03681 · Cited by 149
Lv, X., Lin, Y., Cao, Y., Hou, L., Li, J., Liu, Z., Li, P., & Zhou, J. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.282 · Cited by 148
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT
Poerner, N., Waltinger, U., & Schütze, H. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.71 · Cited by 146
No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence
Wang, C., Yang, Y., Gao, C., Peng, Y., Zhang, H., & Lyu, M. R. (2022) · Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3540250.3549113 · Cited by 146
Dynamic Prefix-Tuning for Generative Template-based Event Extraction
Liu, X., Huang, H., Shi, G., & Wang, B. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.358 · Cited by 146
Lee, J., & Lee, W.-I. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.416 · Cited by 143
ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness
Prasad, A., Saha, S., Zhou, X., & Bansal, M. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.622 · Cited by 142
minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language ModelsPreprint
Misra, K. (2022) · arXiv (Cornell University) · doi:10.48550/arxiv.2203.13112 · Cited by 143
Metadata
- Authors
- Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A.
- Affiliations
- Meta (United States) (US)
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 2463-2473
- OpenAlex
- W2970476646
- Cited by
- 1,834
- References
- 42
- Citations by year
- 2026: 129 · 2025: 230 · 2024: 300 · 2023: 469 · 2022: 334 · 2021: 246 · 2020: 121 · 2019: 5
- FWCI
- 67.54
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- language models, knowledge base, knowledge representation, knowledge extraction, question answering, commonsense knowledge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.