Language Models as Knowledge Bases?
Open accessThe DOI matches its Crossref record; 43 of the 43 references it cites were found in a registry or an index.
Abstract
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Tracing back to the work [53], large language models have shown their impressive power in storing factual information, such as entities and common sense, and then commonsense knowledge can be reliably transferred to downtown tasks.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [53]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“While pure generative LLMs are often capable of answering basic factual questions (Petroni et al., 2019), simply by reciting information memorized from their training sets and stored in the model parameters”
From Deciphering the Interplay of Parametric and Non-parametric Memory in Retrieval-augmented Language Models (2024) · cites this paper as Petroni et al., 2019
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It is well-accepted, even among skeptical researchers, that the parameters of modern LLMs encode a remarkable amount of world knowledge [2, 12, 23, 28, 31, 39, 62, 79], which can be leveraged by an LLM in flexible and robust ways [81, 83, 93].”
From HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models (2024) · cites this paper as [62]
The cited source says what the claim attributes to it.
In the citing paper
“Pre-training of models on a large corpus is known to result in the model memorizing knowledge in its parameters [169, 201, 221].”
From Survey of Hallucination in Natural Language Generation (2023) · cites this paper as [201]
The cited source says what the claim attributes to it.
In the citing paper
“The aforementioned example shows that AI long-term memory can be implicitly stored in the parameters of AI models [248, 249].”
From Human-inspired Perspectives: A Survey on AI Long-term Memory (2024) · cites this paper as [248]
7 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“BERT (base) (Devlin et al., 2018a) Transformer 110M Wikipedia (en) & BookCorpus 3.”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
SQuAD: 100,000+ Questions for Machine Comprehension of Text
Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016) · Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d16-1264 · Cited by 6,208 · cited 1×
“SQuAD (Rajpurkar et al., 2016) is a popular question answering dataset.”
Background(2013) · Cited by 5,726
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. (2018) · Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP · doi:10.18653/v1/w18-5446 · Cited by 4,222
Transformer-XL: Attentive language models beyond a fixed-length context
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q., & Salakhutdinov, R. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1285 · Cited by 3,169 · cited 3×
“…2017), recurrent neural networks (Zaremba et al., 2014; Merity et al., 2016; Melis et al., 2017) or self-attention mechanisms (Radford et al., 2018; Dai et al., 2019; Radford et al., 2019). fairseq-fconv: Instead of commonly used recurrent neural networks, Dauphin et al. (2017) use multiple layers…”
BackgroundMethodInfluential
Research works printed but not linked: 3
- [1]1. Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefenstette. 2019. Learning to understand goal specifications by 7 https://github.com/allenai/allennlp 8 https://github.com/pytorch/fairseq 9 https://github.com/huggingface/ pytorch-transformers modelling reward. In International Conference on Learning Representations (ICLR).
- [12]12. Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh Interna- tional Conference on Language Resources and Eval- uation (LREC-2018).
- [24]24. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word rep- resentations. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, NAACL-HLT 2018, New Or- leans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 2227-2237.
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
“Other examples of hybrid symbolic and sub-symbolic methods where a knowledge-base tool or graph-perspective enhances the neural (e.g., language [308] ) model are in [309,310] .”
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
“Pre-training of models on a large corpus is known to result in the model memorizing knowledge in its parameters [121, 142, 158].”
BackgroundSurvey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y., Chen, D., Dai, W., et al. (2022) · arXiv · doi:10.48550/arxiv.2202.03629 · Cited by 4,981
Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., et al. (2025) · ACM Transactions on Information Systems · doi:10.1145/3703155 · Cited by 4,688
“While significant efforts have been undertaken in exploring their knowledge storage (Geva et al., 2021; Meng et al., 2022)and probing (Petroni et al., 2019; Zhong et al., 2021; Yu et al., 2023c), the exact mechanism by which LLMs capture the factual knowledge remains elusive.”
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361
“For example, the seminal LAMA dataset [100] provides manually created cloze templates to probe knowledge in LMs.”
BackgroundMethodInfluentialLearning to Prompt for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · International Journal of Computer Vision · doi:10.1007/s11263-022-01653-1 · Cited by 3,304
“Knowledge probing for large pre-trained language models, formally defined by Petroni et al. (2019) as “fill-in-the-blank” cloze tests, has recently sparked interest in prompt learning research in NLP (Shin et al., 2020; Jiang et al., 2020; Li and Liang, 2021; Zhong et al., 2021; Lester et al., 2021;…”
BackgroundRetrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W.-T., Rocktäschel, T., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.11401 · Cited by 3,101
“Pre-trained neural language models have been shown to learn a substantial amount of in-depth knowledge from data [41].”
BackgroundSelf-citationA Survey on Knowledge Graphs: Representation, Acquisition, and Applications
Ji, S., Pan, S., Cambria, E., Marttinen, P., & Yu, P. S. (2022) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2021.3070843 · Cited by 2,947
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
A Survey of Large Language Models
Zhao, W. X., Zhou, K., Li, J., Tang, T., Dong, Z., Hou, Y., Zhang, B., Min, Y., Zhang, J., Liu, P., et al. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-026-60308-3 · Cited by 2,093
“Knowledge Completion WikiFact [307], FB15k-237 [308], Freebase [309], WN18RR [310], WordNet [311], LAMA [312], YAGO3-10 [313], YAGO [314]”
Conditional Prompt Learning for Vision-Language Models
Zhou, K., Yang, J., Loy, C. C., & Liu, Z. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01631 · Cited by 1,796
Steering Language Models With Activation EngineeringPreprint
Turner, A., Thiergart, L., Leech, G., Udell, D., Vazquez, J. J., Mini, U., & MacDiarmid, M. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.10248 · Cited by 1,630
Pre-trained models for natural language processing: A survey
Qiu, X., Sun, T., Xu, Y., Shao, Y., Dai, N., & Huang, X. (2020) · Science China Technological Sciences · doi:10.1007/s11431-020-1647-3 · Cited by 1,591
“[138] constructed LAMA (Language Model Analysis) task by manually creating single-token cloze statements (queries) from several knowledge sources.”
BackgroundA primer in BERTology: What we know about how BERT works
Rogers, A., Kovaleva, O., & Rumshisky, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00349 · Cited by 1,552
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey
Min, B., Ross, H., Sulem, E., Veyseh, A. P. B., Nguyen, T. H., Sainz, O., Agirre, E., Heintz, I., & Roth, D. (2024) · ACM Computing Surveys · doi:10.1145/3605943 · Cited by 1,387
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380
Unifying Large Language Models and Knowledge Graphs: A Roadmap
Pan, S., Luo, L., Wang, Y., Chen, C., Wang, J., & Wu, X. (2024) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2024.3352100 · Cited by 1,376
“To improve the interpretability of LLMs, researchers also utilize KGs to interpret the facts [14] and the reasoning process of LLMs [38].”
BackgroundMethodInfluentialExploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference
Schick, T., & Schütze, H. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.20 · Cited by 1,305
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., & Singh, S. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.346 · Cited by 1,279
Parameter-efficient fine-tuning of large-scale pre-trained language models
Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., et al. (2023) · Nature Machine Intelligence · doi:10.1038/s42256-023-00626-4 · Cited by 1,191
Toolformer: Language Models Can Teach Themselves to Use Tools
Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N., & Scialom, T. (2023) · arXiv · doi:10.48550/arxiv.2302.04761 · Cited by 1,150
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
Izacard, G., & Grave, E. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.74 · Cited by 1,109
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data
Bender, E. M., & Koller, A. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.463 · Cited by 1,082
“( Petroni et al., 2019) If the highlighted terms are meant to describe human-analogous understanding, comprehension, or recall of factual knowledge, then these are gross overclaims.”
BackgroundHow Can We Know What Language Models Know?
Jiang, Z., Xu, F. F., Araki, J., & Neubig, G. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00324 · Cited by 1,002
Pre-trained models: Past, present and future
Han, X., Zhang, Z., Ding, N., Gu, Y., Liu, X., Huo, Y., Qiu, J., Yao, Y., Zhang, A., Zhang, L., et al. (2021) · AI Open · doi:10.1016/j.aiopen.2021.08.002 · Cited by 994
Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2021) · arXiv · doi:10.48550/arxiv.2103.10385 · Cited by 908
Structured information extraction from scientific text with large language models
Dagdelen, J., Dunn, A., Lee, S., Walker, N., Rosen, A. S., Ceder, G., Persson, K. A., & Jain, A. (2024) · Nature Communications · doi:10.1038/s41467-024-45563-x · Cited by 771
Retrieval-Augmented Generation for Large Language Models: A Survey
Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, M., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2312.10997 · Cited by 876
“The recursive nature of the process allows for continuous learning and adaptation to the user’s requirements, often resulting in improved satisfaction with the search outcomes.”
BackgroundIt’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.185 · Cited by 747
Extracting Training Data from Large Language Models
Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2020) · arXiv · doi:10.48550/arxiv.2012.07805 · Cited by 727
“Indeed, there is an entire area of research that analyzes neural networks as repositories of (memorized) knowledge [47, 55].”
BackgroundA Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities
Song, Y., Wang, T., Cai, P., Mondal, S. K., & Sahoo, J. P. (2023) · ACM Computing Surveys · doi:10.1145/3582688 · Cited by 721
“[87] EMNLP’19 BERT Manual Prompt Cloze-style Use cloze-style prompt to learn prior knowledge Cui et al.”
BackgroundJin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · Applied Sciences · doi:10.3390/app11146421 · Cited by 718
Jin, D., Pan, E., Oufattole, N., Weng, W.-H., Fang, H., & Szolovits, P. (2021) · doi:10.20944/preprints202105.0498.v1 · Cited by 718
Explainability for Large Language Models: A Survey
Zhao, H., Chen, H., Yang, F., Liu, N., Deng, H., Cai, H., Wang, S., Yin, D., & Du, M. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3639372 · Cited by 681
“Overall, these attribution scores that blend attention and gradients generally perform better than using either alone, as they fuse more information that helps to highlight important features and understand networks.”
BackgroundKEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation
Wang, X., Gao, T., Zhu, Z., Zhang, Z., Liu, Z., Li, J., & Tang, J. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00360 · Cited by 676
How Much Knowledge Can You Pack into the Parameters of a Language Model?
Roberts, A., Raffel, C., & Shazeer, N. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.437 · Cited by 638
Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2021) · arXiv · doi:10.48550/arxiv.2104.08786 · Cited by 615
“…et al., 2019; Peters et al., 2018; Raffel et al., 2020; Liu et al., 2019; Yang et al., 2019; Radford et al., 2019) have shown remarkable performance when conditioned with an appropriate textual context (Petroni et al., 2019, 2020; Jiang et al., 2020; Shin et al., 2020; Davison et al., 2019).”
BackgroundSelf-citationFantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
Lu, Y., Bartolo, M., Moore, A., Riedel, S., & Stenetorp, P. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.556 · Cited by 610
“…et al., 2019; Peters et al., 2018; Raffel et al., 2020; Liu et al., 2019; Yang et al., 2019; Radford et al., 2019) have shown remarkable performance when conditioned with an appropriate textual context (Petroni et al., 2019, 2020; Jiang et al., 2020; Shin et al., 2020; Davison et al., 2019).”
BackgroundSelf-citationGalactica: A Large Language Model for Science
Taylor, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., & Stojnic, R. (2022) · arXiv · doi:10.48550/arxiv.2211.09085 · Cited by 605
Liu, X., Zheng, Y., Du, Z., Ding, M., Qian, Y., Yang, Z., & Tang, J. (2024) · AI Open · doi:10.1016/j.aiopen.2023.08.012 · Cited by 596
Retrieval Augmentation Reduces Hallucination in Conversation
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.320 · Cited by 575
Knowledge Conflicts for LLMs: A Survey
Xu, R., Qi, Z., Guo, Z., Wang, C., Wang, H., Zhang, Y., & Xu, W. (2024) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2024.emnlp-main.486 · Cited by 556
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
Yasunaga, M., Ren, H., Bosselut, A., Liang, P., & Leskovec, J. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.45 · Cited by 522
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352
“For example, the seminal LAMA dataset [100] provides manually created cloze templates to probe knowledge in LMs.”
BackgroundMethodInfluentialA Survey on Automated Fact-Checking
Guo, Z., Schlichtkrull, M., & Vlachos, A. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00454 · Cited by 500
Active Retrieval Augmented Generation
Jiang, Z., Xu, F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., & Neubig, G. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.495 · Cited by 497
Retrieval Augmentation Reduces Hallucination in ConversationPreprint
Shuster, K., Poff, S., Chen, M., Kiela, D., & Weston, J. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2104.07567 · Cited by 495
PTR: Prompt Tuning with Rules for Text Classification
Han, X., Zhao, W., Ding, N., Liu, Z., & Sun, M. (2021) · arXiv · doi:10.48550/arxiv.2105.11259 · Cited by 478
“ries of research work on knowledge probing (Trinh and Le, 2018; Petroni et al., 2019; Davison et al., 2019), language prompts have been proven to effectively stimulate knowledge from PLMs.”
BackgroundEvaluating Large Language Models: A Comprehensive SurveyPreprint
Guo, Z., Jin, R., Chuang, L. I. U., Huang, Y., Shi, D., Supryadi, Yu, L., Liu, Y., Li, J., Xiong, B., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2310.19736 · Cited by 478
PTR: Prompt Tuning with Rules for Text Classification
Han, X., Zhao, W., Ding, N., Liu, Z., & Sun, M. (2022) · AI Open · doi:10.1016/j.aiopen.2022.11.003 · Cited by 467
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
Wang, R., Tang, D., Duan, N., Wei, Z., Huang, X., Ji, J., Cao, G., Jiang, D., & Zhou, M. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.121 · Cited by 456
Pretrained Transformers for Text Ranking: BERT and Beyond
Lin, J., Nogueira, R. F., & Yates, A. (2021) · Synthesis lectures on human language technologies · doi:10.2200/s01123ed1v01y202108hlt053 · Cited by 452
Kakwani, D., Kunchukuttan, A., Golla, S., N.C., G., Bhattacharyya, A., Khapra, M. M., & Kumar, P. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.445 · Cited by 446
Security and Privacy Challenges of Large Language Models: A Survey
Das, B. C., Amini, M. H., & Wu, Y. (2025) · ACM Computing Surveys · doi:10.1145/3712001 · Cited by 445
“Finally, Section 9 concludes the paper.”
Self-citationContinual Learning of Large Language Models: A Comprehensive Survey
Shi, H., Xu, Z., Wang, H., Qin, W., Wang, W., Wang, Y., Wang, Z., Ebrahimi, S., & Wang, H. (2025) · ACM Computing Surveys · doi:10.1145/3735633 · Cited by 443
Yang, Y., Panagopoulou, A., Zhou, S., Jin, D., Callison-Burch, C., & Yatskar, M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01839 · Cited by 435
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language ProcessingPreprint
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2108.05542 · Cited by 430
Large language models in healthcare and medical domain: A review
Nazi, Z. A., & Peng, W. (2024) · Informatics · doi:10.3390/informatics11030057 · Cited by 415
A Comprehensive Study of Knowledge Editing for Large Language ModelsPreprint
Zhang, N., Yao, Y., Tian, B., Wang, P., Deng, S., Wang, M., Xi, Z., Mao, S., Zhang, J., Ni, Y., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.01286 · Cited by 411
A survey on LLM-based multi-agent systems: Workflow, infrastructure, and challenges
Li, X., Wang, S., Zeng, S., Wu, Y., & Yang, Y. (2024) · Vicinagearth · doi:10.1007/s44336-024-00009-2 · Cited by 404
“For example, P-Tuning [288] stabilizes the training process by combining continuous prompt embeddings with discrete prompts and has achieved significant performance improvements in natural language understanding tasks such as LAMA [289] and SuperGLUE [290].”
Self-citationSurvey on Factuality in Large Language Models
Wang, C., Liu, X., Yue, Y., Guo, Q., Hu, X., Tang, X., Zhang, T., Cheng, J., Yao, Y., Hu, X., et al. (2025) · ACM Computing Surveys · doi:10.1145/3742420 · Cited by 399
Large language models (LLMs): survey, technical frameworks, and future challenges
Kumar, P. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10888-y · Cited by 395
Pre-Trained Language Models and Their Applications
Wang, H., Li, J., Wu, H., Hovy, E., & Sun, Y. (2023) · Engineering · doi:10.1016/j.eng.2022.04.024 · Cited by 385
A Comprehensive Survey on Automatic Knowledge Graph Construction
Zhong, L., Wu, J., Li, Q., Peng, H., & Wu, X. (2024) · ACM Computing Surveys · doi:10.1145/3618295 · Cited by 382
Learning To Retrieve Prompts for In-Context Learning
Rubin, O., Herzig, J., & Berant, J. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.191 · Cited by 374
When large language models meet personalization: perspectives of challenges and opportunities
Chen, J., Liu, Z., Huang, X., Wu, C., Liu, Q., Jiang, G., Pu, Y., Lei, Y., Chen, X., Wang, X., et al. (2024) · World Wide Web · doi:10.1007/s11280-024-01276-1 · Cited by 367
“The inherent nature of large language models can introduce ambiguity or inaccurate provenance [66].”
Dissociating language and thought in large language models
Mahowald, K., Ivanova, A. A., Blank, I. A., Kanwisher, N., Tenenbaum, J. B., & Fedorenko, E. (2024) · Trends in Cognitive Sciences · doi:10.1016/j.tics.2024.01.011 · Cited by 345
Template-Based Named Entity Recognition Using BART
Cui, L., Wu, Y., Liu, J., Yang, S., & Zhang, Y. (2021) · arXiv · doi:10.48550/arxiv.2106.01760 · Cited by 342
“Petroni et al. (2019) extract relation between entities from BERT by constructing cloze-style templates.”
MethodSelf-citationZeroGen: Efficient Zero-shot Learning via Dataset Generation
Ye, J., Gao, J., Li, Q., Xu, H., Feng, J., Wu, Z., Yu, C., & Kong, L. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.801 · Cited by 341
Mallen, A., Asai, A., Zhong, V., Das, R., Khashabi, D., & Hajishirzi, H. (2023) · Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2023.acl-long.546 · Cited by 340
“Petroni et al. (2019) demonstrate that large pre-trained LMs such as BERT (Devlin et al., 2019) memorize the significant amount of world knowledge in their parameters ( parametric knowledge ), and Roberts et al. (2020) show that fine-tuned T5 without any reference documents (closed-book QA) can…”
BackgroundPPT: Pre-trained Prompt Tuning for Few-shot Learning
Gu, Y., Han, X., Liu, Z., & Huang, M. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.576 · Cited by 335
LLMs4OL: Large Language Models for Ontology Learning
Giglou, H. B., D’Souza, J., & Auer, S. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47240-4_22 · Cited by 328
LinkBERT: Pretraining Language Models with Document Links
Yasunaga, M., Leskovec, J., & Liang, P. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.551 · Cited by 327
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA
Yang, Z., Gan, Z., Wang, J., Hu, X., Lu, Y., Liu, Z., & Wang, L. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i3.20215 · Cited by 326
Sequence-to-Sequence Knowledge Graph Completion and Question Answering
Saxena, A., Kochsiek, A., & Gemulla, R. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.201 · Cited by 318
Template-Based Named Entity Recognition Using BART
Cui, L., Wu, Y., Liu, J., Yang, S., & Zhang, Y. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.161 · Cited by 314
Self-citationBLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages
Myung, J., Lee, N., Zhou, Y., Jin, J., Putri, R. A., Antypas, D., Borkakoty, H., Kim, E., Perez-Almendros, C., Ayele, A. A., et al. (2024) · neural information processing systems · doi:10.52202/079017-2483 · Cited by 307
LLMs for knowledge graph construction and reasoning: recent capabilities and future opportunities
Zhu, Y., Wang, X., Chen, J., Qiao, S., Ou, Y., Yao, Y., Deng, S., Chen, H., & Zhang, N. (2024) · World Wide Web · doi:10.1007/s11280-024-01297-w · Cited by 294
TRAK: Attributing Model Behavior at ScalePreprint
Park, S., Georgiev, K., Ilyas, A., Leclerc, G., & Mądry, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2303.14186 · Cited by 293
A Practical Review of Mechanistic Interpretability for Transformer-Based Language ModelsPreprint
Rai, D., Zhou, Y., Feng, S., Saparov, A., & Yao, Z. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.02646 · Cited by 296
Large pre-trained language models contain human-like biases of what is right and wrong to do
Schramowski, P., Turan, C., Andersen, N., Rothkopf, C. A., & Kersting, K. (2022) · Nature Machine Intelligence · doi:10.1038/s42256-022-00458-8 · Cited by 291
“, commonsense knowledge by accessing the language model’s memory [13].”
BackgroundMethodResultInfluentialNull It Out: Guarding Protected Attributes by Iterative Nullspace Projection
Ravfogel, S., Elazar, Y., Gonen, H., Twiton, M., & Goldberg, Y. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.647 · Cited by 287
Inner Monologue: Embodied Reasoning through Planning with Language Models
Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2207.05608 · Cited by 287
“One of the remarkable observations in recent machine learning research is that large language models (LLMs) can not only generate fluent textual descriptions, but also appear to have rich internalized knowledge about the world [3, 4, 5, 6, 7].”
BackgroundSymbolic Chain-of-Thought Distillation: Small Models Can Also “Think” Step-by-Step
Li, L. H., Hessel, J., Yu, Y., Ren, X., Chang, K.-W., & Choi, Y. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.150 · Cited by 287
(Comet-) Atomic 2020: On Symbolic and Neural Commonsense Knowledge Graphs
Hwang, J. D., Bhagavatula, C., Le Bras, R., Da, J., Sakaguchi, K., Bosselut, A., & Choi, Y. (2021) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v35i7.16792 · Cited by 286
Information Retrieval: Recent Advances and Beyond
Hambarde, K., & Proença, H. P. M. C. (2023) · IEEE Access · doi:10.1109/access.2023.3295776 · Cited by 284
Tan, Y., Min, D., Li, Y., Li, W., Hu, N., Chen, Y., & Qi, G. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-47240-4_19 · Cited by 282
Cold-start active learning through self-supervised language modeling
Yuan, M., Lin, H.-T., & Boyd-Graber, J. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.637 · Cited by 84
DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models
Ge, Z., Yang, B., Tang, J., Zhou, H.-Y., & Yang, S. (2023) · neural information processing systems · doi:10.52202/075280-0228 · Cited by 279
AMMU: A survey of transformer-based biomedical pretrained language models
Kalyan, K. S., Rajasekharan, A., & Sangeetha, S. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103982 · Cited by 278
mGPT: Few-Shot Learners Go Multilingual
Shliazhko, O., Fenogenova, A., Tikhonova, M., Kozlova, A., Mikhailov, V., & Shavrina, T. (2024) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00633 · Cited by 277
Factual Probing Is [MASK]: Learning vs. Learning to Recall
Zhong, Z., Friedman, D., & Chen, D. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.398 · Cited by 274
Retrieval-augmented generation for natural language processing: a survey
Wu, S., Xiong, Y., Cui, Y., Wu, H., Chen, C., Yuan, Y., Huang, L., Liu, X., Kuo, T., Guan, N., et al. (2026) · Artificial Intelligence Review · doi:10.1007/s10462-026-11605-7 · Cited by 273
oLMpics-On What Language Model Pre-training Captures
Talmor, A., Elazar, Y., Goldberg, Y., & Berant, J. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00342 · Cited by 271
Bulathwela, S., Pérez-Ortiz, M., Holloway, C., Cukurova, M., & Shawe-Taylor, J. (2024) · Sustainability · doi:10.3390/su16020781 · Cited by 269
Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification
Hu, S., Ding, N., Wang, H., Liu, Z., Wang, J., Li, J., Wu, W., & Sun, M. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.158 · Cited by 265
SimKGC: Simple Contrastive Knowledge Graph Completion with Pre-trained Language Models
Wang, L., Zhao, W., Wei, Z., & Liu, J. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.295 · Cited by 264
A Survey of Knowledge-enhanced Text Generation
Yu, W., Zhu, C., Li, Z., Hu, Z., Wang, Q., Ji, H., & Jiang, M. (2022) · ACM Computing Surveys · doi:10.1145/3512467 · Cited by 257
Metadata
- Authors
- Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A.
- Affiliations
- Meta (United States) (US)
- Venue
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)
- Published
- 2019
- Pages
- 2463-2473
- OpenAlex
- W2970476646
- Cited by
- 1,834
- References
- 42
- Citations by year
- 2026: 129 · 2025: 230 · 2024: 300 · 2023: 469 · 2022: 334 · 2021: 246 · 2020: 121 · 2019: 5
- FWCI
- 67.54
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Conference on Empirical Methods in Natural Language Processing (EMNLP) (publishedVersion, cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- language models, knowledge base, knowledge representation, knowledge extraction, question answering, commonsense knowledge
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.