How Can We Know What Language Models Know?
Open accessThe DOI matches its Crossref record; 51 of the 52 references it cites were found in a registry or an index.
Abstract
Recent work has presented intriguing results examining the knowledge contained in language models (LMs) by having the LM fill in the blanks of prompts such as “ Obama is a __ by profession”. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as “ Obama worked as a __ ” may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ability of Large Language Models to retrieve factual knowledge as explicit knowledge bases [53-56, 56-61] has been stirred discussed, which presents an opportunity to construct more comprehensive knowledge graphs within recommender systems.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [56]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“even experienced prompt designers may fail to manually discover optimal prompts [52].”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Beyond this, we found models trained with Toolformer to often be sensitive to the exact wording of their input when deciding whether or not to call an API; this is perhaps unsurprising given that LMs are known to be very sensitive to the prompt they are provided with in both zeroand few-shot settings (Jiang et al., 2020;Schick and Schütze, 2021a).”
From Toolformer: Language Models Can Teach Themselves to Use Tools (2023) · cites this paper as Jiang et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is some evidence to suggest that formats that occur commonly in the training data will lead to better performance (Jiang et al., 2020).”
From The Prompt Report: A Systematic Survey of Prompt Engineering Techniques (2024) · cites this paper as Jiang et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Jiang et al. [52]'s Mine approach is a mining-based method to automatically find templates given a set of training inputs x and outputs y.”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [52]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“Prompts where the hidden vectors learned through a language modeling objective are then used in downstream language understanding systems (Dai and Le, 2015; Melamud et al., 2016; Peters et al., 2018; Devlin et al., 2019).”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1014 · Cited by 1,926
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482 · cited 5×
“Similar extensions have been applied to pre-trained LMs like BERT, where contextualized representations are enhancedwith entity embeddings (Zhang et al., 2019; Peters et al., 2019; Pörner et al., 2019).”
MethodInfluentialOpen information extraction from the web
Banko, M., Cafarella, M., Soderland, S. G., Broadhead, M., & Etzioni, O. (2007) · International Joint Conference on Artificial Intelligence · Cited by 1,398
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307
(2019) · doi:10.18653/v1/p19-1356 · Cited by 1,291 · cited 1×
“…a broad suite of NLP tasks are used to analyze both syntactic and semantic properties, providing evidence that contextualized representations encode linguistic knowledge in different layers (Hewitt and Manning, 2019; Tenney et al., 2019a; Tenney et al., 2019b; Jawahar et al., 2019; Goldberg, 2019).”
BackgroundSnowball: extracting relations from large plain-text collections
Agichtein, E., & Gravano, L. (2000) · Proceedings of the fifth ACM conference on Digital libraries · doi:10.1145/336597.336644 · Cited by 1,147
Research works printed but not linked: 7
- Andrew M. Dai and Quoc V. Le. 2015. Semisupervised sequence learning. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 3079-3087.
- Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversitypromoting objective function for neural conversation models. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016 , pages 110-119.
- Gábor Melis, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings .
- Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing LSTM language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings .
- 1-6, 2018, Volume 1 (Long Papers) , pages 22272237.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog , 1(8).
- Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? Probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 .
Cited by
Items 301–400 of 1,040
Hierarchical Verbalizer for Few-Shot Hierarchical Text Classification
Ji, K., Lian, Y., Gao, J., & Wang, B. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.164 · Cited by 46
Generating Executable Action Plans with Environmentally-Aware Language Models
Gramopadhye, M., & Szafir, D. (2023) · 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) · doi:10.1109/iros55552.2023.10341989 · Cited by 26
Zero-Shot Generative Model Adaptation via Image-Specific Prompt Learning
Guo, J., Wang, C., Wu, Y., Zhang, E., Wang, K., Xu, X., Song, S., Shi, H., & Huang, G. M. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01106 · Cited by 42
Wysocka, M., Wysocki, O., Delmas, M., Mutel, V., & Freitas, A. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104724 · Cited by 44
Few-Shot Adaptation of Medical Vision-Language Models
Shakeri, F., Huang, Y., Silva-Rodríguez, J., Bahig, H., Tang, A., Dolz, J., & Ayed, I. B. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-72390-2_52 · Cited by 55
Mr. DETR: Instructive Multi-Route Training for Detection Transformers
Zhang, C.-B., Zhong, Y., & Han, K. (2025) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52734.2025.00928 · Cited by 54
A Dual Prompt Learning Framework for Few-Shot Dialogue State Tracking
Yang, Y., Lei, W., Huang, P., Cao, J., Li, J., & Chua, T. (2023) · International World Wide Web Conference (WWW) · doi:10.1145/3543507.3583238 · Cited by 36
Twenty Years Beyond the Turing Test: Moving Beyond the Human Judges Too
Hernández‐Orallo, J. (2020) · Minds and Machines · doi:10.1007/s11023-020-09549-0 · Cited by 31
Boosting Static Resource Leak Detection via LLM-based Resource-Oriented Intention Inference
Wang, C., Liu, J., Peng, X., Liu, Y., & Lou, Y. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00131 · Cited by 52
Constructing Taxonomies from Pretrained Language Models
Chen, C., Lin, K., & Klein, D. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.373 · Cited by 48
Pandia, L., & Ettinger, A. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.119 · Cited by 22
The Art of Prompting: Event Detection based on Type Specific Prompts
Wang, S., Yu, M., & Huang, L. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.111 · Cited by 44
Generative AI for Reengineering Variants into Software Product Lines: An Experience Report
Acher, M., & Martinez, J. (2023) · International Software Product Line Conference (SPLC) · doi:10.1145/3579028.3609016 · Cited by 38
Inverse is Better! Fast and Accurate Prompt for Few-shot Slot Tagging
Hou, Y., Chen, C., Luo, X., Li, B., & Che, W. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.53 · Cited by 32
Zero-Shot Commonsense Question Answering with Cloze Translation and Consistency Optimization
Dou, Z.-Y., & Peng, N. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21301 · Cited by 37
Detecting Edit Failures In Large Language Models: An Improved Specificity Benchmark
Hoelscher-Obermaier, J., Persson, J., Kran, E., Konstas, I., & Barez, F. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.733 · Cited by 19
Text Augmented Open Knowledge Graph Completion via Pre-Trained Language Models
Jiang, P., Agarwal, S., Jin, B., Wang, X., Sun, J., & Han, J. (2023) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2023.findings-acl.709 · Cited by 44
Learning Expressive Prompting With Residuals for Vision Transformers
Das, R., Dukler, Y., Ravichandran, A., & Swaminathan, A. D. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00328 · Cited by 40
Can AI Mitigate Bias in Writing Letters of Recommendation?
Leung, T. I., Sagar, A., Shroff, S., & Henry, T. L. (2023) · JMIR Medical Education · doi:10.2196/51494 · Cited by 28
Uncertainty Estimation and Reduction of Pre-trained Models for Text Regression
Wang, Y., Beck, D., Baldwin, T., & Verspoor, K. (2022) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00483 · Cited by 23
“In addition, these models have been empirically demonstrated to have poor calibration—the predictive probability does not reflect the true correctness likelihood, and they are generally over-confident when they make wrong predictions (Guo et al., 2017; Desai and Durrett, 2020; Jiang et al., 2020).”
BackgroundOn the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies
Zhang, T., & Hashimoto, T. (2021) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2021.naacl-main.404 · Cited by 34
Show, Don’t Tell: Demonstrations Outperform Descriptions for Schema-Guided Task-Oriented Dialogue
Gupta, R., Lee, H., Zhao, J., Cao, Y. J., Rastogi, A., & Wu, Y. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.336 · Cited by 42
Prompt Consistency for Multi-Label Textual Emotion Detection
Zhou, Y., Kang, X., & Ren, F. (2023) · IEEE Transactions on Affective Computing · doi:10.1109/taffc.2023.3254883 · Cited by 27
Briakou, E., Cherry, C., & Foster, G. F. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.524 · Cited by 97
MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt Optimization
Chen, Y., Wen, Z., Fan, G., Chen, Z., Wu, W., Liu, D., Li, Z., Liu, B., & Xiao, Y. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.215 · Cited by 65
Comparing Traditional and LLM-based Search for Image Geolocation
Wazzan, A., MacNeil, S., & Souvenir, R. (2024) · Conference on Human Information Interaction and Retrieval · doi:10.1145/3627508.3638305 · Cited by 43
N.A., A. A. T. (2024) · Future Business Journal · doi:10.1186/s43093-024-00368-8 · Cited by 31
Aftab, W., Apostolou, Z., Bouazoune, K., & Straub, T. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05902-7 · Cited by 24
Thinking Aloud: Dynamic Context Generation Improves Zero-Shot Reasoning Performance of GPT-2Preprint
Betz, G., Richardson, K., & Voigt, C. C. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2103.13033 · Cited by 40
CoPL: Contextual Prompt Learning for Vision-Language Understanding
Goswami, K., Karanam, S., Udhayanan, P., Joseph, K. J., & Srinivasan, B. V. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i16.29766 · Cited by 36
Chen, L., Darko, A., Zhang, F., Chan, A. P. C., & Yang, Q. (2025) · Building and Environment · doi:10.1016/j.buildenv.2025.112891 · Cited by 31
Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal View
Cao, B., Lin, H., Han, X., Liu, F., & Sun, L. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/2022.acl-long.398 · Cited by 20
Automatic Multi-Label Prompting: Simple and Interpretable Few-Shot Classification
Wang, H., Xu, C., & McAuley, J. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.401 · Cited by 36
COPEN: Probing Conceptual Knowledge in Pre-trained Language Models
Peng, H. M., Wang, X., Hu, S., Jin, H., Hou, L., Li, J., Liu, Z., & Liu, Q. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.335 · Cited by 67
PSAN: Prompt Semantic Augmented Network for aspect-based sentiment analysis
He, Y., Huang, X., Zou, S., & Zhang, C. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.121632 · Cited by 24
van Dijk, B., Kouwenhoven, T., Spruit, M. R., & van Duijn, M. J. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.779 · Cited by 54
DSI++: Updating Transformer Memory with New Documents
Mehta, S., Gupta, J., Tay, Y., Dehghani, M., Tran, V., Rao, J., Najork, M., Strubell, E., & Metzler, D. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.510 · Cited by 21
SIGFRID: Unsupervised, Platform-Agnostic Interference Detection in IoT Automation Rules
Cimino, G., & Deufemia, V. (2025) · ACM Transactions on Internet of Things · doi:10.1145/3722231 · Cited by 23
Generative AI and the social functions of educational assessment
Broadfoot, P., & Rockey, J. C. (2025) · Oxford Review of Education · doi:10.1080/03054985.2025.2455549 · Cited by 24
The World of an Octopus: How Reporting Bias Influences a Language Model’s Perception of Color
Paik, C., Aroca-Ouellette, S., Roncone, A., & Kann, K. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.63 · Cited by 47
Can Pretrained English Language Models Benefit Non-English NLP Systems in Low-Resource Scenarios?
Chi, Z., Huang, H., Liu, L., Bai, Y., Gao, X., & Mao, X.-L. (2023) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2023.3267618 · Cited by 25
KnowledgeVIS: Interpreting Language Models by Comparing Fill-in-the-Blank Prompts
Coscia, A. J., & Endert, A. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3346713 · Cited by 26
Tem-adapter: Adapting Image-Text Pretraining for Video Question Answer
Chen, G., Liu, X., Wang, G., Zhang, K., Torr, P. H. S., Zhang, X.-P. S., & Tang, Y. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01282 · Cited by 40
Cross-Modal Concept Learning and Inference for Vision-Language Models
Zhang, Y. H., Zhang, C., Tang, Y., & He, Z. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.127530 · Cited by 32
Li, S., Tan, W., Zhang, C., Li, J., Ren, H., Guo, Y., Jia, J., Liu, Y., Pan, X., Guo, J., et al. (2024) · Expert Systems with Applications · doi:10.1016/j.eswa.2024.125920 · Cited by 24
Automated mapping between SDG indicators and open data: An LLM-augmented knowledge graph approach
Benjira, W., Atigui, F., Bucher, B., Grim-Yefsah, M., & Travers, N. (2025) · Data & Knowledge Engineering · doi:10.1016/j.datak.2024.102405 · Cited by 28
Leave It to Large Language Models! Correction and Planning with Memory Integration
Zhang, Y., Wang, C., Qi, J., & Peng, Y. (2023) · Cyborg and Bionic Systems · doi:10.34133/cbsystems.0087 · Cited by 24
Lee, U., Kim, Y., Lee, S., Park, J., Mun, J., Lee, E., Kim, H., Lim, C., & Yoo, Y. J. (2024) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-024-00448-4 · Cited by 30
Reinforced Prompt Personalization for Recommendation with Large Language Models
Mao, W., Wu, J., Chen, W., Gao, C., Wang, X., & He, X. (2025) · ACM Transactions on Information Systems · doi:10.1145/3716320 · Cited by 20
CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Survey
Li, J., Li, Y., Fu, Y., Liu, J., Liu, Y., Yang, M., & King, I. (2026) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2026.3651700 · Cited by 43
Meng, Z., Liu, F., Shareghi, E., Su, Y., Collins, C., & Collier, N. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.329 · Cited by 39
Framing the News: From Human Perception to Large Language Model Inferences
del Barrio, D. A., & Gática-Pérez, D. (2023) · International Conference on Multimedia Retrieval · doi:10.1145/3591106.3592278 · Cited by 39
KILM: Knowledge Injection into Encoder-Decoder Language Models
Xu, Y., Namazifar, M., Hazarika, D., Padmakumar, A., Liu, Y., & Hakkani-Tür, D. Z. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.275 · Cited by 37
Cheng, K., Song, W., Ma, Z., Zhu, W., Zhu, Z., & Zhang, J. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611987 · Cited by 32
Eliciting knowledge from language models with automatically generated continuous prompts
Chen, Y., Yang, G., Wang, D., & Li, D. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.122327 · Cited by 26
ProRLearn: boosting prompt tuning-based vulnerability detection by reinforcement learning
Ren, Z., Ju, X., Chen, X., & Shen, H. (2024) · Automated Software Engineering · doi:10.1007/s10515-024-00438-9 · Cited by 28
SmartLLMSentry: A Comprehensive LLM Based Smart Contract Vulnerability Detection Framework
Zaazaa, O., & Bakkali, H. E. (2024) · Journal of Metaverse · doi:10.57019/jmv.1489060 · Cited by 30
Mousavi, S. M., Stogaitis, M., Gadh, T., Allen, R. M., Barski, A., Bosch, R. P., Robertson, P., Cho, Y., Thiruverahan, N., & Raj, A. (2024) · Geophysical Journal International · doi:10.1093/gji/ggae436 · Cited by 26
Learning to Generate Task-Specific Adapters from Task Description
Ye, Q., & Ren, X. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-short.82 · Cited by 42
(2022) · doi:10.2139/ssrn.4496380 · Cited by 20
Comprehensive Review of Generative artificial Intelligence: Mechanisms, Models, and Applications
Pahuja, S., Kukreja, S., & Singh, A. (2025) · Procedia Computer Science · doi:10.1016/j.procs.2025.04.628 · Cited by 23
How Can the [MASK] Know? The Sources and Limitations of Knowledge in BERT
Podkorytov, M., Biś, D., & Liu, X. (2021) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn52387.2021.9534299 · Cited by 24
Clues Before Answers: Generation-Enhanced Multiple-Choice QA
Huang, Z., Wu, A., Zhou, J., Gu, Y., Zhao, Y., & Cheng, G. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.239 · Cited by 32
Few-Shot Fine-Grained Entity Typing with Automatic Label Interpretation and Instance Generation
Huang, J., Yu, M., & Han, J. (2022) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3534678.3539443 · Cited by 27
Xu, S., Pang, L., Shen, H., & Cheng, X. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557388 · Cited by 25
Domain Prompt Tuning via Meta Relabeling for Unsupervised Adversarial Adaptation
Jin, X., Lan, C., Zeng, W., & Chen, Z. (2023) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2023.3272742 · Cited by 24
Do PLMs Know and Understand Ontological Knowledge?
Wu, W., Jiang, C., Jiang, Y., Xie, P.-J., & Tu, K. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2023.acl-long.173 · Cited by 40
Understanding Transformer Memorization Recall Through Idioms
Haviv, A., Cohen, I., Gidron, J., Schuster, R., Goldberg, Y., & Geva, M. (2023) · Conference of the European Chapter of the Association for Computational Linguistics · doi:10.18653/v1/2023.eacl-main.19 · Cited by 93
Hierarchical Prompt Learning Using CLIP for Multi-label Classification with Single Positive Labels
Wang, A., Chen, H., Lin, Z., Ding, Z., Liu, P., Bao, Y., Yan, W., & Ding, G. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3611988 · Cited by 29
A prompt construction method for the reverse dictionary task of large-scale language models
Tian, S., Huang, S., Li, R., & Wei, C. (2024) · Engineering Applications of Artificial Intelligence · doi:10.1016/j.engappai.2024.108596 · Cited by 23
Regressor-Segmenter Mutual Prompt Learning for Crowd Counting
Guo, M., Yuan, L., Yan, Z., Chen, B., Wang, Y., & Ye, Q. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02681 · Cited by 38
RAPGen: An Approach for Fixing Code Inefficiencies in Zero-Shot
Garg, S., Moghaddam, R. Z., & Sundaresan, N. (2025) · IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00017 · Cited by 47
Automated Scoring of Question Complexity with Transformer Language Models
Raz, T., Luchini, S. A., Beaty, R. E., & Kenett, Y. N. (2025) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2025.102090 · Cited by 24
Acquiring and modeling abstract commonsense knowledge via conceptualization
He, M., Fang, T., Wang, W., & Song, Y. (2024) · Artificial Intelligence · doi:10.1016/j.artint.2024.104149 · Cited by 41
The Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2023) · arXiv · doi:10.48550/arxiv.2303.07616 · Cited by 18
The Life Cycle of Knowledge in Big Language Models: A Survey
Cao, B., Lin, H., Han, X., & Sun, L. (2024) · Machine Intelligence Research · doi:10.1007/s11633-023-1416-x · Cited by 18
Robustness of Large Language Models Against Adversarial Attacks
Tao, Y., Shen, Y., Zhang, H., Shen, Y., Wang, L., Shi, C., & Du, S. (2024) · International Conference on Artificial Intelligence, Robotics, and Communication (ICAIRC) · doi:10.1109/icairc64177.2024.10900215 · Cited by 40
Benchmarking Commonsense Knowledge Base Population with an Effective Evaluation Dataset
Fang, T., Wang, W., Choi, S.-H., Hao, S., Zhang, H., Song, Y., & He, B. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.705 · Cited by 40
Distilling Relation Embeddings from Pretrained Language Models
Ushio, A., Camacho-Collados, J., & Schockaert, S. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.712 · Cited by 40
Factual Consistency of Multilingual Pretrained Language Models
Fierro, C., & Søgaard, A. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.240 · Cited by 62
Prompting Large Language Models to Power Educational Chatbots
Farah, J. C., Ingram, S., Spaenlehauer, B., Lasne, F. K.-L., & Gillet, D. (2023) · Lecture notes in computer science · doi:10.1007/978-981-99-8385-8_14 · Cited by 23
A Causal View of Entity Bias in (Large) Language Models
Wang, F., Mo, W., Wang, Y., Zhou, W., & Chen, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.1013 · Cited by 77
Interpretable Entity Representations through Large-Scale Typing
Onoe, Y., & Durrett, G. (2020) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2020.findings-emnlp.54 · Cited by 26
Good Night at 4 pm?! Time Expressions in Different Cultures
Shwartz, V. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.224 · Cited by 21
Adaptive Prompt Learning-Based Few-Shot Sentiment Analysis
Zhang, P., Chai, T., & Xu, Y. (2023) · Neural Processing Letters · doi:10.1007/s11063-023-11259-4 · Cited by 19
Lu, J., Li, J., Wallace, B., He, Y., & Pergola, G. (2023) · Findings of the Association for Computational Linguistics: EACL · doi:10.18653/v1/2023.findings-eacl.80 · Cited by 32
Active Retrieval Augmented Generation
Jiang, Z., Xu, F. F., Gao, L., Sun, Z., Liu, Q., Dwivedi-Yu, J., Yang, Y., Callan, J., & Neubig, G. (2023) · arXiv · doi:10.48550/arxiv.2305.06983 · Cited by 18
Self-citationPrototypical Prompting for Text-to-image Person Re-identification
Yan, S., Liu, J., Dong, N., Zhang, L., & Tang, J. (2024) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3664647.3681165 · Cited by 40
Multi-modal soft prompt-tuning for Chinese Clickbait Detection
Wang, Y., Zhu, Y. Q., Li, Y., Wei, L., Yuan, Y., & Qiang, J. (2024) · Neurocomputing · doi:10.1016/j.neucom.2024.128829 · Cited by 19
DREAM: Improving Situational QA by First Elaborating the Situation
Gu, Y., Dalvi, B. B., & Clark, P. A. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.82 · Cited by 33
AugPrompt: Knowledgeable augmented-trigger prompt for few-shot event classification
Song, C., Cai, F., Zheng, J., Zhao, X. H., & Shao, T. (2022) · Information Processing & Management · doi:10.1016/j.ipm.2022.103153 · Cited by 19
Yu, S., Dou, Z., & Wang, S. (2023) · Tsinghua Science & Technology · doi:10.26599/tst.2022.9010044 · Cited by 18
Prompt-Guided Level Generation
Sudhakaran, S., González-Duque, M., Glanois, C., Freiberger, M., Najarro, E., & Risi, S. (2023) · Genetic and Evolutionary Computation Conference Companion (GECCO Companion) · doi:10.1145/3583133.3590656 · Cited by 17
LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-Based Code Completion
Wang, C., Huang, K., Zhang, J., Feng, Y., Zhang, L., Liu, Y., & Peng, X. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00245 · Cited by 82
DocTER: Evaluating document-based knowledge editing
Wu, S., Wang, A., Peng, M., Lin, Y., Li, W., Sun, M., & Su, J. (2025) · Information Processing & Management · doi:10.1016/j.ipm.2025.104299 · Cited by 35
Tan, H., Shao, W., Wu, H., Yang, K., & Song, L. (2022) · Findings of the Association for Computational Linguistics: ACL (Findings of ACL) · doi:10.18653/v1/2022.findings-acl.22 · Cited by 28
Active Prompt Learning in Vision Language Models
Bang, J., Ahn, S., & Lee, J.-G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02550 · Cited by 48
A Hitchhiker's Guide to Good Prompting Practices for Large Language Models in Radiology
Tripathi, S., Alkhulaifat, D., Lyo, S. K., Sukumaran, R., Li, B., Acharya, V., McBeth, R., & Cook, T. S. (2025) · Journal of the American College of Radiology · doi:10.1016/j.jacr.2025.02.051 · Cited by 22
LMdiff: A Visual Diff Tool to Compare Language Models
Strobelt, H., Hoover, B., Satyanaryan, A., & Gehrmann, S. (2021) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2021.emnlp-demo.12 · Cited by 26
Metadata
- Authors
- Jiang, Z., Xu, F. F., Araki, J., & Neubig, G.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 423-438
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2991382858
- Cited by
- 1,002
- References
- 52
- Citations by year
- 2026: 76 · 2025: 169 · 2024: 224 · 2023: 285 · 2022: 144 · 2021: 78 · 2020: 25
- FWCI
- 52.35
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1815287)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- prompt generation, ensemble methods
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.