How Can We Know What Language Models Know?
Open accessThe DOI matches its Crossref record; 51 of the 52 references it cites were found in a registry or an index.
Abstract
Recent work has presented intriguing results examining the knowledge contained in language models (LMs) by having the LM fill in the blanks of prompts such as “ Obama is a __ by profession”. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as “ Obama worked as a __ ” may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA .
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ability of Large Language Models to retrieve factual knowledge as explicit knowledge bases [53-56, 56-61] has been stirred discussed, which presents an opportunity to construct more comprehensive knowledge graphs within recommender systems.”
From When large language models meet personalization: perspectives of challenges and opportunities (2024) · cites this paper as [56]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“even experienced prompt designers may fail to manually discover optimal prompts [52].”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Beyond this, we found models trained with Toolformer to often be sensitive to the exact wording of their input when deciding whether or not to call an API; this is perhaps unsurprising given that LMs are known to be very sensitive to the prompt they are provided with in both zeroand few-shot settings (Jiang et al., 2020;Schick and Schütze, 2021a).”
From Toolformer: Language Models Can Teach Themselves to Use Tools (2023) · cites this paper as Jiang et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is some evidence to suggest that formats that occur commonly in the training data will lead to better performance (Jiang et al., 2020).”
From The Prompt Report: A Systematic Survey of Prompt Engineering Techniques (2024) · cites this paper as Jiang et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Jiang et al. [52]'s Mine approach is a mining-based method to automatically find templates given a set of training inputs x and outputs y.”
From Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2023) · cites this paper as [52]
5 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325 · cited 7×
“Prompts where the hidden vectors learned through a language modeling objective are then used in downstream language understanding systems (Dai and Le, 2015; Melamud et al., 2016; Peters et al., 2018; Devlin et al., 2019).”
BackgroundMethodInfluentialDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
A Diversity-Promoting Objective Function for Neural Conversation Models
Li, J., Galley, M., Brockett, C., Gao, J., & Dolan, B. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1014 · Cited by 1,926
Language Models as Knowledge Bases?
Petroni, F., Rocktäschel, T., Riedel, S., Lewis, P., Bakhtin, A., Wu, Y., & Miller, A. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1250 · Cited by 1,834
ERNIE: Enhanced Language Representation with Informative Entities
Zhang, Z., Han, X., Liu, Z., Jiang, X., Sun, M., & Liu, Q. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1139 · Cited by 1,482 · cited 5×
“Similar extensions have been applied to pre-trained LMs like BERT, where contextualized representations are enhancedwith entity embeddings (Zhang et al., 2019; Peters et al., 2019; Pörner et al., 2019).”
MethodInfluentialOpen information extraction from the web
Banko, M., Cafarella, M., Soderland, S. G., Broadhead, M., & Etzioni, O. (2007) · International Joint Conference on Artificial Intelligence · Cited by 1,398
BERT Rediscovers the Classical NLP Pipeline
Tenney, I., Das, D., & Pavlick, E. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/p19-1452 · Cited by 1,307
(2019) · doi:10.18653/v1/p19-1356 · Cited by 1,291 · cited 1×
“…a broad suite of NLP tasks are used to analyze both syntactic and semantic properties, providing evidence that contextualized representations encode linguistic knowledge in different layers (Hewitt and Manning, 2019; Tenney et al., 2019a; Tenney et al., 2019b; Jawahar et al., 2019; Goldberg, 2019).”
BackgroundSnowball: extracting relations from large plain-text collections
Agichtein, E., & Gravano, L. (2000) · Proceedings of the fifth ACM conference on Digital libraries · doi:10.1145/336597.336644 · Cited by 1,147
Research works printed but not linked: 7
- Andrew M. Dai and Quoc V. Le. 2015. Semisupervised sequence learning. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada , pages 3079-3087.
- Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversitypromoting objective function for neural conversation models. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016 , pages 110-119.
- Gábor Melis, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings .
- Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing LSTM language models. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings .
- 1-6, 2018, Volume 1 (Long Papers) , pages 22272237.
- Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog , 1(8).
- Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? Probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 .
Cited by
Items 201–300 of 1,040
Exploring low-resource medical image classification with weakly supervised prompt learning
Zheng, F., Cao, J., Yu, W., Chen, Z., Xiao, N., & Lu, Y. (2024) · Pattern Recognition · doi:10.1016/j.patcog.2024.110250 · Cited by 63
Sclar, M., Choi, Y., Tsvetkov, Y., & Suhr, A. (2023) · arXiv · doi:10.48550/arxiv.2310.11324 · Cited by 52
Reordering Examples Helps during Priming-based Few-Shot Learning
Kumar, S., & Talukdar, P. (2021) · Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · doi:10.18653/v1/2021.findings-acl.395 · Cited by 51
Mobile Foundation Model as Firmware
Yuan, J., Yang, C., Cai, D., Wang, S., Yuan, X., Zhang, Z., Li, X., Zhang, D., Mei, H., Jia, X., et al. (2024) · ACM Annual International Conference on Mobile Computing and Networking (MobiCom) · doi:10.1145/3636534.3649361 · Cited by 96
What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use
Ma, Q., Peng, W., Yang, C., Shen, H., Koedinger, K. R., & Wu, T. (2025) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3731756 · Cited by 113
A Survey of Reasoning with Foundation ModelsPreprint
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Liu, J., Xu, J., Ding, M., Li, H., Geng, M., et al. (2023) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/ac4sp · Cited by 158
He, X., Zannettou, S., Shen, Y., & Zhang, Y. (2024) · 2024 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp54263.2024.00061 · Cited by 35
WorldSmith: Iterative and Expressive Prompting for World Building with a Generative AI
Dang, H., Brudy, F., Fitzmaurice, G., & Anderson, F. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606772 · Cited by 47
Explaining Question Answering Models through Text GenerationPreprint
Latcinnik, V., & Berant, J. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.05569 · Cited by 70
Natural Language to Code Translation with Execution
Shi, F., Fried, D., Ghazvininejad, M., Zettlemoyer, L., & Wang, S. I. (2022) · arXiv · doi:10.48550/arxiv.2204.11454 · Cited by 49
Modifying Memories in Transformer Models
Zhu, C., Rawat, A. S., Zaheer, M., Bhojanapalli, S., Li, D., Yu, F., & Kumar, S. (2020) · arXiv · doi:10.48550/arxiv.2012.00363 · Cited by 48
Learning Approximate Execution Semantics From Traces for Binary Function Similarity
Pei, K., Xuan, Z., Yang, J., Jana, S., & Ray, B. (2022) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2022.3231621 · Cited by 68
It’s not Greek to mBERT: Inducing Word-Level Translations from Multilingual BERT
Gonen, H., Ravfogel, S., Elazar, Y., & Goldberg, Y. (2020) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2020.blackboxnlp-1.5 · Cited by 68
Sun, Y., Zheng, Y., Hao, C., & Qiu, H. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2109.03564 · Cited by 83
Multi-Stage Prompting for Knowledgeable Dialogue Generation
Liu, Z., Patwary, M., Prenger, R., Prabhumoye, S., Ping, W., Shoeybi, M., & Catanzaro, B. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.104 · Cited by 41
Controllable Image Captioning via Prompting
Wang, N., Xie, J., Wu, J., Jia, M., & Li, L. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25360 · Cited by 67
Prompt Learns Prompt: Exploring Knowledge-Aware Generative Prompt Collaboration For Video Captioning
Yan, L., Han, C., Xu, Z., Liu, D., & Wang, Q. (2023) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2023/180 · Cited by 74
ArGue: Attribute-Guided Prompt Tuning for Vision-Language Models
Tian, X., Zou, S., Yang, Z., & Zhang, J. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02700 · Cited by 102
TransPrompt: Towards an Automatic Transferable Prompting Framework for Few-shot Text Classification
Wang, C., Wang, J., Qiu, M., Huang, J., & Gao, M. Z. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.221 · Cited by 68
Piloting Copilot, Codex, and StarCoder2: Hot temperature, cold prompts, or black magic?
Döderlein, J.-B., Kouadio, N. H., Acher, M., Khelladi, D. E., & Combemale, B. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112562 · Cited by 70
AESOP: Paraphrase Generation with Adaptive Syntactic Control
Sun, J., Ma, X., & Peng, N. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.420 · Cited by 66
AdaPrompt: Adaptive Model Training for Prompt-based NLP
Chen, Y., Liu, Y., Li, D., Wang, S., Zhu, C., Zeng, M., & Zhang, Y. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.448 · Cited by 61
Prompt Consistency for Zero-Shot Task Generalization
Zhou, C., He, J., Ma, X., Berg-Kirkpatrick, T., & Neubig, G. (2022) · Findings of the Association for Computational Linguistics: EMNLP 2022 · doi:10.18653/v1/2022.findings-emnlp.192 · Cited by 41
Self-citationLanguage models are not naysayers: an analysis of language models on negation benchmarks
Truong, T. H., Baldwin, T., Verspoor, K. M., & Cohn, T. (2023) · Joint Conference on Lexical and Computational Semantics · doi:10.18653/v1/2023.starsem-1.10 · Cited by 130
Liu, Y., Du, H., Niyato, D. T., Kang, J., Cui, S. R., Shen, X., & Zhang, P. (2023) · IEEE Network · doi:10.1109/mnet.2023.3335255 · Cited by 60
Distribution-Aware Prompt Tuning for Vision-Language Models
Cho, E., Kim, J., & Kim, H. J. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.02011 · Cited by 93
Li, H., Fu, J., & Python, A. (2025) · Journal of Medical Internet Research · doi:10.2196/71916 · Cited by 68
BBTv2: Towards a Gradient-Free Future with Large Language Models
Sun, T., He, Z., Qian, H., Zhou, Y., Huang, X., & Qiu, X. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.259 · Cited by 131
Continual Prompt Tuning for Dialog State Tracking
Zhu, Q., Li, B., Mi, F., Zhu, X., & Huang, M. (2022) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2022.acl-long.80 · Cited by 80
Prompt Tuning in Code Intelligence: An Experimental Evaluation
Wang, C., Yang, Y., Gao, C., Peng, Y., Zhang, H., & Lyu, M. R.-T. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3313881 · Cited by 58
Ultra-Fine Entity Typing with Weak Supervision from a Masked Language Model
Dai, H., Song, Y., & Wang, H. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.141 · Cited by 71
Exploring Universal Intrinsic Task Subspace for Few-Shot Learning via Prompt Tuning
Qin, Y., Wang, X., Su, Y., Lin, Y., Ding, N., Yi, J., Chen, W., Liu, Z., Li, J., Hou, L., et al. (2024) · IEEE/ACM Transactions on Audio Speech and Language Processing · doi:10.1109/taslp.2024.3430545 · Cited by 76
Exploring Structured Semantic Prior for Multi Label Recognition with Incomplete Labels
Ding, Z., Wang, A., Chen, H., Zhang, Q., Liu, P., Bao, Y., Yan, W., & Han, J. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.00331 · Cited by 66
Hierarchical Prompt Learning for Multi-Task Learning
Liu, Y., Lu, Y., Liu, H., An, Y., Xu, Z., Yao, Z., Zhang, B., Xiong, Z., & Gui, C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01048 · Cited by 75
Shi, S., Hu, K., Xie, J., Guo, Y., & Wu, H. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103531 · Cited by 50
The language of prompting: What linguistic properties make a prompt successful?
Leidinger, A., van Rooij, R., & Shutova, E. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.618 · Cited by 109
On the Evolution of Knowledge Graphs: A Survey and Perspective
Jiang, X., Xu, C., Shen, Y., Sun, X., Tang, L., Wang, S., Chen, Z., Wang, Y., & Guo, J. (2023) · arXiv · doi:10.48550/arxiv.2310.04835 · Cited by 42
“Moreover, some follow-up works propose various automatic prompt generation methods to further improve probing performances of LLMs [194], [195], [196], [197], [198].”
Knowledge Graph Generation From Text
Melnyk, I., Dognin, P., & Das, P. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.116 · Cited by 81
Prompt-based and weak-modality enhanced multimodal recommendation
Dong, X., Song, X., Tian, M., & Hu, L. (2023) · Information Fusion · doi:10.1016/j.inffus.2023.101989 · Cited by 48
Bayesian Prompt Learning for Image-Language Model Generalization
Derakhshani, M. M., Sanchez, E., Bulat, A., da Costa, V. G. T., Snoek, C. G. M., Tzimiropoulos, G., & Martínez, B. (2023) · IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.01398 · Cited by 109
A Pedestrian is Worth One Prompt: Towards Language Guidance Person Re- Identification
Yang, Z., Wu, D., Wu, C., Zheng, L. L., Gu, J., & Wang, W. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01642 · Cited by 50
APEER : Automatic Prompt Engineering Enhances Large Language Model Reranking
Jin, C., Peng, H., Zhao, S., Wang, Z. G., Xu, W., Han, L., Zhao, J., Zhong, K.-W., Rajasekaran, S., & Metaxas, D. (2025) · International World Wide Web Conference (WWW) · doi:10.1145/3701716.3717574 · Cited by 85
Analyzing the Forgetting Problem in Pretrain-Finetuning of Open-domain Dialogue Response Models
He, T., Liu, J., Cho, K., Ott, M., Liu, B., Glass, J., & Peng, F. (2021) · Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume · doi:10.18653/v1/2021.eacl-main.95 · Cited by 40
Knowledge-based dynamic prompt learning for multi-label disease diagnosis
Xie, J., Li, X., Yuan, Y., Guan, Y., Jiang, J., Guo, X., & Peng, X. X. (2024) · Knowledge-Based Systems · doi:10.1016/j.knosys.2024.111395 · Cited by 44
GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Models
Yin, D., Bansal, H., Monajatipoor, M., Li, L. H., & Chang, K.-W. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.132 · Cited by 123
An evaluation on large language model outputs: Discourse and memorization
de Wynter, A., Wang, X., Sokolov, A., Gu, Q., & Chen, S. (2023) · Natural Language Processing Journal · doi:10.1016/j.nlp.2023.100024 · Cited by 75
Text-Visual Prompting for Efficient 2D Temporal Video Grounding
Zhang, Y., Chen, X., Jia, J., Liu, S., & Ding, K. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01421 · Cited by 55
Yang, R., Zhu, J., Man, J., Liu, H., Li, F., & Zhou, Y. J. (2024) · Information Fusion · doi:10.1016/j.inffus.2024.102868 · Cited by 55
Language Models in the Loop: Incorporating Prompting into Weak Supervision
Smith, R., Fries, J., Hancock, B. J., & Bach, S. H. (2024) · ACM / IMS Journal of Data Science · doi:10.1145/3617130 · Cited by 96
Zhang, H., Liu, X., Pan, H., Ke, H., Ou, J., Fang, T., & Song, Y. (2022) · Artificial Intelligence · doi:10.1016/j.artint.2022.103740 · Cited by 72
ProGen: Progressive Zero-shot Dataset Generation via In-context Feedback
Ye, J., Gao, J., Wu, Z., Feng, J., Yu, C., & Kong, L. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.269 · Cited by 99
How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advances
Zhang, Z., Fang, M., Chen, L., Namazi‐Rad, M., & Wang, Y.-X. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.516 · Cited by 114
Prompt Guided Transformer for Multi-Task Dense Prediction
Lu, Y., Sirejiding, S., Ding, Y., Wang, C., & Lu, H. (2024) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2024.3349865 · Cited by 49
LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP
Huang, Y., Shakeri, F., Dolz, J., Boudiaf, M., Bahig, H., & Ayed, I. B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02244 · Cited by 111
Relational World Knowledge Representation in Contextual Language Models: A Review
Safavi, T., & Koutra, D. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.81 · Cited by 68
Finding Skill Neurons in Pre-trained Transformer-based Language Models
Wang, X., Wen, K., Zhang, Z., Hou, L., Liu, Z., & Li, J. (2022) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2022.emnlp-main.765 · Cited by 160
Learning Transferable Negative Prompts for Out-of-Distribution Detection
Li, T., Pang, G., Bai, X. P., Miao, W., & Zheng, J. G. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.01665 · Cited by 107
Improving Math Word Problems with Pre-trained Knowledge and Hierarchical Reasoning
Yu, W., Wen, Y., Zheng, F., & Xiao, N. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.272 · Cited by 45
A prompt model with combined semantic refinement for aspect sentiment analysis
Zhu, X., Kuang, Z., & Zhang, L. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103462 · Cited by 40
What Does My QA Model Know? Devising Controlled Probes Using Expert Knowledge
Richardson, K., & Sabharwal, A. (2020) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00331 · Cited by 48
Language models are not naysayers: an analysis of language models on negation benchmarks
Truong, T. H., Baldwin, T., Verspoor, K., & Cohn, T. (2023) · arXiv · doi:10.48550/arxiv.2306.08189 · Cited by 34
Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts
Liang, J. T., Lin, M. M., Rao, N., & Myers, B. A. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729342 · Cited by 73
How Pre-trained Language Models Capture Factual Knowledge? A Causal-Inspired Analysis
Li, S., Li, X., Shang, L., Dong, Z., Sun, C., Liu, B., Ji, Z., Jiang, X., & Liu, Q. (2022) · Findings of the Association for Computational Linguistics: ACL 2022 · doi:10.18653/v1/2022.findings-acl.136 · Cited by 30
Neural Language Generation: Formulation, Methods, and Evaluation
Gârbacea, C., & Mei, Q. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.15780 · Cited by 24
A Simple Long-Tailed Recognition Baseline via Vision-Language ModelPreprint
Ma, T., Geng, S., Wang, M., Shao, J., Lu, J., Li, H., Gao, P., & Qiao, Y. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2111.14745 · Cited by 84
Burford, K. G., Itzkowitz, N. G., Ortega, A. G., Teitler, J. O., & Rundle, A. G. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.25981 · Cited by 33
Evaluating search engines and large language models for answering health questions
Fernández-Pichel, M., Pichel, J. C., & Losada, D. E. (2025) · npj Digital Medicine · doi:10.1038/s41746-025-01546-w · Cited by 33
FAIR Enough: Develop and Assess a FAIR-Compliant Dataset for Large Language Model Training?
Raza, S., Ghuge, S., Ding, C., Dolatabadi, E., & Pandya, D. A. (2024) · Data Intelligence · doi:10.1162/dint_a_00255 · Cited by 48
Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuning
Wei, C., Xie, S. M., & Ma, T. (2021) · arXiv · doi:10.48550/arxiv.2106.09226 · Cited by 32
A comprehensive taxonomy of prompt engineering techniques for large language models
Liu, Y.-Y., Zheng, Z., Zhang, F., Feng, J.-C., Fu, Y.-Y., Zhai, J.-D., He, B.-S., Zhang, X., & Du, X.-Y. (2026) · Frontiers of Computer Science · doi:10.1007/s11704-025-50058-z · Cited by 37
Open-Set Fine-Grained Retrieval via Prompting Vision-Language Evaluator
Wang, S., Chang, J., Li, H., Wang, Z., Ouyang, W., & Tian, Q. C. (2023) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52729.2023.01857 · Cited by 42
Knowledge Prompt-tuning for Sequential Recommendation
Zhai, J., Zheng, X., Wang, C., Li, H., & Tian, Y. (2023) · ACM International Conference on Multimedia (ACM MM) · doi:10.1145/3581783.3612252 · Cited by 49
Shi, Y., Ren, P., Wang, J., Han, B., ValizadehAslani, T., Agbavor, F., Zhang, Y., Hu, M., Zhao, L., & Liang, H. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104533 · Cited by 40
Prompt Tuning of Deep Neural Networks for Speaker-Adaptive Visual Speech Recognition
Kim, M., Kim, H.-I., & Ro, Y. M. (2024) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2024.3484658 · Cited by 41
CINS: Comprehensive Instruction for Few-Shot Learning in Task-Oriented Dialog Systems
Mi, F., Wang, Y., & Li, Y. (2022) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v36i10.21356 · Cited by 56
Contrastive Graph Prompt-tuning for Cross-domain Recommendation
Yi, Z., Ounis, I., & Macdonald, C. (2023) · ACM Transactions on Information Systems · doi:10.1145/3618298 · Cited by 55
DesPrompt: Personality-descriptive prompt tuning for few-shot personality recognition
Wen, Z., Cao, J., Yang, Y., Wang, H., Yang, R., & Liu, S. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103422 · Cited by 38
Zanella, M., & Ayed, I. B. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02245 · Cited by 105
SD4Match: Learning to Prompt Stable Diffusion Model for Semantic Matching
Li, X., Lu, J., Han, K., & Prisacariu, V. A. (2024) · IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings - IEEE Computer Society Conference on Computer Vision and Pattern Recognition/Proceedings · doi:10.1109/cvpr52733.2024.02602 · Cited by 57
Cross-Modal Remote Sensing Image–Text Retrieval via Context and Uncertainty-Aware Prompt
Wang, Y., Tang, X., Ma, J., Zhang, X., Liu, F., & Jiao, L. (2024) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2024.3458898 · Cited by 46
Correia, A., Hickey, S., & Xu, F. (2025) · Theory Into Practice · doi:10.1080/00405841.2025.2528545 · Cited by 42
Self-citationPractices, opportunities and challenges in the fusion of knowledge graphs and large language models
Cai, L., Yu, C., Kang, Y., Fu, Y., Zhang, H., & Zhao, Y. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2025.1590632 · Cited by 46
NOTABLE: Transferable Backdoor Attacks Against Prompt-based NLP Models
Mei, K., Li, Z., Wang, Z., Zhang, Y., & Ma, S. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.867 · Cited by 62
Prompt-and-Align: Prompt-Based Social Alignment for Few-Shot Fake News Detection
Wu, J., Li, S., Deng, A., Xiong, M., & Hooi, B. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3615015 · Cited by 49
(2022) · doi:10.18653/v1/2022.acl-long.168 · Cited by 29
ProQA: Structural Prompt-based Pre-training for Unified Question Answering
Zhong, W., Gao, Y., Ding, N., Qin, Y., Liu, Z., Zhou, M., Wang, J., Yin, J. P., & Duan, N. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.313 · Cited by 41
Multi-Modal Attribute Prompting for Vision-Language Models
Liu, X., Wu, J., Yang, W., Zhou, X., & Zhang, T. (2024) · IEEE Transactions on Circuits and Systems for Video Technology · doi:10.1109/tcsvt.2024.3424566 · Cited by 42
Learning to Rewrite Prompts for Personalized Text Generation
Li, C., Zhang, M., Mei, Q., Kong, W., & Bendersky, M. (2024) · International World Wide Web Conference (WWW) · doi:10.1145/3589334.3645408 · Cited by 88
Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts
Khashabi, D., Lyu, X., Min, S., Qin, L., Richardson, K., Welleck, S., Hajishirzi, H., Khot, T., Sabharwal, A., Singh, S., et al. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.266 · Cited by 25
Storyfier: Exploring Vocabulary Learning Support with Text Generation Models
Peng, Z., Wang, X., Han, Q., Zhu, J., Ma, X., & Qu, H. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606786 · Cited by 27
The language of prompting: What linguistic properties make a prompt successful?
Leidinger, A., van Rooij, R., & Shutova, E. (2023) · arXiv · doi:10.48550/arxiv.2311.01967 · Cited by 29
Meta-prompt based learning for low-resource false information detection
Huang, Y., Gao, M., Wang, J., Yin, J., Shu, K., Fan, Q., & Wen, J. (2023) · Information Processing & Management · doi:10.1016/j.ipm.2023.103279 · Cited by 35
Aspect category sentiment analysis based on prompt-based learning with attention mechanism
Ping, Z., Sang, G., Liu, Z., & Zhang, Y. (2023) · Neurocomputing · doi:10.1016/j.neucom.2023.126994 · Cited by 33
Making Pre-trained Language Models End-to-end Few-shot Learners with Contrastive Prompt Tuning
Xu, Z., Wang, C., Qiu, M., Luo, F., Xu, R., Huang, S., & Huang, J. (2023) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3539597.3570398 · Cited by 44
LogicSolver: Towards Interpretable Math Word Problem Solving with Logical Prompt-enhanced Learning
Yang, Z., Qin, J., Chen, J., Lin, L., & Liang, X. (2022) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2022.findings-emnlp.1 · Cited by 33
KGTS: Contrastive Trajectory Similarity Learning over Prompt Knowledge Graph Embedding
Chen, Z., Zhang, D., Feng, S., Chen, K., Chen, L., Han, P., & Shang, S. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i8.28672 · Cited by 46
Robust Knowledge Graph Completion with Stacked Convolutions and a Student Re-Ranking Network
Lovelace, J., Newman-Griffis, D. R., Vashishth, S., Lehman, J. F., & Rosé, C. P. (2021) · doi:10.18653/v1/2021.acl-long.82 · Cited by 43
Instance-Aware Prompt Learning for Language Understanding and Generation
Jin, F., Lu, J., Zhang, J., & Zong, C. (2023) · ACM Transactions on Asian and Low-Resource Language Information Processing · doi:10.1145/3604613 · Cited by 36
Fine-Grained Retrieval Prompt Tuning
Wang, S., Chang, J., Wang, Z., Li, H., Ouyang, W., & Tian, Q. C. (2023) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v37i2.25363 · Cited by 36
Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuning
Özdayi, M. S., Peris, C., FitzGerald, J., Dupuy, C., Majmudar, J., Khan, H., Parikh, R., & Gupta, R. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-short.129 · Cited by 72
Metadata
- Authors
- Jiang, Z., Xu, F. F., Araki, J., & Neubig, G.
- Affiliations
- Carnegie Mellon University (US)
- Robert Bosch (United States) (US)
- Venue
- Transactions of the Association for Computational Linguistics
- Publisher
- MIT Press
- Published
- 2020-12
- Volume
- 8
- Pages
- 423-438
- ISSN
- 2307-387X (ISSN-L)
- OpenAlex
- W2991382858
- Cited by
- 1,002
- References
- 52
- Citations by year
- 2026: 76 · 2025: 169 · 2024: 224 · 2023: 285 · 2022: 144 · 2021: 78 · 2020: 25
- FWCI
- 52.35
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- cc-by
- Locations
- Transactions of the Association for Computational Linguistics (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1815287)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Keywords
- prompt generation, ensemble methods
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.